Skip to content

http: add support for HTTP 429 rate limit retries #2008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vaidas-shopify wants to merge 2 commits into
base: master
Choose a base branch
from
+750 −57
Open

http: add support for HTTP 429 rate limit retries #2008

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d80ce07
http: add support for HTTP 429 rate limit retries
vaidas-shopify Nov 26, 2025
ad4495f
http: add trace2 logging for retry operations
vaidas-shopify Dec 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions Documentation/config/http.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,30 @@ http.keepAliveCount::
unset, curl's default value is used. Can be overridden by the
Copy link

@gitgitgadget gitgitgadget bot Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Taylor Blau wrote (reply to this):

On Wed, Nov 26, 2025 at 12:30:25PM +0000, Vaidas Pilkauskas via GitGitGadget wrote:
> Retry behavior is controlled by three new configuration options:
>
>   * http.maxRetries: Maximum number of retry attempts (default: 0,
>     meaning retries are disabled by default). Users must explicitly
>     opt-in to retry behavior.
>
>   * http.retryAfter: Default delay in seconds when the server doesn't
>     provide a Retry-After header (default: -1, meaning fail if no
>     header is provided). This serves as a fallback mechanism.
>
>   * http.maxRetryTime: Maximum delay in seconds for a single retry
>     (default: 300). If the server requests a delay exceeding this
>     limit, Git fails immediately rather than waiting. This prevents
>     indefinite blocking on unreasonable server requests.
>
> All three options can be overridden via environment variables:
> GIT_HTTP_MAX_RETRIES, GIT_HTTP_RETRY_AFTER, and
> GIT_HTTP_MAX_RETRY_TIME.

This is great information, and I am glad that it is written down in
http.adoc so that it shows up in git-config(1). I think that it's fine
to omit this level of detail from the commit message, since it
duplicates information from the authoritative source on configuration
knobs.

It might be reasonable to say something like:

    Retry behavior is controlled by three new configuration options
    (http.maxRetries, http.retryAfter, and http.maxRetryTime) which are
    documented in git-config(1).

or something.

> diff --git a/http-push.c b/http-push.c
> index d86ce77119..a602a302ec 100644
> --- a/http-push.c
> +++ b/http-push.c
> @@ -716,6 +716,10 @@ static int fetch_indices(void)
>  	case HTTP_MISSING_TARGET:
>  		ret = 0;
>  		break;
> +	case HTTP_RATE_LIMITED:
> +		error("rate limited by '%s', please try again later", repo->url);
> +		ret = -1;

Other strings in this file aren't marked for translation, but I think
we can/should mark this one like so:

    error(_("rate limited by %s ..."), repo->url);

> diff --git a/http.c b/http.c
> index 41f850db16..212805cad5 100644
> --- a/http.c
> +++ b/http.c
> @@ -22,6 +22,7 @@
>  #include "object-file.h"
>  #include "odb.h"
>  #include "tempfile.h"
> +#include "date.h"
>
>  static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
>  static int trace_curl_data = 1;
> @@ -149,6 +150,14 @@ static char *cached_accept_language;
>  static char *http_ssl_backend;
>
>  static int http_schannel_check_revoke = 1;
> +
> +/* Retry configuration */
> +static long http_retry_after = -1; /* Default retry-after in seconds when header is missing (-1 means not set, exit with 128) */
> +static long http_max_retries = 0; /* Maximum number of retry attempts (0 means retries are disabled) */
> +static long http_max_retry_time = 300; /* Maximum time to wait for a single retry (default 5 minutes) */

These comments should be OK to drop, the variables indicate what Git
configuration they correspond to (e.g., http_retry_after ->
http.retryAfter), so git-config(1) is the authoritative source for
documentation here.

> @@ -257,6 +267,47 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p UN
>  		goto exit;
>  	}
>
> +	/* Parse Retry-After header for rate limiting */
> +	if (skip_iprefix_mem(ptr, size, "retry-after:", &val, &val_len)) {

Makes sense, though I wonder if we should rename this function, since
fwrite_wwwauth is now doing more than just handling WWW-Authenticate
headers.

Perhaps we should have a single top-level function that is registered as
our CURLOPT_HEADERFUNCTION that dispatches calls to header-specific
functions? Otherwise the actual parsing of the Retry-After header looks
good to me.

> @@ -1422,6 +1488,10 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
>  	set_long_from_env(&curl_tcp_keepintvl, "GIT_TCP_KEEPINTVL");
>  	set_long_from_env(&curl_tcp_keepcnt, "GIT_TCP_KEEPCNT");
>
> +	set_long_from_env(&http_retry_after, "GIT_HTTP_RETRY_AFTER");
> +	set_long_from_env(&http_max_retries, "GIT_HTTP_MAX_RETRIES");
> +	set_long_from_env(&http_max_retry_time, "GIT_HTTP_MAX_RETRY_TIME");
> +

The configuration handling and overrides look good to me.

> @@ -2253,19 +2330,36 @@ static int update_url_from_redirect(struct strbuf *base,
>  	return 1;
>  }
>
> +/*
> + * Sleep for the specified number of seconds before retrying.
> + */
> +static void sleep_for_retry(long retry_after)
> +{
> +	if (retry_after > 0) {
> +		unsigned int remaining;
> +		warning(_("rate limited, waiting %ld seconds before retry"), retry_after);
> +		remaining = sleep(retry_after);

What should we do if there are other active request slots? It has been a
couple of years since I have looked at Git's HTTP code, but I imagine
that we should be able to continue processing other requests while
waiting for the retry-after period to elapse here.

> @@ -2302,7 +2396,54 @@ static int http_request_reauth(const char *url,
>  			BUG("Unknown http_request target");
>  		}
>
> -		credential_fill(the_repository, &http_auth, 1);
> +		if (ret == HTTP_RATE_LIMITED) {

Should handling the retry behavior be moved into a separate function? I
think that http_request_reauth() might be clearer if it read:

    if (ret == HTTP_RATE_LIMITED)
      apply_rate_limit(...); /* presumably with a better name */
    else
      credential_fill(...);

, and likewise, should we rename this function as it is no longer just
re-authenticating HTTP requests?

> diff --git a/t/t5584-http-429-retry.sh b/t/t5584-http-429-retry.sh
> new file mode 100755
> index 0000000000..8bcc382763
> --- /dev/null
> +++ b/t/t5584-http-429-retry.sh
> @@ -0,0 +1,429 @@
> +#!/bin/sh
> +
> +test_description='test HTTP 429 Too Many Requests retry logic'
> +
> +. ./test-lib.sh
> +
> +. "$TEST_DIRECTORY"/lib-httpd.sh
> +
> +start_httpd
> +
> +test_expect_success 'setup test repository' '
> +	test_commit initial &&
> +	git clone --bare . "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
> +	git --git-dir="$HTTPD_DOCUMENT_ROOT_PATH/repo.git" config http.receivepack true
> +'
> +
> +test_expect_success 'HTTP 429 with retries disabled (maxRetries=0) fails immediately' '
> +	write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
> +	printf "Status: 429 Too Many Requests\r\n"
> +	printf "Retry-After: 1\r\n"
> +	printf "Content-Type: text/plain\r\n"
> +	printf "\r\n"
> +	printf "Rate limited\n"
> +	cat "$1" >/dev/null
> +	EOF

To avoid having to write this script multiple write, you can write it as
a separate script in t/lib-httpd and then make sure to list it in
prepare_httpd() (from t/lib-httpd.sh).

You can then list it in the apache.conf in the same directory and invoke
it however you like. If you need to take in arguments to the script
(e.g., to change the Retry-After value), you can use a ScriptAliasMatch
instead of a normal ScriptAlias to pass in extra parameters from the URL.

The one-time-script mechanism here will cause the test harness to delete
the script after its first (and only) use, which can be useful for some
cases but I suspect is not necessary for all of these tests.
> +
> +	# Set maxRetries to 0 (disabled)
> +	test_config http.maxRetries 0 &&
> +	test_config http.retryAfter 1 &&
> +
> +	# Should fail immediately without any retry attempt
> +	test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
> +
> +	# Verify no retry happened (no "waiting" message in stderr)
> +	! grep -i "waiting.*retry" err &&

test_grep can be helpful when reading the output of test failures, since
it dumps the contents of the file it was searching. Just make sure to
write "test_grep !" instead of "! test_grep" (there are a few such
instances of the latter that I just wrote patches to clean up).

"! test_grep" isn't *wrong* per-se, but it will pollute the test output
with "couldn't find xyz in abc".

I skimmed through the the remainder of the tests since I imagine that
they will change substantially after writing the script out explicitly
instead of using one-time-script, so I'll hold off on reviewing that
portion in more detail until then.

Thanks,
Taylor

Copy link

@gitgitgadget gitgitgadget bot Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Vaidas Pilkauskas wrote (reply to this):

On Wed, Dec 10, 2025 at 1:15 AM Taylor Blau <me@ttaylorr.com> wrote:
> > +/*
> > + * Sleep for the specified number of seconds before retrying.
> > + */
> > +static void sleep_for_retry(long retry_after)
> > +{
> > +     if (retry_after > 0) {
> > +             unsigned int remaining;
> > +             warning(_("rate limited, waiting %ld seconds before retry"), retry_after);
> > +             remaining = sleep(retry_after);
>
> What should we do if there are other active request slots? It has been a
> couple of years since I have looked at Git's HTTP code, but I imagine
> that we should be able to continue processing other requests while
> waiting for the retry-after period to elapse here.

This is a very good catch - I'll rewrite this to a non-blocking wait.

Thanks for the review, Taylor, I'll work to address this and other
comments in the next version of the patch.

On Wed, Dec 10, 2025 at 1:15 AM Taylor Blau <me@ttaylorr.com> wrote:
>
> On Wed, Nov 26, 2025 at 12:30:25PM +0000, Vaidas Pilkauskas via GitGitGadget wrote:
> > Retry behavior is controlled by three new configuration options:
> >
> >   * http.maxRetries: Maximum number of retry attempts (default: 0,
> >     meaning retries are disabled by default). Users must explicitly
> >     opt-in to retry behavior.
> >
> >   * http.retryAfter: Default delay in seconds when the server doesn't
> >     provide a Retry-After header (default: -1, meaning fail if no
> >     header is provided). This serves as a fallback mechanism.
> >
> >   * http.maxRetryTime: Maximum delay in seconds for a single retry
> >     (default: 300). If the server requests a delay exceeding this
> >     limit, Git fails immediately rather than waiting. This prevents
> >     indefinite blocking on unreasonable server requests.
> >
> > All three options can be overridden via environment variables:
> > GIT_HTTP_MAX_RETRIES, GIT_HTTP_RETRY_AFTER, and
> > GIT_HTTP_MAX_RETRY_TIME.
>
> This is great information, and I am glad that it is written down in
> http.adoc so that it shows up in git-config(1). I think that it's fine
> to omit this level of detail from the commit message, since it
> duplicates information from the authoritative source on configuration
> knobs.
>
> It might be reasonable to say something like:
>
>     Retry behavior is controlled by three new configuration options
>     (http.maxRetries, http.retryAfter, and http.maxRetryTime) which are
>     documented in git-config(1).
>
> or something.
>
> > diff --git a/http-push.c b/http-push.c
> > index d86ce77119..a602a302ec 100644
> > --- a/http-push.c
> > +++ b/http-push.c
> > @@ -716,6 +716,10 @@ static int fetch_indices(void)
> >       case HTTP_MISSING_TARGET:
> >               ret = 0;
> >               break;
> > +     case HTTP_RATE_LIMITED:
> > +             error("rate limited by '%s', please try again later", repo->url);
> > +             ret = -1;
>
> Other strings in this file aren't marked for translation, but I think
> we can/should mark this one like so:
>
>     error(_("rate limited by %s ..."), repo->url);
>
> > diff --git a/http.c b/http.c
> > index 41f850db16..212805cad5 100644
> > --- a/http.c
> > +++ b/http.c
> > @@ -22,6 +22,7 @@
> >  #include "object-file.h"
> >  #include "odb.h"
> >  #include "tempfile.h"
> > +#include "date.h"
> >
> >  static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
> >  static int trace_curl_data = 1;
> > @@ -149,6 +150,14 @@ static char *cached_accept_language;
> >  static char *http_ssl_backend;
> >
> >  static int http_schannel_check_revoke = 1;
> > +
> > +/* Retry configuration */
> > +static long http_retry_after = -1; /* Default retry-after in seconds when header is missing (-1 means not set, exit with 128) */
> > +static long http_max_retries = 0; /* Maximum number of retry attempts (0 means retries are disabled) */
> > +static long http_max_retry_time = 300; /* Maximum time to wait for a single retry (default 5 minutes) */
>
> These comments should be OK to drop, the variables indicate what Git
> configuration they correspond to (e.g., http_retry_after ->
> http.retryAfter), so git-config(1) is the authoritative source for
> documentation here.
>
> > @@ -257,6 +267,47 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p UN
> >               goto exit;
> >       }
> >
> > +     /* Parse Retry-After header for rate limiting */
> > +     if (skip_iprefix_mem(ptr, size, "retry-after:", &val, &val_len)) {
>
> Makes sense, though I wonder if we should rename this function, since
> fwrite_wwwauth is now doing more than just handling WWW-Authenticate
> headers.
>
> Perhaps we should have a single top-level function that is registered as
> our CURLOPT_HEADERFUNCTION that dispatches calls to header-specific
> functions? Otherwise the actual parsing of the Retry-After header looks
> good to me.
>
> > @@ -1422,6 +1488,10 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
> >       set_long_from_env(&curl_tcp_keepintvl, "GIT_TCP_KEEPINTVL");
> >       set_long_from_env(&curl_tcp_keepcnt, "GIT_TCP_KEEPCNT");
> >
> > +     set_long_from_env(&http_retry_after, "GIT_HTTP_RETRY_AFTER");
> > +     set_long_from_env(&http_max_retries, "GIT_HTTP_MAX_RETRIES");
> > +     set_long_from_env(&http_max_retry_time, "GIT_HTTP_MAX_RETRY_TIME");
> > +
>
> The configuration handling and overrides look good to me.
>
> > @@ -2253,19 +2330,36 @@ static int update_url_from_redirect(struct strbuf *base,
> >       return 1;
> >  }
> >
> > +/*
> > + * Sleep for the specified number of seconds before retrying.
> > + */
> > +static void sleep_for_retry(long retry_after)
> > +{
> > +     if (retry_after > 0) {
> > +             unsigned int remaining;
> > +             warning(_("rate limited, waiting %ld seconds before retry"), retry_after);
> > +             remaining = sleep(retry_after);
>
> What should we do if there are other active request slots? It has been a
> couple of years since I have looked at Git's HTTP code, but I imagine
> that we should be able to continue processing other requests while
> waiting for the retry-after period to elapse here.
>
> > @@ -2302,7 +2396,54 @@ static int http_request_reauth(const char *url,
> >                       BUG("Unknown http_request target");
> >               }
> >
> > -             credential_fill(the_repository, &http_auth, 1);
> > +             if (ret == HTTP_RATE_LIMITED) {
>
> Should handling the retry behavior be moved into a separate function? I
> think that http_request_reauth() might be clearer if it read:
>
>     if (ret == HTTP_RATE_LIMITED)
>       apply_rate_limit(...); /* presumably with a better name */
>     else
>       credential_fill(...);
>
> , and likewise, should we rename this function as it is no longer just
> re-authenticating HTTP requests?
>
> > diff --git a/t/t5584-http-429-retry.sh b/t/t5584-http-429-retry.sh
> > new file mode 100755
> > index 0000000000..8bcc382763
> > --- /dev/null
> > +++ b/t/t5584-http-429-retry.sh
> > @@ -0,0 +1,429 @@
> > +#!/bin/sh
> > +
> > +test_description='test HTTP 429 Too Many Requests retry logic'
> > +
> > +. ./test-lib.sh
> > +
> > +. "$TEST_DIRECTORY"/lib-httpd.sh
> > +
> > +start_httpd
> > +
> > +test_expect_success 'setup test repository' '
> > +     test_commit initial &&
> > +     git clone --bare . "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
> > +     git --git-dir="$HTTPD_DOCUMENT_ROOT_PATH/repo.git" config http.receivepack true
> > +'
> > +
> > +test_expect_success 'HTTP 429 with retries disabled (maxRetries=0) fails immediately' '
> > +     write_script "$HTTPD_ROOT_PATH/one-time-script" <<-\EOF &&
> > +     printf "Status: 429 Too Many Requests\r\n"
> > +     printf "Retry-After: 1\r\n"
> > +     printf "Content-Type: text/plain\r\n"
> > +     printf "\r\n"
> > +     printf "Rate limited\n"
> > +     cat "$1" >/dev/null
> > +     EOF
>
> To avoid having to write this script multiple write, you can write it as
> a separate script in t/lib-httpd and then make sure to list it in
> prepare_httpd() (from t/lib-httpd.sh).
>
> You can then list it in the apache.conf in the same directory and invoke
> it however you like. If you need to take in arguments to the script
> (e.g., to change the Retry-After value), you can use a ScriptAliasMatch
> instead of a normal ScriptAlias to pass in extra parameters from the URL.
>
> The one-time-script mechanism here will cause the test harness to delete
> the script after its first (and only) use, which can be useful for some
> cases but I suspect is not necessary for all of these tests.
> > +
> > +     # Set maxRetries to 0 (disabled)
> > +     test_config http.maxRetries 0 &&
> > +     test_config http.retryAfter 1 &&
> > +
> > +     # Should fail immediately without any retry attempt
> > +     test_must_fail git ls-remote "$HTTPD_URL/one_time_script/repo.git" 2>err &&
> > +
> > +     # Verify no retry happened (no "waiting" message in stderr)
> > +     ! grep -i "waiting.*retry" err &&
>
> test_grep can be helpful when reading the output of test failures, since
> it dumps the contents of the file it was searching. Just make sure to
> write "test_grep !" instead of "! test_grep" (there are a few such
> instances of the latter that I just wrote patches to clean up).
>
> "! test_grep" isn't *wrong* per-se, but it will pollute the test output
> with "couldn't find xyz in abc".
>
> I skimmed through the the remainder of the tests since I imagine that
> they will change substantially after writing the script out explicitly
> instead of using one-time-script, so I'll hold off on reviewing that
> portion in more detail until then.
>
> Thanks,
> Taylor

`GIT_HTTP_KEEPALIVE_COUNT` environment variable.

http.retryAfter::
Default wait time in seconds before retrying when a server returns
HTTP 429 (Too Many Requests) without a Retry-After header. If set
to -1 (the default), Git will fail immediately when encountering
a 429 response without a Retry-After header. When a Retry-After
header is present, its value takes precedence over this setting.
Can be overridden by the `GIT_HTTP_RETRY_AFTER` environment variable.
See also `http.maxRetries` and `http.maxRetryTime`.

http.maxRetries::
Maximum number of times to retry after receiving HTTP 429 (Too Many
Requests) responses. Set to 0 (the default) to disable retries.
Can be overridden by the `GIT_HTTP_MAX_RETRIES` environment variable.
See also `http.retryAfter` and `http.maxRetryTime`.

http.maxRetryTime::
Maximum time in seconds to wait for a single retry attempt when
handling HTTP 429 (Too Many Requests) responses. If the server
requests a delay (via Retry-After header) or if `http.retryAfter`
is configured with a value that exceeds this maximum, Git will fail
immediately rather than waiting. Default is 300 seconds (5 minutes).
Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment
variable. See also `http.retryAfter` and `http.maxRetries`.

http.noEPSV::
A boolean which disables using of EPSV ftp command by curl.
This can be helpful with some "poor" ftp servers which don't
Expand Down
8 changes: 8 additions & 0 deletions http-push.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -716,6 +716,10 @@ static int fetch_indices(void)
case HTTP_MISSING_TARGET:
ret = 0;
break;
case HTTP_RATE_LIMITED:
error(_("rate limited by '%s', please try again later"), repo->url);
ret = -1;
break;
default:
ret = -1;
}
Expand Down Expand Up @@ -1548,6 +1552,10 @@ static int remote_exists(const char *path)
case HTTP_MISSING_TARGET:
ret = 0;
break;
case HTTP_RATE_LIMITED:
error(_("rate limited by '%s', please try again later"), url);
ret = -1;
break;
case HTTP_ERROR:
error("unable to access '%s': %s", url, curl_errorstr);
/* fallthrough */
Expand Down
5 changes: 5 additions & 0 deletions http-walker.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,11 @@ static int fetch_indices(struct walker *walker, struct alt_base *repo)
repo->got_indices = 1;
ret = 0;
break;
case HTTP_RATE_LIMITED:
error("rate limited by '%s', please try again later", repo->base);
repo->got_indices = 0;
ret = -1;
break;
default:
repo->got_indices = 0;
ret = -1;
Expand Down
Loading
Loading