[Yum-devel] yum does not retry on 503 errors

[Zdenek Pavlas]

> 3) the failure callback that yum gives to urlgrabber /always/
> re-raises the exception, preventing any possible retries that that
> level
> 
> I found #3 the most surprising since yum goes to the trouble of
> passing the retry option to urlgrabber

Yes, was surprised too when I had discovered this, but the idea
is that it's often better to try next mirror instead of retrying
(think of timing out on a dead mirror and retrying that 20 times).

> 503 errors are usually meant to be retried. Possibly there are other
> http statuses that should be retried as well.

I agree.

> I've attached a overly simple patch to urlgrabber that does solve my
> immediate issue, but I'm not sure if it's the right fix overall. I'd
> be happy to work on a better fix, but I thought I would solicit
> feedback/advice first.

If you return from the Yum failure callback instead, the retry
logic will work.  Could you try this instead?

diff --git a/output.py b/output.py
index b3f2c75..4a55357 100755
--- a/output.py
+++ b/output.py
@@ -465,6 +465,9 @@ class YumOutput:
         :raises: *errobj*.exception
         """
         self.logger.error('%s: %s', errobj.url, errobj.exception)
+        if errorobj.exception.code == 503:
+            errorobj.exception.errno = -1
+            return
         self.logger.error(_('Trying other mirror.'))
         raise errobj.exception
     
Unfortunately, urlgrabber caches errno, we can't change it 
in callback.. Have to patch it too:

diff --git a/urlgrabber/grabber.py b/urlgrabber/grabber.py
index 3127b48..c46f96f 100644
--- a/urlgrabber/grabber.py
+++ b/urlgrabber/grabber.py
@@ -1011,7 +1011,6 @@ class URLGrabber(object):
             # beware of infinite loops :)
             tries = tries + 1
             exception = None
-            retrycode = None
             callback  = None
             if DEBUG: DEBUG.info('attempt %i/%s: %s',
                                  tries, opts.retry, args[0])
@@ -1022,7 +1021,6 @@ class URLGrabber(object):
             except URLGrabError, e:
                 exception = e
                 callback = opts.failure_callback
-                retrycode = e.errno
             except KeyboardInterrupt, e:
                 exception = e
                 callback = opts.interrupt_callback
@@ -1040,6 +1038,7 @@ class URLGrabber(object):
                 if DEBUG: DEBUG.info('retries exceeded, re-raising')
                 raise
 
+            retrycode = getattr(exception, 'errno', None)
             if (retrycode is not None) and (retrycode not in opts.retrycodes):
                 if DEBUG: DEBUG.info('retrycode (%i) not in list %s, re-raising',
                                      retrycode, opts.retrycodes)
@@ -2176,7 +2175,7 @@ def parallel_wait(meter = 'text'):
                 try: _run_callback(opts.failure_callback, opts)
                 except URLGrabError, ug_err:
                     retry = 0 # no retries
-            if opts.tries < retry and ug_err.args[0] in opts.retrycodes:
+            if opts.tries < retry and ug_err.errno in opts.retrycodes:
                 start(opts, opts.tries + 1) # simple retry
                 continue