[Yum-devel] [PATCH] Rename the mirror's connection limit to max_mirror_connections.

Zdenek Pavlas zpavlas at redhat.com
Fri Aug 17 09:34:13 UTC 2012


> > The mirror's 'kwargs' hash is used to update the grab request,
> > and it was overriding the global limit.  Using different names
> > is also much clearer.
> 
>  Is there anyway we can fix this name collision while being
>  compatible?

- opts.max_connections
+ default_grabber.opts.max_connections

In fact, this was probably enough to fix this issue :)
But there's another:

1) We run _mirror_try() once for the 1st mirror.
2) Here we mix mirror's kwargs into the request's kwargs.
3) Multi downloader does not "undo" 2) when moving to next mirror,
   neither does it use any kwargs from the next mirror.

The following patch should fix this, too:

commit 53befbf36b39136b9d2654f921a691706aff4b78
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Aug 17 10:49:16 2012 +0200

    Use mirror kwargs and grabber options properly
    
    Don't modify urlgrab kwargs, since we can't undo that later.
    Instead, push mirror kwargs on top of grabber.opts.
    
    In multi downloader, update the grabber options when moving
    to the next mirror.

diff --git a/urlgrabber/grabber.py b/urlgrabber/grabber.py
index bdcdfe3..d2d1a9c 100644
--- a/urlgrabber/grabber.py
+++ b/urlgrabber/grabber.py
@@ -1069,7 +1069,8 @@ class URLGrabber(object):
         different from the passed-in filename if copy_local == 0.
         """
         url = _to_utf8(url)
-        opts = self.opts.derive(**kwargs)
+        # kwarg 'opts' overrides self.opts
+        opts = kwargs.pop('opts', self.opts).derive(**kwargs)
         if DEBUG: DEBUG.debug('combined options: %s' % repr(opts))
         (url,parts) = opts.urlparser.parse(url, opts) 
         (scheme, host, path, parm, query, frag) = parts
@@ -2213,7 +2214,7 @@ def parallel_wait(meter = 'text'):
             idx += 1
 
             # check global limit
-            while len(dl.running) >= opts.max_connections:
+            while len(dl.running) >= default_grabber.opts.max_connections:
                 perform()
 
             if opts.mirror_group:
@@ -2241,6 +2242,10 @@ def parallel_wait(meter = 'text'):
                     _run_callback(opts.failfunc, opts)
                     continue
 
+                # update the grabber object, apply mirror kwargs
+                grabber = best.get('grabber') or mg.grabber
+                opts.delegate = grabber.opts.derive(**best.get('kwargs', {}))
+
                 # update the current mirror and limit
                 key = best['mirror']
                 limit = best.get('kwargs', {}).get('max_connections', 2)
diff --git a/urlgrabber/mirror.py b/urlgrabber/mirror.py
index ac78b34..edbbbe8 100644
--- a/urlgrabber/mirror.py
+++ b/urlgrabber/mirror.py
@@ -393,13 +393,13 @@ class MirrorGroup:
             tries += 1
             mirrorchoice = self._get_mirror(gr)
             fullurl = self._join_url(mirrorchoice['mirror'], gr.url)
-            kwargs = dict(mirrorchoice.get('kwargs', {}))
-            kwargs.update(kw)
             grabber = mirrorchoice.get('grabber') or self.grabber
+            # apply mirrorchoice kwargs on top of grabber.opts
+            opts = grabber.opts.derive(**mirrorchoice.get('kwargs', {}))
             func_ref = getattr(grabber, func)
             if DEBUG: DEBUG.info('MIRROR: trying %s -> %s', url, fullurl)
             try:
-                return func_ref( *(fullurl,), **kwargs )
+                return func_ref( *(fullurl,), opts=opts, **kw )
             except URLGrabError, e:
                 if DEBUG: DEBUG.info('MIRROR: failed')
                 obj = CallbackObject()


More information about the Yum-devel mailing list