[Yum-devel] [PATCH] Dynamic mirror selection

[James Antill]

On Wed, 2012-01-25 at 11:23 +0100, Zdeněk Pavlas wrote:
> - Dropped the MG's 'max_connections' option, added a global one.
> - Single download queue, mirror selected as late as possible.
> - Merged request dispatch & flushing loops.
> 
> Also, dropped the 'GrabRequest' instance, which originally contained
> a copy of all mirrors.  Replaced that with a set of failed mirrors.
> 
> - No need to clone the mirror list to each request.
> - Now the master list could be shuffled during downloads,
>   and further mirror selections could act on that.

[...]

> +        while True:
> +            if idx >= len(_async_queue):

 Might be worth some comments explaining bits. Eg. _async_queue will
grow as the loop is run.

> +                if not dl.running: break
> +                perform(); continue

 Why do you keep putting things on one line?

> +
> +            # check global limit
> +            opts = _async_queue[idx]; idx += 1

 Again, I had to use search to find where idx was being changed.

> +            limit = opts.max_connections
> +            while len(dl.running) >= limit: perform()

 .

> +            if opts.mirror_group:
> +                first = None
> +                mg, failed = opts.mirror_group
> +                for mirror in mg.mirrors:
> +                    key = mirror['mirror']
> +                    if key in failed: continue
> +                    if not first: first = mirror
> +                    limit = mirror.get('kwargs', {}).get('max_connections', 3)
> +                    if host_con.get(key, 0) < limit: break
> +                else:
> +                    # no mirror with free slots.
> +                    if not first:
> +                        opts.exception = URLGrabError(256, _('No more mirrors to try.'))
> +                        _run_callback(opts.failfunc, opts)
> +                        continue
> +                    mirror = first # fallback
> +                    key = mirror['mirror']
> +                    limit = mirror.get('kwargs', {}).get('max_connections', 3)
> +
> +                # update the request
> +                opts.async = key, limit
> +                opts.url = mg._join_url(key, opts.relative_url)
> +
> +            # check host limit, then start
> +            key, limit = opts.async
> +            while host_con.get(key, 0) >= limit: perform()
> +            start(opts, 1)
>  
>      finally:
>          dl.abort()
>          if meter: meter.end()
> -        _async.clear()
> +        del _async_queue[:]

 Also, why this change? (not that it really matters, I don't think).

 ACK.