[yum-commits] Changes to 'multi-downloader'

[zpavlas at osuosl.org]

New branch 'multi-downloader' available with the following commits:
commit 1936c1582a978b82e4e88953a4dd50cc9dbf1e87
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 24 15:44:35 2012 +0100

    Add internal 'timedhosts' callback.
    
    Gets called when a download (both sync and async ones) finishes.
    May be used to estimate mirror speeds and update their priority.
    
    - true download size is reported
    - checkfunc callback is not timed

commit 7fe55c3615279aced15624cab3776370657e0478
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Jan 20 14:15:06 2012 +0100

    Dynamic mirror selection
    
    - Dropped the MG's 'max_connections' option, added a global one.
    - Single download queue, mirror selected as late as possible.
    - Merged request dispatch & flushing loops.
    
    Also, dropped the 'GrabRequest' instance, which originally contained
    a copy of all mirrors.  Replaced that with a set of failed mirrors.
    
    - No need to clone the mirror list to each request.
    - Now the master list could be shuffled during downloads,
      and further mirror selections could act on that.

commit 158d4c4b709a016c9e9a08a8ba828d61e9c5dd20
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Jan 23 16:54:01 2012 +0100

    Small fixes

commit 2ac493070881b217b381230e187e889919b39727
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 15:29:24 2011 +0100

    max_connections: updated semantics
    
    - mirror's max_connections default changed from 3 to 1.
    
    - added and documented MirrorGroup's max_connections option.
    
    - updated the code that selects initial mirrors for parallel downloads
      to adjust the number of mirrors used, honoring the MG's option.

commit 6adea0e757d92d9cc23021319bba4c9bebeb09c4
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 14:24:08 2011 +0100

    Fix ctrl-c tracebacks from downloader processes
    
    - handle broken pipe case
    
    - handle sigint early, so no KeyboardInterrupt
      inside pycurl callbacks that can't be caught.

commit 5e0700bd6ac23148314f348045111872048bc68b
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 15:13:51 2011 +0100

    Remove unnecessary try/except
    
    In pooled mode, _readlines won't block

commit 6021ebbe23423429dd6445909cc4f97b324fc57a
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:14:47 2011 +0100

    Use first 5 mirrors instead of just the 1st one.

commit 21c8d1f8ab2c0f62d9298108a24d7a3d7221e4de
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:44 2011 +0100

    Revert "move pycurl.error handling to _do_perform_exc()"
    
    This reverts commit 54ee979ada490cdae12fff038a838602fa404075.

commit 80bd911c671b0b93db41eef059cd1c490cc32d77
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:42 2011 +0100

    Revert "move opening of target file to _do_open_fo()."
    
    This reverts commit eb365596a44bd8ca15fe290d50290f0f84a268dd.

commit a9fdac39a0fdae5b8ac2174ca5b656f2e8e81415
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:33 2011 +0100

    Revert "move closing of target file to _do_close_fo()."
    
    This reverts commit 5443754de676ed4b173502522143460ae8a50013.

commit f24d38b29a6ee8ea8c88c11b51b5971e3f1fb0a0
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:11:36 2011 +0100

    assume ug_mode == 'pooled'

commit f9f70d6725571af21c7ed7d380bfc1b158823259
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 16:09:44 2011 +0100

    urlgrabber downloader mode selection
    
    Removed the parallel_wait() argument 'external',
    and the compile-time constant AVOID_CURL_MULTI.
    
    URLGRABBER_MODE selects the downloader mode:
    * 'compat': no parallel downloads
    * 'direct': use CurlMulti directly
    * 'extern': fork+exec a process that uses CurlMulti
    * 'pooled': multiple fork+exec'd processes

commit c34f90a53d33bcd063d89508fc40c6dbc31fc54b
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 15:38:47 2011 +0100

    _DirectDownloader{Single,Multi}
    
    Add some more code to _DirectDownloader class and rename it to
    '_DirectDownloaderMulti'.  Add a similiar class with the same API
    that downloads a single file at a time: _DirectDownloaderSingle.
    Downloader then picks the right class at runtime.

commit 79a2483cf0f391345cca6d41088f7c1ec3dcd168
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Nov 24 15:30:06 2011 +0100

    Initial epoll() support in ExternalDownloaderPool.
    
    Handy if there's 1000+ downloaders to handle :)

commit 9332b9e8cd79f6afe27dd5d1921546395e3b19d9
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Nov 22 14:37:43 2011 +0100

    AVOID_CURL_MULTI flag
    
    When using the 'external' download process it's not strictly necessary
    to use CurlMulti(), as we can use ordinary blocking code and just throw
    more processes on that.
    
    AVOID_CURL_MULTI = True: Each download runs is a separate process.
    Processes are reused when downloading files from the same host.
    
    AVOID_CURL_MULTI = False: Fork a single process that handles all the
    downloading.  Should be somewhat more efficient.

commit f35e3d5575335a7234b2a0c6ff073cc8664d2527
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Nov 7 11:02:33 2011 +0100

    Optional/throttled progress reporting in download_process
    
    CurlFileObject updates progress meter on every write.  _ProxyProgress
    pipes this to the parent, but it's often ignored there.
    
    - make updates conditional
    - throttle update rate at 0.31s

commit 063a7f02763cd22912a846f8d9ffcc07dab314cf
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 09:25:58 2011 +0100

    Improve ctrl-c handling
    
    We don't detach the downloader process from TTY, so it receives
    SIGINT as well, and may even exit sooner than Python raises
    KeyboardInterrupt in the parent process.
    
    - downloader: don't print ctrl-c traceback
    - parent: handle EINTR and EOF as ctrl-c

commit cd60b3d82f52d0230480b357d9b3568292ed70b4
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 21 10:42:59 2011 +0200

    External downloading
    
    Add 'external = True' flag to parallel_wait()
    to relay download requests to external process.

commit fb539deaf116b2b5d4fc7b45c0d7834689103166
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:35:15 2011 +0100

    Downloader process
    
    When executed with a single argument 'DOWNLOADER', grabber.py
    parses download requests on stdin, and reports the results to stdout.
    
    Conflicts:
    
    	urlgrabber/grabber.py

commit f978127f42ce510a7de8b48267a33762fad20ef4
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:17:02 2011 +0100

    _dumps + _loads: custom serializer/parser

commit 06624ba76f58b132ab6449000fcfd5165b1d45cc
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 13:54:43 2011 +0200

    Reuse curl objects (per host)

commit 811ab7ccac581538e294f8c0f2c1790e2f0fef5c
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 4 17:24:20 2011 +0200

    Implement parallel urlgrab()s
    
    opts.async = (key, limit):
        async urlgrab() with conn limiting.
    
    parallel_wait():
        wait untill all grabs have finished.

commit b01de1eabef26c92aa499a5498f703d764c67422
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 7 12:37:00 2011 +0200

    Obsolete the _make_callback() method
    
    Use _run_callback() instead.

commit ae3265686ea13aac5b0e93af75ba806dd146da34
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Sep 6 14:41:27 2011 +0200

    Implement 'failfunc' callback.
    
    This callback is called when urlgrab request fails.
    If grab is wrapped in a mirror group, only the mirror
    group issues the callback.

commit c5026d7eb19bd6f1631c45e43058338729f01b19
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 12:52:23 2011 +0200

    TextMultiFileMeter: minor tweaks
    
    remove _do_end(), because individual finished files were already
    handled in end_meter, and _do_update_meter(None) fails.
    
    remove _do_update_meter() at end of _do_end_meter().
    we already have bumped finished_files counter, and
    _do_update_meter() would report N+1 -th download.