[yum-commits] Changes to 'multi-downloader'

zpavlas at osuosl.org zpavlas at osuosl.org
Thu Mar 15 14:50:42 UTC 2012


New branch 'multi-downloader' available with the following commits:
commit 729448dd2edabafee598b83cfa4e712f2b40c7b1
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 29 17:54:03 2012 +0100

    Add 'dict' support to serializer.

commit b883272b95b0bfc74a1337df8bab7d0b16bbbcff
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 15 16:17:05 2012 +0100

    HTTP error code is optional

commit dee892d45b7c75ee56e4abb045a5b183fb239545
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 1 16:33:20 2012 +0100

    prevent a race on timedhosts file

commit efb112c70d8771609ad6cc86f05e0234ed94b4e7
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 31 12:20:07 2012 +0100

    code reformat

commit dd7329396a6fdb806d3d9aa9c5890fbebb23ddae
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 31 11:57:15 2012 +0100

    'timedhosts' option updates
    
    Global symbols now in a private _TH class.
    HTTP error 404 is not really a host failure.
    Added a new method estimate(url).

commit 32bde497de32e707506af2cba9c6b64aea121b64
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Jan 26 15:08:01 2012 +0100

    timedhosts option, smarter mirror selection
    
    Added the 'timedhosts' option.
    
    (speed, failures, timestamp) tuple is assigned to each host
    we've downloaded from.  Successful downloads update 'speed'
    and reset the failure count to zero.  Failures increment
    the failure counter.  Speed update code:
    
    - assigns lower weight to old information
    - <1MB downloads are assumed not to be 100% accurate
    - simple 10:1 filtering
    
    Updated the mirror selection code.  Try to find the best one,
    using the above mentioned information.
    
    - every failure in a row halves the speed estimate
    - unknown or too old mirrors regress to default speed
    - pending downloads are assumed to eat bandwidth

commit f408e297e43019f926bc2533d98ce9eb38b6a8b2
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Jan 25 16:06:32 2012 +0100

    Move the downloader code to /usr/libexec/urlgrabber-ext-down.
    
    This is much cleaner.  Also exec() the script instead of python,
    to make the api lang-neutral, and more selinux friendly.

commit 85fe7e55bb717854b80a9f9a4d8979c10a2a1415
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 24 15:44:35 2012 +0100

    Add internal 'timedhosts' callback.
    
    Gets called when a download (both sync and async ones) finishes.
    May be used to estimate mirror speeds and update their priority.
    
    - true download size is reported
    - checkfunc callback is not timed

commit 505ee43f04a8c470316ab926a75dc7f7039b635d
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Jan 20 14:15:06 2012 +0100

    Dynamic mirror selection
    
    - Dropped the MG's 'max_connections' option, added a global one.
    - Single download queue, mirror selected as late as possible.
    - Merged request dispatch & flushing loops.
    
    Also, dropped the 'GrabRequest' instance, which originally contained
    a copy of all mirrors.  Replaced that with a set of failed mirrors.
    
    - No need to clone the mirror list to each request.
    - Now the master list could be shuffled during downloads,
      and further mirror selections could act on that.

commit 52d57ae46743f5ab0bf2554f447a5299c0cd26c6
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Jan 23 16:54:01 2012 +0100

    Small fixes

commit e9491c6f593630db92fdad4a8907ba312ebbc586
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 15:29:24 2011 +0100

    max_connections: updated semantics
    
    - mirror's max_connections default changed from 3 to 1.
    
    - added and documented MirrorGroup's max_connections option.
    
    - updated the code that selects initial mirrors for parallel downloads
      to adjust the number of mirrors used, honoring the MG's option.

commit 3be5fb9b3479c03cd815f63e53bb3e18489b1e3b
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 14:24:08 2011 +0100

    Fix ctrl-c tracebacks from downloader processes
    
    - handle broken pipe case
    
    - handle sigint early, so no KeyboardInterrupt
      inside pycurl callbacks that can't be caught.

commit 255fbd5e85a16b66dc150c9461956b1f5ea7a670
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 15:13:51 2011 +0100

    Remove unnecessary try/except
    
    In pooled mode, _readlines won't block

commit a75bf4d5c7a0a7b97c3d6049a6fc8659de030e97
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:14:47 2011 +0100

    Use first 5 mirrors instead of just the 1st one.

commit b67724c9f61109c2879b029b9d0955d13979c7b3
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:44 2011 +0100

    Revert "move pycurl.error handling to _do_perform_exc()"
    
    This reverts commit 54ee979ada490cdae12fff038a838602fa404075.

commit 11cac9c4bbeea57b8cdd0b2103aa4588a1015665
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:42 2011 +0100

    Revert "move opening of target file to _do_open_fo()."
    
    This reverts commit eb365596a44bd8ca15fe290d50290f0f84a268dd.

commit 6b84b9c66e5e7216e2a478775bc9595b82a5c783
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:33 2011 +0100

    Revert "move closing of target file to _do_close_fo()."
    
    This reverts commit 5443754de676ed4b173502522143460ae8a50013.

commit 07b5cfc2fee5a35b44c4118d4064e76f0e503bf8
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:11:36 2011 +0100

    assume ug_mode == 'pooled'

commit 25ce104203acce0f54642e115a596b100c49484b
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 16:09:44 2011 +0100

    urlgrabber downloader mode selection
    
    Removed the parallel_wait() argument 'external',
    and the compile-time constant AVOID_CURL_MULTI.
    
    URLGRABBER_MODE selects the downloader mode:
    * 'compat': no parallel downloads
    * 'direct': use CurlMulti directly
    * 'extern': fork+exec a process that uses CurlMulti
    * 'pooled': multiple fork+exec'd processes

commit c316adea07eaf4bcbae676ca6d413919da099b63
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 15:38:47 2011 +0100

    _DirectDownloader{Single,Multi}
    
    Add some more code to _DirectDownloader class and rename it to
    '_DirectDownloaderMulti'.  Add a similiar class with the same API
    that downloads a single file at a time: _DirectDownloaderSingle.
    Downloader then picks the right class at runtime.

commit 6682584d7c91d21d62a77a6d30036e09acfc7be2
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Nov 24 15:30:06 2011 +0100

    Initial epoll() support in ExternalDownloaderPool.
    
    Handy if there's 1000+ downloaders to handle :)

commit 1244f2758e3cc2ea17b6e7d2e61a6ff54ae9f2aa
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Nov 22 14:37:43 2011 +0100

    AVOID_CURL_MULTI flag
    
    When using the 'external' download process it's not strictly necessary
    to use CurlMulti(), as we can use ordinary blocking code and just throw
    more processes on that.
    
    AVOID_CURL_MULTI = True: Each download runs is a separate process.
    Processes are reused when downloading files from the same host.
    
    AVOID_CURL_MULTI = False: Fork a single process that handles all the
    downloading.  Should be somewhat more efficient.

commit f7d6737c80bd61119e11a0b578e4e8260e6455de
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Nov 7 11:02:33 2011 +0100

    Optional/throttled progress reporting in download_process
    
    CurlFileObject updates progress meter on every write.  _ProxyProgress
    pipes this to the parent, but it's often ignored there.
    
    - make updates conditional
    - throttle update rate at 0.31s

commit a88674cda98a7eb05eab9b8b1c6d9773524ebc09
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 09:25:58 2011 +0100

    Improve ctrl-c handling
    
    We don't detach the downloader process from TTY, so it receives
    SIGINT as well, and may even exit sooner than Python raises
    KeyboardInterrupt in the parent process.
    
    - downloader: don't print ctrl-c traceback
    - parent: handle EINTR and EOF as ctrl-c

commit 2a4bedc6774e16f3dceebe9bfcbe013978138191
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 21 10:42:59 2011 +0200

    External downloading
    
    Add 'external = True' flag to parallel_wait()
    to relay download requests to external process.

commit 7807c2f4f5fcd2d758b483ad50a814f738c173ed
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:35:15 2011 +0100

    Downloader process
    
    When executed with a single argument 'DOWNLOADER', grabber.py
    parses download requests on stdin, and reports the results to stdout.
    
    Conflicts:
    
    	urlgrabber/grabber.py

commit c1773c4e912bc9092c856cf743fa7912cf656282
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:17:02 2011 +0100

    _dumps + _loads: custom serializer/parser

commit e0edac874e2289180434a29d1d007a4c0940c3d9
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 13:54:43 2011 +0200

    Reuse curl objects (per host)

commit ad7ea54698b49f6018a9a4afc6af83ba2b0eb22d
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 4 17:24:20 2011 +0200

    Implement parallel urlgrab()s
    
    opts.async = (key, limit):
        async urlgrab() with conn limiting.
    
    parallel_wait():
        wait untill all grabs have finished.

commit 9ee06565f032d50e2a0dae9713b75e913ffcdd23
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 7 12:37:00 2011 +0200

    Obsolete the _make_callback() method
    
    Use _run_callback() instead.

commit 6de324d7ef5787c28b2f7b95c56b0ea6ef8ad7a0
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Sep 6 14:41:27 2011 +0200

    Implement 'failfunc' callback.
    
    This callback is called when urlgrab request fails.
    If grab is wrapped in a mirror group, only the mirror
    group issues the callback.

commit e5c03d5a46659e968f12db4b9e1f88be40aa1775
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 12:52:23 2011 +0200

    TextMultiFileMeter: minor tweaks
    
    remove _do_end(), because individual finished files were already
    handled in end_meter, and _do_update_meter(None) fails.
    
    remove _do_update_meter() at end of _do_end_meter().
    we already have bumped finished_files counter, and
    _do_update_meter() would report N+1 -th download.



More information about the Yum-commits mailing list