[yum-commits] Changes to 'multi-downloader'

[zpavlas at osuosl.org]

New branch 'multi-downloader' available with the following commits:
commit cd75b47d9527fbd55dfbadc96fc3ceaaefdbc75e
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Mar 16 15:19:36 2012 +0100

    Remove "proxies" and "quote" options in downloader
    (already handled in parent process)

commit fc09903a5aff03c11734d1f41ecd4c4507925f83
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Mar 16 15:16:00 2012 +0100

    Revert "Add 'dict' support to serializer."
    
    This reverts commit a4abdabf71451e03ea4a5f7b7ba3661cca129892.

commit a4abdabf71451e03ea4a5f7b7ba3661cca129892
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 29 17:54:03 2012 +0100

    Add 'dict' support to serializer.

commit a6ae5620d24cee40e0e28978f6b57e382db4f186
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 15 16:17:05 2012 +0100

    HTTP error code is optional

commit 9dc15dbff18deee87b14cc14af6d656d11ccf3ed
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 1 16:33:20 2012 +0100

    prevent a race on timedhosts file

commit d8025c401d99dde82f66032a59b9a8392afe045e
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 31 12:20:07 2012 +0100

    code reformat

commit ed61442502f8df1a57ba45ba0d7b9d4440b2eabb
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 31 11:57:15 2012 +0100

    'timedhosts' option updates
    
    Global symbols now in a private _TH class.
    HTTP error 404 is not really a host failure.
    Added a new method estimate(url).

commit 8f46880a1f40585b36fa3459c3c8f63d4d3ad829
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Jan 26 15:08:01 2012 +0100

    timedhosts option, smarter mirror selection
    
    Added the 'timedhosts' option.
    
    (speed, failures, timestamp) tuple is assigned to each host
    we've downloaded from.  Successful downloads update 'speed'
    and reset the failure count to zero.  Failures increment
    the failure counter.  Speed update code:
    
    - assigns lower weight to old information
    - <1MB downloads are assumed not to be 100% accurate
    - simple 10:1 filtering
    
    Updated the mirror selection code.  Try to find the best one,
    using the above mentioned information.
    
    - every failure in a row halves the speed estimate
    - unknown or too old mirrors regress to default speed
    - pending downloads are assumed to eat bandwidth

commit 6d10fd0561a05143dd1d4b373ea8557154149b7d
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Jan 25 16:06:32 2012 +0100

    Move the downloader code to /usr/libexec/urlgrabber-ext-down.
    
    This is much cleaner.  Also exec() the script instead of python,
    to make the api lang-neutral, and more selinux friendly.

commit 139b73068cf1f62158d58bce9c40db98e377d3f9
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 24 15:44:35 2012 +0100

    Add internal 'timedhosts' callback.
    
    Gets called when a download (both sync and async ones) finishes.
    May be used to estimate mirror speeds and update their priority.
    
    - true download size is reported
    - checkfunc callback is not timed

commit da176a1e7b61be86f899070320ccd9fae3dfef3a
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Jan 20 14:15:06 2012 +0100

    Dynamic mirror selection
    
    - Dropped the MG's 'max_connections' option, added a global one.
    - Single download queue, mirror selected as late as possible.
    - Merged request dispatch & flushing loops.
    
    Also, dropped the 'GrabRequest' instance, which originally contained
    a copy of all mirrors.  Replaced that with a set of failed mirrors.
    
    - No need to clone the mirror list to each request.
    - Now the master list could be shuffled during downloads,
      and further mirror selections could act on that.

commit a3c4135a98338647291bc7bf964e86cbb96b5196
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Jan 23 16:54:01 2012 +0100

    Small fixes

commit e7c6e0565a6064163140c272140910e197afd2a1
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 15:29:24 2011 +0100

    max_connections: updated semantics
    
    - mirror's max_connections default changed from 3 to 1.
    
    - added and documented MirrorGroup's max_connections option.
    
    - updated the code that selects initial mirrors for parallel downloads
      to adjust the number of mirrors used, honoring the MG's option.

commit be78c8fe2beedb7032353b0716a3d05fde4c9a38
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 14:24:08 2011 +0100

    Fix ctrl-c tracebacks from downloader processes
    
    - handle broken pipe case
    
    - handle sigint early, so no KeyboardInterrupt
      inside pycurl callbacks that can't be caught.

commit c7bff4aeae9506b4728eb949f31fbc4862aafc42
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 15:13:51 2011 +0100

    Remove unnecessary try/except
    
    In pooled mode, _readlines won't block

commit f6a8a61969ed0f3946b4dc8b2a5eebcc5f2c60b2
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:14:47 2011 +0100

    Use first 5 mirrors instead of just the 1st one.

commit 4ec597507152c455893a941ef9f344c95bda1b33
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:44 2011 +0100

    Revert "move pycurl.error handling to _do_perform_exc()"
    
    This reverts commit 54ee979ada490cdae12fff038a838602fa404075.

commit f9100560f1176e6ffdc6fa816ac2f55ceb0294b4
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:42 2011 +0100

    Revert "move opening of target file to _do_open_fo()."
    
    This reverts commit eb365596a44bd8ca15fe290d50290f0f84a268dd.

commit e22053a26c5a6bb21f0a786f18ddf9ceb8133232
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:33 2011 +0100

    Revert "move closing of target file to _do_close_fo()."
    
    This reverts commit 5443754de676ed4b173502522143460ae8a50013.

commit df297eaa832749abf31bc3f54bba33c485bc2f91
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:11:36 2011 +0100

    assume ug_mode == 'pooled'

commit 4f8914e253b8de2544cbf50c856f63870c486809
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 16:09:44 2011 +0100

    urlgrabber downloader mode selection
    
    Removed the parallel_wait() argument 'external',
    and the compile-time constant AVOID_CURL_MULTI.
    
    URLGRABBER_MODE selects the downloader mode:
    * 'compat': no parallel downloads
    * 'direct': use CurlMulti directly
    * 'extern': fork+exec a process that uses CurlMulti
    * 'pooled': multiple fork+exec'd processes

commit 1e60545a2cf929da3327b0c7a652bf1e3db51ae3
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 15:38:47 2011 +0100

    _DirectDownloader{Single,Multi}
    
    Add some more code to _DirectDownloader class and rename it to
    '_DirectDownloaderMulti'.  Add a similiar class with the same API
    that downloads a single file at a time: _DirectDownloaderSingle.
    Downloader then picks the right class at runtime.

commit be8f5d559fd58587702d0193dd47793fc1eb3fbb
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Nov 24 15:30:06 2011 +0100

    Initial epoll() support in ExternalDownloaderPool.
    
    Handy if there's 1000+ downloaders to handle :)

commit 4df84f8152788c9fc58b73f7b181194c5dbf822c
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Nov 22 14:37:43 2011 +0100

    AVOID_CURL_MULTI flag
    
    When using the 'external' download process it's not strictly necessary
    to use CurlMulti(), as we can use ordinary blocking code and just throw
    more processes on that.
    
    AVOID_CURL_MULTI = True: Each download runs is a separate process.
    Processes are reused when downloading files from the same host.
    
    AVOID_CURL_MULTI = False: Fork a single process that handles all the
    downloading.  Should be somewhat more efficient.

commit 7f5a77ac8772a5965b8b5932ed4ae57579c7cdf3
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Nov 7 11:02:33 2011 +0100

    Optional/throttled progress reporting in download_process
    
    CurlFileObject updates progress meter on every write.  _ProxyProgress
    pipes this to the parent, but it's often ignored there.
    
    - make updates conditional
    - throttle update rate at 0.31s

commit 87609be67be6f7699f87996b1325f0a94b486366
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 09:25:58 2011 +0100

    Improve ctrl-c handling
    
    We don't detach the downloader process from TTY, so it receives
    SIGINT as well, and may even exit sooner than Python raises
    KeyboardInterrupt in the parent process.
    
    - downloader: don't print ctrl-c traceback
    - parent: handle EINTR and EOF as ctrl-c

commit a05b354eb1a3a6a3761a54b9892667d849edb835
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 21 10:42:59 2011 +0200

    External downloading
    
    Add 'external = True' flag to parallel_wait()
    to relay download requests to external process.

commit 7b103d443eeb8feb78ed258d353298b26b795a46
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:35:15 2011 +0100

    Downloader process
    
    When executed with a single argument 'DOWNLOADER', grabber.py
    parses download requests on stdin, and reports the results to stdout.
    
    Conflicts:
    
    	urlgrabber/grabber.py

commit 4f9d7c0b5624eeac87887d0b4dcbeb3aa322d05f
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:17:02 2011 +0100

    _dumps + _loads: custom serializer/parser

commit 4fa58857708b12b91686732b27ceb04e21a1da01
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 13:54:43 2011 +0200

    Reuse curl objects (per host)

commit 57ad1bdd9ce0eb9edddefa040acf12524124b336
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 4 17:24:20 2011 +0200

    Implement parallel urlgrab()s
    
    opts.async = (key, limit):
        async urlgrab() with conn limiting.
    
    parallel_wait():
        wait untill all grabs have finished.

commit e0ea7ed945ec8e95b16238730ce8b7ac8aad31a8
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 7 12:37:00 2011 +0200

    Obsolete the _make_callback() method
    
    Use _run_callback() instead.

commit 6ba0df2699ee24796870bcb5883460534a5aac16
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Sep 6 14:41:27 2011 +0200

    Implement 'failfunc' callback.
    
    This callback is called when urlgrab request fails.
    If grab is wrapped in a mirror group, only the mirror
    group issues the callback.

commit cc4643d4ca8b28be24493e76ce04c3883cc2fff7
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 12:52:23 2011 +0200

    TextMultiFileMeter: minor tweaks
    
    remove _do_end(), because individual finished files were already
    handled in end_meter, and _do_update_meter(None) fails.
    
    remove _do_update_meter() at end of _do_end_meter().
    we already have bumped finished_files counter, and
    _do_update_meter() would report N+1 -th download.