[yum-commits] Changes to 'multi-downloader'

[zpavlas at osuosl.org]

New branch 'multi-downloader' available with the following commits:
commit 0fccfd96271941caae644dc755c5fd5678d8c1ce
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Mar 23 16:24:29 2012 +0100

    yum compatibility: raise KeyboardInterrupt instead of IOError(EINTR)

commit 4db684bfecb429e3b9a14c156491a84a79d5a2f1
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Mar 22 14:20:17 2012 +0100

    Parallel downloader: parse URL and update proxy

commit be218c38320077b1a3cca9854c7ac1a198c99b5b
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Mar 16 15:19:36 2012 +0100

    Remove "proxies" and "quote" options in downloader
    (already handled in parent process). Add a new option 'proxy'.

commit fb16e00b99f0f19733383b898735f5f75cb1d250
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Mar 16 15:16:00 2012 +0100

    Revert "Add 'dict' support to serializer."
    
    This reverts commit a4abdabf71451e03ea4a5f7b7ba3661cca129892.

commit 253f518c61251e52e2d231435afd0950ecdeed02
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 29 17:54:03 2012 +0100

    Add 'dict' support to serializer.

commit f42241d58108756724924ac87d046b3e4b993646
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 15 16:17:05 2012 +0100

    HTTP error code is optional

commit 62ef972c5a939f4138e302639ecd201650a6a52b
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Feb 1 16:33:20 2012 +0100

    prevent a race on timedhosts file

commit 8187852e2a848cff0f76ecc5c6a98aa3feb151dc
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 31 12:20:07 2012 +0100

    code reformat

commit e5d8b9920c387a2f40b12c4219ea1d33e43b4ea1
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 31 11:57:15 2012 +0100

    'timedhosts' option updates
    
    Global symbols now in a private _TH class.
    HTTP error 404 is not really a host failure.
    Added a new method estimate(url).

commit 2bff4dd242b78b597336877c97d2ab0be1e80c29
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Jan 26 15:08:01 2012 +0100

    timedhosts option, smarter mirror selection
    
    Added the 'timedhosts' option.
    
    (speed, failures, timestamp) tuple is assigned to each host
    we've downloaded from.  Successful downloads update 'speed'
    and reset the failure count to zero.  Failures increment
    the failure counter.  Speed update code:
    
    - assigns lower weight to old information
    - <1MB downloads are assumed not to be 100% accurate
    - simple 10:1 filtering
    
    Updated the mirror selection code.  Try to find the best one,
    using the above mentioned information.
    
    - every failure in a row halves the speed estimate
    - unknown or too old mirrors regress to default speed
    - pending downloads are assumed to eat bandwidth

commit bc212e3517ccf46298e2262d5ad0c1b273bee663
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Jan 25 16:06:32 2012 +0100

    Move the downloader code to /usr/libexec/urlgrabber-ext-down.
    
    This is much cleaner.  Also exec() the script instead of python,
    to make the api lang-neutral, and more selinux friendly.

commit 3df1684618c79b1c608cb4b3ac60568133da1d76
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Jan 24 15:44:35 2012 +0100

    Add internal 'timedhosts' callback.
    
    Gets called when a download (both sync and async ones) finishes.
    May be used to estimate mirror speeds and update their priority.
    
    - true download size is reported
    - checkfunc callback is not timed

commit 7cd5188791a087dd811e702590e779109c5af2e8
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Jan 20 14:15:06 2012 +0100

    Dynamic mirror selection
    
    - Dropped the MG's 'max_connections' option, added a global one.
    - Single download queue, mirror selected as late as possible.
    - Merged request dispatch & flushing loops.
    
    Also, dropped the 'GrabRequest' instance, which originally contained
    a copy of all mirrors.  Replaced that with a set of failed mirrors.
    
    - No need to clone the mirror list to each request.
    - Now the master list could be shuffled during downloads,
      and further mirror selections could act on that.

commit 04d6d5b7d4b7ee1379b54849aaff9acec492b5ff
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Jan 23 16:54:01 2012 +0100

    Small fixes

commit 59bc4b7be9f95fe02972977592f6d3aaaadf3a48
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 15:29:24 2011 +0100

    max_connections: updated semantics
    
    - mirror's max_connections default changed from 3 to 1.
    
    - added and documented MirrorGroup's max_connections option.
    
    - updated the code that selects initial mirrors for parallel downloads
      to adjust the number of mirrors used, honoring the MG's option.

commit d2c58a177129f2fda8dc613459c978902641a5d9
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Wed Dec 14 14:24:08 2011 +0100

    Fix ctrl-c tracebacks from downloader processes
    
    - handle broken pipe case
    
    - handle sigint early, so no KeyboardInterrupt
      inside pycurl callbacks that can't be caught.

commit c7c11419bb7258e7e03a68d4dbba41cd0f6ca2fd
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 15:13:51 2011 +0100

    Remove unnecessary try/except
    
    In pooled mode, _readlines won't block

commit 67ebe6deaf8f9563343aeca72d9f6418356f7b64
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:14:47 2011 +0100

    Use first 5 mirrors instead of just the 1st one.

commit 06b13c7adde5f6c369a1e2fa0d88f3a88a2b03eb
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:44 2011 +0100

    Revert "move pycurl.error handling to _do_perform_exc()"
    
    This reverts commit 54ee979ada490cdae12fff038a838602fa404075.

commit 4bd8d5be743b22ab93e3e7692442e63a85dd9f85
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:42 2011 +0100

    Revert "move opening of target file to _do_open_fo()."
    
    This reverts commit eb365596a44bd8ca15fe290d50290f0f84a268dd.

commit 2a38f74f868717f01c7c5285da1ce28625a6c6cb
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:13:33 2011 +0100

    Revert "move closing of target file to _do_close_fo()."
    
    This reverts commit 5443754de676ed4b173502522143460ae8a50013.

commit c7886bd605fe9407aa59b9e77915cf7b6cd3afd8
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Dec 9 14:11:36 2011 +0100

    assume ug_mode == 'pooled'

commit 559093778b9d097227b297891c8820d29ad2b613
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 16:09:44 2011 +0100

    urlgrabber downloader mode selection
    
    Removed the parallel_wait() argument 'external',
    and the compile-time constant AVOID_CURL_MULTI.
    
    URLGRABBER_MODE selects the downloader mode:
    * 'compat': no parallel downloads
    * 'direct': use CurlMulti directly
    * 'extern': fork+exec a process that uses CurlMulti
    * 'pooled': multiple fork+exec'd processes

commit 71689e1fea0a5932faf1c313dae333db262e684f
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Dec 6 15:38:47 2011 +0100

    _DirectDownloader{Single,Multi}
    
    Add some more code to _DirectDownloader class and rename it to
    '_DirectDownloaderMulti'.  Add a similiar class with the same API
    that downloads a single file at a time: _DirectDownloaderSingle.
    Downloader then picks the right class at runtime.

commit 6774c7381924edfd20aaa3c4daeb197c0f99cdbf
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Thu Nov 24 15:30:06 2011 +0100

    Initial epoll() support in ExternalDownloaderPool.
    
    Handy if there's 1000+ downloaders to handle :)

commit 3e220142575927e94baab6eca276e6ed214f5377
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Nov 22 14:37:43 2011 +0100

    AVOID_CURL_MULTI flag
    
    When using the 'external' download process it's not strictly necessary
    to use CurlMulti(), as we can use ordinary blocking code and just throw
    more processes on that.
    
    AVOID_CURL_MULTI = True: Each download runs is a separate process.
    Processes are reused when downloading files from the same host.
    
    AVOID_CURL_MULTI = False: Fork a single process that handles all the
    downloading.  Should be somewhat more efficient.

commit 3a77c4c5fbcbb22d9a4c068b53ab405d1693a042
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Mon Nov 7 11:02:33 2011 +0100

    Optional/throttled progress reporting in download_process
    
    CurlFileObject updates progress meter on every write.  _ProxyProgress
    pipes this to the parent, but it's often ignored there.
    
    - make updates conditional
    - throttle update rate at 0.31s

commit f150fbc013c5a110d479221980d4da800fedf62f
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 09:25:58 2011 +0100

    Improve ctrl-c handling
    
    We don't detach the downloader process from TTY, so it receives
    SIGINT as well, and may even exit sooner than Python raises
    KeyboardInterrupt in the parent process.
    
    - downloader: don't print ctrl-c traceback
    - parent: handle EINTR and EOF as ctrl-c

commit 4e829c4ed70f1f30049c818aa18c169c44f39670
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 21 10:42:59 2011 +0200

    External downloading
    
    Add 'external = True' flag to parallel_wait()
    to relay download requests to external process.

commit 094f3f48f8c84c8e5d95ca26fcd8be224f17fca3
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:35:15 2011 +0100

    Downloader process
    
    When executed with a single argument 'DOWNLOADER', grabber.py
    parses download requests on stdin, and reports the results to stdout.
    
    Conflicts:
    
    	urlgrabber/grabber.py

commit 91d9cd4b5dab2d9690e38f9d0c3c56ab828ad6e1
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Nov 4 08:17:02 2011 +0100

    _dumps + _loads: custom serializer/parser

commit 8fbef65d402120b9d8af766ea1b3ad608b00309a
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 13:54:43 2011 +0200

    Reuse curl objects (per host)

commit 03f42faf37f2cd0f8f402c44b9a106c0fdf94f20
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 4 17:24:20 2011 +0200

    Implement parallel urlgrab()s
    
    opts.async = (key, limit):
        async urlgrab() with conn limiting.
    
    parallel_wait():
        wait untill all grabs have finished.

commit 98fb6d5de23d4fea914ffba633fff328fff84068
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Fri Oct 7 12:37:00 2011 +0200

    Obsolete the _make_callback() method
    
    Use _run_callback() instead.

commit 392d79a5f679e45a260f8cd9238f97687f9aed9e
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Sep 6 14:41:27 2011 +0200

    Implement 'failfunc' callback.
    
    This callback is called when urlgrab request fails.
    If grab is wrapped in a mirror group, only the mirror
    group issues the callback.

commit 3d65dc137b5bc5f4fe83e63ccaa81fc14348f88c
Author: Zdeněk Pavlas <zpavlas at redhat.com>
Date:   Tue Oct 25 12:52:23 2011 +0200

    TextMultiFileMeter: minor tweaks
    
    remove _do_end(), because individual finished files were already
    handled in end_meter, and _do_update_meter(None) fails.
    
    remove _do_update_meter() at end of _do_end_meter().
    we already have bumped finished_files counter, and
    _do_update_meter() would report N+1 -th download.