[yum-commits] Changes to 'multi-downloader'
zpavlas at osuosl.org
zpavlas at osuosl.org
Thu Mar 15 14:50:42 UTC 2012
New branch 'multi-downloader' available with the following commits:
commit 729448dd2edabafee598b83cfa4e712f2b40c7b1
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Wed Feb 29 17:54:03 2012 +0100
Add 'dict' support to serializer.
commit b883272b95b0bfc74a1337df8bab7d0b16bbbcff
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Wed Feb 15 16:17:05 2012 +0100
HTTP error code is optional
commit dee892d45b7c75ee56e4abb045a5b183fb239545
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Wed Feb 1 16:33:20 2012 +0100
prevent a race on timedhosts file
commit efb112c70d8771609ad6cc86f05e0234ed94b4e7
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Jan 31 12:20:07 2012 +0100
code reformat
commit dd7329396a6fdb806d3d9aa9c5890fbebb23ddae
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Jan 31 11:57:15 2012 +0100
'timedhosts' option updates
Global symbols now in a private _TH class.
HTTP error 404 is not really a host failure.
Added a new method estimate(url).
commit 32bde497de32e707506af2cba9c6b64aea121b64
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Thu Jan 26 15:08:01 2012 +0100
timedhosts option, smarter mirror selection
Added the 'timedhosts' option.
(speed, failures, timestamp) tuple is assigned to each host
we've downloaded from. Successful downloads update 'speed'
and reset the failure count to zero. Failures increment
the failure counter. Speed update code:
- assigns lower weight to old information
- <1MB downloads are assumed not to be 100% accurate
- simple 10:1 filtering
Updated the mirror selection code. Try to find the best one,
using the above mentioned information.
- every failure in a row halves the speed estimate
- unknown or too old mirrors regress to default speed
- pending downloads are assumed to eat bandwidth
commit f408e297e43019f926bc2533d98ce9eb38b6a8b2
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Wed Jan 25 16:06:32 2012 +0100
Move the downloader code to /usr/libexec/urlgrabber-ext-down.
This is much cleaner. Also exec() the script instead of python,
to make the api lang-neutral, and more selinux friendly.
commit 85fe7e55bb717854b80a9f9a4d8979c10a2a1415
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Jan 24 15:44:35 2012 +0100
Add internal 'timedhosts' callback.
Gets called when a download (both sync and async ones) finishes.
May be used to estimate mirror speeds and update their priority.
- true download size is reported
- checkfunc callback is not timed
commit 505ee43f04a8c470316ab926a75dc7f7039b635d
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Jan 20 14:15:06 2012 +0100
Dynamic mirror selection
- Dropped the MG's 'max_connections' option, added a global one.
- Single download queue, mirror selected as late as possible.
- Merged request dispatch & flushing loops.
Also, dropped the 'GrabRequest' instance, which originally contained
a copy of all mirrors. Replaced that with a set of failed mirrors.
- No need to clone the mirror list to each request.
- Now the master list could be shuffled during downloads,
and further mirror selections could act on that.
commit 52d57ae46743f5ab0bf2554f447a5299c0cd26c6
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Mon Jan 23 16:54:01 2012 +0100
Small fixes
commit e9491c6f593630db92fdad4a8907ba312ebbc586
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Wed Dec 14 15:29:24 2011 +0100
max_connections: updated semantics
- mirror's max_connections default changed from 3 to 1.
- added and documented MirrorGroup's max_connections option.
- updated the code that selects initial mirrors for parallel downloads
to adjust the number of mirrors used, honoring the MG's option.
commit 3be5fb9b3479c03cd815f63e53bb3e18489b1e3b
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Wed Dec 14 14:24:08 2011 +0100
Fix ctrl-c tracebacks from downloader processes
- handle broken pipe case
- handle sigint early, so no KeyboardInterrupt
inside pycurl callbacks that can't be caught.
commit 255fbd5e85a16b66dc150c9461956b1f5ea7a670
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Dec 9 15:13:51 2011 +0100
Remove unnecessary try/except
In pooled mode, _readlines won't block
commit a75bf4d5c7a0a7b97c3d6049a6fc8659de030e97
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Dec 9 14:14:47 2011 +0100
Use first 5 mirrors instead of just the 1st one.
commit b67724c9f61109c2879b029b9d0955d13979c7b3
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Dec 9 14:13:44 2011 +0100
Revert "move pycurl.error handling to _do_perform_exc()"
This reverts commit 54ee979ada490cdae12fff038a838602fa404075.
commit 11cac9c4bbeea57b8cdd0b2103aa4588a1015665
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Dec 9 14:13:42 2011 +0100
Revert "move opening of target file to _do_open_fo()."
This reverts commit eb365596a44bd8ca15fe290d50290f0f84a268dd.
commit 6b84b9c66e5e7216e2a478775bc9595b82a5c783
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Dec 9 14:13:33 2011 +0100
Revert "move closing of target file to _do_close_fo()."
This reverts commit 5443754de676ed4b173502522143460ae8a50013.
commit 07b5cfc2fee5a35b44c4118d4064e76f0e503bf8
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Dec 9 14:11:36 2011 +0100
assume ug_mode == 'pooled'
commit 25ce104203acce0f54642e115a596b100c49484b
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Dec 6 16:09:44 2011 +0100
urlgrabber downloader mode selection
Removed the parallel_wait() argument 'external',
and the compile-time constant AVOID_CURL_MULTI.
URLGRABBER_MODE selects the downloader mode:
* 'compat': no parallel downloads
* 'direct': use CurlMulti directly
* 'extern': fork+exec a process that uses CurlMulti
* 'pooled': multiple fork+exec'd processes
commit c316adea07eaf4bcbae676ca6d413919da099b63
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Dec 6 15:38:47 2011 +0100
_DirectDownloader{Single,Multi}
Add some more code to _DirectDownloader class and rename it to
'_DirectDownloaderMulti'. Add a similiar class with the same API
that downloads a single file at a time: _DirectDownloaderSingle.
Downloader then picks the right class at runtime.
commit 6682584d7c91d21d62a77a6d30036e09acfc7be2
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Thu Nov 24 15:30:06 2011 +0100
Initial epoll() support in ExternalDownloaderPool.
Handy if there's 1000+ downloaders to handle :)
commit 1244f2758e3cc2ea17b6e7d2e61a6ff54ae9f2aa
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Nov 22 14:37:43 2011 +0100
AVOID_CURL_MULTI flag
When using the 'external' download process it's not strictly necessary
to use CurlMulti(), as we can use ordinary blocking code and just throw
more processes on that.
AVOID_CURL_MULTI = True: Each download runs is a separate process.
Processes are reused when downloading files from the same host.
AVOID_CURL_MULTI = False: Fork a single process that handles all the
downloading. Should be somewhat more efficient.
commit f7d6737c80bd61119e11a0b578e4e8260e6455de
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Mon Nov 7 11:02:33 2011 +0100
Optional/throttled progress reporting in download_process
CurlFileObject updates progress meter on every write. _ProxyProgress
pipes this to the parent, but it's often ignored there.
- make updates conditional
- throttle update rate at 0.31s
commit a88674cda98a7eb05eab9b8b1c6d9773524ebc09
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Nov 4 09:25:58 2011 +0100
Improve ctrl-c handling
We don't detach the downloader process from TTY, so it receives
SIGINT as well, and may even exit sooner than Python raises
KeyboardInterrupt in the parent process.
- downloader: don't print ctrl-c traceback
- parent: handle EINTR and EOF as ctrl-c
commit 2a4bedc6774e16f3dceebe9bfcbe013978138191
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Oct 21 10:42:59 2011 +0200
External downloading
Add 'external = True' flag to parallel_wait()
to relay download requests to external process.
commit 7807c2f4f5fcd2d758b483ad50a814f738c173ed
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Nov 4 08:35:15 2011 +0100
Downloader process
When executed with a single argument 'DOWNLOADER', grabber.py
parses download requests on stdin, and reports the results to stdout.
Conflicts:
urlgrabber/grabber.py
commit c1773c4e912bc9092c856cf743fa7912cf656282
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Nov 4 08:17:02 2011 +0100
_dumps + _loads: custom serializer/parser
commit e0edac874e2289180434a29d1d007a4c0940c3d9
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Oct 25 13:54:43 2011 +0200
Reuse curl objects (per host)
commit ad7ea54698b49f6018a9a4afc6af83ba2b0eb22d
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Oct 4 17:24:20 2011 +0200
Implement parallel urlgrab()s
opts.async = (key, limit):
async urlgrab() with conn limiting.
parallel_wait():
wait untill all grabs have finished.
commit 9ee06565f032d50e2a0dae9713b75e913ffcdd23
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Fri Oct 7 12:37:00 2011 +0200
Obsolete the _make_callback() method
Use _run_callback() instead.
commit 6de324d7ef5787c28b2f7b95c56b0ea6ef8ad7a0
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Sep 6 14:41:27 2011 +0200
Implement 'failfunc' callback.
This callback is called when urlgrab request fails.
If grab is wrapped in a mirror group, only the mirror
group issues the callback.
commit e5c03d5a46659e968f12db4b9e1f88be40aa1775
Author: ZdenÄk Pavlas <zpavlas at redhat.com>
Date: Tue Oct 25 12:52:23 2011 +0200
TextMultiFileMeter: minor tweaks
remove _do_end(), because individual finished files were already
handled in end_meter, and _do_update_meter(None) fails.
remove _do_update_meter() at end of _do_end_meter().
we already have bumped finished_files counter, and
_do_update_meter() would report N+1 -th download.
More information about the Yum-commits
mailing list