[yum-commits] Changes to 'refs/tags/urlgrabber-3-9-0'

skvidal at osuosl.org skvidal at osuosl.org
Fri Jul 31 20:21:41 UTC 2009


Tag 'urlgrabber-3-9-0' created by Seth Vidal <skvidal at fedoraproject.org> at 2009-07-31 20:19 +0000

urlgrabber 3.9.0

Changes since the dawn of time:
James Antill (2):
      Import progress patches from Fedora.
      Import fix for ftp ports in old urilib code (probably worthless now, but meh)

Seth Vidal (16):
      modify urlgraberror so it has a url attribute and includes the url in all
      make it use *args instead of silly if statements
      implement PyCurlFileObject. This makes the default and forklifts all the code to
      get the content-length/size for ftp pkgs too
      implement throttle/bandwidth controls in pycurl
      - add range support
      - make simple/most proxies work
      - make makefile work again without using cvs
      - clean up some unused files
      when I first started this I hacked something into URLGrabberFileObject - this reverts that hack
      add git2cl
      - make basic posts work
      - make readlines() work for mirrorlists in yum (which probably shouldn't be using it anyway)
      update changelog
      add a few more things to be cleaned out
      changelog + release date touchup

mstenner (113):
      Initial revision
      created test_urlgrabber.py and started moving tests into it
      These are the changes from when ryan was working in a separate repo.
      Edited the cvs repo to restructure a little bit.  All module python
      Added mirror code and associated test code.
      [TODO] added not about restructuring import style
      Updated MANIFEST.in and README.  Created 'maint' directory containing
      Added first attempt at reget support.  Created test suite for http.
      Fixed a typo in mirror.py.  Added reget docs and reformatted module-level
      Updated TODO.
      Added .cvsignore to urlgrabber/ directory
      Fixed a stupid typo in the splituser fix (from 5 minutes ago)
      added keepalive tests; fixed test_bad_reget_type
      added "failure_callback" options to both MirrorGroup and URLGrabber.  These
      added callback test to test_mirror.py.  Restructuring the test code a
      making tests a little nicer
      updated TODO
      converted test_byterange.py to use base_test_code
      improved mirror.py docs, added MGRandomStart (which is equivalent to yum's
      Put version, date, url, and author information into __init__.py and
      updated TODO
      Added Id: strings to distributed python files, made grabber.py use the
      updated TODO after alpha-huddle
      Changed test base_url according to new web page.  Added prefix
      Made the control of MirrorGroup failover action WAY cooler and more flexible.
      Added a little more documentation.
      Made tests prettier (again... I do like pretty tests) and created the test
      By popular demand, I broke out the unittest coolness into a separate
      Added file and ftp tests for reget.
      In grabber.py, fixed a bug with urlread when no limit is specified.
      Made tests a little cleaner and added "none" test, which is a
      OK, I THINK I've made keepalive thread-friendly.  The interface is
      Rearranged keepalive a little to make it nicer.  Also made it deal
      added a simple threading test
      Updated ChangeLog, TODO.  Played with the batchgrabber stuff a little.
      Tidied up some of the callback code.
      Changed the wait time for the dropped-connection test.  The apache
      Fixed documentation error in the range docs.  Temporarily disabled the
      Added "make test" target.
      Modified "make ChangeLog" target so that it doesn't print the times.  If
      updated TODO
      Fixed a few bugs related to python 2.3.
      fixed minor typo in opener option docs
      Replace LICENSE with the LGPL. Added maint/license-notice, which
      Tidied up the grabber tests so we're not always skipping those ftp reget
      Committing new progress meter code.  This new version includes an emulation
      fixed documentation typos
      Fixed a bug in URLGrabber._retry that prevented failure callbacks from
      fixed typo in _make_callback and added testcase
      enhanced the failover test to ensure that the bad mirror is tried and fails
      Changed the callback syntax.  This affects failure_callback use in
      Updated callback tests to accomodate the new callback semantics.  Also added
      Added "release" and "daily" targets to makefile for automated releases.
      release 2.9.0
      updated ChangeLog
      Fixed a typo in the new 'release' target.  Reformatted the doc in __init__.py
      Added "MANIFEST" to MANIFEST.in.  Without this, one couldn't correctly
      Added Ryan's progress changes in.  That still has some quirks I need
      Fixed python 2.4 bug - added .code attribute to returned file objects.
      release 2.9.1
      updated ChangeLog
      slight cleanup - no new code
      Added http character encoding patch from Chris Lumens.
      release 2.9.2
      updated ChangeLog
      Fixed minor exception bug.
      updated project url
      updated README with instructions for building rpms and notes about
      release 2.9.3
      updated ChangeLog
      Added data and instructions for setting up ftp and proxy servers for
      Added http_headers and ftp_headers options.
      release 2.9.4
      updated ChangeLog
      Fixed a bug caused by the recent proxy fix for python 2.2.  This new
      release 2.9.5
      updated ChangeLog
      Changed development status to Beta an fixed a small build bug.  It
      release 2.9.6
      updated ChangeLog
      Added tests to check proxy format.  Otherwise, missing proto could
      Changed the version checking so SUSE's python "2.3+" version string
      Fixed a typo in that last change.  Always remember to do "make test" before
      started using sys.version_info rather than parsing sys.version
      Fixed two namespace-related bugs.
      except HTTPError, which basically means all HTTP return codes that are
      Added Menno's idea for catching KeyboardInterrupt.  This allows the
      Fixed a minor formatting bug, helpfully pointed out by the nice folks
      Another minor simplification offered by the WTF folks.
      Added some slick logging support to urlgrabber.  It can be controlled
      Updated TODO and preparing for release.
      updated ChangeLog
      Added a reget progress bar patch from Menno, and fixed
      release 2.9.8
      updated ChangeLog
      Significant improvement to URL parsing.  Parsing is now broken out into
      Added tests to make sure that the "quote" option works as advertised.
      release 2.9.9
      updated ChangeLog
      Patch from Jeremy Katz to catch read errors after the file has been opened.
      Fixed a minor error reporting bug due to changes in python 2.4.
      Added patch from James Bowes to make keepalive, byteranges, etc. work
      release 2.9.10
      updated ChangeLog
      release 3.0.0
      updated ChangeLog
      Added James Bowes' SSL patch to use M2Crypto when available.
      release 3.1.0
      updated ChangeLog
      Minor fix to make byteranges work with some servers.  _do_grab now only
      Minor doc updates and error handling in grabber.py.  Complete rewrite of
      lots of changes... improved clarity of cached objects, improved debugging
      more debugging code to expose options

rtomayko (48):
      Mass commit/merge of changes made for urlgrabber restructuring.. See ChangeLog for details.
      Restructured TODO list based on recent email from Michael.
      Added item about handling KeyboardInterupt exceptions.
      Changed all variables/args named 'range' to rangetup or brange to avoid possible python keyword clash.
      Added checkfunc support to urlread.
      Brought in fix from yum for user:pass escaping in URLs.
      Added reget and batch stuff under the 'maybe' section.
      Clean up temp files after testing urlgrab.
      First cut at a module to run through some performance tests for grabber.py
      Removed speed test from grabber.py. This test has been moved to test/grabberperf.py
      Performance tweak to bypass URLGrabberFileObject.read when not using progress_obj or throttle.
      Module for testing urlgrabber w/ multiple threads.
      Some URLs for batchgrabber.py.
      Confirmed FTP connections are being closed properly when using ranges.
      urllib2 conditionals were cleaned up a while ago..
      Always copy_local when a byte range is specified to urlgrab.
      raising RangeError anytime a range is non-satisfiable.
      Fixed try/except without an explicit exception. that's a no-no.
      Added support for RangeError to URLGrabError handling (errno 9). Also, default is _not_ to retry grabs resulting in RangeError.
      raise exceptions for invalid ranges DONE.
      Fixed up for new cvs structure. Tried to simplify a bit while I was here.
      Fixed up tests to work with new directory layout.
      Added mirror tests to main test suite.
      Removed ugly code that was suppose to allow a file-object to be passed to urlgrab as the output file.
      Using a tempfile for urlgrab test instead of named file in cwd.
      Added urlgrab, urlopen, urlread as package level exports.
      Grabber tests now call package level urlgrab, urlopen, urlread functions.
      Caching urllib2 ProxyHandler and OpenerDirectors instead of creating new ones for each request.
      Added proxies to performance tests to ensure that ProxyHandlers are being added to the urllib2 OpenerDirector.
      Removed item on reducing urllib2 opener director creation.
      Fixed real small typo in mirror doc string.
      Modified URLGrabberOptions to use "instance-inheritance" pattern instead of copying all options.
      Removed items from TODO list related to options handling as well as the item on reget not working due to how options are copied.
      Added opener kwarg/option to allow overriding of urllib2.OpenerDirector.
      Added support for specifying verbosity of unit tests when using runtests.py. 'python runtests.py --help' for usage.
      Fixed bug with urllib2.OpenerDirector caching.
      Added distutils trove classifiers to setup.py for eventual inclusion in PyPI.
      opener related items for ALPHA are completed.
      Ignoring kdevelop, kate, and anjuta project files.
      Updates for Michael's multifile progress bar changes.
      Added timeout support.
      Fixed AttributeError problem that was pretty common after additions for timeout support.
      Added except for socket resource error when reading from urllib2 file object.
      Fix for RH bug #140387
      Applied urlgrabber-add-text.patch from Terje Rosten
      Fixed proftpd test (was raising error when it shouldn't have been)
      Fixed RH bug #147124.
      Added Python 2.4 to lists of pythons to test..



More information about the Yum-commits mailing list