[yum-commits] 2 commits - urlgrabber/grabber.py

[zpavlas at osuosl.org]

 urlgrabber/grabber.py |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

New commits:
commit 4374b6b4c6196f1c770954d2ceb136daa62fbb66
Author: Zdenek Pavlas <zpavlas at redhat.com>
Date:   Mon Aug 26 08:56:13 2013 +0200

    A tiny bit more robust "Content-Length" parsing. BZ 1000841
    
    RFC2616 does not allow anything between header name and the colon.

diff --git a/urlgrabber/grabber.py b/urlgrabber/grabber.py
index 8bb8e58..30a8bdb 100644
--- a/urlgrabber/grabber.py
+++ b/urlgrabber/grabber.py
@@ -1310,7 +1310,7 @@ class PyCurlFileObject(object):
             # but we can't do that w/o making it do 2 connects, which sucks
             # so we cheat and stuff it in here in the hdr_retrieve
             if self.scheme in ['http','https']:
-                if buf.lower().find('content-length') != -1:
+                if buf.lower().find('content-length:') != -1:
                     length = buf.split(':')[1]
                     self.size = int(length)
                 elif self.append and self._hdr_dump == '' and ' 200 ' in buf:
commit 656f364e287e876cd30060c99603da59b497a987
Author: Zdenek Pavlas <zpavlas at redhat.com>
Date:   Mon Aug 19 10:32:09 2013 +0200

    docs: throttling is per-connection, suggest max_connections=1. BZ 998263

diff --git a/urlgrabber/grabber.py b/urlgrabber/grabber.py
index 711215f..8bb8e58 100644
--- a/urlgrabber/grabber.py
+++ b/urlgrabber/grabber.py
@@ -447,6 +447,11 @@ BANDWIDTH THROTTLING
   is a float and bandwidth == 0, throttling is disabled.  If None, the
   module-level default (which can be set with set_bandwidth) is used.
 
+  Note that when multiple downloads run simultaneously (multiprocessing
+  or the parallel urlgrab() feature is used) the total bandwidth might
+  exceed the throttle limit. You may want to also set max_connections=1
+  or scale your throttle option down accordingly.
+
   THROTTLING EXAMPLES:
 
   Lets say you have a 100 Mbps connection.  This is (about) 10^8 bits