[Yum-devel] URLGrabber Batch Mode
David Farning
dfarning at sbcglobal.net
Sat Feb 28 13:23:17 UTC 2004
On Sat, 2004-02-28 at 06:19 -0500, Ryan Tomayko wrote:
> None of this would be easy and there's a ton of stuff to think about.
> What I'm wondering is 1.) do we feel URLGrabber is the right place to be
> doing this kind of work? and 2.) does the idea of using a
> BatchedURLGrabber class make sense?
To be truly useful, fail over logic and leach prevention would need to
be handled within the batchedUrlGrabber. What a pain, how much gain?
How about download failure? Updating is atomic, if one dl fails should
the batch dl abort or return a problemList? OTOH there are several good
thread pool examples here.
http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Threads
Also, several users have requested a download resume feature.
Here is an example. I found it in a python mailing list a few weeks ago
can't remember where so I can't give the author credit.
import httplib, os, sys
def resumeDL(local, remote):
"""resumeDL(local, remote) - Resumes the download of the file specified
in
<remote> in the form of (<server>, <path>) into the local file <local>. If
the local file is not found, it is created and the download is started."""
existSize = 0
if os.path.exists(local):
outputFile = open(local, 'ab')
existSize = os.path.getsize(local)
else:
outputFile = open(local, 'wb')
h = httplib.HTTP(remote[0])
h.putrequest('GET', remote[1])
h.putheader('Accept', '*/*')
h.putheader('User-Agent', 'PythonHTTPResumer/0.1')
h.putheader('Host', remote[0])
if existSize > 0:
h.putheader('Range', 'bytes=%d-' % (existSize, ))
h.endheaders()
errcode, errmsg, headers = h.getreply()
print 'Response: %d (%s)' % (errcode, errmsg)
if errcode == 416:
## HTTP error 416 = Request Range not Satisiable
print 'File already exists and is larger than the remote file'
h.getfile().close()
del h
return
f = h.getfile()
try:
while 1:
temp = f.read(8192) ## 8192 == 8kb. Saves 8kb at a time
if temp == '':
break
outputFile.write(temp)
sys.stdout.write('.')
finally:
outputFile.close()
f.close()
sys.stdout.write('\n')
if __name__ == '__main__':
resumeDL('sm.tar.gz', ('127.0.0.1', '/hugeTestFile'))
Dave Farning
More information about the Yum-devel
mailing list