[Yum-devel] URLGrabber Batch Mode

David Farning dfarning at sbcglobal.net
Sat Feb 28 13:23:17 UTC 2004


On Sat, 2004-02-28 at 06:19 -0500, Ryan Tomayko wrote: 
> None of this would be easy and there's a ton of stuff to think about.
> What I'm wondering is 1.) do we feel URLGrabber is the right place to be
> doing this kind of work? and 2.) does the idea of using a
> BatchedURLGrabber class make sense?


To be truly useful, fail over logic and leach prevention would need to
be handled within the batchedUrlGrabber. What a pain, how much gain?

How about download failure?  Updating is atomic, if one dl fails should
the batch dl abort or return a problemList?  OTOH there are several good
thread pool examples here.

http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Threads

Also, several users have requested a download resume feature.
Here is an example.  I found it in a python mailing list a few weeks ago
can't remember where so I can't give the author credit.

import httplib, os, sys

def resumeDL(local, remote):
    """resumeDL(local, remote) - Resumes the download of the file specified
in
<remote> in the form of (<server>, <path>) into the local file <local>.  If
the local file is not found, it is created and the download is started."""

    existSize = 0
    if os.path.exists(local):
        outputFile = open(local, 'ab')
        existSize = os.path.getsize(local)
    else:
        outputFile = open(local, 'wb')

    h = httplib.HTTP(remote[0])
    h.putrequest('GET', remote[1])
    h.putheader('Accept', '*/*')
    h.putheader('User-Agent', 'PythonHTTPResumer/0.1')
    h.putheader('Host', remote[0])
    if existSize > 0:
        h.putheader('Range', 'bytes=%d-' % (existSize, ))
    h.endheaders()
    errcode, errmsg, headers = h.getreply()
    print 'Response: %d (%s)' % (errcode, errmsg)
    if errcode == 416:
        ## HTTP error 416 = Request Range not Satisiable
        print 'File already exists and is larger than the remote file'
        h.getfile().close()
        del h
        return

    f = h.getfile()
    try:
        while 1:
            temp = f.read(8192)  ## 8192 == 8kb.  Saves 8kb at a time
            if temp == '':
                break
            outputFile.write(temp)
            sys.stdout.write('.')
    finally:
        outputFile.close()
        f.close()
        sys.stdout.write('\n')

if __name__ == '__main__':
    resumeDL('sm.tar.gz', ('127.0.0.1', '/hugeTestFile'))


Dave Farning




More information about the Yum-devel mailing list