[Yum-devel] URLGrabber Batch Mode

[Michael Stenner]

On Sat, Feb 28, 2004 at 07:23:17AM -0600, David Farning wrote:
> On Sat, 2004-02-28 at 06:19 -0500, Ryan Tomayko wrote: 
> > None of this would be easy and there's a ton of stuff to think about.
> > What I'm wondering is 1.) do we feel URLGrabber is the right place to be
> > doing this kind of work? and 2.) does the idea of using a
> > BatchedURLGrabber class make sense?
> 
> To be truly useful, fail over logic and leach prevention would need to
> be handled within the batchedUrlGrabber. What a pain, how much gain?

What is "leach prevention"?  Making sure you don't spin off multiple
threads for the same server?

failover logic is already going into urlgrabber along with the
"MirrorGroup" functionality that Ryan alluded to.  The basic concept
behind MirrorGroup is that you have an object that acts like a
URLGrabber instance that you pass relative urls to.  It chooses a
mirror by some algorithm and then downloads the file.  The calling
application doesn't have to know or care where it came from.

> How about download failure?  Updating is atomic, if one dl fails should
> the batch dl abort or return a problemList?

Well, that seems application dependent, so I think the best general
answer would be "either".

> OTOH there are several good thread pool examples here.

I'm not yet convinced that threading is the right way to handle that,
though.  A select loop might be better.  I think this is one of those
"when all you've got is a hammer" sort of situations.  Threading is a
very popular hammer, but this problem might be a screw.

> http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Threads
> 
> Also, several users have requested a download resume feature.
> Here is an example.  I found it in a python mailing list a few weeks ago
> can't remember where so I can't give the author credit.

That is also being implemented (hell, it might be done... Ryan?) in
the current urlgrabber.  It's a little more complex than the example
you provide simply because of the other constraints in place (http,
ftp, file, independent need for byte-ranges, etc).

					-Michael
-- 
  Michael D. Stenner                            mstenner at ece.arizona.edu
  ECE Department, the University of Arizona                 520-626-1619
  1230 E. Speedway Blvd., Tucson, AZ 85721-0104                 ECE 524G