[Yum] [UG] parallelizing downloading

Michael Stenner mstenner at ece.arizona.edu
Wed Jun 29 21:05:46 UTC 2005


On Wed, Jun 29, 2005 at 02:35:59PM -0600, Greg Knaddison wrote:
> This came up in the BT/yum discussions and it got me thinking...
> 
> What are the thoughts on parallel downloading?  I resubjected this to
> be a UG issue, but maybe it should be handled in yum and not UG.

It would definitely require changes in yum, and to be done most
cleanly, would require changes urlgrabber.  Currently, all of the
grabber, mirrorgroups, and keepalive code is threadsafe, so there are
really two ways to approach it:

  1) threads
  2) select loops

It turns out that neither of these are trivial.  They each have their
strengths and weaknesses.  Ryan and I put some serious thought into
this a while back but basically found that it's a non-trivial problem
given all the other stuff urlgrabber does and then we got busy :)

I'm actually open to picking this issue up again if there's interest.
We imagined a "batch grabber" which would be a grabber wrapper object
much like the mirrorgroup stuff, but which would take a list of files
(perhaps a queue for pipelining applications) and go to town on them.

					-Michael
-- 
  Michael D. Stenner                            mstenner at ece.arizona.edu
  ECE Department, the University of Arizona                 520-626-1619
  1230 E. Speedway Blvd., Tucson, AZ 85721-0104                 ECE 524G



More information about the Yum mailing list