[Yum-devel] URLGrabber Batch Mode

[Ryan Tomayko]

Two separate requests have come up that have me thinking about being
able to use URLGrabber in a sort-of batch mode: 

1. Grabbing in a forked sandbox type thing for security reasons.
2. Grabbing files in parallel using multiple threads.

I was thinking one way of accomplishing this might be to add some kind
of BatchedURLGrabber/BatchedMirrorGroup class that would have pretty
much the same interface as URLGrabber/MirrorGroup but instead of
actually grabbing the files when the methods were called, it would keep
a list of the url/kwargs. Once you ran through calling urlXXX for each
file you would then call a "batchgrab" method that could do a variety of
things including:

1. Fork, setuid, and pull all files in the list. The synchronization
between processes (via sockets or whatever) could be handled in the
batchgrab method so it would feel as if it were a normal synchronous
in-process call.

2. The batchgrab method could handle splitting off some number of
threads to use when grabbing files and orchestrating mirror usage (to be
sure only n connections were open to a single server at a time).

None of this would be easy and there's a ton of stuff to think about.
What I'm wondering is 1.) do we feel URLGrabber is the right place to be
doing this kind of work? and 2.) does the idea of using a
BatchedURLGrabber class make sense?

- Ryan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.baseurl.org/pipermail/yum-devel/attachments/20040228/9fe18680/attachment.pgp