[Yum-devel] External downloader
Zdenek Pavlas
zpavlas at redhat.com
Mon Nov 21 09:57:11 UTC 2011
Hi,
> This still uses curl multi, again given the giant bugs we've had
> with basic curl downloading I don't want to introduce this for 1.0 ...
> it's just asking for everything to not work properly.
Ok, I'm going to implement multiple downloader processes, and their
management and polling in the parent.
> > [PATCH 10/10] Optional/throttled progress reporting
>
> This last one isn't the correct solution, you can't just drop
> updates.
Why not? The size reported is cumulative, and when download finishes
it's sent unconditionally. Do we have users relying on faster than
0.3s updates? Do we have to 1:1 match write() and update()?
I've noticed curl (both multi and single) processes data in 1348-byte
chunks and easily does few hundred updates/s.
> Doesn't it matter a lot to fix this for 1.0? If so then the better
> solution would be to work out why we are doing a lot of small reads
> +writes and try to merge them at that layer.
I believe urlgrabber users never see individual reads and writes
(there's no download-to-fileobject API), so merging is not necessary.
> You have access to push now? Can you remove the curl multi bits and
> the last patch and push to a branch?
Yes, but avoiding curl multi is not trivial, as it complicates
the ExternalDownloader class.
More information about the Yum-devel
mailing list