[Yum-devel] [PATCH] Use multiple applydeltarpm workers

[James Antill]

On Fri, 2013-02-22 at 16:36 +0100, Zdenek Pavlas wrote:
> Did some real-world testing.

 Seems fine, although I'm still confused about what is
calling .rebuild() and how it handles the pkg being in an intermediate
state after this patch.

 Also note sure "wait" is the correct function name, as what happens is
that:

self.wait(conf.limit) == wait until we can add a job to hit "limit".
                      == wait until only limit-1 jobs are running.
                      == (when "limit" jobs are running)
                         wait for a single job to finish.
self.wait(1) == wait for all jobs to finish.

>   Haven't been updating my F17
> for some time, and now 88 updates are available.  Skipped
> applying updates with --downloadonly so only the setup, DL,
> and rebuild times are reported:
> 
> 1) download RPMs only
> Total 235 MB
> real    0m35.433s
> user    0m9.543s
> sys 0m2.865s

 Yeh, most people aren't downloading at almost 10MB/s :).

> 2) use yum-presto-0.7.3-1.fc17.noarch
> Presto reduced the update size by 88% (from 197 M to 25 M).
> real    2m50.171s
> user    2m1.482s
> sys 0m4.633s
> 
> 3) use the new drpm code
> Delta RPMs reduced 197 M of updates to 25 M (87% saved)
> 
> 1 worker:
> real	2m17.530s
> user	2m6.819s
> sys	0m3.539s
> 
> 2 workers:
> real	1m21.588s
> user	2m20.474s
> sys	0m2.965s
> 
> 4 workers:
> real	1m10.760s
> user	2m40.738s
> sys	0m3.699s
>
> 6 workers:
> real	1m23.832s
> user	4m33.390s
> sys	0m6.313s

 How many CPUs do you have? What kind of disk, and IO do you have? Maybe
defaulting to -1 == os.sysconf("SC_NPROCESSORS_ONLN") is better?

> There's only a little gain in parallelizing downloads, since
> I use a single fast local mirror and only one concurrent
> connection is used.  Spawning more workers helps a lot.

 Also we don't have progress for drpm => rpm conversion anymore ...
given the numbers above I'm guessing that yum just stopped after
downloading, showing nothing for over a minute?
 Getting data on what that's going to look like for normal users is
probably good ... as I'm very sure we'll get bug reports about yum
"hanging" for a minute (we did before, as we didn't have progress on the
plugin to start with).

 Do you have a plan to add extra progress data, or merge it into the
download progress in some way?

 How hard would it be to start downloading .rpms if we get to the end of
the download stage and we still have N .drpm => .rpm conversions to do?