Parallel deltarpm creation

[Ales Kozumplik]

CC Tomas who is the author of createrepo_c which we plan to use instead 
of the old and slow createrepo.

Ales

On 02/20/2014 06:26 PM, Ian Mcleod wrote:
> Posting here at James Antill's suggestion.
>
> In his talk at devconf Dennis Gilmore discussed the current bottlenecks
> in the Fedora compose/release process.  One thing he mentioned was
> deltarpm creation.
>
> The current upstream createrepo is single-threaded/single-process for
> all deltarpm actions.  I've written some code to allow parallel workers
> for these tasks, similar to the multi-process workers that can be used
> in the initial package XML parsing tasks.
>
> GIT -
> https://github.com/imcleod/createrepo/tree/feature/parallel_deltas_full
>
> RPMS -
> http://imcleod.fedorapeople.org/createrepo/
>
> The patch adds two options to the command line createrepo and the
> associated config object:
>
> --delta-workers - The number of worker processes to use for delta
> related tasks
>
> --max-concurrent-delta-rpm-size - The maximum total size of uncompressed
> rpm payloads that are actively being processed by makedeltarpm at any
> given time.
>
> The deltarpm documentation suggests that its peak RAM use is typically
> typically 4x the uncompressed RPM payload size.  This is consistent with
> my experience.  So, a reasonable use case is to set --delta-workers to
> the number of CPU cores and --max-concurrent-delta-rpm-size to ~25% of
> RAM size (or whatever quantity of memory you want to devote to the
> parallel deltas).
>
> For my development stress-test-case I re-created an F20 x86_64
> Everything repo with F19 Everything as the "old" rpm source for deltas.
> On a 32 core test system this task ran in 8 hours with a single deltarpm
> worker versus 20 minutes when all 32 cores were used with a concurrent
> size limit of 16 GB.  In total this creates about 32,000 drpms.  So,
> this helps.
>
> Thoughts?
>
> -Ian
>
> _______________________________________________
> Yum-devel mailing list
> Yum-devel at lists.baseurl.org
> http://lists.baseurl.org/mailman/listinfo/yum-devel
>