[Rpm-metadata] createrepo: huge memory consumption?
skvidal at fedoraproject.org
Mon Aug 24 02:54:58 UTC 2009
On Sun, 23 Aug 2009, Ryan Lynch wrote:
> I've noticed that running 'createrepo' against large repositories
> takes up an enormous amount of memory, on my machines, at least.
> I'm currently using the following options:
> * --cachedir cache
> * --database
> * --update
> * --unique-md-filenames
> I see createrepo's 'genpkgmetadata.py' script eating up ~1.5 GB of
> memory, or 37% physical memory (runtime ~1 min). I wouldn't have
> noticed this, except that it usually triggers the OOM killer and
> randomly whacks firefox, eclipse, etc.
> So, is this normal/expected? Are other users seeing this happen with
> large repositories? I've noticed that dropping the '--update' option
> takes a lot longer to run (~8-10 min), but with a much lower peak
> memory utilization. Is it just a case of weighing memory usage vs.
> runtime, and making my choice?
--update reads in ALL of the old metadata into memory so it can add/delete
from it and dump it back out quickly.
the speed up you get from reading that in is at the cost of memory.
code to improve that is welcome.
More information about the Rpm-metadata