[Yum-devel] Yum startup speed

[Menno Smits]

Hi all,

After following the long thread on fedora-devel, "why doesn't yum cache 
anything?", I started prodding around as this was something that's been 
bugging me too. (Thread is at: 
http://www.redhat.com/archives/fedora-devel-list/2004-December/msg01057.html)

Based on some profiling I found that much of yum's startup time is in 
reconstruction of the big YumPackageSack object from the XML metadata. 
This is probably no suprise to most people on this list.

So I thought, "why doesn't yum just cache the whole thing as a pickle 
rather recreating it every time"? I roughly hacked up 
YumBase.doSackSetup() so that after building the YumPackageSack object 
it writes it to a pickle and if this pickle exists in future calls it 
blindly uses it.

Next I benchmarked successive yum runs both with the pickle present and 
not present, using "times yum -C list mtr". Here's the result:

Uncached
19.225 31.197 17.653 17.208 (avg = 21.32s)

Cached
11.469 8.546 9.233 9.853 (avg = 9.775s)

The execution time is almost halved when the pickle is used, which is a 
pretty decent improvement.

The problem with this idea is knowing when the pickle can be used and 
when it needs to be rebuilt. The YumPackageSack object relies on the XML 
repo metadata, the repos that are enabled and the package 
exclusions/inclusions from the config files and command line. If any of 
these change the pickle needs to be rebuilt or can't be used.

My proposal is:
* keep a pickle of the YumPackageSack object

* if any repository metadata changes, rebuild YumPackageSack and 
overwrite existing pickle

* if any config file changes (check via checksums), rebuild 
YumPackageSack and overwrite existing pickle

* if package exclusions are given on the command line, don't use the 
YumPackageSack pickle and just build a correct one on the fly


I'm happy to develop this further if people here think it might be 
worthwhile. Is this a reasonable idea or do I need to get more sleep? :)

Regards,
Menno











Scanned by the NetBox from NetBox Blue
(http://netboxblue.com/)