[Yum-devel] Yum startup speed

[Menno Smits]

Hi Seth,

>>My proposal is:
>>* keep a pickle of the YumPackageSack object
>>
>>* if any repository metadata changes, rebuild YumPackageSack and 
>>overwrite existing pickle
> 
> 
> How much of a time hit is this? B/c any repo metadata is going to
> change.

There's no extra time hit because there'll be nothing extra happening 
then what is already currently happening now. My proposal doesn't add 
any extra computation. It only involves pickling of the YumPackageSack 
instance which is currently created every time yum is invoked anyway.

>>* if any config file changes (check via checksums), rebuild 
>>YumPackageSack and overwrite existing pickle
> 
> again - not an uncommon occurrence, even less uncommon in a graphical
> interface.

The config file(s) is unlikely to change between successive commands. 
This is what most people are complaining about. You run yum several 
times to say search for a package, waiting a significant time each run. 
Then you run yum to install the package and wait again. My proposal 
would mean that only perhaps the first run of the yum command would 
impose a longer delay (if the pickle needs updating). The following runs 
all use the pickle and therefore are quicker.

> What's the benefit from having a single big pickle of all the package
> metadata for all repos instead of having individual pickles for each?
> B/c you're going to have read in the metadata if anything has changed
> and, as is the case for fedora core 3,  the updates-released and 3rd
> party repos change quite a bit. 

The current pickles are Python representations of the XML metadata files.

The pickle I'm proposing is of the data structure that results once the 
XML data are combined (creating the YumPackageSack instance). This 
combination of package data takes a significant amount of time and it 
seems to be what a lot of people have a problem with.

As far as I can see, this object only needs to change if there's 
metadata updates or a config file changes. These things are unlikely to 
change between successive commands within the one session.

> In addition, you're taking a memory hit by loading up stuff you may not
> want. And if you have to do another lookup you're going to need to read
> in the filelists data which may not be there.

My proposed optimisation involves no extra lookups over the way yum 
currently works.

> Remember the metadata is more than just primary.xml - and reading in ALL
> the metadata is a memory hit you may not want to deal with.

Doens't yum load pretty much all the metadata anyway? I'm not proposing 
that any extra data is loaded.

> I think you're talking about optimization for the ideal but not terribly
> common case.
> 
> Gijs has suggested using something other than a python pickle to speed
> up access of the data.  That might make things simpler in some ways.

Yep, I saw that post. I think that's also a good idea worth pursuing.

> Another set of options might be to generate a smaller-than-primary-but-
> really-common lookup file on the repo-side to make startup faster. I'm
> not really sure what the best tack is here, then again, I've done the
> benchmarks and timing on this and I know how long it takes to read in
> the data. Many of the folks who were complaining in that thread were
> suffering b/c the mirrors were out of sync and they weren't using the
> metadata pickles at all.

Like you mention, I think that being selective about which data yum 
loads depending on the context is the better long term solution because 
that's what the real problem is here. What I'm proposing is quite 
effective and simple to implement now with minimal impact on the rest of 
yum.

I've taken the time to address your feedback in detail because I still 
think the idea has merit. If you really think it just won't work or 
isn't worth the effort, then I'll drop it.

Regards,
Menno

Scanned by the NetBox from NetBox Blue
(http://netboxblue.com/)