[Yum-devel] Yum startup speed

[Menno Smits]

seth vidal wrote:
>>There's no extra time hit because there'll be nothing extra happening 
>>then what is already currently happening now. My proposal doesn't add 
>>any extra computation. It only involves pickling of the YumPackageSack 
>>instance which is currently created every time yum is invoked anyway.
> 
> 
> The pickling would be happening and that's not a zero-time process.

Ok fine, but (in theory) the pickling happens far less often than the 
unpickling. The overall effect is quicker execution in most cases.

>>>What's the benefit from having a single big pickle of all the package
>>>metadata for all repos instead of having individual pickles for each?
>>>B/c you're going to have read in the metadata if anything has changed
>>>and, as is the case for fedora core 3,  the updates-released and 3rd
>>>party repos change quite a bit. 
>>
>>The current pickles are Python representations of the XML metadata files.
> 
> No they aren't - they're a dict structure of the data that used to be in
> the xml. But it's not just a dump in of the xml.

That's what I meant by "Python representations". We're on the same page 
here. I should have been clearer.

>>As far as I can see, this object only needs to change if there's 
>>metadata updates or a config file changes. These things are unlikely to 
>>change between successive commands within the one session.
> 
> unless the mirror they're using changes (which happens often with
> mirrorlists) and the mirrors are out of sync (which happens even more
> often, sadly)

ok, here's where my proposal starts to fall apart :)
I didn't think too much about this because I don't tend to use mirrors....

>>>Remember the metadata is more than just primary.xml - and reading in ALL
>>>the metadata is a memory hit you may not want to deal with.
>>
>>Doens't yum load pretty much all the metadata anyway? I'm not proposing 
>>that any extra data is loaded.
> 
> 
> no.
> 
> Try this out - if you want to see ALL the metadata loaded run this
> command:
> 
> yum makecache
> 
> then compare that to:
> yum list updates foo\*
> 
> in most cases filelists.xml and other.xml never get parsed. That's
> intentional.

Point taken. Looks like I missed something when I was looking through 
the yum code. I didn't realise that the YumPackageSack instance varies 
in different contexts.

>>>Gijs has suggested using something other than a python pickle to speed
>>>up access of the data.  That might make things simpler in some ways.
>>
>>Yep, I saw that post. I think that's also a good idea worth pursuing.
> 
> 
> Indeed. I suggested maybe looking at sqlite b/c metakit, while cool
> looking, hasn't been updated in a year and i'm afraid of maintaining
> complex, potentially abandoned code. :)

Something like this would be cool. Definitely the most flexible solution 
and keeps stuff on disk unless you need it so it should help with yum's 
memory footprint.

>>Like you mention, I think that being selective about which data yum 
>>loads depending on the context is the better long term solution because 
>>that's what the real problem is here. What I'm proposing is quite 
>>effective and simple to implement now with minimal impact on the rest of 
>>yum.
> 
> I'd love to see how simple it is to implement - I must be visualizing
> something wrong but it seems like it wouldn't be that simple to
> implement - but go for it - let's see a patch for it.

The planned proposal involved a simple change to doSackSetup() to do the 
pickling/unpicking with some other changes elsewhere to decide whether 
or no the pickle needs to be rebuilt (this would be passed into 
doSackSetup() as a flag).

That said, the mirrors out of sync thing really kills it though :)

Also, as I said above, I didn't realise that the object doesn't always 
hold all the parts of the metadata. Perhaps different YumPackageSack 
pickles could be cached for different contexts but that starts to get ugly.

> To be clear, i'm not against the idea  - but I'm wondering if we're
> optimizing earlier than we should be. That's really my only concern, I
> just don't want to make the code uglier with early optimization.

I'm going to leave it. It's really just a workaround for the real issues 
and will end up getting ugly. Better to spend the time working on the 
underlying problems.

Thanks for taking the time to think through the idea anyway.

Menno


Scanned by the NetBox from NetBox Blue
(http://netboxblue.com/)