[Yum-devel] yum on an olpc machine (slooooooooooow)

[Panu Matilainen]

On Sat, 16 Dec 2006, seth vidal wrote:

> On Sat, 2006-12-16 at 20:56 +0200, Panu Matilainen wrote:
>>>
>>> Or maybe I'm being crazy, but I don't see it as another copy of the
>>> metadata. I see it as the same copy, just removing the extra processing
>>> of it from every person's machine and having the processing only occur
>>> once.
>>
>> On a second thought...
>>
>> What do we need the xml for then, anyway? Even on fast computers the
>> repodata format is very expensive to process (otherwise we wouldn't have
>> had several generations of pickle and sqlite caches).
>
> If we were going to get rid of the xml format altogether I would
> recommend:
> 1. using the sqlitedb's as an optimization and non-api-breaking test in
> 3.0.X or so
>
> 2. figure out what improvements we could make to the db format to make
> searching faster or to make it smaller on disk. This would mean working
> out the right indexes, etc.

I wrote a quick-n-dirty sqlite "backend" for apt to see how things would 
look like for another implementation (with totally different usage 
patterns) of repodata.

The initial version performs so badly you wouldn't believe it :D

On 2.4GHz AMD64 the creation of dependency cache (which involves walking 
over all packages and recording the dependency data to it's the memory 
mapped dependency cache) takes over 3 MINUTES (!) for just FC6 core 
data. With xml repodata, that's ~6.5s operation on this system, fully 
reading filelists.xml info as well, which the sqlite version doesn't do at 
all at this point.

After a bit of investigation, the major bottleneck here is that the 
provides, requires, conflicts and obsoletes tables don't have index on 
pkgKey. After creating those indexes, it's back to ~6.5s even with the 
naive initial implementation.

I haven't done any timings on how those indexes would affect yum's usage 
patterns, probably not *that* dramatic but it might be something to look 
at.

 	- Panu -