[Yum-devel] yum on an olpc machine (slooooooooooow)

[Panu Matilainen]

On Thu, 21 Dec 2006, seth vidal wrote:
> On Thu, 2006-12-21 at 09:30 +0200, Panu Matilainen wrote:
>> On Thu, 21 Dec 2006, seth vidal wrote:
>>
>>> On Wed, 2006-12-20 at 10:47 +0200, Panu Matilainen wrote:
>>>
>>>> I wrote a quick-n-dirty sqlite "backend" for apt to see how things would
>>>> look like for another implementation (with totally different usage
>>>> patterns) of repodata.
>>>>
>>>> The initial version performs so badly you wouldn't believe it :D
>>>>
>>>> On 2.4GHz AMD64 the creation of dependency cache (which involves walking
>>>> over all packages and recording the dependency data to it's the memory
>>>> mapped dependency cache) takes over 3 MINUTES (!) for just FC6 core
>>>> data. With xml repodata, that's ~6.5s operation on this system, fully
>>>> reading filelists.xml info as well, which the sqlite version doesn't do at
>>>> all at this point.
>>>>
>>>> After a bit of investigation, the major bottleneck here is that the
>>>> provides, requires, conflicts and obsoletes tables don't have index on
>>>> pkgKey. After creating those indexes, it's back to ~6.5s even with the
>>>> naive initial implementation.
>>>>
>>>> I haven't done any timings on how those indexes would affect yum's usage
>>>> patterns, probably not *that* dramatic but it might be something to look
>>>> at.
>>>
>>> I'm sure that better indexing would help some lookups rather a lot.
>>
>> Curiously enough, the pkgKey indexes on prco data don't affect yum at all,
>> even though one would think eg returnPrco() would benefit a great deal.
>
> odd, I'd thought it would have benefitted, too.

Indeed. It's so "obvious" that I feel I must be overlooked something but I 
just don't see what.

>>> out of curiosity - how much does the size of the sqlite file go up when
>>> the indexes are added?
>>
>> For FC-development primary db:
>> -rw-r--r-- 1 pmatilai pmatilai 5074944 Dec 21 09:17 primary.xml.gz.sqlite.orig
>> -rw-r--r-- 1 pmatilai pmatilai 5650432 Dec 21 09:18 primary.xml.gz.sqlite
>>
>> So it's certainly a non-trivial amount of metadata on metadata which yum
>> doesn't even seem to care about :)
>
> How well they do compare, compressed? Do the indexes compress better
> than we might think? :)

Heh, I forgot about compression completely :) That way things seem a whole 
lot better:
1619533 Dec 21 09:17 primary.xml.gz.sqlite.orig.gz
1741906 Dec 21 09:18 primary.xml.gz.sqlite.gz

1223076 Dec 21 09:17 primary.xml.gz.sqlite.orig.bz2
1299333 Dec 21 09:18 primary.xml.gz.sqlite.bz2

 	- Panu -