[Yum-devel] yum on an olpc machine (slooooooooooow)

[seth vidal]

On Fri, 2006-12-15 at 23:07 -0500, seth vidal wrote:
> So, the things we will need to figure out:
> 1. can we store less data in primary.xml and still get what we need?
> primary is fairly trimmed down, actually, so I'm not sure how'd we get
> there from here. Also changing the format would be a big pain in the
> arse for a variety of reasons. Not impossible, just not a quicker thing
> to fix.
> 
> The only way I can think of would be a different format so we don't have
> to parse the xml or pre-parsing the metadata into a sqlite db. This
> would make downloads of the metadata larger but maybe it would be faster
> for operations.
> 
> For example - fedora extras:
> -rw-r--r--  1 root root 1.6M Dec 16 03:06 primary.xml.gz
> -rw-r--r--  1 root root 2.2M Dec 16 03:09 primary.xml.sqlite.bz2
> 
> bzipped the primary xml sqlite db is 2.2M vs 1.6M for the xml itself.

okay doing a bit more research on this one:

1. sqlite is endian-safe and arch-safe so we should be able to create
the dbs wherever and use them wherever
2. generating them when createrepo makes the repository shouldn't be the
end of the world and is very little code using the yum-metadata-parser
module:

import sqlitecachec
p = sqlitecachec.RepodataParserSqlite('/tmp/t/', 'foo', None)
p.getPrimary('/tmp/t/primary.xml.gz',
'c56e0342822811c2517d0039700ce6a293c79461')

3. If we have createrepo treat the db-files as yet-another piece of
metadata to add to repomd.xml then we should be able to grab them the
way we already have in the code and use them instead of grabbing the xml
if they are available.

4. They're bigger by about 1/2-again the size when compressed versus
downloading the xml, so there is a size-cost to these files.

Another perk of this is that it could be implemented w/o breaking the
api. If we did it right then any code using yum would work the same and
just take advantage of this.

Am I missing something or does this seem like a useful weekend project?

-sv