[Yum-devel] Sqlite improvements

[seth vidal]

On Sun, 2005-01-16 at 21:17 +0100, Gijs Hollestelle wrote:
> Hi all,
> 
> I've been working a bit more on my sqlite patches, they look a lot
> cleaner now and seem to work quite well for the stuff I use yum for
> (install, update and list)
> 
> What is working:
> - yum list [updates] [obsoletes] [extras]
> - yum install
> - yum update
> - all primary.xml.gz metadata is imported 




> What is not working (yet):
> - yum info (all the necessary data is in the sqlite database, it just
> isn't passed down to yum yet)


> - yum whatprovides (full regexp support would be very hard to
> implement using sqlite whatprovides /filename or feature is possible
> but not implemented yet)
> - Support for other.xml.gz and filelist.xml.gz
> - Probably a lot more :-(
> 

What matching capabilities does sqlite offer? Is it just sql 'like'
matches or anything else?

If push comes to shove it shouldn't be that hard to iterate the sql db
in the same way we do it with the xml in memory. Just do a match on each
one, sadly.

> Some numbers to impress you all:
> 
> yum -C list updates: (dag,base,updates,freshrpms) (5226 packages)
> 
> Sqlite: 1.4 seconds 13MB memory usage
> Normal: 8.6 seconds 65 MB memory usage
> 
> yum -C install xen: (dag,base,updates,freshrpms,development) (8917
> packages and a lot of deps (this pulls in python-2.3 and all other
> kinds of stuff), all headers are cached and it is run upto the point
> where it asks for confirmation)
> 
> sqlite: 19 seconds 38MB memory usage 
> Normal: 67 seconds 105MB memory usage
> 
> There is a drawback however: creating the sqlite caches takes some time:
> Running sqlite yum list updates as above for the first time needs to
> create the sqlite caches and takes 20 seconds, memory usage is 40MB (I
> believe both these numbers can be improved)

I've checked out the patch you sent, now. The times you're getting are
great and they map out similarly on my system. Frequently changing
metadata will hurt those times but for lots of consecutive commands it
will be a fantastic improvement.

Have you run this under hotshot or the other profiler to see where all
the time is going on import?

I'm going to be releasing 2.1.13 soon. Once that's done I'd like to see
about getting the sqlite patches in to cvs and seeing if we can flesh
out the rest of the problem areas you've mentioned.

This is great work, thanks.

-sv