[Yum-devel] Speed improvements to sqlite filelists

[Menno Smits]

Hi Gijs,

I'm having a little trouble with this version and I don't have time to 
debug right now. I'm getting this output from a clean checkout:

....
Reading repository metadata in from local files
Primary sqlite cache needs updating, reading original meta data
Added 2622 new packages, deleted 0 old in 11.01 seconds
Primary sqlite cache needs updating, reading original meta data
Added 1694 new packages, deleted 0 old in 7.46 seconds
Primary sqlite cache needs updating, reading original meta data
Added 136 new packages, deleted 0 old in 0.44 seconds
Primary sqlite cache needs updating, reading original meta data
Added 649 new packages, deleted 0 old in 4.49 seconds
Primary sqlite cache needs updating, reading original meta data
Added 477 new packages, deleted 0 old in 1.33 seconds
Primary sqlite cache needs updating, reading original meta data
Added 723 new packages, deleted 0 old in 2.41 seconds
Excluding Packages in global exclude list
Finished
Traceback (most recent call last):
   File "/usr/bin/yum-dev", line 10, in ?
     yummain.main(sys.argv[1:])
   File "/home/msmits/projects/yum/yum-sqlite/trunk/yummain.py", line 
72, in main
     result, resultmsgs = do()
   File "/home/msmits/projects/yum/yum-sqlite/trunk/cli.py", line 635, 
in doCommands
     return self.provides()
   File "/home/msmits/projects/yum/yum-sqlite/trunk/cli.py", line 1233, 
in provides
     matching = self.searchPackageProvides(args, 
callback=self.matchcallback)
   File "/home/msmits/projects/yum/yum-sqlite/trunk/yum/__init__.py", 
line 909, in searchPackageProvides
     for filetype in po.returnFileTypes():
   File "/home/msmits/projects/yum/yum-sqlite/trunk/yum/sqlitesack.py", 
line 42, in returnFileTypes
     return YumAvailablePackage.returnFileTypes(self)
   File 
"/home/msmits/projects/yum/yum-sqlite/trunk/repomd/packageObject.py", 
line 299, in returnFileTypes
     return self.files.keys()
AttributeError: 'NoneType' object has no attribute 'keys'

These new performance figures sound great. I'm looking forward to trying 
this out.

Regards,
Menno


Gijs Hollestelle wrote:
> Hi all,
> 
> Today I found some spare time to work on the yum sqlite patches. I've
> implemented the following now for the sqlite filelists caches:
> 
> Instead creating a database entry per file I now create a database
> entry per directory in a package and I put the files and filetypes in
> that directory into a single TEXT database field. This reduces cache
> generation times by a factor +/- 4 and diskusage by a factor 2.5
> 
> See my subversion repository:
> https://svn.win.tue.nl/viewcvs/yum-sqlite
> 
> Greets,
>   Gijs
> 
> Notes:
> - Currently I'm using |file1|file2|file3| to put them in, but I guess
> this should be changed to something like pickle's dumps, but I havent
> tested that yet.
> - Filetypes can be stored more compactly than |file|file|file|dir|
> maybe something like fffd for this example, because a type can be
> encoded as a single character.
> _______________________________________________
> Yum-devel mailing list
> Yum-devel at linux.duke.edu
> https://lists.dulug.duke.edu/mailman/listinfo/yum-devel
> 
> Scanned by the NetBox from NetBox Blue
> (http://netboxblue.com/)
> 


Scanned by the NetBox from NetBox Blue
(http://netboxblue.com/)