[Yum-devel] Yum speed

[seth vidal]

On Thu, 2006-08-24 at 23:38 +0200, Gijs Hollestelle wrote:
> Hi all,
> 
> Being quite impressed by the speed of the current version of yum I'm
> still wondering if we can make it even faster.
> 
> In order to get a measure of the time 'wasted' in yum I wrote a very
> simple program that gives the same output as yum -C list updates in
> the simplest way I could think about:
> - Obtain the list of installed packages from rpmdb
> - 'Parse' /etc/yum.repos.d/*.repo files to find all enabled repositories
> - Read the package info in the corresponding sqlite databases for
> these repositories
> - Find the best package that is supported on this arch and report it
> as an update in case it's not already installed
> 
> I was able to get it to run in 1.2 seconds on my machine where the
> real yum -C list updates takes 4.4 seconds (output is the same but it
> probably won't handle all the corner cases).
> 
> So in case of -C list updates there is 3.2 seconds of  parsing config
> files, juggling between tuples and objects,  setting up of (in this
> simple case unnecesarry) holder classes, dealing with multi-arch, etc.
> 
> I think we should be able to come close to the 1.2 seconds for the
> simple case of yum list commands. I hope to have some time in the near
> future to digg into this more deeply and actualy contribute some
> substantial speed-up patches. One of the things I noticed was that
> importing config takes almost a second and pulls in a lot of extra
> classes. That's why my speed-test implementation contains its own
> 'config parser' ;-)

This is an interesting test. I'd be curious where we're spending the
most time.

When I ran both I got 1.5s for your script and 4.5s for the same output
using yum.
So if we knock off a second from that b/c of configparser then the code
we still have about 2seconds to work out. Unless you can think of a nice
way of not eating so much time inside ConfigParser.

We've chewed through a number of the object->tuple conversions this week
but the place where we might be able to win some more time is in
rpmUtils.updates. I've considered rewriting that using just package
objects all around to see if any of the conversions become unnecessary.

I know Menno mentioned that importFromDict() in YumAvailablePackage
seems a bit hurky. That might be another place to look.

Can anyone else think of any obvious places where looking might be
useful?

-sv