[Yum-devel] metadata parser in C

Tambet Ingo tambet at ximian.com
Wed May 10 13:53:12 UTC 2006


On Wed, 2006-05-10 at 09:34 -0400, Russell Harrison wrote:
> I will say that the biggest complaint I get from people about yum is
> speed.  I know some people on older machines that are reporting
> initial parsing times of over an hour.

You're probably confusing parsing and "downloading and parsing". The
original primary.xml parse took ~9 seconds on my machine, it sounds a
bit unlikely to take hours.

> I'm assuming (haven't looked at the code) that since his changes
> require db changes his parsing algorithm is different from the current
> python implementation.  If using C is a concern can some of the same
> improvements be made in python?  Even if they're not as dramatic any
> speed increase for yum is a big win. 

The algorithm is exactly the same, the output is exactly the same. The
only thing that's done differently is the removing of old content. The
speed improvement is only because of "C is faster than python".

Tambet

> 
> On 5/10/06, seth vidal <skvidal at linux.duke.edu> wrote:
>         On Wed, 2006-05-10 at 15:45 +0300, Tambet Ingo wrote:
>         > email message attachment, "Forwarded message - [Yum]
>         metadata parser
>         > in C"
>         > > -------- Forwarded Message --------
>         > > From: Tambet Ingo < tambet at ximian.com>
>         > > Reply-To: Yellowdog Updater, Modified
>         <yum at lists.dulug.duke.edu>
>         > > To: Yellowdog Updater, Modified <
>         yum at lists.dulug.duke.edu>
>         > > Subject: [Yum] metadata parser in C
>         > > Date: Wed, 10 May 2006 14:03:01 +0300
>         > >
>         > > Hey,
>         > >
>         > > Attached is a yum metadata parser written in C. It should
>         produce
>         > > identical results with the sqlitecache implementation in
>         yum-2.6.1 but
>         > > should be quite a bit faster. Here are the numbers parsing
>         FC5 core 
>         > > metadata (2207 packages):
>         > >
>         
>         Hi Tambet,
>         Thanks for fwding this onto this list - it's just a better
>         place for
>         the discussion and a number of folks who are developers aren't
>         on
>         yum-list b/c it's mostly for user questions. 
>         
>         Also thanks for working on this, it looks cool. I'm going to
>         give you
>         some initial thoughts and since I've just jumped out of the
>         shower some
>         extended thoughts, too. So take all of it inclusively and
>         we'll see what 
>         comes out, okay?
>         
>         Initial thoughts:
>         Yay Speed!!!
>         Ugh! C.
>         Hey, look, someone from ximian contributing, cool.
>         
>         I like the initial speed results you've posted - those look
>         great but
>         all of the parsing being in C concerns me some if only due to 
>         maintenance issues.  It's not necessarily a show-stopper just
>         a
>         show-complicator. :)
>         
>         
>         Post-shower thoughts:
>         - If we have the speed up granted by your code then we can
>         rework the
>         sqlite db format (especially for filelists) to vastly simplify
>         and speed 
>         up our all of the 'whatProvides' calls that have to look up
>         files. This
>         makes depsolving faster.
>         - the metadata parsing code is, for obvious reasons, some of
>         the least
>         changed code in yum. So modifying it wouldn't be a massively
>         frequent 
>         concern
>         - this makes all of the metadata importing code obsolete (to
>         some
>         extent) which does get rid of a bunch of code that has needed
>         some
>         quailty time with an editor. In short it is a parallel thought
>         to a goal 
>         for the development series.
>         
>         
>         So that's the majority of what I've been thinking of right
>         now. What do
>         other folks think?
>         
>         -sv
>         
>         
>         
>         _______________________________________________
>         Yum-devel mailing list 
>         Yum-devel at linux.duke.edu
>         https://lists.dulug.duke.edu/mailman/listinfo/yum-devel
> 
> _______________________________________________
> Yum-devel mailing list
> Yum-devel at linux.duke.edu
> https://lists.dulug.duke.edu/mailman/listinfo/yum-devel




More information about the Yum-devel mailing list