[Yum-devel] metadata parser in C

Russell Harrison rtlm10 at gmail.com
Wed May 10 15:52:55 UTC 2006


On 5/10/06, Tambet Ingo <tambet at ximian.com> wrote:
>
> On Wed, 2006-05-10 at 09:34 -0400, Russell Harrison wrote:
> > I will say that the biggest complaint I get from people about yum is
> > speed.  I know some people on older machines that are reporting
> > initial parsing times of over an hour.
>
> You're probably confusing parsing and "downloading and parsing". The
> original primary.xml parse took ~9 seconds on my machine, it sounds a
> bit unlikely to take hours.


That was my first thought actually.  Their machines are on a pretty big pipe
though.  Downloading the files with wget takes in the neighborhood of 2
seconds max for all of them.


> I'm assuming (haven't looked at the code) that since his changes
> > require db changes his parsing algorithm is different from the current
> > python implementation.  If using C is a concern can some of the same
> > improvements be made in python?  Even if they're not as dramatic any
> > speed increase for yum is a big win.
>
> The algorithm is exactly the same, the output is exactly the same. The
> only thing that's done differently is the removing of old content. The
> speed improvement is only because of "C is faster than python".
>
> Tambet
>
> >
> > On 5/10/06, seth vidal <skvidal at linux.duke.edu> wrote:
> >         On Wed, 2006-05-10 at 15:45 +0300, Tambet Ingo wrote:
> >         > email message attachment, "Forwarded message - [Yum]
> >         metadata parser
> >         > in C"
> >         > > -------- Forwarded Message --------
> >         > > From: Tambet Ingo < tambet at ximian.com>
> >         > > Reply-To: Yellowdog Updater, Modified
> >         <yum at lists.dulug.duke.edu>
> >         > > To: Yellowdog Updater, Modified <
> >         yum at lists.dulug.duke.edu>
> >         > > Subject: [Yum] metadata parser in C
> >         > > Date: Wed, 10 May 2006 14:03:01 +0300
> >         > >
> >         > > Hey,
> >         > >
> >         > > Attached is a yum metadata parser written in C. It should
> >         produce
> >         > > identical results with the sqlitecache implementation in
> >         yum-2.6.1 but
> >         > > should be quite a bit faster. Here are the numbers parsing
> >         FC5 core
> >         > > metadata (2207 packages):
> >         > >
> >
> >         Hi Tambet,
> >         Thanks for fwding this onto this list - it's just a better
> >         place for
> >         the discussion and a number of folks who are developers aren't
> >         on
> >         yum-list b/c it's mostly for user questions.
> >
> >         Also thanks for working on this, it looks cool. I'm going to
> >         give you
> >         some initial thoughts and since I've just jumped out of the
> >         shower some
> >         extended thoughts, too. So take all of it inclusively and
> >         we'll see what
> >         comes out, okay?
> >
> >         Initial thoughts:
> >         Yay Speed!!!
> >         Ugh! C.
> >         Hey, look, someone from ximian contributing, cool.
> >
> >         I like the initial speed results you've posted - those look
> >         great but
> >         all of the parsing being in C concerns me some if only due to
> >         maintenance issues.  It's not necessarily a show-stopper just
> >         a
> >         show-complicator. :)
> >
> >
> >         Post-shower thoughts:
> >         - If we have the speed up granted by your code then we can
> >         rework the
> >         sqlite db format (especially for filelists) to vastly simplify
> >         and speed
> >         up our all of the 'whatProvides' calls that have to look up
> >         files. This
> >         makes depsolving faster.
> >         - the metadata parsing code is, for obvious reasons, some of
> >         the least
> >         changed code in yum. So modifying it wouldn't be a massively
> >         frequent
> >         concern
> >         - this makes all of the metadata importing code obsolete (to
> >         some
> >         extent) which does get rid of a bunch of code that has needed
> >         some
> >         quailty time with an editor. In short it is a parallel thought
> >         to a goal
> >         for the development series.
> >
> >
> >         So that's the majority of what I've been thinking of right
> >         now. What do
> >         other folks think?
> >
> >         -sv
> >
> >
> >
> >         _______________________________________________
> >         Yum-devel mailing list
> >         Yum-devel at linux.duke.edu
> >         https://lists.dulug.duke.edu/mailman/listinfo/yum-devel
> >
> > _______________________________________________
> > Yum-devel mailing list
> > Yum-devel at linux.duke.edu
> > https://lists.dulug.duke.edu/mailman/listinfo/yum-devel
>
> _______________________________________________
> Yum-devel mailing list
> Yum-devel at linux.duke.edu
> https://lists.dulug.duke.edu/mailman/listinfo/yum-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.baseurl.org/pipermail/yum-devel/attachments/20060510/e57709db/attachment.htm 


More information about the Yum-devel mailing list