[Yum-devel] clean up sooner

[David Lutterkort]

On Tue, 2006-02-21 at 00:07 -0500, seth vidal wrote:
> On Mon, 2006-02-20 at 12:38 -0800, David Lutterkort wrote:
> > On Mon, 2006-02-20 at 10:48 -0500, seth vidal wrote:
> > > > I don't think there's anything wrong with using a sqlite DB, but it
> > > > might be simpler to just use flat files: before actually starting the
> > > > transaction, write out what needs to be done into a file in a format
> > > > that is convenient to parse back in. 
> > > 
> > > xml. No point in maintaining N file formats just for fun, if we don't
> > > have to.
> > 
> > It's a new file format either way. Do you think that XML is needed
> > because the structure of the file will require nontrivial parsing or
> > that these files might be interesting to other programs ? Also, XML is
> > not suited for writing out data where the stream could be interrupted at
> > any moment; if yum crashes half way through writing, the XML is not
> > well-formed.
> 
> fair enough - but that's true of any text format, isn't it?
> who knows when the write will die?

If you write plain text files where each line corresponds to one record
as I described in my initial mail, it's pretty easy to deal with teh
incomplete file. At the very least, you can always read all but the last
record. With XML, the parser will start complaining at some point; it
probably depends on the parser how much of the incomplete file you get
to see before the parser gives up.

> > The only difference between your and my approach is when you do the work
> > you describe above - during the transaction or when yum starts. With
> > files, you can avoid parsing the file altogether if you maintain a
> > separate lockfile during the transaction and delete it afterwards.
> 
> So you'd be maintaining two files, one of them simply as a lock file -
> which can also be done how I was describing, can't it?

Absolutely. I am not saying that you couldn't do all these things with
sqlite.

> > 
> > > I'm curious if writing out an sqlite db is that much less cheap for this
> > > little an amount of data.
> > 
> > Only profiling can tell us that; though I would assume that there is
> > some nontrivial startup cost for any sqlite DB.
> 
> not much - have you ever used sqlite?

Yes, but I never really looked at how fast it is compared to the rest of
what the program is doing. Do you have some profiling data ?

> > > > If you use one file per transaction, you could even keep them around as
> > > > an audit log of what yum did on a given system, and people could simply
> > > > truncate the logs with an appropriate tmpwatch job. That kinda thing is
> > > > a little more hairy if you use a DB.
> > > 
> > > You can do the same with a sqlite db per transaction, too.
> > 
> > But you have to write special-purpose code to let the user indicate how
> > much of an audit log to keep - with tmpwatch it's a small configfile,
> > that gives the user a lot of flexibility in describing how much to keep.
> 
> I'm talking about using tmpwatch.
> 
> sqlite is just a db in a file.
> 
> so you make one per transaction and use tmpwatch the exact same way you
> were describing.

Good point; I had thought you wanted one DB for all transactions.

I really just wanted to throw the flat file biz out there as another
idea - after all, I am not doing the actual work to implement this;
Dennis will know much better which will be the best way to do all this.

David