[Yum-devel] urlgrabber fixes and other stuff...

[Michael Stenner]

On Sun, Jul 18, 2004 at 07:39:42PM -0400, Ryan Tomayko wrote:
> I'm finally starting to free up after a whole bunch of life altering
> events (babies, houses, strikes, etc.).

I've been looking for something to poke at me, too.  I've gotten out
of the habit of coding and I'd like to get back in to it.  Work got
crazy for a while.  It still takes lots of my time, but I have enough
to do some useful coding, so I should.

> I had a chance to dig into the urlgrabber issue [1,2] that I left
> out there a few months ago related to urllib2 opener caching. I
> believe I have the caching piece fixed.  Caching the openers speeds
> things up considerably and maybe more importantly, seems to cut mem
> usage in half; at least it does during the grabberperf.py tests. I'm
> feeling good about committing this to the urlgrabber module but I
> wanted to send something out first to make sure things aren't frozen
> for anything. Are there any restrictions on committing right now? If
> not, I can commit this immediately.

No.  No restrictions.  I think you and I dropped off the planet at
pretty much the same time.  Nothing special has been happening.  I
just haven't been working on it much.

> Also, Michael, how do you feel about urlgrabber right now? I'm feeling
> like we're probably solid enough to put out an 0.1 tarball and make a
> formal announcement.

I think I'd prever 2.9.x until we get the threaded stuff working, but
otherwise, I think I agree with you.  It's a shame to let that
threading stuff hold everything else up.

> We could submit it to PyPI at the very least.

Agreed.

> You had mentioned previously that there were a few other places you
> would like to make an announcement...  I can't seem to find it now.

Yeah.  I have a list around here somewhere.

> Anyway, I have some time to put toward getting a grabber release out
> if you can think of things to do. Having a few more eyeballs on
> grabber should help yum out a bit too as we might be able to get
> through a couple bug fix cycles before yum HEAD goes out.

Basically, I think it's pretty close to ready to go.  Lets hold off on
the threading stuff for this release.  I'll need to think a little
about how to handle the progress meter thing: whether to keep the old
thing or to try and put in the new one which contains some of the
threading code.  I'll try and take a look at that tonight.

Otherwise, you can just look around and try to find things that need
tidying or fixing.  I don't think there's much, though.

Looking forward to getting back into it.

					-Michael

-- 
  Michael D. Stenner                            mstenner at ece.arizona.edu
  ECE Department, the University of Arizona                 520-626-1619
  1230 E. Speedway Blvd., Tucson, AZ 85721-0104                 ECE 524G