[Yum-devel] [UG] keepalive

[James Antill]

On Sun, 2009-02-15 at 23:57 -0500, Andrew Trusty wrote:
> So I've been using urlgrabber for awhile in a number of projects and I
> love it because it usually works great.  In my most recent project I'm
> using it to access Google's ajax translation api.  Since I'm
> translating a lot of stuff I keep urlgrabber a lot busier than in my
> previous projects.  I noticed though, that after fetching translations
> for around 1000 words my python process would run out of available
> file descriptors to use and urlgrabber and other parts of the code
> that used files would fail.  I figured out that urlgrabber was keeping
> a socket open for each word I translated since keepalive is on by
> default.  So my current simple fix is to pass the close_connection=1
> keyword argument to make keepalive close its sockets.
> 
> But I'm curious why I have to do this.  Isn't the point of keepalive
> that further requests (all of which go to the google translation
> servers) should use the same socket and not open new sockets?  Or do I
> need to use urlgrabber in a different way (currently I'm just using
> the urlread with the default_grabber)?

 Search bugzilla.redhat.com ... there are "bugs" in python/urllib2 which
leaks file descriptors.