[Yum-devel] urlgrabber caching dns

Ryan Tomayko rtomayko at naeblis.cx
Sun Mar 28 20:14:04 UTC 2004


On Sat, 2004-03-27 at 15:22 -0500, seth vidal wrote:
> Hey,
>  I was wondering if urlgrabber is caching dns look ups internally
> anywhere? 

There's definitely not any name caching at the urlgrabber level today.
We would have to dig into urllib2 or maybe even python's low level
socket stuff to see if this is happening at all. 

I thought name caching was handled at the kernel level in most
circumstances? (NOTE: I'm out of my realm of real knowledge here, just
regurgitating stuff I've read.)

> I noticed when I had a fairly dumb resolv.conf configuration
> the other day that the lookup time for each connection was very slow and
> it repeated for each download. This surprised me so I was wondering
> about that for the new urlgrabber.

Hmmm. Are you thinking that it was cycling through the resolv.conf
"search" list each time or something like that? If you put IPs into yum.
conf do you see a significant increase in name resolution time? 

Here's the thing, I can't think of where we would add name caching if we
wanted to. We pretty much just hand URLs off to urllib2. urllib2 breaks
the URLs down and hands the info off to httplib or ftplib to perform all
the heavy lifting wrt resolving names and establishing connections and
whatnot. The takeaway is that if this is a name resolution caching
problem, it's probably out of our hands and into python core land. 

- Ryan





More information about the Yum-devel mailing list