[Yum-devel] URLGrabber package layout and [FW: Yum usage of URLGrabber]
Ryan Tomayko
rtomayko at naeblis.cx
Sun Feb 22 06:42:49 UTC 2004
I've looked at the urlgrabber usage on yum HEAD config.py and it looks
pretty good. One thing I noticed was that you brought the new urlgrabber
sources into a directory named "urlgrabber". The distutils script for
urlgrabber installs it under a package named "URLGrabber" (case). So,
right now there's a slight mismatch in package names between yum and
urlgrabber proper. This isn't a big deal if yum will always embed
urlgrabber but if you ever wanted to split the two off, we would have to
change some yum code.
Before I suggest changing stuff in yum, I figured this might be a good
time to revisit the package naming discussions we had a while back for
urlgrabber. Partially because it would be good to have it recorded on
the list and partially because we might want to evaluate it to see if we
want to change it a little before switching yum around. The current
package layout is:
URLGrabber
__init__.py
grabber.py
progress.py
keepalive.py
byterange.py
Some notes on how this came about:
* The case on the URLGrabber package name was a flip-a-coin thing if I
remember correctly. I think we chose camelcase based on looking at other
packages under site-packages. (Am I forgetting something Michael?)
* grabber.py was renamed from urlgrabber.py for some very good reasons
given by Michael. Mostly to distinguish it from the package. import
URLGrabber.urlgrabber looks kind of silly.
One thing I've found weird with this layout is that both the package and
the main class are named URLGrabber. So, you get weird looking imports
some times:
import URLGrabber.grabber.URLGrabber
from URLGrabber.grabber import URLGrabber
Not a huge deal by any means but a little weird looking.
So, what do you guys think? Do we need to discuss the urlgrabber package
layout further or does that look okay? If we like what we have, we
should probably move some things around in yum.
- Ryan
I figured I would forward the following so we have it recorded.
-----Forwarded Message-----
From: Ryan Tomayko <rtomayko at naeblis.cx>
To: Seth Vidal <skvidal at phy.duke.edu>
Cc: Michael Stenner <mstenner at ece.arizona.edu>
Subject: Yum usage of URLGrabber
Date: Sat, 14 Feb 2004 23:58:48 -0500
Seth,
I wanted to respond to a couple of questions you've asked about
urlgrabber wrt usage in yum. I figure now would be a good time to ramble
a bit on usage. Maybe we can fold some of this into a doc for urlgrabber
somewhere.
There have been a couple of questions like "How do I set XXX option?"
and "Is XXX option a module level option or instance level option?".
Basically, the way things are working right now, all kwargs can be set
and overridden at the module level, URLGrabber instance level, and/or
request (urlXXX function call) level. The kwargs are documented in the
module level __doc__ in grabber.py and cover pretty much everything that
is configurable. I'm going to run through various usage scenarios and
then suggest what I think might be the best option for yum.
First, the grabber.URLGrabber class is pretty much the center of the
urlgrabber universe now. The module level urlgrab, urlopen, and urlread
are one line functions that call a default/global instance of URLGrabber
(default_grabber). The default values for all kwargs can be found in the
grabber.URLGrabberOptions class (look in __init__). Here are the current
values for your convenience:
self.progress_obj = None
self.throttle = 1.0
self.bandwidth = 0
self.retry = None
self.retrycodes = [-1,2,4,5,6,7]
self.checkfunc = None
self.copy_local = 0
self.close_connection = 0
self.range = None
self.user_agent = 'urlgrabber/%s' % VERSION
self.keepalive = 1
self.proxies = None
To set any of these at the module level--meaning for module level urlXXX
functions--you would use something like this:
<<
import URLGrabber.grabber as urlgrabber
urlgrabber.default_grabber.opts.throttle = 0.3
urlgrabber.default_grabber.opts.retry = 10
urlgrabber.default_grabber.opts.user_agent = 'Yum!'
>>
At this point, calling the following types of functions will use the
defaults you set. You can override any of the defaults using kwargs to
the method calls.
# use all defaults
urlgrabber.urlgrab('http://bla.com/file', 'localfile')
# override some defaults
urlgrabber.urlgrab('http://bla.com/file', 'localfile', throttle=0.5,
retry=7)
One thing to note is that setting options on the default_grabber
instance does not effect new instances of the URLGrabber class. New
instances always start with the defaults listed above.
So, now forget all that about module level functions and defaults
because I think the best thing to do in yum would be to create your own
URLGrabber instance and use it in place of the module level functions.
What you want to do here is have config.py or yummain.py or some other
module that is loaded really early create a URLGrabber instance, set
defaults, and then use that from the rest of yum. For example, there are
some urlgrabber related options set in yum.conf so let's say we create
this in config.py as follows:
<<
import URLGrabber.grabber as urlgrabber
# create the URLGrabber
yumgrabber = urlgrabber.URLGrabber(user_agent='Yum', progress_obj=yumpo)
# read some values from config file:
yumgrabber.opts.throttle = config.get('throttle', 1.0)
yumgrabber.opts.retry = config.get('retries', 1.0)
yumgrabber.opts.keepalive = config.get('keepalive', 1.0)
>>
Next, you want to import this "yumgrabber" instance into other modules
instead of using the URLGrabber module directly. So, if yumgrabber was a
global in config.py, from yummain.py you would do something like:
<<
from config import yumgrabber
yumgrabber.urlgrab(url, filename)
# or
yumgrabber.urlgrab(url, filename, range=(200,500))
>>
The benefit of this approach, IMO, is that you have your own instance of
URLGrabber that does not effect (and is not effected by) the module
level options.
Also, I wanted to note after running through some of those examples that
these are all hypothetical wrt yum code. i.e. config.py probably isn't
the right place to create the URLGrabber instance, I used it as an
first-guess example. I still need to take a peek at what you have
cooking on the HEAD for yum. I might be able to make some real
suggestions at that point.
Sorry if this was a little disconnected. I wanted to get that out before
you got to deep into integrating this stuff into yum. Also, some of this
might change based on what Michael wants to do after he sweeps the code.
I think the interface is pretty close to what was envisioned but I don't
want to make any promises.
Hope this helps,
- Ryan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.baseurl.org/pipermail/yum-devel/attachments/20040222/20aa1e02/attachment.pgp
More information about the Yum-devel
mailing list