[Yum-devel] urlgrabber and mirrors and failover

[Michael Stenner]

Replying to ryan's reply in order to keep this a linear thread.

On Thu, Mar 18, 2004 at 09:00:51AM -0500, Ryan Tomayko wrote:
> On Thu, 2004-03-18 at 01:51 -0500, seth vidal wrote:
> > So I figure we can let urlgrabber handle the mirrorgroups and failover
> > stuff internally as we talked about the other night. so that being the
> > case then I need to work on the repos stuff to create a urlgrabber
> > instance for each repo instance. Then each repo can call to it's very
> > own urlgrabber instance. Disadvantage of this is, of course, that for
> > the same server but different repos we lose out a bit on the keepalive -
> > b/c they'll be a different connection but to the same server.
> 
> It should still work I think. We reuse the same keepalive instance for
> all URLGrabber instances.

I hadn't thought of that, but it is of course correct.  I was going to
say that you can use the same grabber in multiple mirror groups (which
is also correct) and may save a little overhead.  The only
disadvantage of this is that you can't as easily use different grabber
options for each of them, then.

You can do crazy shit like this, though:

defaultgrabber = URLGrabber(reget=None)
foograbber = URLGrabber(reget='simple')
bargrabber = URLGrabber(reget='check_timestamp'

base_mirror = MirrorGroup(defaultgrabber, \
  [ {mirror="http://foo.com/", grabber=foograbber},
    {mirror="http://bar.com/", grabber=bargrabber}])

update_mirror = MirrorGroup(defaultgrabber, \
  [ {mirror="http://foo.com/update/", grabber=foograbber},
    {mirror="http://bar.com/update/", grabber=bargrabber}])

This will use the same grabber (foograbber) for all connections to
foo.com, and will use bargrabber for all connections to bar.com.

It's pretty flexible :)

> > Also - what are y'alls feelings on how we should deal with the
> > urlgrabber dependency from yum.
> >
> > should urlgrabber go out with yum or should yum just require it like any
> > other dep?
> 
> I imagine you would have to package w/ yum unless we can get grabber
> into core. It might not be bad to have a separate package but I don't
> know what the politics are there.

I think that's the way to go, at least for now.  Let me confirm.
We're not talking about FC2, right?  Yum head won't be ready for that,
and neither will urlgrabber 3.0.  So, if we're talking FC3, it would
probably be reasonable to get urlgrabber into core.  So....

If you think it's a good idea to have urlgrabber in core (which will
be a lot more reasonable if something other than yum wants to use it)
then I would say
  a) include urlgrabber IN the yum package, but simultaneously
  b) lobby to have it included
  c) when/if it gets included, start requiring it and remove it from
     yum

How's that sound?

Hmm.. one more thought (and now we're dipping into politics): if seth
said "you NEED to include this in core because yum now requires it",
that might be more persuasive :)

					-Michael
-- 
  Michael D. Stenner                            mstenner at ece.arizona.edu
  ECE Department, the University of Arizona                 520-626-1619
  1230 E. Speedway Blvd., Tucson, AZ 85721-0104                 ECE 524G