[Yum-devel] [UG] openers and keepalive

[Michael Stenner]

OK, I mentioned yesterday that I was having problems with some of the
reget stuff working and with keepalive.

1) keepalive

   Well, the keepalive thing turned out to be a non-issue.  The apache
   that comes with FC1 has keepalive off by default.  Since we just
   upgraded www.linux.duke.edu (the test server, among other things)
   keepalive stopped working.  Not a problem in urlgrabber.

2) opener-caching broken

   The opener-caching behavior seems to not quite be working right
   yet, so I disabled it for now.  It's a very simple and obvious
   modification.  So, for now, a new opener is built for EVERY fetch.

   I only explored this enough to find that the problem was related to
   the caching.  I found it illuminating to dump the contents of
   opener.handle_errors and opener.handlers before each fetch.  That
   clearly showed that somehow, support for the right stuff found
   itself missing.  This only showed up in the test suites where LOTS
   of stuff was happening.  All my simple tests worked.

   This is one of the downsides of using global structures, I guess.
   We should also keep the threadsafety of such things in mind.

   Ryan, if you pursue this, let me know.  I'll do the same.  That
   way, we won't duplicate effort.  For now, I'm going to keep working
   on the progress meter stuff.

3) custom openers
   
   In talking with a potential user on irc, I was reminded that it
   would be nice to allow custom openers/handlers.  I figured we
   should address this at the same time since it's basically the same
   code.  Here are the reasons I can think of why someone might want
   to customize things:

    * introduce support for another protocol or option
    * REMOVE support for a protocol or option (the user I chatted with
      wants to REMOVE file:// support for security reasons)

   Now, the biggest problem I can think of is that there doesn't seem
   to be a tidy way to ADD handlers to an existing opener, and there
   is almost certainly no tidy way to REMOVE handlers from one.

   The simplest thing we could do would be to provide a "opener"
   option.  Then, one could specify a custom opener for the grabber,
   or per-fetch, as usual.  If not provided, we could fall back our
   grabber building/caching policy.  The biggest problem with this is
   that the caller would be completely responsible for building it
   with support for all the necessary stuff: byteranges, keepalive,
   proxies.  Therefore, the presence options like "proxy" and
   "keepalive" would have no effect if "opener" is used.  Of these,
   the most annoying is the proxy thing.

   Thoughts?

					-Michael

-- 
  Michael D. Stenner                            mstenner at ece.arizona.edu
  ECE Department, the University of Arizona                 520-626-1619
  1230 E. Speedway Blvd., Tucson, AZ 85721-0104                 ECE 524G