[Yum-devel] [UG] HTTP POST with Urlgrabber?

William McVey wam at cisco.com
Tue Oct 25 21:41:34 UTC 2005


I'm using urlgrabber to build a python API to a bugzilla's CGI scripts
(not the most elegant method of interfacing with a bugzilla instance,
but lacking a SOAP or XML-RPC backend to bugzilla, it's what is
available). Anyway, I'm having real difficulty getting urlgrabber to do
POST method http transactions. For example:

	$ nc -l -p 9090 # Setup a fake web server to see what gets sent

In another window:
        
        $ python
        Python 2.4 (#2, Feb 12 2005, 00:29:46) 
        [GCC 3.4.3 (Mandrakelinux 10.2 3.4.3-3mdk)] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> from urlgrabber.grabber import URLGrabber
        >>> grabber = URLGrabber(timeout=10.0)
        >>> grabber.urlopen("http://localhost:9090/foo", data="parameter=en&email=foo%40email.com")

And back in the other window:
	
        GET /foo HTTP/1.1
        Host: localhost:9090
        Accept-Encoding: identity
        User-agent: urlgrabber/2.9.6

Documentation at:
http://linux.duke.edu/projects/urlgrabber/help/urlgrabber.grabber.html
indicates that URLGrabber objects as well as urlopen() methods on
URLGrabber objects can take kwargs and that 'data' option:
    
    Only relevant for the HTTP family (and ignored for other
    protocols), this allows HTTP POSTs.  When the data kwarg is
    present (and not None), an HTTP request will automatically become
    a POST rather than GET.  This is done by direct passthrough to
    urllib2.  If you use this, you may also want to set the
    'Content-length' and 'Content-type' headers with the http_headers
    option.  Note that python 2.2 handles the case of these
    badly and if you do not use the proper case (shown here), your
    values will be overridden with the defaults.

Note that *something* isn't being passed through properly to urllib2
since if I execute:
        >>> import urllib2
        >>> urllib2.urlopen("http://localhost:9090/foo", data="parameter=en&email=you%40yourmail.com")
        
My netcat shows:
        $ nc -l -p 9090 
        POST /foo HTTP/1.1
        Accept-Encoding: identity
        Content-length: 37
        Connection: close
        Content-type: application/x-www-form-urlencoded
        Host: localhost:9090
        User-agent: Python-urllib/2.4
        
        parameter=en&email=you%40yourmail.com

I tried going through the unittests that are distributed with
urlgrabber, and but I couldn't find any testcases that issue a 'POST'.
Any help would be very much appreciated.

  -- William



More information about the Yum-devel mailing list