diff options
Diffstat (limited to 'Doc/howto/urllib2.rst')
| -rw-r--r-- | Doc/howto/urllib2.rst | 578 |
1 files changed, 578 insertions, 0 deletions
diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst new file mode 100644 index 0000000..dc20b02 --- /dev/null +++ b/Doc/howto/urllib2.rst @@ -0,0 +1,578 @@ +************************************************ + HOWTO Fetch Internet Resources Using urllib2 +************************************************ + +:Author: `Michael Foord <http://www.voidspace.org.uk/python/index.shtml>`_ + +.. note:: + + There is an French translation of an earlier revision of this + HOWTO, available at `urllib2 - Le Manuel manquant + <http://www.voidspace/python/articles/urllib2_francais.shtml>`_. + + + +Introduction +============ + +.. sidebar:: Related Articles + + You may also find useful the following article on fetching web resources + with Python : + + * `Basic Authentication <http://www.voidspace.org.uk/python/articles/authentication.shtml>`_ + + A tutorial on *Basic Authentication*, with examples in Python. + +**urllib2** is a `Python <http://www.python.org>`_ module for fetching URLs +(Uniform Resource Locators). It offers a very simple interface, in the form of +the *urlopen* function. This is capable of fetching URLs using a variety of +different protocols. It also offers a slightly more complex interface for +handling common situations - like basic authentication, cookies, proxies and so +on. These are provided by objects called handlers and openers. + +urllib2 supports fetching URLs for many "URL schemes" (identified by the string +before the ":" in URL - for example "ftp" is the URL scheme of +"ftp://python.org/") using their associated network protocols (e.g. FTP, HTTP). +This tutorial focuses on the most common case, HTTP. + +For straightforward situations *urlopen* is very easy to use. But as soon as you +encounter errors or non-trivial cases when opening HTTP URLs, you will need some +understanding of the HyperText Transfer Protocol. The most comprehensive and +authoritative reference to HTTP is :rfc:`2616`. This is a technical document and +not intended to be easy to read. This HOWTO aims to illustrate using *urllib2*, +with enough detail about HTTP to help you through. It is not intended to replace +the :mod:`urllib2` docs, but is supplementary to them. + + +Fetching URLs +============= + +The simplest way to use urllib2 is as follows:: + + import urllib2 + response = urllib2.urlopen('http://python.org/') + html = response.read() + +Many uses of urllib2 will be that simple (note that instead of an 'http:' URL we +could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the +purpose of this tutorial to explain the more complicated cases, concentrating on +HTTP. + +HTTP is based on requests and responses - the client makes requests and servers +send responses. urllib2 mirrors this with a ``Request`` object which represents +the HTTP request you are making. In its simplest form you create a Request +object that specifies the URL you want to fetch. Calling ``urlopen`` with this +Request object returns a response object for the URL requested. This response is +a file-like object, which means you can for example call ``.read()`` on the +response:: + + import urllib2 + + req = urllib2.Request('http://www.voidspace.org.uk') + response = urllib2.urlopen(req) + the_page = response.read() + +Note that urllib2 makes use of the same Request interface to handle all URL +schemes. For example, you can make an FTP request like so:: + + req = urllib2.Request('ftp://example.com/') + +In the case of HTTP, there are two extra things that Request objects allow you +to do: First, you can pass data to be sent to the server. Second, you can pass +extra information ("metadata") *about* the data or the about request itself, to +the server - this information is sent as HTTP "headers". Let's look at each of +these in turn. + +Data +---- + +Sometimes you want to send data to a URL (often the URL will refer to a CGI +(Common Gateway Interface) script [#]_ or other web application). With HTTP, +this is often done using what's known as a **POST** request. This is often what +your browser does when you submit a HTML form that you filled in on the web. Not +all POSTs have to come from forms: you can use a POST to transmit arbitrary data +to your own application. In the common case of HTML forms, the data needs to be +encoded in a standard way, and then passed to the Request object as the ``data`` +argument. The encoding is done using a function from the ``urllib`` library +*not* from ``urllib2``. :: + + import urllib + import urllib2 + + url = 'http://www.someserver.com/cgi-bin/register.cgi' + values = {'name' : 'Michael Foord', + 'location' : 'Northampton', + 'language' : 'Python' } + + data = urllib.urlencode(values) + req = urllib2.Request(url, data) + response = urllib2.urlopen(req) + the_page = response.read() + +Note that other encodings are sometimes required (e.g. for file upload from HTML +forms - see `HTML Specification, Form Submission +<http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_ for more +details). + +If you do not pass the ``data`` argument, urllib2 uses a **GET** request. One +way in which GET and POST requests differ is that POST requests often have +"side-effects": they change the state of the system in some way (for example by +placing an order with the website for a hundredweight of tinned spam to be +delivered to your door). Though the HTTP standard makes it clear that POSTs are +intended to *always* cause side-effects, and GET requests *never* to cause +side-effects, nothing prevents a GET request from having side-effects, nor a +POST requests from having no side-effects. Data can also be passed in an HTTP +GET request by encoding it in the URL itself. + +This is done as follows:: + + >>> import urllib2 + >>> import urllib + >>> data = {} + >>> data['name'] = 'Somebody Here' + >>> data['location'] = 'Northampton' + >>> data['language'] = 'Python' + >>> url_values = urllib.urlencode(data) + >>> print url_values + name=Somebody+Here&language=Python&location=Northampton + >>> url = 'http://www.example.com/example.cgi' + >>> full_url = url + '?' + url_values + >>> data = urllib2.open(full_url) + +Notice that the full URL is created by adding a ``?`` to the URL, followed by +the encoded values. + +Headers +------- + +We'll discuss here one particular HTTP header, to illustrate how to add headers +to your HTTP request. + +Some websites [#]_ dislike being browsed by programs, or send different versions +to different browsers [#]_ . By default urllib2 identifies itself as +``Python-urllib/x.y`` (where ``x`` and ``y`` are the major and minor version +numbers of the Python release, +e.g. ``Python-urllib/2.5``), which may confuse the site, or just plain +not work. The way a browser identifies itself is through the +``User-Agent`` header [#]_. When you create a Request object you can +pass a dictionary of headers in. The following example makes the same +request as above, but identifies itself as a version of Internet +Explorer [#]_. :: + + import urllib + import urllib2 + + url = 'http://www.someserver.com/cgi-bin/register.cgi' + user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)' + values = {'name' : 'Michael Foord', + 'location' : 'Northampton', + 'language' : 'Python' } + headers = { 'User-Agent' : user_agent } + + data = urllib.urlencode(values) + req = urllib2.Request(url, data, headers) + response = urllib2.urlopen(req) + the_page = response.read() + +The response also has two useful methods. See the section on `info and geturl`_ +which comes after we have a look at what happens when things go wrong. + + +Handling Exceptions +=================== + +*urlopen* raises ``URLError`` when it cannot handle a response (though as usual +with Python APIs, builtin exceptions such as ValueError, TypeError etc. may also +be raised). + +``HTTPError`` is the subclass of ``URLError`` raised in the specific case of +HTTP URLs. + +URLError +-------- + +Often, URLError is raised because there is no network connection (no route to +the specified server), or the specified server doesn't exist. In this case, the +exception raised will have a 'reason' attribute, which is a tuple containing an +error code and a text error message. + +e.g. :: + + >>> req = urllib2.Request('http://www.pretend_server.org') + >>> try: urllib2.urlopen(req) + >>> except URLError, e: + >>> print e.reason + >>> + (4, 'getaddrinfo failed') + + +HTTPError +--------- + +Every HTTP response from the server contains a numeric "status code". Sometimes +the status code indicates that the server is unable to fulfil the request. The +default handlers will handle some of these responses for you (for example, if +the response is a "redirection" that requests the client fetch the document from +a different URL, urllib2 will handle that for you). For those it can't handle, +urlopen will raise an ``HTTPError``. Typical errors include '404' (page not +found), '403' (request forbidden), and '401' (authentication required). + +See section 10 of RFC 2616 for a reference on all the HTTP error codes. + +The ``HTTPError`` instance raised will have an integer 'code' attribute, which +corresponds to the error sent by the server. + +Error Codes +~~~~~~~~~~~ + +Because the default handlers handle redirects (codes in the 300 range), and +codes in the 100-299 range indicate success, you will usually only see error +codes in the 400-599 range. + +``BaseHTTPServer.BaseHTTPRequestHandler.responses`` is a useful dictionary of +response codes in that shows all the response codes used by RFC 2616. The +dictionary is reproduced here for convenience :: + + # Table mapping response codes to messages; entries have the + # form {code: (shortmessage, longmessage)}. + responses = { + 100: ('Continue', 'Request received, please continue'), + 101: ('Switching Protocols', + 'Switching to new protocol; obey Upgrade header'), + + 200: ('OK', 'Request fulfilled, document follows'), + 201: ('Created', 'Document created, URL follows'), + 202: ('Accepted', + 'Request accepted, processing continues off-line'), + 203: ('Non-Authoritative Information', 'Request fulfilled from cache'), + 204: ('No Content', 'Request fulfilled, nothing follows'), + 205: ('Reset Content', 'Clear input form for further input.'), + 206: ('Partial Content', 'Partial content follows.'), + + 300: ('Multiple Choices', + 'Object has several resources -- see URI list'), + 301: ('Moved Permanently', 'Object moved permanently -- see URI list'), + 302: ('Found', 'Object moved temporarily -- see URI list'), + 303: ('See Other', 'Object moved -- see Method and URL list'), + 304: ('Not Modified', + 'Document has not changed since given time'), + 305: ('Use Proxy', + 'You must use proxy specified in Location to access this ' + 'resource.'), + 307: ('Temporary Redirect', + 'Object moved temporarily -- see URI list'), + + 400: ('Bad Request', + 'Bad request syntax or unsupported method'), + 401: ('Unauthorized', + 'No permission -- see authorization schemes'), + 402: ('Payment Required', + 'No payment -- see charging schemes'), + 403: ('Forbidden', + 'Request forbidden -- authorization will not help'), + 404: ('Not Found', 'Nothing matches the given URI'), + 405: ('Method Not Allowed', + 'Specified method is invalid for this server.'), + 406: ('Not Acceptable', 'URI not available in preferred format.'), + 407: ('Proxy Authentication Required', 'You must authenticate with ' + 'this proxy before proceeding.'), + 408: ('Request Timeout', 'Request timed out; try again later.'), + 409: ('Conflict', 'Request conflict.'), + 410: ('Gone', + 'URI no longer exists and has been permanently removed.'), + 411: ('Length Required', 'Client must specify Content-Length.'), + 412: ('Precondition Failed', 'Precondition in headers is false.'), + 413: ('Request Entity Too Large', 'Entity is too large.'), + 414: ('Request-URI Too Long', 'URI is too long.'), + 415: ('Unsupported Media Type', 'Entity body in unsupported format.'), + 416: ('Requested Range Not Satisfiable', + 'Cannot satisfy request range.'), + 417: ('Expectation Failed', + 'Expect condition could not be satisfied.'), + + 500: ('Internal Server Error', 'Server got itself in trouble'), + 501: ('Not Implemented', + 'Server does not support this operation'), + 502: ('Bad Gateway', 'Invalid responses from another server/proxy.'), + 503: ('Service Unavailable', + 'The server cannot process the request due to a high load'), + 504: ('Gateway Timeout', + 'The gateway server did not receive a timely response'), + 505: ('HTTP Version Not Supported', 'Cannot fulfill request.'), + } + +When an error is raised the server responds by returning an HTTP error code +*and* an error page. You can use the ``HTTPError`` instance as a response on the +page returned. This means that as well as the code attribute, it also has read, +geturl, and info, methods. :: + + >>> req = urllib2.Request('http://www.python.org/fish.html') + >>> try: + >>> urllib2.urlopen(req) + >>> except URLError, e: + >>> print e.code + >>> print e.read() + >>> + 404 + <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" + "http://www.w3.org/TR/html4/loose.dtd"> + <?xml-stylesheet href="./css/ht2html.css" + type="text/css"?> + <html><head><title>Error 404: File Not Found</title> + ...... etc... + +Wrapping it Up +-------------- + +So if you want to be prepared for ``HTTPError`` *or* ``URLError`` there are two +basic approaches. I prefer the second approach. + +Number 1 +~~~~~~~~ + +:: + + + from urllib2 import Request, urlopen, URLError, HTTPError + req = Request(someurl) + try: + response = urlopen(req) + except HTTPError, e: + print 'The server couldn\'t fulfill the request.' + print 'Error code: ', e.code + except URLError, e: + print 'We failed to reach a server.' + print 'Reason: ', e.reason + else: + # everything is fine + + +.. note:: + + The ``except HTTPError`` *must* come first, otherwise ``except URLError`` + will *also* catch an ``HTTPError``. + +Number 2 +~~~~~~~~ + +:: + + from urllib2 import Request, urlopen, URLError + req = Request(someurl) + try: + response = urlopen(req) + except URLError, e: + if hasattr(e, 'reason'): + print 'We failed to reach a server.' + print 'Reason: ', e.reason + elif hasattr(e, 'code'): + print 'The server couldn\'t fulfill the request.' + print 'Error code: ', e.code + else: + # everything is fine + + +info and geturl +=============== + +The response returned by urlopen (or the ``HTTPError`` instance) has two useful +methods ``info`` and ``geturl``. + +**geturl** - this returns the real URL of the page fetched. This is useful +because ``urlopen`` (or the opener object used) may have followed a +redirect. The URL of the page fetched may not be the same as the URL requested. + +**info** - this returns a dictionary-like object that describes the page +fetched, particularly the headers sent by the server. It is currently an +``httplib.HTTPMessage`` instance. + +Typical headers include 'Content-length', 'Content-type', and so on. See the +`Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_ +for a useful listing of HTTP headers with brief explanations of their meaning +and use. + + +Openers and Handlers +==================== + +When you fetch a URL you use an opener (an instance of the perhaps +confusingly-named :class:`urllib2.OpenerDirector`). Normally we have been using +the default opener - via ``urlopen`` - but you can create custom +openers. Openers use handlers. All the "heavy lifting" is done by the +handlers. Each handler knows how to open URLs for a particular URL scheme (http, +ftp, etc.), or how to handle an aspect of URL opening, for example HTTP +redirections or HTTP cookies. + +You will want to create openers if you want to fetch URLs with specific handlers +installed, for example to get an opener that handles cookies, or to get an +opener that does not handle redirections. + +To create an opener, instantiate an ``OpenerDirector``, and then call +``.add_handler(some_handler_instance)`` repeatedly. + +Alternatively, you can use ``build_opener``, which is a convenience function for +creating opener objects with a single function call. ``build_opener`` adds +several handlers by default, but provides a quick way to add more and/or +override the default handlers. + +Other sorts of handlers you might want to can handle proxies, authentication, +and other common but slightly specialised situations. + +``install_opener`` can be used to make an ``opener`` object the (global) default +opener. This means that calls to ``urlopen`` will use the opener you have +installed. + +Opener objects have an ``open`` method, which can be called directly to fetch +urls in the same way as the ``urlopen`` function: there's no need to call +``install_opener``, except as a convenience. + + +Basic Authentication +==================== + +To illustrate creating and installing a handler we will use the +``HTTPBasicAuthHandler``. For a more detailed discussion of this subject -- +including an explanation of how Basic Authentication works - see the `Basic +Authentication Tutorial +<http://www.voidspace.org.uk/python/articles/authentication.shtml>`_. + +When authentication is required, the server sends a header (as well as the 401 +error code) requesting authentication. This specifies the authentication scheme +and a 'realm'. The header looks like : ``Www-authenticate: SCHEME +realm="REALM"``. + +e.g. :: + + Www-authenticate: Basic realm="cPanel Users" + + +The client should then retry the request with the appropriate name and password +for the realm included as a header in the request. This is 'basic +authentication'. In order to simplify this process we can create an instance of +``HTTPBasicAuthHandler`` and an opener to use this handler. + +The ``HTTPBasicAuthHandler`` uses an object called a password manager to handle +the mapping of URLs and realms to passwords and usernames. If you know what the +realm is (from the authentication header sent by the server), then you can use a +``HTTPPasswordMgr``. Frequently one doesn't care what the realm is. In that +case, it is convenient to use ``HTTPPasswordMgrWithDefaultRealm``. This allows +you to specify a default username and password for a URL. This will be supplied +in the absence of you providing an alternative combination for a specific +realm. We indicate this by providing ``None`` as the realm argument to the +``add_password`` method. + +The top-level URL is the first URL that requires authentication. URLs "deeper" +than the URL you pass to .add_password() will also match. :: + + # create a password manager + password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() + + # Add the username and password. + # If we knew the realm, we could use it instead of ``None``. + top_level_url = "http://example.com/foo/" + password_mgr.add_password(None, top_level_url, username, password) + + handler = urllib2.HTTPBasicAuthHandler(password_mgr) + + # create "opener" (OpenerDirector instance) + opener = urllib2.build_opener(handler) + + # use the opener to fetch a URL + opener.open(a_url) + + # Install the opener. + # Now all calls to urllib2.urlopen use our opener. + urllib2.install_opener(opener) + +.. note:: + + In the above example we only supplied our ``HHTPBasicAuthHandler`` to + ``build_opener``. By default openers have the handlers for normal situations + -- ``ProxyHandler``, ``UnknownHandler``, ``HTTPHandler``, + ``HTTPDefaultErrorHandler``, ``HTTPRedirectHandler``, ``FTPHandler``, + ``FileHandler``, ``HTTPErrorProcessor``. + +``top_level_url`` is in fact *either* a full URL (including the 'http:' scheme +component and the hostname and optionally the port number) +e.g. "http://example.com/" *or* an "authority" (i.e. the hostname, +optionally including the port number) e.g. "example.com" or "example.com:8080" +(the latter example includes a port number). The authority, if present, must +NOT contain the "userinfo" component - for example "joe@password:example.com" is +not correct. + + +Proxies +======= + +**urllib2** will auto-detect your proxy settings and use those. This is through +the ``ProxyHandler`` which is part of the normal handler chain. Normally that's +a good thing, but there are occasions when it may not be helpful [#]_. One way +to do this is to setup our own ``ProxyHandler``, with no proxies defined. This +is done using similar steps to setting up a `Basic Authentication`_ handler : :: + + >>> proxy_support = urllib2.ProxyHandler({}) + >>> opener = urllib2.build_opener(proxy_support) + >>> urllib2.install_opener(opener) + +.. note:: + + Currently ``urllib2`` *does not* support fetching of ``https`` locations + through a proxy. However, this can be enabled by extending urllib2 as + shown in the recipe [#]_. + + +Sockets and Layers +================== + +The Python support for fetching resources from the web is layered. urllib2 uses +the httplib library, which in turn uses the socket library. + +As of Python 2.3 you can specify how long a socket should wait for a response +before timing out. This can be useful in applications which have to fetch web +pages. By default the socket module has *no timeout* and can hang. Currently, +the socket timeout is not exposed at the httplib or urllib2 levels. However, +you can set the default timeout globally for all sockets using :: + + import socket + import urllib2 + + # timeout in seconds + timeout = 10 + socket.setdefaulttimeout(timeout) + + # this call to urllib2.urlopen now uses the default timeout + # we have set in the socket module + req = urllib2.Request('http://www.voidspace.org.uk') + response = urllib2.urlopen(req) + + +------- + + +Footnotes +========= + +This document was reviewed and revised by John Lee. + +.. [#] For an introduction to the CGI protocol see + `Writing Web Applications in Python <http://www.pyzine.com/Issue008/Section_Articles/article_CGIOne.html>`_. +.. [#] Like Google for example. The *proper* way to use google from a program + is to use `PyGoogle <http://pygoogle.sourceforge.net>`_ of course. See + `Voidspace Google <http://www.voidspace.org.uk/python/recipebook.shtml#google>`_ + for some examples of using the Google API. +.. [#] Browser sniffing is a very bad practise for website design - building + sites using web standards is much more sensible. Unfortunately a lot of + sites still send different versions to different browsers. +.. [#] The user agent for MSIE 6 is + *'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'* +.. [#] For details of more HTTP request headers, see + `Quick Reference to HTTP Headers`_. +.. [#] In my case I have to use a proxy to access the internet at work. If you + attempt to fetch *localhost* URLs through this proxy it blocks them. IE + is set to use the proxy, which urllib2 picks up on. In order to test + scripts with a localhost server, I have to prevent urllib2 from using + the proxy. +.. [#] urllib2 opener for SSL proxy (CONNECT method): `ASPN Cookbook Recipe + <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195>`_. + |