diff options
| -rw-r--r-- | Doc/howto/urllib2.rst | 38 | ||||
| -rw-r--r-- | Doc/library/contextlib.rst | 4 | ||||
| -rw-r--r-- | Doc/library/http.client.rst | 3 | ||||
| -rw-r--r-- | Doc/library/robotparser.rst | 73 | ||||
| -rw-r--r-- | Doc/library/urllib.error.rst | 42 | ||||
| -rw-r--r-- | Doc/library/urllib.parse.rst | 58 | ||||
| -rw-r--r-- | Doc/library/urllib.request.rst | 22 | ||||
| -rw-r--r-- | Doc/library/urllib.robotparser.rst | 12 | ||||
| -rw-r--r-- | Doc/tutorial/stdlib.rst | 4 |
9 files changed, 90 insertions, 166 deletions
diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst index 6342b6e..5d32d4a 100644 --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -1,12 +1,12 @@ -***************************************************** - HOWTO Fetch Internet Resources Using urllib package -***************************************************** +*********************************************************** + HOWTO Fetch Internet Resources Using The urllib Package +*********************************************************** :Author: `Michael Foord <http://www.voidspace.org.uk/python/index.shtml>`_ .. note:: - There is an French translation of an earlier revision of this + There is a French translation of an earlier revision of this HOWTO, available at `urllib2 - Le Manuel manquant <http://www.voidspace.org.uk/python/articles/urllib2_francais.shtml>`_. @@ -18,7 +18,7 @@ Introduction .. sidebar:: Related Articles You may also find useful the following article on fetching web resources - with Python : + with Python: * `Basic Authentication <http://www.voidspace.org.uk/python/articles/authentication.shtml>`_ @@ -94,8 +94,8 @@ your browser does when you submit a HTML form that you filled in on the web. Not all POSTs have to come from forms: you can use a POST to transmit arbitrary data to your own application. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the ``data`` -argument. The encoding is done using a function from the ``urllib.parse`` library -*not* from ``urllib.request``. :: +argument. The encoding is done using a function from the :mod:`urllib.parse` +library. :: import urllib.parse import urllib.request @@ -115,7 +115,7 @@ forms - see `HTML Specification, Form Submission <http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_ for more details). -If you do not pass the ``data`` argument, urllib.request uses a **GET** request. One +If you do not pass the ``data`` argument, urllib uses a **GET** request. One way in which GET and POST requests differ is that POST requests often have "side-effects": they change the state of the system in some way (for example by placing an order with the website for a hundredweight of tinned spam to be @@ -182,13 +182,15 @@ which comes after we have a look at what happens when things go wrong. Handling Exceptions =================== -*urllib.error* raises ``URLError`` when it cannot handle a response (though as usual +*urlopen* raises ``URLError`` when it cannot handle a response (though as usual with Python APIs, builtin exceptions such as ValueError, TypeError etc. may also be raised). ``HTTPError`` is the subclass of ``URLError`` raised in the specific case of HTTP URLs. +The exception classes are exported from the :mod:`urllib.error` module. + URLError -------- @@ -214,7 +216,7 @@ Every HTTP response from the server contains a numeric "status code". Sometimes the status code indicates that the server is unable to fulfil the request. The default handlers will handle some of these responses for you (for example, if the response is a "redirection" that requests the client fetch the document from -a different URL, urllib.request will handle that for you). For those it can't handle, +a different URL, urllib will handle that for you). For those it can't handle, urlopen will raise an ``HTTPError``. Typical errors include '404' (page not found), '403' (request forbidden), and '401' (authentication required). @@ -380,7 +382,7 @@ info and geturl The response returned by urlopen (or the ``HTTPError`` instance) has two useful methods ``info`` and ``geturl`` and is defined in the module -``urllib.response``. +:mod:`urllib.response`. **geturl** - this returns the real URL of the page fetched. This is useful because ``urlopen`` (or the opener object used) may have followed a @@ -388,7 +390,7 @@ redirect. The URL of the page fetched may not be the same as the URL requested. **info** - this returns a dictionary-like object that describes the page fetched, particularly the headers sent by the server. It is currently an -``http.client.HTTPMessage`` instance. +:class:`http.client.HTTPMessage` instance. Typical headers include 'Content-length', 'Content-type', and so on. See the `Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_ @@ -508,7 +510,7 @@ not correct. Proxies ======= -**urllib.request** will auto-detect your proxy settings and use those. This is through +**urllib** will auto-detect your proxy settings and use those. This is through the ``ProxyHandler`` which is part of the normal handler chain. Normally that's a good thing, but there are occasions when it may not be helpful [#]_. One way to do this is to setup our own ``ProxyHandler``, with no proxies defined. This @@ -528,8 +530,8 @@ is done using similar steps to setting up a `Basic Authentication`_ handler : :: Sockets and Layers ================== -The Python support for fetching resources from the web is layered. -urllib.request uses the http.client library, which in turn uses the socket library. +The Python support for fetching resources from the web is layered. urllib uses +the :mod:`http.client` library, which in turn uses the socket library. As of Python 2.3 you can specify how long a socket should wait for a response before timing out. This can be useful in applications which have to fetch web @@ -573,9 +575,9 @@ This document was reviewed and revised by John Lee. `Quick Reference to HTTP Headers`_. .. [#] In my case I have to use a proxy to access the internet at work. If you attempt to fetch *localhost* URLs through this proxy it blocks them. IE - is set to use the proxy, which urllib2 picks up on. In order to test - scripts with a localhost server, I have to prevent urllib2 from using + is set to use the proxy, which urllib picks up on. In order to test + scripts with a localhost server, I have to prevent urllib from using the proxy. -.. [#] urllib2 opener for SSL proxy (CONNECT method): `ASPN Cookbook Recipe +.. [#] urllib opener for SSL proxy (CONNECT method): `ASPN Cookbook Recipe <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195>`_. diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index 2cd97c2..74a68cf 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -98,9 +98,9 @@ Functions provided: And lets you write code like this:: from contextlib import closing - import urllib.request + from urllib.request import urlopen - with closing(urllib.request.urlopen('http://www.python.org')) as page: + with closing(urlopen('http://www.python.org')) as page: for line in page: print(line) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 1ea3576..bcda4c9 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -13,8 +13,7 @@ This module defines classes which implement the client side of the HTTP and HTTPS protocols. It is normally not used directly --- the module -:mod:`urllib.request` -uses it to handle URLs that use HTTP and HTTPS. +:mod:`urllib.request` uses it to handle URLs that use HTTP and HTTPS. .. note:: diff --git a/Doc/library/robotparser.rst b/Doc/library/robotparser.rst deleted file mode 100644 index cce7966..0000000 --- a/Doc/library/robotparser.rst +++ /dev/null @@ -1,73 +0,0 @@ - -:mod:`robotparser` --- Parser for robots.txt -============================================= - -.. module:: robotparser - :synopsis: Loads a robots.txt file and answers questions about - fetchability of other URLs. -.. sectionauthor:: Skip Montanaro <skip@pobox.com> - - -.. index:: - single: WWW - single: World Wide Web - single: URL - single: robots.txt - -This module provides a single class, :class:`RobotFileParser`, which answers -questions about whether or not a particular user agent can fetch a URL on the -Web site that published the :file:`robots.txt` file. For more details on the -structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. - - -.. class:: RobotFileParser() - - This class provides a set of methods to read, parse and answer questions - about a single :file:`robots.txt` file. - - - .. method:: set_url(url) - - Sets the URL referring to a :file:`robots.txt` file. - - - .. method:: read() - - Reads the :file:`robots.txt` URL and feeds it to the parser. - - - .. method:: parse(lines) - - Parses the lines argument. - - - .. method:: can_fetch(useragent, url) - - Returns ``True`` if the *useragent* is allowed to fetch the *url* - according to the rules contained in the parsed :file:`robots.txt` - file. - - - .. method:: mtime() - - Returns the time the ``robots.txt`` file was last fetched. This is - useful for long-running web spiders that need to check for new - ``robots.txt`` files periodically. - - - .. method:: modified() - - Sets the time the ``robots.txt`` file was last fetched to the current - time. - -The following example demonstrates basic use of the RobotFileParser class. :: - - >>> import robotparser - >>> rp = robotparser.RobotFileParser() - >>> rp.set_url("http://www.musi-cal.com/robots.txt") - >>> rp.read() - >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco") - False - >>> rp.can_fetch("*", "http://www.musi-cal.com/") - True - diff --git a/Doc/library/urllib.error.rst b/Doc/library/urllib.error.rst index 1cbfe7d..bd76860 100644 --- a/Doc/library/urllib.error.rst +++ b/Doc/library/urllib.error.rst @@ -2,47 +2,47 @@ ================================================================== .. module:: urllib.error - :synopsis: Next generation URL opening library. + :synopsis: Exception classes raised by urllib.request. .. moduleauthor:: Jeremy Hylton <jhylton@users.sourceforge.net> .. sectionauthor:: Senthil Kumaran <orsenthil@gmail.com> -The :mod:`urllib.error` module defines exception classes raise by -urllib.request. The base exception class is URLError, which inherits from -IOError. +The :mod:`urllib.error` module defines the exception classes for exceptions +raised by :mod:`urllib.request`. The base exception class is :exc:`URLError`, +which inherits from :exc:`IOError`. The following exceptions are raised by :mod:`urllib.error` as appropriate: - .. exception:: URLError - The handlers raise this exception (or derived exceptions) when they run into a - problem. It is a subclass of :exc:`IOError`. + The handlers raise this exception (or derived exceptions) when they run into + a problem. It is a subclass of :exc:`IOError`. .. attribute:: reason - The reason for this error. It can be a message string or another exception - instance (:exc:`socket.error` for remote URLs, :exc:`OSError` for local - URLs). + The reason for this error. It can be a message string or another + exception instance (:exc:`socket.error` for remote URLs, :exc:`OSError` + for local URLs). .. exception:: HTTPError - Though being an exception (a subclass of :exc:`URLError`), an :exc:`HTTPError` - can also function as a non-exceptional file-like return value (the same thing - that :func:`urlopen` returns). This is useful when handling exotic HTTP - errors, such as requests for authentication. + Though being an exception (a subclass of :exc:`URLError`), an + :exc:`HTTPError` can also function as a non-exceptional file-like return + value (the same thing that :func:`urlopen` returns). This is useful when + handling exotic HTTP errors, such as requests for authentication. .. attribute:: code - An HTTP status code as defined in `RFC 2616 <http://www.faqs.org/rfcs/rfc2616.html>`_. - This numeric value corresponds to a value found in the dictionary of - codes as found in :attr:`http.server.BaseHTTPRequestHandler.responses`. + An HTTP status code as defined in `RFC 2616 + <http://www.faqs.org/rfcs/rfc2616.html>`_. This numeric value corresponds + to a value found in the dictionary of codes as found in + :attr:`http.server.BaseHTTPRequestHandler.responses`. .. exception:: ContentTooShortError(msg[, content]) - This exception is raised when the :func:`urlretrieve` function detects that the - amount of the downloaded data is less than the expected amount (given by the - *Content-Length* header). The :attr:`content` attribute stores the downloaded - (and supposedly truncated) data. + This exception is raised when the :func:`urlretrieve` function detects that + the amount of the downloaded data is less than the expected amount (given by + the *Content-Length* header). The :attr:`content` attribute stores the + downloaded (and supposedly truncated) data. diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index affa406..a5463e6 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -20,13 +20,12 @@ to an absolute URL given a "base URL." The module has been designed to match the Internet RFC on Relative Uniform Resource Locators (and discovered a bug in an earlier draft!). It supports the following URL schemes: ``file``, ``ftp``, ``gopher``, ``hdl``, ``http``, -``https``, ``imap``, ``mailto``, ``mms``, ``news``, ``nntp``, ``prospero``, -``rsync``, ``rtsp``, ``rtspu``, ``sftp``, ``shttp``, ``sip``, ``sips``, -``snews``, ``svn``, ``svn+ssh``, ``telnet``, ``wais``. +``https``, ``imap``, ``mailto``, ``mms``, ``news``, ``nntp``, ``prospero``, +``rsync``, ``rtsp``, ``rtspu``, ``sftp``, ``shttp``, ``sip``, ``sips``, +``snews``, ``svn``, ``svn+ssh``, ``telnet``, ``wais``. The :mod:`urllib.parse` module defines the following functions: - .. function:: urlparse(urlstring[, default_scheme[, allow_fragments]]) Parse a URL into six components, returning a 6-tuple. This corresponds to the @@ -92,11 +91,11 @@ The :mod:`urllib.parse` module defines the following functions: .. function:: urlunparse(parts) - Construct a URL from a tuple as returned by ``urlparse()``. The *parts* argument - can be any six-item iterable. This may result in a slightly different, but - equivalent URL, if the URL that was parsed originally had unnecessary delimiters - (for example, a ? with an empty query; the RFC states that these are - equivalent). + Construct a URL from a tuple as returned by ``urlparse()``. The *parts* + argument can be any six-item iterable. This may result in a slightly + different, but equivalent URL, if the URL that was parsed originally had + unnecessary delimiters (for example, a ``?`` with an empty query; the RFC + states that these are equivalent). .. function:: urlsplit(urlstring[, default_scheme[, allow_fragments]]) @@ -140,19 +139,19 @@ The :mod:`urllib.parse` module defines the following functions: .. function:: urlunsplit(parts) - Combine the elements of a tuple as returned by :func:`urlsplit` into a complete - URL as a string. The *parts* argument can be any five-item iterable. This may - result in a slightly different, but equivalent URL, if the URL that was parsed - originally had unnecessary delimiters (for example, a ? with an empty query; the - RFC states that these are equivalent). + Combine the elements of a tuple as returned by :func:`urlsplit` into a + complete URL as a string. The *parts* argument can be any five-item + iterable. This may result in a slightly different, but equivalent URL, if the + URL that was parsed originally had unnecessary delimiters (for example, a ? + with an empty query; the RFC states that these are equivalent). .. function:: urljoin(base, url[, allow_fragments]) Construct a full ("absolute") URL by combining a "base URL" (*base*) with another URL (*url*). Informally, this uses components of the base URL, in - particular the addressing scheme, the network location and (part of) the path, - to provide missing components in the relative URL. For example: + particular the addressing scheme, the network location and (part of) the + path, to provide missing components in the relative URL. For example: >>> from urllib.parse import urljoin >>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html') @@ -178,10 +177,10 @@ The :mod:`urllib.parse` module defines the following functions: .. function:: urldefrag(url) - If *url* contains a fragment identifier, returns a modified version of *url* - with no fragment identifier, and the fragment identifier as a separate string. - If there is no fragment identifier in *url*, returns *url* unmodified and an - empty string. + If *url* contains a fragment identifier, return a modified version of *url* + with no fragment identifier, and the fragment identifier as a separate + string. If there is no fragment identifier in *url*, return *url* unmodified + and an empty string. .. function:: quote(string[, safe]) @@ -195,9 +194,10 @@ The :mod:`urllib.parse` module defines the following functions: .. function:: quote_plus(string[, safe]) - Like :func:`quote`, but also replaces spaces by plus signs, as required for - quoting HTML form values. Plus signs in the original string are escaped unless - they are included in *safe*. It also does not have *safe* default to ``'/'``. + Like :func:`quote`, but also replace spaces by plus signs, as required for + quoting HTML form values. Plus signs in the original string are escaped + unless they are included in *safe*. It also does not have *safe* default to + ``'/'``. .. function:: unquote(string) @@ -209,7 +209,7 @@ The :mod:`urllib.parse` module defines the following functions: .. function:: unquote_plus(string) - Like :func:`unquote`, but also replaces plus signs by spaces, as required for + Like :func:`unquote`, but also replace plus signs by spaces, as required for unquoting HTML form values. @@ -254,7 +254,6 @@ The result objects from the :func:`urlparse` and :func:`urlsplit` functions are subclasses of the :class:`tuple` type. These subclasses add the attributes described in those functions, as well as provide an additional method: - .. method:: ParseResult.geturl() Return the re-combined version of the original URL as a string. This may differ @@ -279,13 +278,12 @@ described in those functions, as well as provide an additional method: The following classes provide the implementations of the parse results:: - .. class:: BaseResult - Base class for the concrete result classes. This provides most of the attribute - definitions. It does not provide a :meth:`geturl` method. It is derived from - :class:`tuple`, but does not override the :meth:`__init__` or :meth:`__new__` - methods. + Base class for the concrete result classes. This provides most of the + attribute definitions. It does not provide a :meth:`geturl` method. It is + derived from :class:`tuple`, but does not override the :meth:`__init__` or + :meth:`__new__` methods. .. class:: ParseResult(scheme, netloc, path, params, query, fragment) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 4262836..d124d9a 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -7,9 +7,9 @@ .. sectionauthor:: Moshe Zadka <moshez@users.sourceforge.net> -The :mod:`urllib.request` module defines functions and classes which help in opening -URLs (mostly HTTP) in a complex world --- basic and digest authentication, -redirections, cookies and more. +The :mod:`urllib.request` module defines functions and classes which help in +opening URLs (mostly HTTP) in a complex world --- basic and digest +authentication, redirections, cookies and more. The :mod:`urllib.request` module defines the following functions: @@ -180,7 +180,7 @@ The following classes are provided: the ``User-Agent`` header, which is used by a browser to identify itself -- some HTTP servers only allow requests coming from common browsers as opposed to scripts. For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0 - (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib2`'s + (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib`'s default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). The final two arguments are only of interest for correct handling of third-party @@ -1005,10 +1005,11 @@ HTTPErrorProcessor Objects For non-200 error codes, this simply passes the job on to the :meth:`protocol_error_code` handler methods, via :meth:`OpenerDirector.error`. - Eventually, :class:`urllib2.HTTPDefaultErrorHandler` will raise an + Eventually, :class:`HTTPDefaultErrorHandler` will raise an :exc:`HTTPError` if no other handler handles the error. -.. _urllib2-examples: + +.. _urllib-request-examples: Examples -------- @@ -1180,15 +1181,18 @@ The following example uses no proxies at all, overriding environment settings:: using the :mod:`ftplib` module, subclassing :class:`FancyURLOpener`, or changing *_urlopener* to meet your needs. + + :mod:`urllib.response` --- Response classes used by urllib. =========================================================== + .. module:: urllib.response :synopsis: Response classes used by urllib. The :mod:`urllib.response` module defines functions and classes which define a -minimal file like interface, including read() and readline(). The typical -response object is an addinfourl instance, which defines and info() method and -that returns headers and a geturl() method that returns the url. +minimal file like interface, including ``read()`` and ``readline()``. The +typical response object is an addinfourl instance, which defines and ``info()`` +method and that returns headers and a ``geturl()`` method that returns the url. Functions defined by this module are used internally by the :mod:`urllib.request` module. diff --git a/Doc/library/urllib.robotparser.rst b/Doc/library/urllib.robotparser.rst index e351c56..0cac2ad 100644 --- a/Doc/library/urllib.robotparser.rst +++ b/Doc/library/urllib.robotparser.rst @@ -1,9 +1,8 @@ - :mod:`urllib.robotparser` --- Parser for robots.txt ==================================================== .. module:: urllib.robotparser - :synopsis: Loads a robots.txt file and answers questions about + :synopsis: Load a robots.txt file and answer questions about fetchability of other URLs. .. sectionauthor:: Skip Montanaro <skip@pobox.com> @@ -25,42 +24,37 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. This class provides a set of methods to read, parse and answer questions about a single :file:`robots.txt` file. - .. method:: set_url(url) Sets the URL referring to a :file:`robots.txt` file. - .. method:: read() Reads the :file:`robots.txt` URL and feeds it to the parser. - .. method:: parse(lines) Parses the lines argument. - .. method:: can_fetch(useragent, url) Returns ``True`` if the *useragent* is allowed to fetch the *url* according to the rules contained in the parsed :file:`robots.txt` file. - .. method:: mtime() Returns the time the ``robots.txt`` file was last fetched. This is useful for long-running web spiders that need to check for new ``robots.txt`` files periodically. - .. method:: modified() Sets the time the ``robots.txt`` file was last fetched to the current time. -The following example demonstrates basic use of the RobotFileParser class. :: + +The following example demonstrates basic use of the RobotFileParser class. >>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index b0c6e8e..9bc0890 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -150,8 +150,8 @@ There are a number of modules for accessing the internet and processing internet protocols. Two of the simplest are :mod:`urllib.request` for retrieving data from urls and :mod:`smtplib` for sending mail:: - >>> import urllib.request - >>> for line in urllib.request.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'): + >>> from urllib.request import urlopen + >>> for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'): ... if 'EST' in line or 'EDT' in line: # look for Eastern Time ... print(line) |