diff options
| author | Skip Montanaro <skip@pobox.com> | 2008-04-28 03:25:37 (GMT) |
|---|---|---|
| committer | Skip Montanaro <skip@pobox.com> | 2008-04-28 03:25:37 (GMT) |
| commit | dfd982715bc81103dfcb3eecdccff32675a772a3 (patch) | |
| tree | dbc76dfcbc46b50d47382cfb5481d177485b7fcc /Doc/library/robotparser.rst | |
| parent | 141534e56fccd9feb049c0c4115ab7e66688196c (diff) | |
| download | cpython-dfd982715bc81103dfcb3eecdccff32675a772a3.zip cpython-dfd982715bc81103dfcb3eecdccff32675a772a3.tar.gz cpython-dfd982715bc81103dfcb3eecdccff32675a772a3.tar.bz2 | |
Wrap some long lines.
Diffstat (limited to 'Doc/library/robotparser.rst')
| -rw-r--r-- | Doc/library/robotparser.rst | 21 |
1 files changed, 12 insertions, 9 deletions
diff --git a/Doc/library/robotparser.rst b/Doc/library/robotparser.rst index b3a9a60..11ee0bd 100644 --- a/Doc/library/robotparser.rst +++ b/Doc/library/robotparser.rst @@ -3,7 +3,8 @@ ============================================= .. module:: robotparser - :synopsis: Loads a robots.txt file and answers questions about fetchability of other URLs. + :synopsis: Loads a robots.txt file and answers questions about + fetchability of other URLs. .. sectionauthor:: Skip Montanaro <skip@pobox.com> @@ -21,8 +22,8 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. .. class:: RobotFileParser() - This class provides a set of methods to read, parse and answer questions about a - single :file:`robots.txt` file. + This class provides a set of methods to read, parse and answer questions + about a single :file:`robots.txt` file. .. method:: set_url(url) @@ -42,20 +43,22 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. .. method:: can_fetch(useragent, url) - Returns ``True`` if the *useragent* is allowed to fetch the *url* according to - the rules contained in the parsed :file:`robots.txt` file. + Returns ``True`` if the *useragent* is allowed to fetch the *url* + according to the rules contained in the parsed :file:`robots.txt` + file. .. method:: mtime() - Returns the time the ``robots.txt`` file was last fetched. This is useful for - long-running web spiders that need to check for new ``robots.txt`` files - periodically. + Returns the time the ``robots.txt`` file was last fetched. This is + useful for long-running web spiders that need to check for new + ``robots.txt`` files periodically. .. method:: modified() - Sets the time the ``robots.txt`` file was last fetched to the current time. + Sets the time the ``robots.txt`` file was last fetched to the current + time. The following example demonstrates basic use of the RobotFileParser class. :: |