diff options
Diffstat (limited to 'Doc/library/urllib.robotparser.rst')
| -rw-r--r-- | Doc/library/urllib.robotparser.rst | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/Doc/library/urllib.robotparser.rst b/Doc/library/urllib.robotparser.rst index f179de2..ba701c3 100644 --- a/Doc/library/urllib.robotparser.rst +++ b/Doc/library/urllib.robotparser.rst @@ -4,8 +4,10 @@ .. module:: urllib.robotparser :synopsis: Load a robots.txt file and answer questions about fetchability of other URLs. + .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/urllib/robotparser.py` .. index:: single: WWW @@ -13,6 +15,8 @@ single: URL single: robots.txt +-------------- + This module provides a single class, :class:`RobotFileParser`, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the :file:`robots.txt` file. For more details on the |