diff options
| author | Christopher Beacham <mcscope@gmail.com> | 2018-05-16 14:52:07 (GMT) |
|---|---|---|
| committer | Ned Deily <nad@python.org> | 2018-05-16 14:52:07 (GMT) |
| commit | 5db5c0669e624767375593cc1a01f32092c91c58 (patch) | |
| tree | 0172f5c0c9ae6879ca21c82c175be1c2b829b4c9 /Doc/library/urllib.robotparser.rst | |
| parent | 7a1c02750171d9895754da5d560700aaba93da56 (diff) | |
| download | cpython-5db5c0669e624767375593cc1a01f32092c91c58.zip cpython-5db5c0669e624767375593cc1a01f32092c91c58.tar.gz cpython-5db5c0669e624767375593cc1a01f32092c91c58.tar.bz2 | |
bpo-21475: Support the Sitemap extension in robotparser (GH-6883)
Diffstat (limited to 'Doc/library/urllib.robotparser.rst')
| -rw-r--r-- | Doc/library/urllib.robotparser.rst | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/Doc/library/urllib.robotparser.rst b/Doc/library/urllib.robotparser.rst index e3b90e6..544f502 100644 --- a/Doc/library/urllib.robotparser.rst +++ b/Doc/library/urllib.robotparser.rst @@ -76,6 +76,15 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. .. versionadded:: 3.6 + .. method:: site_maps() + + Returns the contents of the ``Sitemap`` parameter from + ``robots.txt`` in the form of a :func:`list`. If there is no such + parameter or the ``robots.txt`` entry for this parameter has + invalid syntax, return ``None``. + + .. versionadded:: 3.8 + The following example demonstrates basic use of the :class:`RobotFileParser` class:: |