summaryrefslogtreecommitdiffstats
path: root/Lib/robotparser.py
Commit message (Collapse)AuthorAgeFilesLines
* Correctly set default entry in all cases.Martin v. Löwis2002-03-181-6/+9
|
* Patch #499513: use readline() instead of readlines(). Removed theMartin v. Löwis2002-03-181-16/+6
| | | | unnecessary redirection limit code which is already in FancyURLopener.
* Correct various errors:Martin v. Löwis2002-02-281-6/+16
| | | | | | | - Use substring search, not re search for user-agent and paths. - Consider * entry last. Unquote, then requote URLs. - Treat empty Disallow as "allow everything". Add test cases. Fixes #523041
* Remove unused import (PyChecker)Andrew M. Kuchling2001-08-131-1/+0
|
* Whitespace normalization.Tim Peters2001-02-151-1/+1
|
* The bulk of the credit for these changes goes to Bastian KleineidamSkip Montanaro2001-02-121-34/+89
| | | | | | | * restores urllib as the file fetcher (closes bug #132000) * allows checking URLs with empty paths (closes patches #103511 and 103721) * properly handle user agents with versions (e.g., SpamMeister/1.5) * added several more tests
* String method conversion.Eric S. Raymond2001-02-091-8/+8
|
* Whitespace normalization.Tim Peters2001-01-211-10/+10
|
* added __all__ lists to a number of Python modulesSkip Montanaro2001-01-201-0/+2
| | | | | | | | added test script and expected output file as well this closes patch 103297. __all__ attributes will be added to other modules without first submitting a patch, just adding the necessary line to the test script to verify more-or-less correct implementation.
* rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.Skip Montanaro2001-01-201-60/+179
|
* Skip Montanaro:Guido van Rossum2000-03-271-17/+17
| | | | | | | | | | | | The robotparser.py module currently lives in Tools/webchecker. In preparation for its migration to Lib, I made the following changes: * renamed the test() function _test * corrected the URLs in _test() so they refer to actual documents * added an "if __name__ == '__main__'" catcher to invoke _test() when run as a main program * added doc strings for the two main methods, parse and can_fetch * replaced usage of regsub and regex with corresponding re code
* Give in to tabnannyGuido van Rossum1998-04-061-60/+60
|
* Skip Montanaro's robots.txt parser.Guido van Rossum1997-01-301-0/+97
generated by cgit v0.12 at 2025-11-04 15:09:44 (GMT)