summaryrefslogtreecommitdiffstats
path: root/Tools/webchecker/robotparser.py
Commit message (Collapse)AuthorAgeFilesLines
* Moved robotparser.py to the Lib directory.Guido van Rossum2000-03-291-97/+0
| | | | If you do a "cvs update" in the Lib directory, it will pop up there.
* Skip Montanaro:Guido van Rossum2000-03-271-17/+17
| | | | | | | | | | | | The robotparser.py module currently lives in Tools/webchecker. In preparation for its migration to Lib, I made the following changes: * renamed the test() function _test * corrected the URLs in _test() so they refer to actual documents * added an "if __name__ == '__main__'" catcher to invoke _test() when run as a main program * added doc strings for the two main methods, parse and can_fetch * replaced usage of regsub and regex with corresponding re code
* Give in to tabnannyGuido van Rossum1998-04-061-60/+60
|
* Skip Montanaro's robots.txt parser.Guido van Rossum1997-01-301-0/+97