summaryrefslogtreecommitdiffstats
path: root/Lib/robotparser.py
Commit message (Collapse)AuthorAgeFilesLines
* Close issue 3437 - missing state change when Allow lines are processed.Skip Montanaro2008-07-271-0/+5
| | | | Adds test cases which use Allow: as well.
* #1778443 robotparser fixes from Aristotelis MikropoulosBenjamin Peterson2008-07-121-6/+3
|
* Get rid of _test(), _main(), _debug() and _check(). Tests are no longerSkip Montanaro2008-04-281-93/+12
| | | | | | needed (better set available in Lib/test/test_robotparser.py). Clean up a few PEP 8 nits (compound statements on a single line, whitespace around operators).
* fixes 813986Skip Montanaro2007-08-281-0/+5
|
* Patch #1555098: use str.join() instead of repeated stringGeorg Brandl2007-03-131-9/+6
| | | | concatenation in robotparser.
* Patch #1014237: Consistently return booleans throughout.Martin v. Löwis2004-08-231-10/+10
|
* Replace str.find()!=1 with the more readable "in" operator.Raymond Hettinger2004-05-041-1/+1
|
* SF patch #911431: robot.txt must be robots.txtRaymond Hettinger2004-03-131-2/+2
| | | | (Contributed by George Yoshida.)
* Get rid of many apply() calls.Guido van Rossum2003-02-271-1/+1
|
* Remove import of re, it is not usedNeal Norwitz2002-05-311-1/+1
|
* Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were ↵Raymond Hettinger2002-05-291-0/+17
| | | | not updated after 2.2).
* Convert a pile of obvious "yes/no" functions to return bool.Tim Peters2002-04-041-6/+6
|
* Correctly set default entry in all cases.Martin v. Löwis2002-03-181-6/+9
|
* Patch #499513: use readline() instead of readlines(). Removed theMartin v. Löwis2002-03-181-16/+6
| | | | unnecessary redirection limit code which is already in FancyURLopener.
* Correct various errors:Martin v. Löwis2002-02-281-6/+16
| | | | | | | - Use substring search, not re search for user-agent and paths. - Consider * entry last. Unquote, then requote URLs. - Treat empty Disallow as "allow everything". Add test cases. Fixes #523041
* Remove unused import (PyChecker)Andrew M. Kuchling2001-08-131-1/+0
|
* Whitespace normalization.Tim Peters2001-02-151-1/+1
|
* The bulk of the credit for these changes goes to Bastian KleineidamSkip Montanaro2001-02-121-34/+89
| | | | | | | * restores urllib as the file fetcher (closes bug #132000) * allows checking URLs with empty paths (closes patches #103511 and 103721) * properly handle user agents with versions (e.g., SpamMeister/1.5) * added several more tests
* String method conversion.Eric S. Raymond2001-02-091-8/+8
|
* Whitespace normalization.Tim Peters2001-01-211-10/+10
|
* added __all__ lists to a number of Python modulesSkip Montanaro2001-01-201-0/+2
| | | | | | | | added test script and expected output file as well this closes patch 103297. __all__ attributes will be added to other modules without first submitting a patch, just adding the necessary line to the test script to verify more-or-less correct implementation.
* rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.Skip Montanaro2001-01-201-60/+179
|
* Skip Montanaro:Guido van Rossum2000-03-271-17/+17
| | | | | | | | | | | | The robotparser.py module currently lives in Tools/webchecker. In preparation for its migration to Lib, I made the following changes: * renamed the test() function _test * corrected the URLs in _test() so they refer to actual documents * added an "if __name__ == '__main__'" catcher to invoke _test() when run as a main program * added doc strings for the two main methods, parse and can_fetch * replaced usage of regsub and regex with corresponding re code
* Give in to tabnannyGuido van Rossum1998-04-061-60/+60
|
* Skip Montanaro's robots.txt parser.Guido van Rossum1997-01-301-0/+97