summaryrefslogtreecommitdiffstats
path: root/Tools/webchecker
Commit message (Collapse)AuthorAgeFilesLines
* [Bug #512799] urllib.splittype() returns a 2-tuple. (Reported by seb bacon)Andrew M. Kuchling2002-03-081-1/+1
|
* Fix SF bug #482171: webchecker dies on file: URLs w/o robots.txtGuido van Rossum2001-12-111-2/+2
| | | | | | The cause seems to be that when a file URL doesn't exist, urllib.urlopen() raises OSError instead of IOError. Simply add this to the except clause. Not elegant, but effective. :-)
* Only catch NameError and TypeError when attempting to subclass anFred Drake2001-05-111-1/+1
| | | | exception (for compatibility with old versions of Python).
* Added more link attributes based on additonal information from ChrisFred Drake2001-04-051-1/+13
| | | | | | | McCafferty <christopher.mccafferty@csg.ch>, and a bit of experimentation with Navigator 4.7. HTML-as-deployed is evil!
* A number of improvements based on a discussion with Chris McCaffertyFred Drake2001-04-041-2/+24
| | | | | | | | | <christopher.mccafferty@csg.ch>: Add javascript: and telnet: to the types of URLs we ignore. Add support for several additional URL-valued attributes on the BODY, FRAME, IFRAME, LINK, OBJECT, and SCRIPT elements.
* Patch inspired by Just van Rossum: on the Mac, in savefilename(), makeGuido van Rossum2000-04-251-1/+3
| | | | | the path to save a relative path by prefixing it with os.sep (':'). Also fix an indent inconsistency in the same function.
* Moved robotparser.py to the Lib directory.Guido van Rossum2000-03-291-97/+0
| | | | If you do a "cvs update" in the Lib directory, it will pop up there.
* Fix suggested by Magnus Kessler: in class Page, it is possible forGuido van Rossum2000-03-281-1/+4
| | | | | self.parser to be None; in that case don't dereference it in getnames().
* Skip Montanaro:Guido van Rossum2000-03-271-17/+17
| | | | | | | | | | | | The robotparser.py module currently lives in Tools/webchecker. In preparation for its migration to Lib, I made the following changes: * renamed the test() function _test * corrected the URLs in _test() so they refer to actual documents * added an "if __name__ == '__main__'" catcher to invoke _test() when run as a main program * added doc strings for the two main methods, parse and can_fetch * replaced usage of regsub and regex with corresponding re code
* Complete the integration of Sam Bayer's fixes.Guido van Rossum1999-11-172-912/+10
|
* Changed fron importing wcnew back to webchecker.Guido van Rossum1999-11-172-6/+2
|
* Integrated Sam Bayer's wcnew.py code. It seems silly to keep twoGuido van Rossum1999-11-171-46/+185
| | | | | files. Removed Sam's "SLB" change comments; otherwise this is the same as wcnew.py.
* # *NOT* by Sam Bayer: reindented to use 4 spaces like the rest here,Guido van Rossum1999-11-171-204/+203
| | | | # and removed trailing whitespace.
* Samuel L. Bayer:Guido van Rossum1999-11-171-4/+12
| | | | | | | | | - same trick with "import wcnew; webchecker = wcnew" as above - updated readhtml() method to handle pair representation; used new name suppression infrastructure from wcnew.py to suppress processing name anchors [And untabified --GvR]
* Samuel L. Bayer:Guido van Rossum1999-11-171-17/+46
| | | | | | | | | | | | | | | | | | | - added -t and -a arguments - added "import wcnew; webchecker = wcnew" in place of "import webchecker" (I assume that if you're happy with the changes, you'll just replace webchecker.py with wcnew.py, but if I were to do that, the diffs would be incomprehensible) - fixed buggy -v argument (I think you got out of sync with the way verbosity was handled in webchecker vs. wcgui between 1.5 and 1.5.2) - made -v actually do something by adding a call to c.setflags() (probably the same problem as above) - updated references to URLs to accommodate wcnew.py's pair representation; added appropriate calls to format_url() to handle display; added argument to ListPanel() initialization to provide access to format_url() [And untabified --GvR]
* Samuel L. Bayer:Guido van Rossum1999-11-171-154/+178
| | | | | | | | | | - same fixes from webchecker.py - incorporated small diff between current webchecker.py and 1.5.2 - fixed bug where "extra roots" added with the -t argument were being checked as real roots, not just as possible continuations - added -a argument to suppress checking of name anchors [And untabified --GvR]
* Samuel L. Bayer:Guido van Rossum1999-11-171-1/+6
| | | | | | | | - forced new done origins to set errors if they're in self.bad (fixes bug where only the first of a number of errorful references to a link is reported under some circumstances) - suppressed adding duplicates to self.todo list (cleans up printout in wcgui details)
* Some changes (maybe not enough?) to make it work on Windows with localGuido van Rossum1999-04-261-3/+3
| | | | file URLs.
* Added Samuel Bayer's new webchecker.Guido van Rossum1999-03-241-0/+884
| | | | | | | | Unfortunately his code breaks wcgui.py in a way that's not easy to fix. I expect that this is a temporary situation -- eventually Sam's changes will be merged back in. (The changes add a -t option to specify exceptions to the -x option, and explicit checking for #foo style fragment ids.)
* Recover from failed saves; when a file turns out to be a directory,Guido van Rossum1999-01-031-5/+17
| | | | create a directory and moer the original file to the index.html.
* Added note() message to Page class -- this was used but didn't exist.Guido van Rossum1998-08-061-0/+9
| | | | | (The alternative would be to call self.checker.note() but since self.checker might be None that's not quite right.
* Rewrite to support multiple suckers, each with their own thread.Guido van Rossum1998-07-081-102/+140
|
* Instead of printint, use self.message() or self.note().Guido van Rossum1998-07-082-72/+63
|
* # This is a new module I wrote over the weekend. Again, you missed theGuido van Rossum1998-06-151-16/+37
| | | | | | | # checkin email because my PC doesn't have the "Mail" command. Add threading (now that it works). Also some small adaptations to Unix again.
* Primitive GUI for websucker.Guido van Rossum1998-06-151-0/+185
|
* Fix the way a trailing / is changed to /index.html so that itGuido van Rossum1998-06-151-2/+3
| | | | doesn't depend on the value of os.sep. (I.e. ported to Windows :-)
* sort the urls in the todo listGuido van Rossum1998-06-151-1/+3
|
* Use a try-except so that the pickle file is written even when we dieGuido van Rossum1998-04-271-14/+18
| | | | because of an unexpected exception.
* Give in to tabnannyGuido van Rossum1998-04-066-1041/+850
|
* Use a better way to bind the checkext instance variable to a checkGuido van Rossum1998-03-051-9/+8
| | | | | button widget, not involving a __getattr__() method but a callback on the widget.
* Adapt to new webchecker structure. Due to better structure ofGuido van Rossum1998-02-211-59/+33
| | | | | getpage(), much less duplicate code is needed -- we only need to override readhtml().
* Major overhaul. Don't use global variable (e.g. verbose); useGuido van Rossum1998-02-211-130/+191
| | | | | | | instance variables. Make all global functions methods, for easy overriding. Restructure getpage() for easy overriding. Add save_pickle() method and load_pickle() global function to make it easier for other programs to emulate the toplevel interface.
* Map .shtml to text/html.Guido van Rossum1997-10-071-0/+1
|
* A variant on webchecker that creates a mirror copy of a remote site.Guido van Rossum1997-10-061-0/+131
|
* Several changes:Guido van Rossum1997-10-061-6/+24
| | | | | | | | | | - Change the code that looks for robots.txt to always look in /, even if the "root" path is somewhere deep down below. - Add link processing in <AREA> tags. - Change safeclose() to avoid crashing when the file has no geturl() method.
* Tiny script to play with it on a Mac.Guido van Rossum1997-05-281-0/+7
|
* Scroll to top of info window when done.Guido van Rossum1997-05-091-0/+1
|
* Avoid the fancy handler for error 401 (request authentication).Guido van Rossum1997-05-071-4/+7
|
* Restructured Checker class to get rid of 'ext' table.Guido van Rossum1997-02-022-177/+165
| | | | | | | | | | | | | | | | | | | Links are now either in 'todo' or 'done', and ext links are hadled more like local links except that no further links are gathered (and sometimes they aren't checked, e.g. for mailto and news URLs). The -x option reverses its meaning: it disables checking of ext links (they are moved to 'done' without checking). A new 'errors' table collects pages with bad links as we go -- redundant, but useful for the GUI version which needs to report this as we go. Some new methods, including reset(). New checkpoint format. Adapted the GUI to the changes in the Checker class. Added Quit and "Start over" buttons, and a checkbox to disable checking external links. The details window now also shows bad links emanating from the selected page. Miscellaneous small chages.
* Add root URL entry box, separate start/stop/step buttons.Guido van Rossum1997-02-011-54/+131
| | | | If the users selects an item in 'To check', start checking there.
* Process <img> and <frame> tags. Don't bother skipping second href.Guido van Rossum1997-02-011-3/+12
|
* Check in another copy of tktools.py...Guido van Rossum1997-01-311-0/+367
|
* Tk interface to webchecker. Not fully featured yet, but usable.Guido van Rossum1997-01-311-0/+329
|
* Spin off checking of external page in a subroutine.Guido van Rossum1997-01-311-17/+20
| | | | | Increase MAXPAGE to 150K. Add back printing of __doc__ for usage message.
* Many misc changes.Guido van Rossum1997-01-311-95/+142
| | | | | | | | | | | | | | | | | | | | | | | - Faster HTML parser derivede from SGMLparser (Fred Gansevles). - All manipulations of todo, done, ext, bad are done via methods, so a derived class can override. Also moved the 'done' marking to dopage(), so run() is much simpler. - Added a method status() which returns a string containing the summary counts; added a "total" count. - Drop the guessing of the file type before opening the document -- we still need to check those links for validity! - Added a subroutine to close a connection which first slurps up the remaining data when it's an ftp URL -- apparently closing an ftp connection without reading till the end makes it hang. - Added -n option to skip running (only useful with -R). - The Checker object now has an instance variable which is set to 1 when it is changed. This is not pickled.
* Set proper User-agent header (Python-webchecker/<version>).Guido van Rossum1997-01-301-14/+21
| | | | | When -x is combined with -q, still do the checking, but don't print the error in this phase -- they are reported by report_errors().
* Some refinements of the external-link checking code: insert the errorsGuido van Rossum1997-01-301-9/+22
| | | | | | in the 'bad' dictionary (sanitize them so they are picklable; the sanitation code is now a subroutine); don't check mailto: URLs; omit colon in Error message.
* Added -x option to check external links. Slooooow!Guido van Rossum1997-01-301-10/+32
|
* Catch I/O errors when parsing robots.txt file.Guido van Rossum1997-01-301-5/+13
| | | | Add version number, printed at startup in non-quited mode.
* Basic README fileGuido van Rossum1997-01-301-0/+14
|
generated by cgit v0.12 at 2024-12-04 06:59:26 (GMT)