diff options
39 files changed, 840 insertions, 233 deletions
diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 8bd3ca5..2164ec0 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -351,6 +351,11 @@ Note that most parent parsers will specify ``add_help=False``. Otherwise, the :class:`ArgumentParser` will see two ``-h/--help`` options (one in the parent and one in the child) and raise an error. +.. note:: + You must fully initialize the parsers before passing them via ``parents=``. + If you change the parent parsers after the child parser, those changes will + not be reflected in the child. + formatter_class ^^^^^^^^^^^^^^^ @@ -1314,13 +1319,24 @@ of :data:`sys.argv`. This can be accomplished by passing a list of strings to Namespace(accumulate=<built-in function sum>, integers=[1, 2, 3, 4]) -Custom namespaces -^^^^^^^^^^^^^^^^^ +The Namespace object +^^^^^^^^^^^^^^^^^^^^ + +By default, :meth:`parse_args` will return a new object of type :class:`Namespace` +where the necessary attributes have been set. This class is deliberately simple, +just an :class:`object` subclass with a readable string representation. If you +prefer to have dict-like view of the attributes, you can use the standard Python +idiom via :func:`vars`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> args = parser.parse_args(['--foo', 'BAR']) + >>> vars(args) + {'foo': 'BAR'} It may also be useful to have an :class:`ArgumentParser` assign attributes to an -already existing object, rather than the newly-created :class:`Namespace` object -that is normally used. This can be achieved by specifying the ``namespace=`` -keyword argument:: +already existing object, rather than a new :class:`Namespace` object. This can +be achieved by specifying the ``namespace=`` keyword argument:: >>> class C: ... pass diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 602f513..5f67759 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -964,10 +964,10 @@ attribute. ABCs - abstract base classes ---------------------------- -The collections module offers the following ABCs: +The collections module offers the following :term:`ABCs <abstract base class>`: ========================= ===================== ====================== ==================================================== -ABC Inherits Abstract Methods Mixin Methods +ABC Inherits from Abstract Methods Mixin Methods ========================= ===================== ====================== ==================================================== :class:`Container` ``__contains__`` :class:`Hashable` ``__hash__`` @@ -980,15 +980,15 @@ ABC Inherits Abstract Methods Mixin :class:`Iterable`, ``index``, and ``count`` :class:`Container` -:class:`MutableSequence` :class:`Sequence` ``__setitem__`` Inherited Sequence methods and +:class:`MutableSequence` :class:`Sequence` ``__setitem__``, Inherited :class:`Sequence` methods and ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``, - and ``insert`` ``remove``, and ``__iadd__`` + ``insert`` ``remove``, and ``__iadd__`` :class:`Set` :class:`Sized`, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, :class:`Iterable`, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, :class:`Container` ``__sub__``, ``__xor__``, and ``isdisjoint`` -:class:`MutableSet` :class:`Set` ``add`` and Inherited Set methods and +:class:`MutableSet` :class:`Set` ``add``, Inherited :class:`Set` methods and ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``, ``__iand__``, ``__ixor__``, and ``__isub__`` @@ -996,19 +996,61 @@ ABC Inherits Abstract Methods Mixin :class:`Iterable`, ``get``, ``__eq__``, and ``__ne__`` :class:`Container` -:class:`MutableMapping` :class:`Mapping` ``__setitem__`` and Inherited Mapping methods and +:class:`MutableMapping` :class:`Mapping` ``__setitem__``, Inherited :class:`Mapping` methods and ``__delitem__`` ``pop``, ``popitem``, ``clear``, ``update``, and ``setdefault`` :class:`MappingView` :class:`Sized` ``__len__`` -:class:`KeysView` :class:`MappingView`, ``__contains__``, - :class:`Set` ``__iter__`` :class:`ItemsView` :class:`MappingView`, ``__contains__``, :class:`Set` ``__iter__`` +:class:`KeysView` :class:`MappingView`, ``__contains__``, + :class:`Set` ``__iter__`` :class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` ========================= ===================== ====================== ==================================================== + +.. class:: Container + Hashable + Sized + Callable + + ABCs for classes that provide respectively the methods :meth:`__contains__`, + :meth:`__hash__`, :meth:`__len__`, and :meth:`__call__`. + +.. class:: Iterable + + ABC for classes that provide the :meth:`__iter__` method. + See also the definition of :term:`iterable`. + +.. class:: Iterator + + ABC for classes that provide the :meth:`__iter__` and :meth:`next` methods. + See also the definition of :term:`iterator`. + +.. class:: Sequence + MutableSequence + + ABCs for read-only and mutable :term:`sequences <sequence>`. + +.. class:: Set + MutableSet + + ABCs for read-only and mutable sets. + +.. class:: Mapping + MutableMapping + + ABCs for read-only and mutable :term:`mappings <mapping>`. + +.. class:: MappingView + ItemsView + KeysView + ValuesView + + ABCs for mapping, items, keys, and values :term:`views <view>`. + + These ABCs allow us to ask classes or instances if they provide particular functionality, for example:: diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 55dd958..cb7a09c 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -68,6 +68,9 @@ compile Python sources. .. versionchanged:: 3.2 Added the ``-i``, ``-b`` and ``-h`` options. +There is no command-line option to control the optimization level used by the +:func:`compile` function, because the Python interpreter itself already +provides the option: :program:`python -O -m compileall`. Public functions ---------------- diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index e3af41f..9771496 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -722,7 +722,7 @@ cookies (assumes Unix/Netscape convention for location of the cookies file):: import os, http.cookiejar, urllib.request cj = http.cookiejar.MozillaCookieJar() - cj.load(os.path.join(os.environ["HOME"], ".netscape/cookies.txt")) + cj.load(os.path.join(os.path.expanduser("~"), ".netscape", "cookies.txt")) opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) r = opener.open("http://example.com/") diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index c667317..ab55197 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -196,7 +196,7 @@ normally be executed automatically during interactive sessions from the user's import os import readline - histfile = os.path.join(os.environ["HOME"], ".pyhist") + histfile = os.path.join(os.path.expanduser("~"), ".pyhist") try: readline.read_history_file(histfile) except IOError: diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 88655bb..e03ac8a 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -239,33 +239,22 @@ always available. The struct sequence *flags* exposes the status of command line flags. The attributes are read only. - +------------------------------+------------------------------------------+ - | attribute | flag | - +==============================+==========================================+ - | :const:`debug` | -d | - +------------------------------+------------------------------------------+ - | :const:`division_warning` | -Q | - +------------------------------+------------------------------------------+ - | :const:`inspect` | -i | - +------------------------------+------------------------------------------+ - | :const:`interactive` | -i | - +------------------------------+------------------------------------------+ - | :const:`optimize` | -O or -OO | - +------------------------------+------------------------------------------+ - | :const:`dont_write_bytecode` | -B | - +------------------------------+------------------------------------------+ - | :const:`no_user_site` | -s | - +------------------------------+------------------------------------------+ - | :const:`no_site` | -S | - +------------------------------+------------------------------------------+ - | :const:`ignore_environment` | -E | - +------------------------------+------------------------------------------+ - | :const:`verbose` | -v | - +------------------------------+------------------------------------------+ - | :const:`bytes_warning` | -b | - +------------------------------+------------------------------------------+ - | :const:`quiet` | -q | - +------------------------------+------------------------------------------+ + ============================= ============================= + attribute flag + ============================= ============================= + :const:`debug` :option:`-d` + :const:`division_warning` :option:`-Q` + :const:`inspect` :option:`-i` + :const:`interactive` :option:`-i` + :const:`optimize` :option:`-O` or :option:`-OO` + :const:`dont_write_bytecode` :option:`-B` + :const:`no_user_site` :option:`-s` + :const:`no_site` :option:`-S` + :const:`ignore_environment` :option:`-E` + :const:`verbose` :option:`-v` + :const:`bytes_warning` :option:`-b` + :const:`quiet` :option:`-q` + ============================= ============================= .. versionchanged:: 3.2 Added ``quiet`` attribute for the new :option:`-q` flag. diff --git a/Doc/tutorial/interactive.rst b/Doc/tutorial/interactive.rst index ca0cfaf..5faaf96 100644 --- a/Doc/tutorial/interactive.rst +++ b/Doc/tutorial/interactive.rst @@ -123,10 +123,7 @@ interpreter. :: # bound to the Esc key by default (you can change it - see readline docs). # # Store the file in ~/.pystartup, and set an environment variable to point - # to it: "export PYTHONSTARTUP=/home/user/.pystartup" in bash. - # - # Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the - # full path to your home directory. + # to it: "export PYTHONSTARTUP=~/.pystartup" in bash. import atexit import os diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst index 20ce228..d411d85 100644 --- a/Doc/whatsnew/3.2.rst +++ b/Doc/whatsnew/3.2.rst @@ -1499,11 +1499,11 @@ filenames: >>> os.fsencode(filename) b'Sehensw\xc3\xbcrdigkeiten' -Some operating systems allow direct access to the unencoded bytes in the +Some operating systems allow direct access to encoded bytes in the environment. If so, the :attr:`os.supports_bytes_environ` constant will be true. -For direct access to unencoded environment variables (if available), +For direct access to encoded environment variables (if available), use the new :func:`os.getenvb` function or use :data:`os.environb` which is a bytes version of :data:`os.environ`. diff --git a/Lib/argparse.py b/Lib/argparse.py index de3cd11..0ef9f4e 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -82,6 +82,7 @@ __all__ = [ ] +import collections as _collections import copy as _copy import os as _os import re as _re @@ -1041,7 +1042,7 @@ class _SubParsersAction(Action): self._prog_prefix = prog self._parser_class = parser_class - self._name_parser_map = {} + self._name_parser_map = _collections.OrderedDict() self._choices_actions = [] super(_SubParsersAction, self).__init__( @@ -1294,6 +1295,13 @@ class _ActionsContainer(object): if not _callable(type_func): raise ValueError('%r is not callable' % type_func) + # raise an error if the metavar does not match the type + if hasattr(self, "_get_formatter"): + try: + self._get_formatter()._format_args(action, None) + except TypeError: + raise ValueError("length of metavar tuple does not match nargs") + return self._add_action(action) def add_argument_group(self, *args, **kwargs): diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py index 79c60c3..a899d5f 100644 --- a/Lib/concurrent/futures/process.py +++ b/Lib/concurrent/futures/process.py @@ -66,28 +66,17 @@ import weakref # workers to exit when their work queues are empty and then waits until the # threads/processes finish. -_thread_references = set() +_threads_queues = weakref.WeakKeyDictionary() _shutdown = False def _python_exit(): global _shutdown _shutdown = True - for thread_reference in _thread_references: - thread = thread_reference() - if thread is not None: - thread.join() - -def _remove_dead_thread_references(): - """Remove inactive threads from _thread_references. - - Should be called periodically to prevent memory leaks in scenarios such as: - >>> while True: - >>> ... t = ThreadPoolExecutor(max_workers=5) - >>> ... t.map(int, ['1', '2', '3', '4', '5']) - """ - for thread_reference in set(_thread_references): - if thread_reference() is None: - _thread_references.discard(thread_reference) + items = list(_threads_queues.items()) + for t, q in items: + q.put(None) + for t, q in items: + t.join() # Controls how many more calls than processes will be queued in the call queue. # A smaller number will mean that processes spend more time idle waiting for @@ -130,11 +119,15 @@ def _process_worker(call_queue, result_queue, shutdown): """ while True: try: - call_item = call_queue.get(block=True, timeout=0.1) + call_item = call_queue.get(block=True) except queue.Empty: if shutdown.is_set(): return else: + if call_item is None: + # Wake up queue management thread + result_queue.put(None) + return try: r = call_item.fn(*call_item.args, **call_item.kwargs) except BaseException as e: @@ -209,40 +202,56 @@ def _queue_manangement_worker(executor_reference, process workers that they should exit when their work queue is empty. """ + nb_shutdown_processes = 0 + def shutdown_one_process(): + """Tell a worker to terminate, which will in turn wake us again""" + nonlocal nb_shutdown_processes + call_queue.put(None) + nb_shutdown_processes += 1 while True: _add_call_item_to_queue(pending_work_items, work_ids_queue, call_queue) try: - result_item = result_queue.get(block=True, timeout=0.1) + result_item = result_queue.get(block=True) except queue.Empty: - executor = executor_reference() - # No more work items can be added if: - # - The interpreter is shutting down OR - # - The executor that owns this worker has been collected OR - # - The executor that owns this worker has been shutdown. - if _shutdown or executor is None or executor._shutdown_thread: - # Since no new work items can be added, it is safe to shutdown - # this thread if there are no pending work items. - if not pending_work_items: - shutdown_process_event.set() - - # If .join() is not called on the created processes then - # some multiprocessing.Queue methods may deadlock on Mac OS - # X. - for p in processes: - p.join() - return - del executor + pass else: - work_item = pending_work_items[result_item.work_id] - del pending_work_items[result_item.work_id] - - if result_item.exception: - work_item.future.set_exception(result_item.exception) + if result_item is not None: + work_item = pending_work_items[result_item.work_id] + del pending_work_items[result_item.work_id] + + if result_item.exception: + work_item.future.set_exception(result_item.exception) + else: + work_item.future.set_result(result_item.result) + continue + # If we come here, we either got a timeout or were explicitly woken up. + # In either case, check whether we should start shutting down. + executor = executor_reference() + # No more work items can be added if: + # - The interpreter is shutting down OR + # - The executor that owns this worker has been collected OR + # - The executor that owns this worker has been shutdown. + if _shutdown or executor is None or executor._shutdown_thread: + # Since no new work items can be added, it is safe to shutdown + # this thread if there are no pending work items. + if not pending_work_items: + shutdown_process_event.set() + + while nb_shutdown_processes < len(processes): + shutdown_one_process() + # If .join() is not called on the created processes then + # some multiprocessing.Queue methods may deadlock on Mac OS + # X. + for p in processes: + p.join() + return else: - work_item.future.set_result(result_item.result) + # Start shutting down by telling a process it can exit. + shutdown_one_process() + del executor _system_limits_checked = False _system_limited = None @@ -279,7 +288,6 @@ class ProcessPoolExecutor(_base.Executor): worker processes will be created as the machine has processors. """ _check_system_limits() - _remove_dead_thread_references() if max_workers is None: self._max_workers = multiprocessing.cpu_count() @@ -304,10 +312,14 @@ class ProcessPoolExecutor(_base.Executor): self._pending_work_items = {} def _start_queue_management_thread(self): + # When the executor gets lost, the weakref callback will wake up + # the queue management thread. + def weakref_cb(_, q=self._result_queue): + q.put(None) if self._queue_management_thread is None: self._queue_management_thread = threading.Thread( target=_queue_manangement_worker, - args=(weakref.ref(self), + args=(weakref.ref(self, weakref_cb), self._processes, self._pending_work_items, self._work_ids, @@ -316,7 +328,7 @@ class ProcessPoolExecutor(_base.Executor): self._shutdown_process_event)) self._queue_management_thread.daemon = True self._queue_management_thread.start() - _thread_references.add(weakref.ref(self._queue_management_thread)) + _threads_queues[self._queue_management_thread] = self._result_queue def _adjust_process_count(self): for _ in range(len(self._processes), self._max_workers): @@ -339,6 +351,8 @@ class ProcessPoolExecutor(_base.Executor): self._pending_work_items[self._queue_count] = w self._work_ids.put(self._queue_count) self._queue_count += 1 + # Wake up queue management thread + self._result_queue.put(None) self._start_queue_management_thread() self._adjust_process_count() @@ -348,8 +362,10 @@ class ProcessPoolExecutor(_base.Executor): def shutdown(self, wait=True): with self._shutdown_lock: self._shutdown_thread = True - if wait: - if self._queue_management_thread: + if self._queue_management_thread: + # Wake up queue management thread + self._result_queue.put(None) + if wait: self._queue_management_thread.join() # To reduce the risk of openning too many files, remove references to # objects that use file descriptors. diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py index 15736da..93b495f 100644 --- a/Lib/concurrent/futures/thread.py +++ b/Lib/concurrent/futures/thread.py @@ -25,28 +25,17 @@ import weakref # workers to exit when their work queues are empty and then waits until the # threads finish. -_thread_references = set() +_threads_queues = weakref.WeakKeyDictionary() _shutdown = False def _python_exit(): global _shutdown _shutdown = True - for thread_reference in _thread_references: - thread = thread_reference() - if thread is not None: - thread.join() - -def _remove_dead_thread_references(): - """Remove inactive threads from _thread_references. - - Should be called periodically to prevent memory leaks in scenarios such as: - >>> while True: - ... t = ThreadPoolExecutor(max_workers=5) - ... t.map(int, ['1', '2', '3', '4', '5']) - """ - for thread_reference in set(_thread_references): - if thread_reference() is None: - _thread_references.discard(thread_reference) + items = list(_threads_queues.items()) + for t, q in items: + q.put(None) + for t, q in items: + t.join() atexit.register(_python_exit) @@ -72,18 +61,23 @@ def _worker(executor_reference, work_queue): try: while True: try: - work_item = work_queue.get(block=True, timeout=0.1) + work_item = work_queue.get(block=True) except queue.Empty: - executor = executor_reference() - # Exit if: - # - The interpreter is shutting down OR - # - The executor that owns the worker has been collected OR - # - The executor that owns the worker has been shutdown. - if _shutdown or executor is None or executor._shutdown: - return - del executor + pass else: - work_item.run() + if work_item is not None: + work_item.run() + continue + executor = executor_reference() + # Exit if: + # - The interpreter is shutting down OR + # - The executor that owns the worker has been collected OR + # - The executor that owns the worker has been shutdown. + if _shutdown or executor is None or executor._shutdown: + # Notice other workers + work_queue.put(None) + return + del executor except BaseException as e: _base.LOGGER.critical('Exception in worker', exc_info=True) @@ -95,8 +89,6 @@ class ThreadPoolExecutor(_base.Executor): max_workers: The maximum number of threads that can be used to execute the given calls. """ - _remove_dead_thread_references() - self._max_workers = max_workers self._work_queue = queue.Queue() self._threads = set() @@ -117,19 +109,25 @@ class ThreadPoolExecutor(_base.Executor): submit.__doc__ = _base.Executor.submit.__doc__ def _adjust_thread_count(self): + # When the executor gets lost, the weakref callback will wake up + # the worker threads. + def weakref_cb(_, q=self._work_queue): + q.put(None) # TODO(bquinlan): Should avoid creating new threads if there are more # idle threads than items in the work queue. if len(self._threads) < self._max_workers: t = threading.Thread(target=_worker, - args=(weakref.ref(self), self._work_queue)) + args=(weakref.ref(self, weakref_cb), + self._work_queue)) t.daemon = True t.start() self._threads.add(t) - _thread_references.add(weakref.ref(t)) + _threads_queues[t] = self._work_queue def shutdown(self, wait=True): with self._shutdown_lock: self._shutdown = True + self._work_queue.put(None) if wait: for t in self._threads: t.join() diff --git a/Lib/ctypes/test/test_as_parameter.py b/Lib/ctypes/test/test_as_parameter.py index 835398f..475d595 100644 --- a/Lib/ctypes/test/test_as_parameter.py +++ b/Lib/ctypes/test/test_as_parameter.py @@ -187,6 +187,18 @@ class BasicWrapTestCase(unittest.TestCase): self.assertEqual((s8i.a, s8i.b, s8i.c, s8i.d, s8i.e, s8i.f, s8i.g, s8i.h), (9*2, 8*3, 7*4, 6*5, 5*6, 4*7, 3*8, 2*9)) + def test_recursive_as_param(self): + from ctypes import c_int + + class A(object): + pass + + a = A() + a._as_parameter_ = a + with self.assertRaises(RuntimeError): + c_int.from_param(a) + + #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ class AsParamWrapper(object): diff --git a/Lib/email/header.py b/Lib/email/header.py index 8c32514..2562b30 100644 --- a/Lib/email/header.py +++ b/Lib/email/header.py @@ -66,9 +66,15 @@ def decode_header(header): otherwise a lower-case string containing the name of the character set specified in the encoded string. + header may be a string that may or may not contain RFC2047 encoded words, + or it may be a Header object. + An email.errors.HeaderParseError may be raised when certain decoding error occurs (e.g. a base64 decoding exception). """ + # If it is a Header object, we can just return the chunks. + if hasattr(header, '_chunks'): + return list(header._chunks) # If no encoding, just return the header with no charset. if not ecre.search(header): return [(header, None)] diff --git a/Lib/email/test/test_email.py b/Lib/email/test/test_email.py index 250a644..245f659 100644 --- a/Lib/email/test/test_email.py +++ b/Lib/email/test/test_email.py @@ -3918,6 +3918,20 @@ A very long line that must get split to something other than at the h.append(x, errors='replace') eq(str(h), e) + def test_escaped_8bit_header(self): + x = b'Ynwp4dUEbay Auction Semiar- No Charge \x96 Earn Big' + x = x.decode('ascii', 'surrogateescape') + h = Header(x, charset=email.charset.UNKNOWN8BIT) + self.assertEqual(str(h), + 'Ynwp4dUEbay Auction Semiar- No Charge \uFFFD Earn Big') + self.assertEqual(email.header.decode_header(h), [(x, 'unknown-8bit')]) + + def test_modify_returned_list_does_not_change_header(self): + h = Header('test') + chunks = email.header.decode_header(h) + chunks.append(('ascii', 'test2')) + self.assertEqual(str(h), 'test') + def test_encoded_adjacent_nonencoded(self): eq = self.assertEqual h = Header() diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py index a37cb1d..ffc4e88 100644 --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -303,13 +303,13 @@ class EditorWindow(object): return "break" def home_callback(self, event): - if (event.state & 12) != 0 and event.keysym == "Home": - # state&1==shift, state&4==control, state&8==alt - return # <Modifier-Home>; fall back to class binding - + if (event.state & 4) != 0 and event.keysym == "Home": + # state&4==Control. If <Control-Home>, use the Tk binding. + return if self.text.index("iomark") and \ self.text.compare("iomark", "<=", "insert lineend") and \ self.text.compare("insert linestart", "<=", "iomark"): + # In Shell on input line, go to just after prompt insertpt = int(self.text.index("iomark").split(".")[1]) else: line = self.text.get("insert linestart", "insert lineend") @@ -318,30 +318,27 @@ class EditorWindow(object): break else: insertpt=len(line) - lineat = int(self.text.index("insert").split('.')[1]) - if insertpt == lineat: insertpt = 0 - dest = "insert linestart+"+str(insertpt)+"c" - if (event.state&1) == 0: - # shift not pressed + # shift was not pressed self.text.tag_remove("sel", "1.0", "end") else: if not self.text.index("sel.first"): - self.text.mark_set("anchor","insert") - + self.text.mark_set("my_anchor", "insert") # there was no previous selection + else: + if self.text.compare(self.text.index("sel.first"), "<", self.text.index("insert")): + self.text.mark_set("my_anchor", "sel.first") # extend back + else: + self.text.mark_set("my_anchor", "sel.last") # extend forward first = self.text.index(dest) - last = self.text.index("anchor") - + last = self.text.index("my_anchor") if self.text.compare(first,">",last): first,last = last,first - self.text.tag_remove("sel", "1.0", "end") self.text.tag_add("sel", first, last) - self.text.mark_set("insert", dest) self.text.see("insert") return "break" diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 037b02a..deeb4a6 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,4 +1,15 @@ -What's New in IDLE 3.1? +What's New in IDLE 3.1.4? +========================= + +*Release date: XX-XXX-XX* + +- <Home> toggle failing on Tk 8.5, causing IDLE exits and strange selection + behavior. Issue 4676. Improve selection extension behaviour. +- <Home> toggle non-functional when NumLock set on Windows. Issue 3851. + + + +What's New in IDLE 3.1b1? ========================= *Release date: 27-Jun-09* diff --git a/Lib/inspect.py b/Lib/inspect.py index aa951d8..4899cbf 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -944,8 +944,14 @@ def getcallargs(func, *positional, **named): f_name, 'at most' if defaults else 'exactly', num_args, 'arguments' if num_args > 1 else 'argument', num_total)) elif num_args == 0 and num_total: - raise TypeError('%s() takes no arguments (%d given)' % - (f_name, num_total)) + if varkw or kwonlyargs: + if num_pos: + # XXX: We should use num_pos, but Python also uses num_total: + raise TypeError('%s() takes exactly 0 positional arguments ' + '(%d given)' % (f_name, num_total)) + else: + raise TypeError('%s() takes no arguments (%d given)' % + (f_name, num_total)) for arg in itertools.chain(args, kwonlyargs): if arg in named: diff --git a/Lib/msilib/__init__.py b/Lib/msilib/__init__.py index bd3e1f2..6eff89c 100644 --- a/Lib/msilib/__init__.py +++ b/Lib/msilib/__init__.py @@ -172,11 +172,10 @@ def add_tables(db, module): add_data(db, table, getattr(module, table)) def make_id(str): - #str = str.replace(".", "_") # colons are allowed - str = str.replace(" ", "_") - str = str.replace("-", "_") - if str[0] in string.digits: - str = "_"+str + identifier_chars = string.ascii_letters + string.digits + "._" + str = "".join([c if c in identifier_chars else "_" for c in str]) + if str[0] in (string.digits + "."): + str = "_" + str assert re.match("^[A-Za-z_][A-Za-z0-9_.]*$", str), "FILE"+str return str @@ -284,19 +283,28 @@ class Directory: [(feature.id, component)]) def make_short(self, file): + oldfile = file + file = file.replace('+', '_') + file = ''.join(c for c in file if not c in ' "/\[]:;=,') parts = file.split(".") - if len(parts)>1: + if len(parts) > 1: + prefix = "".join(parts[:-1]).upper() suffix = parts[-1].upper() + if not prefix: + prefix = suffix + suffix = None else: + prefix = file.upper() suffix = None - prefix = parts[0].upper() - if len(prefix) <= 8 and (not suffix or len(suffix)<=3): + if len(parts) < 3 and len(prefix) <= 8 and file == oldfile and ( + not suffix or len(suffix) <= 3): if suffix: file = prefix+"."+suffix else: file = prefix - assert file not in self.short_names else: + file = None + if file is None or file in self.short_names: prefix = prefix[:6] if suffix: suffix = suffix[:3] diff --git a/Lib/multiprocessing/sharedctypes.py b/Lib/multiprocessing/sharedctypes.py index e83ce58..1e694da 100644 --- a/Lib/multiprocessing/sharedctypes.py +++ b/Lib/multiprocessing/sharedctypes.py @@ -80,7 +80,9 @@ def RawArray(typecode_or_type, size_or_initializer): type_ = typecode_to_type.get(typecode_or_type, typecode_or_type) if isinstance(size_or_initializer, int): type_ = type_ * size_or_initializer - return _new_value(type_) + obj = _new_value(type_) + ctypes.memset(ctypes.addressof(obj), 0, ctypes.sizeof(obj)) + return obj else: type_ = type_ * len(size_or_initializer) result = _new_value(type_) diff --git a/Lib/pydoc.py b/Lib/pydoc.py index 9d3cdd5..dc398e3 100755 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -165,7 +165,7 @@ def _split_list(s, predicate): no.append(x) return yes, no -def visiblename(name, all=None): +def visiblename(name, all=None, obj=None): """Decide whether to show documentation on a variable.""" # Certain special names are redundant. _hidden_names = ('__builtins__', '__doc__', '__file__', '__path__', @@ -175,6 +175,9 @@ def visiblename(name, all=None): if name in _hidden_names: return 0 # Private names are hidden, but special names are displayed. if name.startswith('__') and name.endswith('__'): return 1 + # Namedtuples have public fields and methods with a single leading underscore + if name.startswith('_') and hasattr(obj, '_fields'): + return True if all is not None: # only document that which the programmer exported in __all__ return name in all @@ -642,7 +645,7 @@ class HTMLDoc(Doc): # if __all__ exists, believe it. Otherwise use old heuristic. if (all is not None or (inspect.getmodule(value) or object) is object): - if visiblename(key, all): + if visiblename(key, all, object): classes.append((key, value)) cdict[key] = cdict[value] = '#' + key for key, value in classes: @@ -658,13 +661,13 @@ class HTMLDoc(Doc): # if __all__ exists, believe it. Otherwise use old heuristic. if (all is not None or inspect.isbuiltin(value) or inspect.getmodule(value) is object): - if visiblename(key, all): + if visiblename(key, all, object): funcs.append((key, value)) fdict[key] = '#-' + key if inspect.isfunction(value): fdict[value] = fdict[key] data = [] for key, value in inspect.getmembers(object, isdata): - if visiblename(key, all): + if visiblename(key, all, object): data.append((key, value)) doc = self.markup(getdoc(object), self.preformat, fdict, cdict) @@ -789,7 +792,7 @@ class HTMLDoc(Doc): attrs = [(name, kind, cls, value) for name, kind, cls, value in classify_class_attrs(object) - if visiblename(name)] + if visiblename(name, obj=object)] mdict = {} for key, kind, homecls, value in attrs: @@ -1056,18 +1059,18 @@ doubt, consult the module reference at the location listed above. # if __all__ exists, believe it. Otherwise use old heuristic. if (all is not None or (inspect.getmodule(value) or object) is object): - if visiblename(key, all): + if visiblename(key, all, object): classes.append((key, value)) funcs = [] for key, value in inspect.getmembers(object, inspect.isroutine): # if __all__ exists, believe it. Otherwise use old heuristic. if (all is not None or inspect.isbuiltin(value) or inspect.getmodule(value) is object): - if visiblename(key, all): + if visiblename(key, all, object): funcs.append((key, value)) data = [] for key, value in inspect.getmembers(object, isdata): - if visiblename(key, all): + if visiblename(key, all, object): data.append((key, value)) modpkgs = [] @@ -1206,7 +1209,7 @@ doubt, consult the module reference at the location listed above. attrs = [(name, kind, cls, value) for name, kind, cls, value in classify_class_attrs(object) - if visiblename(name)] + if visiblename(name, obj=object)] while attrs: if mro: @@ -224,8 +224,7 @@ def escape(pattern): if isinstance(pattern, str): alphanum = _alphanum_str s = list(pattern) - for i in range(len(pattern)): - c = pattern[i] + for i, c in enumerate(pattern): if c not in alphanum: if c == "\000": s[i] = "\\000" diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 03c95fa..e187653 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -2837,16 +2837,22 @@ class TestHelpFormattingMetaclass(type): parser = argparse.ArgumentParser( *tester.parser_signature.args, **tester.parser_signature.kwargs) - for argument_sig in tester.argument_signatures: + for argument_sig in getattr(tester, 'argument_signatures', []): parser.add_argument(*argument_sig.args, **argument_sig.kwargs) - group_signatures = tester.argument_group_signatures - for group_sig, argument_sigs in group_signatures: + group_sigs = getattr(tester, 'argument_group_signatures', []) + for group_sig, argument_sigs in group_sigs: group = parser.add_argument_group(*group_sig.args, **group_sig.kwargs) for argument_sig in argument_sigs: group.add_argument(*argument_sig.args, **argument_sig.kwargs) + subparsers_sigs = getattr(tester, 'subparsers_signatures', []) + if subparsers_sigs: + subparsers = parser.add_subparsers() + for subparser_sig in subparsers_sigs: + subparsers.add_parser(*subparser_sig.args, + **subparser_sig.kwargs) return parser def _test(self, tester, parser_text): @@ -3940,6 +3946,77 @@ class TestHelpVersionAction(HelpTestCase): ''' version = '' +class TestHelpSubparsersOrdering(HelpTestCase): + """Test ordering of subcommands in help matches the code""" + parser_signature = Sig(prog='PROG', + description='display some subcommands', + version='0.1') + + subparsers_signatures = [Sig(name=name) + for name in ('a', 'b', 'c', 'd', 'e')] + + usage = '''\ + usage: PROG [-h] [-v] {a,b,c,d,e} ... + ''' + + help = usage + '''\ + + display some subcommands + + positional arguments: + {a,b,c,d,e} + + optional arguments: + -h, --help show this help message and exit + -v, --version show program's version number and exit + ''' + + version = '''\ + 0.1 + ''' + +class TestHelpSubparsersWithHelpOrdering(HelpTestCase): + """Test ordering of subcommands in help matches the code""" + parser_signature = Sig(prog='PROG', + description='display some subcommands', + version='0.1') + + subcommand_data = (('a', 'a subcommand help'), + ('b', 'b subcommand help'), + ('c', 'c subcommand help'), + ('d', 'd subcommand help'), + ('e', 'e subcommand help'), + ) + + subparsers_signatures = [Sig(name=name, help=help) + for name, help in subcommand_data] + + usage = '''\ + usage: PROG [-h] [-v] {a,b,c,d,e} ... + ''' + + help = usage + '''\ + + display some subcommands + + positional arguments: + {a,b,c,d,e} + a a subcommand help + b b subcommand help + c c subcommand help + d d subcommand help + e e subcommand help + + optional arguments: + -h, --help show this help message and exit + -v, --version show program's version number and exit + ''' + + version = '''\ + 0.1 + ''' + + # ===================================== # Optional/Positional constructor tests # ===================================== @@ -4394,6 +4471,177 @@ class TestParseKnownArgs(TestCase): self.assertEqual(NS(v=3, spam=True, badger="B"), args) self.assertEqual(["C", "--foo", "4"], extras) +# ========================== +# add_argument metavar tests +# ========================== + +class TestAddArgumentMetavar(TestCase): + + EXPECTED_MESSAGE = "length of metavar tuple does not match nargs" + + def do_test_no_exception(self, nargs, metavar): + parser = argparse.ArgumentParser() + parser.add_argument("--foo", nargs=nargs, metavar=metavar) + + def do_test_exception(self, nargs, metavar): + parser = argparse.ArgumentParser() + with self.assertRaises(ValueError) as cm: + parser.add_argument("--foo", nargs=nargs, metavar=metavar) + self.assertEqual(cm.exception.args[0], self.EXPECTED_MESSAGE) + + # Unit tests for different values of metavar when nargs=None + + def test_nargs_None_metavar_string(self): + self.do_test_no_exception(nargs=None, metavar="1") + + def test_nargs_None_metavar_length0(self): + self.do_test_exception(nargs=None, metavar=tuple()) + + def test_nargs_None_metavar_length1(self): + self.do_test_no_exception(nargs=None, metavar=("1")) + + def test_nargs_None_metavar_length2(self): + self.do_test_exception(nargs=None, metavar=("1", "2")) + + def test_nargs_None_metavar_length3(self): + self.do_test_exception(nargs=None, metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=? + + def test_nargs_optional_metavar_string(self): + self.do_test_no_exception(nargs="?", metavar="1") + + def test_nargs_optional_metavar_length0(self): + self.do_test_exception(nargs="?", metavar=tuple()) + + def test_nargs_optional_metavar_length1(self): + self.do_test_no_exception(nargs="?", metavar=("1")) + + def test_nargs_optional_metavar_length2(self): + self.do_test_exception(nargs="?", metavar=("1", "2")) + + def test_nargs_optional_metavar_length3(self): + self.do_test_exception(nargs="?", metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=* + + def test_nargs_zeroormore_metavar_string(self): + self.do_test_no_exception(nargs="*", metavar="1") + + def test_nargs_zeroormore_metavar_length0(self): + self.do_test_exception(nargs="*", metavar=tuple()) + + def test_nargs_zeroormore_metavar_length1(self): + self.do_test_no_exception(nargs="*", metavar=("1")) + + def test_nargs_zeroormore_metavar_length2(self): + self.do_test_no_exception(nargs="*", metavar=("1", "2")) + + def test_nargs_zeroormore_metavar_length3(self): + self.do_test_exception(nargs="*", metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=+ + + def test_nargs_oneormore_metavar_string(self): + self.do_test_no_exception(nargs="+", metavar="1") + + def test_nargs_oneormore_metavar_length0(self): + self.do_test_exception(nargs="+", metavar=tuple()) + + def test_nargs_oneormore_metavar_length1(self): + self.do_test_no_exception(nargs="+", metavar=("1")) + + def test_nargs_oneormore_metavar_length2(self): + self.do_test_no_exception(nargs="+", metavar=("1", "2")) + + def test_nargs_oneormore_metavar_length3(self): + self.do_test_exception(nargs="+", metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=... + + def test_nargs_remainder_metavar_string(self): + self.do_test_no_exception(nargs="...", metavar="1") + + def test_nargs_remainder_metavar_length0(self): + self.do_test_no_exception(nargs="...", metavar=tuple()) + + def test_nargs_remainder_metavar_length1(self): + self.do_test_no_exception(nargs="...", metavar=("1")) + + def test_nargs_remainder_metavar_length2(self): + self.do_test_no_exception(nargs="...", metavar=("1", "2")) + + def test_nargs_remainder_metavar_length3(self): + self.do_test_no_exception(nargs="...", metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=A... + + def test_nargs_parser_metavar_string(self): + self.do_test_no_exception(nargs="A...", metavar="1") + + def test_nargs_parser_metavar_length0(self): + self.do_test_exception(nargs="A...", metavar=tuple()) + + def test_nargs_parser_metavar_length1(self): + self.do_test_no_exception(nargs="A...", metavar=("1")) + + def test_nargs_parser_metavar_length2(self): + self.do_test_exception(nargs="A...", metavar=("1", "2")) + + def test_nargs_parser_metavar_length3(self): + self.do_test_exception(nargs="A...", metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=1 + + def test_nargs_1_metavar_string(self): + self.do_test_no_exception(nargs=1, metavar="1") + + def test_nargs_1_metavar_length0(self): + self.do_test_exception(nargs=1, metavar=tuple()) + + def test_nargs_1_metavar_length1(self): + self.do_test_no_exception(nargs=1, metavar=("1")) + + def test_nargs_1_metavar_length2(self): + self.do_test_exception(nargs=1, metavar=("1", "2")) + + def test_nargs_1_metavar_length3(self): + self.do_test_exception(nargs=1, metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=2 + + def test_nargs_2_metavar_string(self): + self.do_test_no_exception(nargs=2, metavar="1") + + def test_nargs_2_metavar_length0(self): + self.do_test_exception(nargs=2, metavar=tuple()) + + def test_nargs_2_metavar_length1(self): + self.do_test_no_exception(nargs=2, metavar=("1")) + + def test_nargs_2_metavar_length2(self): + self.do_test_no_exception(nargs=2, metavar=("1", "2")) + + def test_nargs_2_metavar_length3(self): + self.do_test_exception(nargs=2, metavar=("1", "2", "3")) + + # Unit tests for different values of metavar when nargs=3 + + def test_nargs_3_metavar_string(self): + self.do_test_no_exception(nargs=3, metavar="1") + + def test_nargs_3_metavar_length0(self): + self.do_test_exception(nargs=3, metavar=tuple()) + + def test_nargs_3_metavar_length1(self): + self.do_test_no_exception(nargs=3, metavar=("1")) + + def test_nargs_3_metavar_length2(self): + self.do_test_exception(nargs=3, metavar=("1", "2")) + + def test_nargs_3_metavar_length3(self): + self.do_test_no_exception(nargs=3, metavar=("1", "2", "3")) + # ============================ # from argparse import * tests # ============================ diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 331d247..7d7e16e 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -632,6 +632,16 @@ class TestGetcallargsFunctions(unittest.TestCase): self.assertEqualCallArgs(f, '2, c=4, **collections.UserDict(b=3)') self.assertEqualCallArgs(f, 'b=2, **collections.UserDict(a=3, c=4)') + def test_varkw_only(self): + # issue11256: + f = self.makeCallable('**c') + self.assertEqualCallArgs(f, '') + self.assertEqualCallArgs(f, 'a=1') + self.assertEqualCallArgs(f, 'a=1, b=2') + self.assertEqualCallArgs(f, 'c=3, **{"a": 1, "b": 2}') + self.assertEqualCallArgs(f, '**collections.UserDict(a=1, b=2)') + self.assertEqualCallArgs(f, 'c=3, **collections.UserDict(a=1, b=2)') + def test_keyword_only(self): f = self.makeCallable('a=3, *, c, d=2') self.assertEqualCallArgs(f, 'c=3') @@ -643,6 +653,11 @@ class TestGetcallargsFunctions(unittest.TestCase): self.assertEqualException(f, 'a=3') self.assertEqualException(f, 'd=4') + f = self.makeCallable('*, c, d=2') + self.assertEqualCallArgs(f, 'c=3') + self.assertEqualCallArgs(f, 'c=3, d=4') + self.assertEqualCallArgs(f, 'd=4, c=3') + def test_multiple_features(self): f = self.makeCallable('a, b=2, *f, **g') self.assertEqualCallArgs(f, '2, 3, 7') @@ -656,6 +671,17 @@ class TestGetcallargsFunctions(unittest.TestCase): '(4,[5,6])]), **collections.UserDict(' 'y=9, z=10)') + f = self.makeCallable('a, b=2, *f, x, y=99, **g') + self.assertEqualCallArgs(f, '2, 3, x=8') + self.assertEqualCallArgs(f, '2, 3, x=8, *[(4,[5,6]), 7]') + self.assertEqualCallArgs(f, '2, x=8, *[3, (4,[5,6]), 7], y=9, z=10') + self.assertEqualCallArgs(f, 'x=8, *[2, 3, (4,[5,6])], y=9, z=10') + self.assertEqualCallArgs(f, 'x=8, *collections.UserList(' + '[2, 3, (4,[5,6])]), q=0, **{"y":9, "z":10}') + self.assertEqualCallArgs(f, '2, x=8, *collections.UserList([3, ' + '(4,[5,6])]), q=0, **collections.UserDict(' + 'y=9, z=10)') + def test_errors(self): f0 = self.makeCallable('') f1 = self.makeCallable('a, b') @@ -692,6 +718,13 @@ class TestGetcallargsFunctions(unittest.TestCase): # - for functions and bound methods: unexpected keyword 'c' # - for unbound methods: multiple values for keyword 'a' #self.assertEqualException(f, '1, c=3, a=2') + # issue11256: + f3 = self.makeCallable('**c') + self.assertEqualException(f3, '1, 2') + self.assertEqualException(f3, '1, 2, a=1, b=2') + f4 = self.makeCallable('*, a, b=0') + self.assertEqualException(f3, '1, 2') + self.assertEqualException(f3, '1, 2, a=1, b=2') class TestGetcallargsMethods(TestGetcallargsFunctions): diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py index 1e4f887..45abcc8 100644 --- a/Lib/test/test_mailbox.py +++ b/Lib/test/test_mailbox.py @@ -869,8 +869,6 @@ class TestMaildir(TestMailbox): self.assertFalse((perms & 0o111)) # Execute bits should all be off. def test_reread(self): - # Wait for 2 seconds - time.sleep(2) # Initially, the mailbox has not been read and the time is null. assert getattr(self._box, '_last_read', None) is None @@ -879,15 +877,21 @@ class TestMaildir(TestMailbox): self._box._refresh() assert getattr(self._box, '_last_read', None) is not None - # Try calling _refresh() again; the modification times shouldn't have - # changed, so the mailbox should not be re-reading. Re-reading causes - # the ._toc attribute to be assigned a new dictionary object, so - # we'll check that the ._toc attribute isn't a different object. + # Put the last modified times more than one second into the past + # (because mtime has a one second granularity, a refresh is done + # unconditionally if called for within the same second, just in case + # the mbox has changed). + for subdir in ('cur', 'new'): + os.utime(os.path.join(self._box._path, subdir), + (time.time()-5,)*2) + + # Re-reading causes the ._toc attribute to be assigned a new dictionary + # object, so we'll check that the ._toc attribute isn't a different + # object. orig_toc = self._box._toc def refreshed(): return self._box._toc is not orig_toc - time.sleep(1) # Wait 1sec to ensure time.time()'s value changes self._box._refresh() assert not refreshed() diff --git a/Lib/test/test_msilib.py b/Lib/test/test_msilib.py new file mode 100644 index 0000000..ccdaec7 --- /dev/null +++ b/Lib/test/test_msilib.py @@ -0,0 +1,46 @@ +""" Test suite for the code in msilib """ +import unittest +import os +from test.support import run_unittest, import_module +msilib = import_module('msilib') + +class Test_make_id(unittest.TestCase): + #http://msdn.microsoft.com/en-us/library/aa369212(v=vs.85).aspx + """The Identifier data type is a text string. Identifiers may contain the + ASCII characters A-Z (a-z), digits, underscores (_), or periods (.). + However, every identifier must begin with either a letter or an + underscore. + """ + + def test_is_no_change_required(self): + self.assertEqual( + msilib.make_id("short"), "short") + self.assertEqual( + msilib.make_id("nochangerequired"), "nochangerequired") + self.assertEqual( + msilib.make_id("one.dot"), "one.dot") + self.assertEqual( + msilib.make_id("_"), "_") + self.assertEqual( + msilib.make_id("a"), "a") + #self.assertEqual( + # msilib.make_id(""), "") + + def test_invalid_first_char(self): + self.assertEqual( + msilib.make_id("9.short"), "_9.short") + self.assertEqual( + msilib.make_id(".short"), "_.short") + + def test_invalid_any_char(self): + self.assertEqual( + msilib.make_id(".s\x82ort"), "_.s_ort") + self.assertEqual ( + msilib.make_id(".s\x82o?*+rt"), "_.s_o___rt") + + +def test_main(): + run_unittest(__name__) + +if __name__ == '__main__': + test_main() diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py index 55b8cdd..4e48944 100644 --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -917,6 +917,21 @@ class _TestArray(BaseTestCase): self.assertEqual(list(arr[:]), seq) @unittest.skipIf(c_int is None, "requires _ctypes") + def test_array_from_size(self): + size = 10 + # Test for zeroing (see issue #11675). + # The repetition below strengthens the test by increasing the chances + # of previously allocated non-zero memory being used for the new array + # on the 2nd and 3rd loops. + for _ in range(3): + arr = self.Array('i', size) + self.assertEqual(len(arr), size) + self.assertEqual(list(arr), [0] * size) + arr[:] = range(10) + self.assertEqual(list(arr), list(range(10))) + del arr + + @unittest.skipIf(c_int is None, "requires _ctypes") def test_rawarray(self): self.test_array(raw=True) diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py index 1d575cd..92f7963 100644 --- a/Lib/test/test_pydoc.py +++ b/Lib/test/test_pydoc.py @@ -12,9 +12,10 @@ import unittest import xml.etree import textwrap from io import StringIO +from collections import namedtuple from contextlib import contextmanager from test.support import TESTFN, forget, rmtree, EnvironmentVarGuard, \ - reap_children, captured_output + reap_children, captured_output, captured_stdout from test import pydoc_mod @@ -373,6 +374,15 @@ class PydocDocTest(unittest.TestCase): finally: pydoc.getpager = getpager_old + def test_namedtuple_public_underscore(self): + NT = namedtuple('NT', ['abc', 'def'], rename=True) + with captured_stdout() as help_io: + help(NT) + helptext = help_io.getvalue() + self.assertIn('_1', helptext) + self.assertIn('_replace', helptext) + self.assertIn('_asdict', helptext) + class TestDescriptions(unittest.TestCase): diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index e4b33c9..fe8bc34 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1,7 +1,9 @@ from test.support import verbose, run_unittest import re from re import Scanner -import sys, traceback +import sys +import string +import traceback from weakref import proxy # Misc tests from Tim Peters' re.doc @@ -411,31 +413,62 @@ class ReTests(unittest.TestCase): self.assertEqual(re.search("\s(b)", " b").group(1), "b") self.assertEqual(re.search("a\s", "a ").group(0), "a ") + def assertMatch(self, pattern, text, match=None, span=None, + matcher=re.match): + if match is None and span is None: + # the pattern matches the whole text + match = text + span = (0, len(text)) + elif match is None or span is None: + raise ValueError('If match is not None, span should be specified ' + '(and vice versa).') + m = matcher(pattern, text) + self.assertTrue(m) + self.assertEqual(m.group(), match) + self.assertEqual(m.span(), span) + def test_re_escape(self): - p="" - self.assertEqual(re.escape(p), p) - for i in range(0, 256): - p = p + chr(i) - self.assertEqual(re.match(re.escape(chr(i)), chr(i)) is not None, - True) - self.assertEqual(re.match(re.escape(chr(i)), chr(i)).span(), (0,1)) - - pat=re.compile(re.escape(p)) - self.assertEqual(pat.match(p) is not None, True) - self.assertEqual(pat.match(p).span(), (0,256)) + alnum_chars = string.ascii_letters + string.digits + p = ''.join(chr(i) for i in range(256)) + for c in p: + if c in alnum_chars: + self.assertEqual(re.escape(c), c) + elif c == '\x00': + self.assertEqual(re.escape(c), '\\000') + else: + self.assertEqual(re.escape(c), '\\' + c) + self.assertMatch(re.escape(c), c) + self.assertMatch(re.escape(p), p) def test_re_escape_byte(self): - p=b"" - self.assertEqual(re.escape(p), p) - for i in range(0, 256): + alnum_chars = (string.ascii_letters + string.digits).encode('ascii') + p = bytes(range(256)) + for i in p: b = bytes([i]) - p += b - self.assertEqual(re.match(re.escape(b), b) is not None, True) - self.assertEqual(re.match(re.escape(b), b).span(), (0,1)) - - pat=re.compile(re.escape(p)) - self.assertEqual(pat.match(p) is not None, True) - self.assertEqual(pat.match(p).span(), (0,256)) + if b in alnum_chars: + self.assertEqual(re.escape(b), b) + elif i == 0: + self.assertEqual(re.escape(b), b'\\000') + else: + self.assertEqual(re.escape(b), b'\\' + b) + self.assertMatch(re.escape(b), b) + self.assertMatch(re.escape(p), p) + + def test_re_escape_non_ascii(self): + s = 'xxx\u2620\u2620\u2620xxx' + s_escaped = re.escape(s) + self.assertEqual(s_escaped, 'xxx\\\u2620\\\u2620\\\u2620xxx') + self.assertMatch(s_escaped, s) + self.assertMatch('.%s+.' % re.escape('\u2620'), s, + 'x\u2620\u2620\u2620x', (2, 7), re.search) + + def test_re_escape_non_ascii_bytes(self): + b = 'y\u2620y\u2620y'.encode('utf-8') + b_escaped = re.escape(b) + self.assertEqual(b_escaped, b'y\\\xe2\\\x98\\\xa0y\\\xe2\\\x98\\\xa0y') + self.assertMatch(b_escaped, b) + res = re.findall(re.escape('\u2620'.encode('utf-8')), b) + self.assertEqual(len(res), 2) def pickle_test(self, pickle): oldpat = re.compile('a(?:b|(c|e){1,2}?|d)+?(.)') diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 01e670e..7ca3d92 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1226,6 +1226,7 @@ class POSIXProcessTestCase(BaseTestCase): stdout=subprocess.PIPE, bufsize=0) f = p.stdout + self.addCleanup(f.close) try: self.assertEqual(f.read(4), b"appl") self.assertIn(f, select.select([f], [], [], 0.0)[0]) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py index c885103..32efb2b 100644 --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -12,6 +12,7 @@ import time class URLTimeoutTest(unittest.TestCase): + # XXX this test doesn't seem to test anything useful. TIMEOUT = 30.0 @@ -24,7 +25,7 @@ class URLTimeoutTest(unittest.TestCase): def testURLread(self): with support.transient_internet("www.python.org"): f = urllib.request.urlopen("http://www.python.org/") - x = f.read() + x = f.read() class urlopenNetworkTests(unittest.TestCase): """Tests urllib.reqest.urlopen using the network. @@ -43,8 +44,10 @@ class urlopenNetworkTests(unittest.TestCase): def urlopen(self, *args, **kwargs): resource = args[0] - with support.transient_internet(resource): - return urllib.request.urlopen(*args, **kwargs) + cm = support.transient_internet(resource) + cm.__enter__() + self.addCleanup(cm.__exit__, None, None, None) + return urllib.request.urlopen(*args, **kwargs) def test_basic(self): # Simple test expected to pass. @@ -135,8 +138,10 @@ class urlretrieveNetworkTests(unittest.TestCase): def urlretrieve(self, *args): resource = args[0] - with support.transient_internet(resource): - return urllib.request.urlretrieve(*args) + cm = support.transient_internet(resource) + cm.__enter__() + self.addCleanup(cm.__exit__, None, None, None) + return urllib.request.urlretrieve(*args) def test_basic(self): # Test basic functionality. diff --git a/Lib/test/test_xdrlib.py b/Lib/test/test_xdrlib.py index 073448c..6004c9f 100644 --- a/Lib/test/test_xdrlib.py +++ b/Lib/test/test_xdrlib.py @@ -12,6 +12,7 @@ class XDRTest(unittest.TestCase): a = [b'what', b'is', b'hapnin', b'doctor'] p.pack_int(42) + p.pack_int(-17) p.pack_uint(9) p.pack_bool(True) p.pack_bool(False) @@ -29,6 +30,7 @@ class XDRTest(unittest.TestCase): self.assertEqual(up.get_position(), 0) self.assertEqual(up.unpack_int(), 42) + self.assertEqual(up.unpack_int(), -17) self.assertEqual(up.unpack_uint(), 9) self.assertTrue(up.unpack_bool() is True) diff --git a/Lib/xdrlib.py b/Lib/xdrlib.py index b293e06..4e48677 100644 --- a/Lib/xdrlib.py +++ b/Lib/xdrlib.py @@ -50,7 +50,9 @@ class Packer: def pack_uint(self, x): self.__buf.write(struct.pack('>L', x)) - pack_int = pack_uint + def pack_int(self, x): + self.__buf.write(struct.pack('>l', x)) + pack_enum = pack_int def pack_bool(self, x): @@ -331,6 +331,7 @@ Duncan Grisby Fabian Groffen Eric Groo Dag Gruneau +Filip Gruszczyński Michael Guravage Lars Gustäbel Thomas Güttler @@ -354,6 +355,7 @@ Brian Harring Larry Hastings Shane Hathaway Rycharde Hawkes +Ben Hayden Jochen Hayek Christian Heimes Thomas Heller @@ -568,6 +570,7 @@ Chris McDonough Greg McFarlane Alan McIntyre Michael McLay +Mark Mc Mahon Gordon McMillan Caolan McNamara Andrew McNamara @@ -49,11 +49,34 @@ Core and Builtins Library ------- -- Issue #11662: Make urllib and urllib2 ignore redirections if the - scheme is not HTTP, HTTPS or FTP (CVE-2011-1521). +- Issue #11256: Fix inspect.getcallargs on functions that take only keyword + arguments. + +- Issue #11696: Fix ID generation in msilib. + +- Issue #9696: Fix exception incorrectly raised by xdrlib.Packer.pack_int when + trying to pack a negative (in-range) integer. + +- Issue #11675: multiprocessing.[Raw]Array objects created from an integer size + are now zeroed on creation. This matches the behaviour specified by the + documentation. + +- Issue #7639: Fix short file name generation in bdist_msi + +- Issue #11659: Fix ResourceWarning in test_subprocess introduced by #11459. + Patch by Ben Hayden. + +- Issue #11635: Don't use polling in worker threads and processes launched by + concurrent.futures. - Issue #11628: cmp_to_key generated class should use __slots__ +- Issue #11666: let help() display named tuple attributes and methods + that start with a leading underscore. + +- Issue #11662: Make urllib and urllib2 ignore redirections if the + scheme is not HTTP, HTTPS or FTP (CVE-2011-1521). + - Issue #5537: Fix time2isoz() and time2netscape() functions of httplib.cookiejar for expiration year greater than 2038 on 32-bit systems. @@ -148,6 +171,15 @@ Library - Issue #10276: Fix the results of zlib.crc32() and zlib.adler32() on buffers larger than 4GB. Patch by Nadeem Vawda. +- Issue #9348: Raise an early error if argparse nargs and metavar don't match. + +- Issue #8982: Improve the documentation for the argparse Namespace object. + +- Issue #9343: Document that argparse parent parsers must be configured before + their children. + +- Issue #9026: Fix order of argparse sub-commands in help messages. + Build ----- @@ -161,6 +193,12 @@ Tools/Demos - Issue #11179: Make ccbench work under Python 3.1 and 2.7 again. +Extensions +---------- + +- Issue #1838: Prevent segfault in ctypes, when _as_parameter_ on a class is set + to an instance of the class. + Tests ----- diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index f613613..8e85980 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -585,7 +585,10 @@ static PyObject * CDataType_from_param(PyObject *type, PyObject *value) { PyObject *as_parameter; - if (1 == PyObject_IsInstance(value, type)) { + int res = PyObject_IsInstance(value, type); + if (res == -1) + return NULL; + if (res) { Py_INCREF(value); return value; } @@ -598,10 +601,14 @@ CDataType_from_param(PyObject *type, PyObject *value) /* If we got a PyCArgObject, we must check if the object packed in it is an instance of the type's dict->proto */ - if(dict && ob - && PyObject_IsInstance(ob, dict->proto)) { - Py_INCREF(value); - return value; + if(dict && ob) { + res = PyObject_IsInstance(ob, dict->proto); + if (res == -1) + return NULL; + if (res) { + Py_INCREF(value); + return value; + } } ob_name = (ob) ? Py_TYPE(ob)->tp_name : "???"; PyErr_Format(PyExc_TypeError, @@ -951,8 +958,7 @@ PyCPointerType_from_param(PyObject *type, PyObject *value) Py_INCREF(value); /* _byref steals a refcount */ return _byref(value); case -1: - PyErr_Clear(); - break; + return NULL; default: break; } @@ -1431,6 +1437,7 @@ static PyObject * c_wchar_p_from_param(PyObject *type, PyObject *value) { PyObject *as_parameter; + int res; if (value == Py_None) { Py_INCREF(Py_None); return Py_None; @@ -1451,7 +1458,10 @@ c_wchar_p_from_param(PyObject *type, PyObject *value) } return (PyObject *)parg; } - if (PyObject_IsInstance(value, type)) { + res = PyObject_IsInstance(value, type); + if (res == -1) + return NULL; + if (res) { Py_INCREF(value); return value; } @@ -1492,6 +1502,7 @@ static PyObject * c_char_p_from_param(PyObject *type, PyObject *value) { PyObject *as_parameter; + int res; if (value == Py_None) { Py_INCREF(Py_None); return Py_None; @@ -1512,7 +1523,10 @@ c_char_p_from_param(PyObject *type, PyObject *value) } return (PyObject *)parg; } - if (PyObject_IsInstance(value, type)) { + res = PyObject_IsInstance(value, type); + if (res == -1) + return NULL; + if (res) { Py_INCREF(value); return value; } @@ -1554,6 +1568,7 @@ c_void_p_from_param(PyObject *type, PyObject *value) { StgDictObject *stgd; PyObject *as_parameter; + int res; /* None */ if (value == Py_None) { @@ -1631,7 +1646,10 @@ c_void_p_from_param(PyObject *type, PyObject *value) return (PyObject *)parg; } /* c_void_p instance (or subclass) */ - if (PyObject_IsInstance(value, type)) { + res = PyObject_IsInstance(value, type); + if (res == -1) + return NULL; + if (res) { /* c_void_p instances */ Py_INCREF(value); return value; @@ -1990,10 +2008,14 @@ PyCSimpleType_from_param(PyObject *type, PyObject *value) PyCArgObject *parg; struct fielddesc *fd; PyObject *as_parameter; + int res; /* If the value is already an instance of the requested type, we can use it as is */ - if (1 == PyObject_IsInstance(value, type)) { + res = PyObject_IsInstance(value, type); + if (res == -1) + return NULL; + if (res) { Py_INCREF(value); return value; } @@ -2022,7 +2044,12 @@ PyCSimpleType_from_param(PyObject *type, PyObject *value) as_parameter = PyObject_GetAttrString(value, "_as_parameter_"); if (as_parameter) { + if (Py_EnterRecursiveCall("while processing _as_parameter_")) { + Py_DECREF(as_parameter); + return NULL; + } value = PyCSimpleType_from_param(type, as_parameter); + Py_LeaveRecursiveCall(); Py_DECREF(as_parameter); return value; } @@ -2714,6 +2741,7 @@ _PyCData_set(CDataObject *dst, PyObject *type, SETFUNC setfunc, PyObject *value, Py_ssize_t size, char *ptr) { CDataObject *src; + int err; if (setfunc) return setfunc(ptr, value, size); @@ -2754,7 +2782,10 @@ _PyCData_set(CDataObject *dst, PyObject *type, SETFUNC setfunc, PyObject *value, } src = (CDataObject *)value; - if (PyObject_IsInstance(value, type)) { + err = PyObject_IsInstance(value, type); + if (err == -1) + return NULL; + if (err) { memcpy(ptr, src->b_ptr, size); @@ -4749,14 +4780,17 @@ Pointer_set_contents(CDataObject *self, PyObject *value, void *closure) stgdict = PyObject_stgdict((PyObject *)self); assert(stgdict); /* Cannot be NULL fr pointer instances */ assert(stgdict->proto); - if (!CDataObject_Check(value) - || 0 == PyObject_IsInstance(value, stgdict->proto)) { - /* XXX PyObject_IsInstance could return -1! */ - PyErr_Format(PyExc_TypeError, - "expected %s instead of %s", - ((PyTypeObject *)(stgdict->proto))->tp_name, - Py_TYPE(value)->tp_name); - return -1; + if (!CDataObject_Check(value)) { + int res = PyObject_IsInstance(value, stgdict->proto); + if (res == -1) + return -1; + if (!res) { + PyErr_Format(PyExc_TypeError, + "expected %s instead of %s", + ((PyTypeObject *)(stgdict->proto))->tp_name, + Py_TYPE(value)->tp_name); + return -1; + } } dst = (CDataObject *)value; diff --git a/Modules/getbuildinfo.c b/Modules/getbuildinfo.c index ac51881..7069b6e 100644 --- a/Modules/getbuildinfo.c +++ b/Modules/getbuildinfo.c @@ -42,9 +42,9 @@ const char * Py_GetBuildInfo(void) { - static char buildinfo[50 + sizeof HGVERSION + - ((sizeof HGTAG > sizeof HGBRANCH) ? - sizeof HGTAG : sizeof HGBRANCH)]; + static char buildinfo[50 + sizeof(HGVERSION) + + ((sizeof(HGTAG) > sizeof(HGBRANCH)) ? + sizeof(HGTAG) : sizeof(HGBRANCH))]; const char *revision = _Py_hgversion(); const char *sep = *revision ? ":" : ""; const char *hgid = _Py_hgidentifier(); diff --git a/Objects/typeslots.inc b/Objects/typeslots.inc index 5186dcf..0494a32 100644 --- a/Objects/typeslots.inc +++ b/Objects/typeslots.inc @@ -1,4 +1,4 @@ -/* Generated by typeslots.py $Revision$ */ +/* Generated by typeslots.py $Revision: 87806 $ */ 0, 0, offsetof(PyHeapTypeObject, as_mapping.mp_ass_subscript), diff --git a/Tools/buildbot/clean-amd64.bat b/Tools/buildbot/clean-amd64.bat index 9fb35e9..715805a 100644 --- a/Tools/buildbot/clean-amd64.bat +++ b/Tools/buildbot/clean-amd64.bat @@ -1,7 +1,10 @@ @rem Used by the buildbot "clean" step. call "%VS90COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 -cd PCbuild @echo Deleting .pyc/.pyo files ... del /s Lib\*.pyc Lib\*.pyo +@echo Deleting test leftovers ... +rmdir /s /q build +cd PCbuild vcbuild /clean pcbuild.sln "Release|x64" vcbuild /clean pcbuild.sln "Debug|x64" +cd .. diff --git a/Tools/buildbot/clean.bat b/Tools/buildbot/clean.bat index ec71804..5c8f33e 100644 --- a/Tools/buildbot/clean.bat +++ b/Tools/buildbot/clean.bat @@ -2,6 +2,9 @@ call "%VS90COMNTOOLS%vsvars32.bat" @echo Deleting .pyc/.pyo files ... del /s Lib\*.pyc Lib\*.pyo +@echo Deleting test leftovers ... +rmdir /s /q build cd PCbuild vcbuild /clean pcbuild.sln "Release|Win32" vcbuild /clean pcbuild.sln "Debug|Win32" +cd .. |