From 73dd030c8b71a7080648554652912982054b1177 Mon Sep 17 00:00:00 2001 From: Antoine Pitrou Date: Sun, 11 Jan 2015 15:05:29 +0100 Subject: Issue #22952: improve multiprocessing doc introduction and defer notes until appropriate. Patch by Davin Potts. --- Doc/library/multiprocessing.rst | 86 ++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 32 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index dc0370a..ecac5b0 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -16,41 +16,27 @@ to this, the :mod:`multiprocessing` module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows. -.. note:: +The :mod:`multiprocessing` module also introduces APIs which do not have +analogs in the :mod:`threading` module. A prime example of this is the +:class:`~multiprocessing.pool.Pool` object which offers a convenient means of +parallelizing the execution of a function across multiple input values, +distributing the input data across processes (data parallelism). The following +example demonstrates the common practice of defining such functions in a module +so that child processes can successfully import that module. This basic example +of data parallelism using :class:`~multiprocessing.pool.Pool`, :: - Some of this package's functionality requires a functioning shared semaphore - implementation on the host operating system. Without one, the - :mod:`multiprocessing.synchronize` module will be disabled, and attempts to - import it will result in an :exc:`ImportError`. See - :issue:`3770` for additional information. + from multiprocessing import Pool -.. note:: + def f(x): + return x*x + + if __name__ == '__main__': + with Pool(5) as p: + print(p.map(f, [1, 2, 3])) - Functionality within this package requires that the ``__main__`` module be - importable by the children. This is covered in :ref:`multiprocessing-programming` - however it is worth pointing out here. This means that some examples, such - as the :class:`multiprocessing.pool.Pool` examples will not work in the - interactive interpreter. For example:: - - >>> from multiprocessing import Pool - >>> p = Pool(5) - >>> def f(x): - ... return x*x - ... - >>> p.map(f, [1,2,3]) - Process PoolWorker-1: - Process PoolWorker-2: - Process PoolWorker-3: - Traceback (most recent call last): - Traceback (most recent call last): - Traceback (most recent call last): - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - - (If you try this it will actually output three full tracebacks - interleaved in a semi-random fashion, and then you may have to - stop the master process somehow.) +will print to standard output :: + + [1, 4, 9] The :class:`Process` class @@ -276,6 +262,14 @@ that only one process prints to standard output at a time:: Without using the lock output from the different processes is liable to get all mixed up. +.. note:: + + Some of this package's functionality requires a functioning shared semaphore + implementation on the host operating system. Without one, the + :mod:`multiprocessing.synchronize` module will be disabled, and attempts to + import it will result in an :exc:`ImportError`. See + :issue:`3770` for additional information. + Sharing state between processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -406,6 +400,34 @@ For example:: Note that the methods of a pool should only ever be used by the process which created it. +.. note:: + + Functionality within this package requires that the ``__main__`` module be + importable by the children. This is covered in :ref:`multiprocessing-programming` + however it is worth pointing out here. This means that some examples, such + as the :class:`multiprocessing.pool.Pool` examples will not work in the + interactive interpreter. For example:: + + >>> from multiprocessing import Pool + >>> p = Pool(5) + >>> def f(x): + ... return x*x + ... + >>> p.map(f, [1,2,3]) + Process PoolWorker-1: + Process PoolWorker-2: + Process PoolWorker-3: + Traceback (most recent call last): + Traceback (most recent call last): + Traceback (most recent call last): + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + + (If you try this it will actually output three full tracebacks + interleaved in a semi-random fashion, and then you may have to + stop the master process somehow.) + Reference --------- -- cgit v0.12