diff options
Diffstat (limited to 'Doc/library')
36 files changed, 2914 insertions, 106 deletions
diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst index 2eced45..3e38cb4 100644 --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -1,3 +1,5 @@ +.. _abstract-base-classes: + :mod:`abc` --- Abstract Base Classes ==================================== @@ -12,7 +14,7 @@ -------------- This module provides the infrastructure for defining an :term:`abstract base -class` (ABCs) in Python, as outlined in :pep:`3119`; see the PEP for why this +class` (ABC) in Python, as outlined in :pep:`3119`; see the PEP for why this was added to Python. (See also :pep:`3141` and the :mod:`numbers` module regarding a type hierarchy for numbers based on ABCs.) diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 2ccdb51..87f2cf3 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -37,14 +37,18 @@ All of the classes in this module may safely be accessed from multiple threads. *fileobj*), or operate directly on a named file (named by *filename*). Exactly one of these two parameters should be provided. - The *mode* argument can be either ``'r'`` for reading (default), or ``'w'`` - for writing. + The *mode* argument can be either ``'r'`` for reading (default), ``'w'`` for + overwriting, or ``'a'`` for appending. If *fileobj* is provided, a mode of + ``'w'`` does not truncate the file, and is instead equivalent to ``'a'``. The *buffering* argument is ignored. Its use is deprecated. - If *mode* is ``'w'``, *compresslevel* can be a number between ``1`` and - ``9`` specifying the level of compression: ``1`` produces the least - compression, and ``9`` (default) produces the most compression. + If *mode* is ``'w'`` or ``'a'``, *compresslevel* can be a number between + ``1`` and ``9`` specifying the level of compression: ``1`` produces the + least compression, and ``9`` (default) produces the most compression. + + If *mode* is ``'r'``, the input file may be the concatenation of multiple + compressed streams. :class:`BZ2File` provides all of the members specified by the :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. @@ -70,6 +74,10 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionchanged:: 3.3 The *fileobj* argument to the constructor was added. + .. versionchanged:: 3.3 + The ``'a'`` (append) mode was added, along with support for reading + multi-stream files. + Incremental (de)compression --------------------------- @@ -106,14 +114,20 @@ Incremental (de)compression incrementally. For one-shot compression, use the :func:`decompress` function instead. + .. note:: + This class does not transparently handle inputs containing multiple + compressed streams, unlike :func:`decompress` and :class:`BZ2File`. If + you need to decompress a multi-stream input with :class:`BZ2Decompressor`, + you must use a new decompressor for each stream. + .. method:: decompress(data) Provide data to the decompressor object. Returns a chunk of decompressed data if possible, or an empty byte string otherwise. - Attempting to decompress data after the end of stream is reached raises - an :exc:`EOFError`. If any data is found after the end of the stream, it - is ignored and saved in the :attr:`unused_data` attribute. + Attempting to decompress data after the end of the current stream is + reached raises an :exc:`EOFError`. If any data is found after the end of + the stream, it is ignored and saved in the :attr:`unused_data` attribute. .. attribute:: eof @@ -127,6 +141,9 @@ Incremental (de)compression Data found after the end of the compressed stream. + If this attribute is accessed before the end of the stream has been + reached, its value will be ``b''``. + One-shot (de)compression ------------------------ @@ -145,5 +162,11 @@ One-shot (de)compression Decompress *data*. + If *data* is the concatenation of multiple compressed streams, decompress + all of the streams. + For incremental decompression, use a :class:`BZ2Decompressor` instead. + .. versionchanged:: 3.3 + Support for multi-stream inputs was added. + diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 4d5058e..90bd0dd 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -458,7 +458,8 @@ define in order to be compatible with the Python codec registry. .. method:: reset() - Reset the encoder to the initial state. + Reset the encoder to the initial state. The output is discarded: call + ``.encode('', final=True)`` to reset the encoder and to get the output. .. method:: IncrementalEncoder.getstate() diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 58354f8..1d451d1 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -23,7 +23,7 @@ example, whether it is hashable or whether it is a mapping. .. versionchanged:: 3.3 Formerly, this module was part of the :mod:`collections` module. -.. _abstract-base-classes: +.. _collections-abstract-base-classes: Collections Abstract Base Classes --------------------------------- diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index f54af32..6c9b1e5 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -34,7 +34,7 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`, ===================== ==================================================================== .. versionchanged:: 3.3 - Moved :ref:`abstract-base-classes` to the :mod:`collections.abc` module. + Moved :ref:`collections-abstract-base-classes` to the :mod:`collections.abc` module. For backwards compatibility, they continue to be visible in this module as well. diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index ec5a4b0..1ba2ed3 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -29,6 +29,8 @@ this module. Hashing Methods --------------- +.. versionadded:: 3.3 + The :mod:`crypt` module defines the list of hashing methods (not all methods are available on all platforms): @@ -37,33 +39,26 @@ are available on all platforms): A Modular Crypt Format method with 16 character salt and 86 character hash. This is the strongest method. - .. versionadded:: 3.3 - .. data:: METHOD_SHA256 Another Modular Crypt Format method with 16 character salt and 43 character hash. - .. versionadded:: 3.3 - .. data:: METHOD_MD5 Another Modular Crypt Format method with 8 character salt and 22 character hash. - .. versionadded:: 3.3 - .. data:: METHOD_CRYPT The traditional method with a 2 character salt and 13 characters of hash. This is the weakest method. - .. versionadded:: 3.3 - Module Attributes ----------------- +.. versionadded:: 3.3 .. attribute:: methods @@ -71,8 +66,6 @@ Module Attributes ``crypt.METHOD_*`` objects. This list is sorted from strongest to weakest, and is guaranteed to have at least ``crypt.METHOD_CRYPT``. - .. versionadded:: 3.3 - Module Functions ---------------- @@ -108,9 +101,8 @@ The :mod:`crypt` module defines the following functions: different sizes in the *salt*, it is recommended to use the full crypted password as salt when checking for a password. -.. versionchanged:: 3.3 - Before version 3.3, *salt* must be specified as a string and cannot - accept ``crypt.METHOD_*`` values (which don't exist anyway). + .. versionchanged:: 3.3 + Accept ``crypt.METHOD_*`` values in addition to strings for *salt*. .. function:: mksalt(method=None) @@ -124,25 +116,27 @@ The :mod:`crypt` module defines the following functions: 16 random characters from the set ``[./a-zA-Z0-9]``, suitable for passing as the *salt* argument to :func:`crypt`. -.. versionadded:: 3.3 + .. versionadded:: 3.3 Examples -------- A simple example illustrating typical use:: - import crypt, getpass, pwd + import pwd + import crypt + import getpass def login(): - username = input('Python login:') + username = input('Python login: ') cryptedpasswd = pwd.getpwnam(username)[1] if cryptedpasswd: if cryptedpasswd == 'x' or cryptedpasswd == '*': - raise "Sorry, currently no support for shadow passwords" + raise ValueError('no support for shadow passwords') cleartext = getpass.getpass() return crypt.crypt(cleartext, cryptedpasswd) == cryptedpasswd else: - return 1 + return True To generate a hash of a password using the strongest available method and check it against the original:: @@ -151,4 +145,4 @@ check it against the original:: hashed = crypt.crypt(plaintext) if hashed != crypt.crypt(plaintext, hashed): - raise "Hashed version doesn't validate against original" + raise ValueError("hashed version doesn't validate against original") diff --git a/Doc/library/depgraph-output.png b/Doc/library/depgraph-output.png Binary files differnew file mode 100644 index 0000000..960bb1b --- /dev/null +++ b/Doc/library/depgraph-output.png diff --git a/Doc/library/distutils.rst b/Doc/library/distutils.rst index 238b79d..53a69ae 100644 --- a/Doc/library/distutils.rst +++ b/Doc/library/distutils.rst @@ -12,18 +12,26 @@ additional modules into a Python installation. The new modules may be either 100%-pure Python, or may be extension modules written in C, or may be collections of Python packages which include modules coded in both Python and C. -This package is discussed in two separate chapters: +.. deprecated:: 3.3 + :mod:`packaging` replaces Distutils. See :ref:`packaging-index` and + :ref:`packaging-install-index`. +User documentation and API reference are provided in another document: + .. seealso:: :ref:`distutils-index` The manual for developers and packagers of Python modules. This describes how to prepare :mod:`distutils`\ -based packages so that they may be - easily installed into an existing Python installation. + easily installed into an existing Python installation. If also contains + instructions for end-users wanting to install a distutils-based package, + :ref:`install-index`. + + +.. trick to silence a Sphinx warning - :ref:`install-index` - An "administrators" manual which includes information on installing - modules into an existing Python installation. You do not need to be a - Python programmer to read this manual. +.. toctree:: + :hidden: + ../distutils/index diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 8b7eef5..1c1c167 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -290,19 +290,18 @@ are always available. They are listed here in alphabetical order. The resulting list is sorted alphabetically. For example: >>> import struct - >>> dir() # doctest: +SKIP + >>> dir() # show the names in the module namespace ['__builtins__', '__doc__', '__name__', 'struct'] - >>> dir(struct) # doctest: +NORMALIZE_WHITESPACE + >>> dir(struct) # show the names in the struct module ['Struct', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_clearcache', 'calcsize', 'error', 'pack', 'pack_into', 'unpack', 'unpack_from'] - >>> class Foo: - ... def __dir__(self): - ... return ["kan", "ga", "roo"] - ... - >>> f = Foo() - >>> dir(f) - ['ga', 'kan', 'roo'] + >>> class Shape(object): + def __dir__(self): + return ['area', 'perimeter', 'location'] + >>> s = Shape() + >>> dir(s) + ['area', 'perimeter', 'location'] .. note:: @@ -333,15 +332,21 @@ are always available. They are listed here in alphabetical order. :meth:`__next__` method of the iterator returned by :func:`enumerate` returns a tuple containing a count (from *start* which defaults to 0) and the corresponding value obtained from iterating over *iterable*. - :func:`enumerate` is useful for obtaining an indexed series: ``(0, seq[0])``, - ``(1, seq[1])``, ``(2, seq[2])``, .... For example: - >>> for i, season in enumerate(['Spring', 'Summer', 'Fall', 'Winter']): - ... print(i, season) - 0 Spring - 1 Summer - 2 Fall - 3 Winter + >>> for i, season in enumerate('Spring Summer Fall Winter'.split(), start=1): + print(i, season) + 1 Spring + 2 Summer + 3 Fall + 4 Winter + + Equivalent to:: + + def enumerate(sequence, start=0): + n = start + for elem in sequence: + yield n, elem + n += 1 .. function:: eval(expression, globals=None, locals=None) @@ -580,7 +585,7 @@ are always available. They are listed here in alphabetical order. Two objects with non-overlapping lifetimes may have the same :func:`id` value. - .. impl-detail:: This is the address of the object. + .. impl-detail:: This is the address of the object in memory. .. function:: input([prompt]) @@ -652,10 +657,10 @@ are always available. They are listed here in alphabetical order. One useful application of the second form of :func:`iter` is to read lines of a file until a certain line is reached. The following example reads a file - until ``"STOP"`` is reached: :: + until the :meth:`readline` method returns an empty string:: - with open("mydata.txt") as fp: - for line in iter(fp.readline, "STOP"): + with open('mydata.txt') as fp: + for line in iter(fp.readline, ''): process_line(line) @@ -1169,8 +1174,9 @@ are always available. They are listed here in alphabetical order. It can be called either on the class (such as ``C.f()``) or on an instance (such as ``C().f()``). The instance is ignored except for its class. - Static methods in Python are similar to those found in Java or C++. For a more - advanced concept, see :func:`classmethod` in this section. + Static methods in Python are similar to those found in Java or C++. Also see + :func:`classmethod` for a variant that is useful for creating alternate class + constructors. For more information on static methods, consult the documentation on the standard type hierarchy in :ref:`types`. @@ -1270,6 +1276,10 @@ are always available. They are listed here in alphabetical order. references. The zero argument form automatically searches the stack frame for the class (``__class__``) and the first argument. + For practical suggestions on how to design cooperative classes using + :func:`super`, see `guide to using super() + <http://rhettinger.wordpress.com/2011/05/26/super-considered-super/>`_. + .. function:: tuple([iterable]) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 6ef6d9d..c62a00d 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1019,6 +1019,19 @@ as internal buffering of data. Availability: Unix, Windows. +.. function:: pipe2(flags=0) + + Create a pipe with *flags* set atomically. + *flags* is optional and can be constructed by ORing together zero or more of + these values: :data:`O_NONBLOCK`, :data:`O_CLOEXEC`. + Return a pair of file descriptors ``(r, w)`` usable for reading and writing, + respectively. + + Availability: some flavors of Unix. + + .. versionadded:: 3.3 + + .. function:: posix_fallocate(fd, offset, len) Ensures that enough disk space is allocated for the file specified by *fd* diff --git a/Doc/library/packaging-misc.rst b/Doc/library/packaging-misc.rst new file mode 100644 index 0000000..5e56247 --- /dev/null +++ b/Doc/library/packaging-misc.rst @@ -0,0 +1,27 @@ +.. temporary file for modules that don't need a dedicated file yet + +:mod:`packaging.errors` --- Packaging exceptions +================================================ + +.. module:: packaging.errors + :synopsis: Packaging exceptions. + + +Provides exceptions used by the Packaging modules. Note that Packaging modules +may raise standard exceptions; in particular, SystemExit is usually raised for +errors that are obviously the end-user's fault (e.g. bad command-line arguments). + +This module is safe to use in ``from ... import *`` mode; it only exports +symbols whose names start with ``Packaging`` and end with ``Error``. + + +:mod:`packaging.manifest` --- The Manifest class +================================================ + +.. module:: packaging.manifest + :synopsis: The Manifest class, used for poking about the file system and + building lists of files. + + +This module provides the :class:`Manifest` class, used for poking about the +filesystem and building lists of files. diff --git a/Doc/library/packaging.command.rst b/Doc/library/packaging.command.rst new file mode 100644 index 0000000..98835c0 --- /dev/null +++ b/Doc/library/packaging.command.rst @@ -0,0 +1,111 @@ +:mod:`packaging.command` --- Standard Packaging commands +======================================================== + +.. module:: packaging.command + :synopsis: Standard packaging commands. + + +This subpackage contains one module for each standard Packaging command, such as +:command:`build` or :command:`upload`. Each command is implemented as a +separate module, with the command name as the name of the module and of the +class defined therein. + + + +:mod:`packaging.command.cmd` --- Abstract base class for Packaging commands +=========================================================================== + +.. module:: packaging.command.cmd + :synopsis: Abstract base class for commands. + + +This module supplies the abstract base class :class:`Command`. This class is +subclassed by the modules in the packaging.command subpackage. + + +.. class:: Command(dist) + + Abstract base class for defining command classes, the "worker bees" of the + Packaging. A useful analogy for command classes is to think of them as + subroutines with local variables called *options*. The options are declared + in :meth:`initialize_options` and defined (given their final values) in + :meth:`finalize_options`, both of which must be defined by every command + class. The distinction between the two is necessary because option values + might come from the outside world (command line, config file, ...), and any + options dependent on other options must be computed after these outside + influences have been processed --- hence :meth:`finalize_options`. The body + of the subroutine, where it does all its work based on the values of its + options, is the :meth:`run` method, which must also be implemented by every + command class. + + The class constructor takes a single argument *dist*, a + :class:`~packaging.dist.Distribution` instance. + + +Creating a new Packaging command +-------------------------------- + +This section outlines the steps to create a new Packaging command. + +.. XXX the following paragraph is focused on the stdlib; expand it to document + how to write and register a command in third-party projects + +A new command lives in a module in the :mod:`packaging.command` package. There +is a sample template in that directory called :file:`command_template`. Copy +this file to a new module with the same name as the new command you're +implementing. This module should implement a class with the same name as the +module (and the command). So, for instance, to create the command +``peel_banana`` (so that users can run ``setup.py peel_banana``), you'd copy +:file:`command_template` to :file:`packaging/command/peel_banana.py`, then edit +it so that it's implementing the class :class:`peel_banana`, a subclass of +:class:`Command`. It must define the following methods: + +.. method:: Command.initialize_options() + + Set default values for all the options that this command supports. Note that + these defaults may be overridden by other commands, by the setup script, by + config files, or by the command line. Thus, this is not the place to code + dependencies between options; generally, :meth:`initialize_options` + implementations are just a bunch of ``self.foo = None`` assignments. + + +.. method:: Command.finalize_options() + + Set final values for all the options that this command supports. This is + always called as late as possible, i.e. after any option assignments from the + command line or from other commands have been done. Thus, this is the place + to to code option dependencies: if *foo* depends on *bar*, then it is safe to + set *foo* from *bar* as long as *foo* still has the same value it was + assigned in :meth:`initialize_options`. + + +.. method:: Command.run() + + A command's raison d'etre: carry out the action it exists to perform, + controlled by the options initialized in :meth:`initialize_options`, + customized by other commands, the setup script, the command line, and config + files, and finalized in :meth:`finalize_options`. All terminal output and + filesystem interaction should be done by :meth:`run`. + + +Command classes may define this attribute: + + +.. attribute:: Command.sub_commands + + *sub_commands* formalizes the notion of a "family" of commands, + e.g. ``install_dist`` as the parent with sub-commands ``install_lib``, + ``install_headers``, etc. The parent of a family of commands defines + *sub_commands* as a class attribute; it's a list of 2-tuples ``(command_name, + predicate)``, with *command_name* a string and *predicate* a function, a + string or ``None``. *predicate* is a method of the parent command that + determines whether the corresponding command is applicable in the current + situation. (E.g. ``install_headers`` is only applicable if we have any C + header files to install.) If *predicate* is ``None``, that command is always + applicable. + + *sub_commands* is usually defined at the *end* of a class, because + predicates can be methods of the class, so they must already have been + defined. The canonical example is the :command:`install_dist` command. + +.. XXX document how to add a custom command to another one's subcommands diff --git a/Doc/library/packaging.compiler.rst b/Doc/library/packaging.compiler.rst new file mode 100644 index 0000000..dac6263 --- /dev/null +++ b/Doc/library/packaging.compiler.rst @@ -0,0 +1,672 @@ +:mod:`packaging.compiler` --- Compiler classes +============================================== + +.. module:: packaging.compiler + :synopsis: Compiler classes to build C/C++ extensions or libraries. + + +This subpackage contains an abstract base class representing a compiler and +concrete implementations for common compilers. The compiler classes should not +be instantiated directly, but created using the :func:`new_compiler` factory +function. Compiler types provided by Packaging are listed in +:ref:`packaging-standard-compilers`. + + +Public functions +---------------- + +.. function:: new_compiler(plat=None, compiler=None, verbose=0, dry_run=0, force=0) + + Factory function to generate an instance of some + :class:`~.ccompiler.CCompiler` subclass for the requested platform or + compiler type. + + If no argument is given for *plat* and *compiler*, the default compiler type + for the platform (:attr:`os.name`) will be used: ``'unix'`` for Unix and + Mac OS X, ``'msvc'`` for Windows. + + If *plat* is given, it must be one of ``'posix'``, ``'darwin'`` or ``'nt'``. + An invalid value will not raise an exception but use the default compiler + type for the current platform. + + .. XXX errors should never pass silently; this behavior is particularly + harmful when a compiler type is given as first argument + + If *compiler* is given, *plat* will be ignored, allowing you to get for + example a ``'unix'`` compiler object under Windows or an ``'msvc'`` compiler + under Unix. However, not all compiler types can be instantiated on every + platform. + + +.. function:: customize_compiler(compiler) + + Do any platform-specific customization of a CCompiler instance. Mainly + needed on Unix to plug in the information that varies across Unices and is + stored in CPython's Makefile. + + +.. function:: gen_lib_options(compiler, library_dirs, runtime_library_dirs, libraries) + + Generate linker options for searching library directories and linking with + specific libraries. *libraries* and *library_dirs* are, respectively, lists + of library names (not filenames!) and search directories. Returns a list of + command-line options suitable for use with some compiler (depending on the + two format strings passed in). + + +.. function:: gen_preprocess_options(macros, include_dirs) + + Generate C preprocessor options (:option:`-D`, :option:`-U`, :option:`-I`) as + used by at least two types of compilers: the typical Unix compiler and Visual + C++. *macros* is the usual thing, a list of 1- or 2-tuples, where ``(name,)`` + means undefine (:option:`-U`) macro *name*, and ``(name, value)`` means + define (:option:`-D`) macro *name* to *value*. *include_dirs* is just a list + of directory names to be added to the header file search path (:option:`-I`). + Returns a list of command-line options suitable for either Unix compilers or + Visual C++. + + +.. function:: get_default_compiler(osname, platform) + + Determine the default compiler to use for the given platform. + + *osname* should be one of the standard Python OS names (i.e. the ones + returned by ``os.name``) and *platform* the common value returned by + ``sys.platform`` for the platform in question. + + The default values are ``os.name`` and ``sys.platform``. + + +.. function:: set_compiler(location) + + Add or change a compiler + + +.. function:: show_compilers() + + Print list of available compilers (used by the :option:`--help-compiler` + options to :command:`build`, :command:`build_ext`, :command:`build_clib`). + + +.. _packaging-standard-compilers: + +Standard compilers +------------------ + +Concrete subclasses of :class:`~.ccompiler.CCompiler` are provided in submodules +of the :mod:`packaging.compiler` package. You do not need to import them, using +:func:`new_compiler` is the public API to use. This table documents the +standard compilers; be aware that they can be replaced by other classes on your +platform. + +=============== ======================================================== ======= +name description notes +=============== ======================================================== ======= +``'unix'`` typical Unix-style command-line C compiler [#]_ +``'msvc'`` Microsoft compiler [#]_ +``'bcpp'`` Borland C++ compiler +``'cygwin'`` Cygwin compiler (Windows port of GCC) +``'mingw32'`` Mingw32 port of GCC (same as Cygwin in no-Cygwin mode) +=============== ======================================================== ======= + + +.. [#] The Unix compiler class assumes this behavior: + + * macros defined with :option:`-Dname[=value]` + + * macros undefined with :option:`-Uname` + + * include search directories specified with :option:`-Idir` + + * libraries specified with :option:`-llib` + + * library search directories specified with :option:`-Ldir` + + * compile handled by :program:`cc` (or similar) executable with + :option:`-c` option: compiles :file:`.c` to :file:`.o` + + * link static library handled by :program:`ar` command (possibly with + :program:`ranlib`) + + * link shared library handled by :program:`cc` :option:`-shared` + + +.. [#] On Windows, extension modules typically need to be compiled with the same + compiler that was used to compile CPython (for example Microsoft Visual + Studio .NET 2003 for CPython 2.4 and 2.5). The AMD64 and Itanium + binaries are created using the Platform SDK. + + Under the hood, there are actually two different subclasses of + :class:`~.ccompiler.CCompiler` defined: one is compatible with MSVC 2005 + and 2008, the other works with older versions. This should not be a + concern for regular use of the functions in this module. + + Packaging will normally choose the right compiler, linker etc. on its + own. To override this choice, the environment variables + *DISTUTILS_USE_SDK* and *MSSdk* must be both set. *MSSdk* indicates that + the current environment has been setup by the SDK's ``SetEnv.Cmd`` + script, or that the environment variables had been registered when the + SDK was installed; *DISTUTILS_USE_SDK* indicates that the user has made + an explicit choice to override the compiler selection done by Packaging. + + .. TODO document the envvars in Doc/using and the man page + + +:mod:`packaging.compiler.ccompiler` --- CCompiler base class +============================================================ + +.. module:: packaging.compiler.ccompiler + :synopsis: Abstract CCompiler class. + + +This module provides the abstract base class for the :class:`CCompiler` +classes. A :class:`CCompiler` instance can be used for all the compile and +link steps needed to build a single project. Methods are provided to set +options for the compiler --- macro definitions, include directories, link path, +libraries and the like. + +.. class:: CCompiler([verbose=0, dry_run=0, force=0]) + + The abstract base class :class:`CCompiler` defines the interface that must be + implemented by real compiler classes. The class also has some utility + methods used by several compiler classes. + + The basic idea behind a compiler abstraction class is that each instance can + be used for all the compile/link steps in building a single project. Thus, + attributes common to all of those compile and link steps --- include + directories, macros to define, libraries to link against, etc. --- are + attributes of the compiler instance. To allow for variability in how + individual files are treated, most of those attributes may be varied on a + per-compilation or per-link basis. + + The constructor for each subclass creates an instance of the Compiler object. + Flags are *verbose* (show verbose output), *dry_run* (don't actually execute + the steps) and *force* (rebuild everything, regardless of dependencies). All + of these flags default to ``0`` (off). Note that you probably don't want to + instantiate :class:`CCompiler` or one of its subclasses directly - use the + :func:`packaging.CCompiler.new_compiler` factory function instead. + + The following methods allow you to manually alter compiler options for the + instance of the Compiler class. + + + .. method:: CCompiler.add_include_dir(dir) + + Add *dir* to the list of directories that will be searched for header + files. The compiler is instructed to search directories in the order in + which they are supplied by successive calls to :meth:`add_include_dir`. + + + .. method:: CCompiler.set_include_dirs(dirs) + + Set the list of directories that will be searched to *dirs* (a list of + strings). Overrides any preceding calls to :meth:`add_include_dir`; + subsequent calls to :meth:`add_include_dir` add to the list passed to + :meth:`set_include_dirs`. This does not affect any list of standard + include directories that the compiler may search by default. + + + .. method:: CCompiler.add_library(libname) + + Add *libname* to the list of libraries that will be included in all links + driven by this compiler object. Note that *libname* should *not* be the + name of a file containing a library, but the name of the library itself: + the actual filename will be inferred by the linker, the compiler, or the + compiler class (depending on the platform). + + The linker will be instructed to link against libraries in the order they + were supplied to :meth:`add_library` and/or :meth:`set_libraries`. It is + perfectly valid to duplicate library names; the linker will be instructed + to link against libraries as many times as they are mentioned. + + + .. method:: CCompiler.set_libraries(libnames) + + Set the list of libraries to be included in all links driven by this + compiler object to *libnames* (a list of strings). This does not affect + any standard system libraries that the linker may include by default. + + + .. method:: CCompiler.add_library_dir(dir) + + Add *dir* to the list of directories that will be searched for libraries + specified to :meth:`add_library` and :meth:`set_libraries`. The linker + will be instructed to search for libraries in the order they are supplied + to :meth:`add_library_dir` and/or :meth:`set_library_dirs`. + + + .. method:: CCompiler.set_library_dirs(dirs) + + Set the list of library search directories to *dirs* (a list of strings). + This does not affect any standard library search path that the linker may + search by default. + + + .. method:: CCompiler.add_runtime_library_dir(dir) + + Add *dir* to the list of directories that will be searched for shared + libraries at runtime. + + + .. method:: CCompiler.set_runtime_library_dirs(dirs) + + Set the list of directories to search for shared libraries at runtime to + *dirs* (a list of strings). This does not affect any standard search path + that the runtime linker may search by default. + + + .. method:: CCompiler.define_macro(name[, value=None]) + + Define a preprocessor macro for all compilations driven by this compiler + object. The optional parameter *value* should be a string; if it is not + supplied, then the macro will be defined without an explicit value and the + exact outcome depends on the compiler used (XXX true? does ANSI say + anything about this?) + + + .. method:: CCompiler.undefine_macro(name) + + Undefine a preprocessor macro for all compilations driven by this compiler + object. If the same macro is defined by :meth:`define_macro` and + undefined by :meth:`undefine_macro` the last call takes precedence + (including multiple redefinitions or undefinitions). If the macro is + redefined/undefined on a per-compilation basis (i.e. in the call to + :meth:`compile`), then that takes precedence. + + + .. method:: CCompiler.add_link_object(object) + + Add *object* to the list of object files (or analogues, such as explicitly + named library files or the output of "resource compilers") to be included + in every link driven by this compiler object. + + + .. method:: CCompiler.set_link_objects(objects) + + Set the list of object files (or analogues) to be included in every link + to *objects*. This does not affect any standard object files that the + linker may include by default (such as system libraries). + + The following methods implement methods for autodetection of compiler + options, providing some functionality similar to GNU :program:`autoconf`. + + + .. method:: CCompiler.detect_language(sources) + + Detect the language of a given file, or list of files. Uses the instance + attributes :attr:`language_map` (a dictionary), and :attr:`language_order` + (a list) to do the job. + + + .. method:: CCompiler.find_library_file(dirs, lib[, debug=0]) + + Search the specified list of directories for a static or shared library file + *lib* and return the full path to that file. If *debug* is true, look for a + debugging version (if that makes sense on the current platform). Return + ``None`` if *lib* wasn't found in any of the specified directories. + + + .. method:: CCompiler.has_function(funcname [, includes=None, include_dirs=None, libraries=None, library_dirs=None]) + + Return a boolean indicating whether *funcname* is supported on the current + platform. The optional arguments can be used to augment the compilation + environment by providing additional include files and paths and libraries and + paths. + + + .. method:: CCompiler.library_dir_option(dir) + + Return the compiler option to add *dir* to the list of directories searched for + libraries. + + + .. method:: CCompiler.library_option(lib) + + Return the compiler option to add *dir* to the list of libraries linked into the + shared library or executable. + + + .. method:: CCompiler.runtime_library_dir_option(dir) + + Return the compiler option to add *dir* to the list of directories searched for + runtime libraries. + + + .. method:: CCompiler.set_executables(**args) + + Define the executables (and options for them) that will be run to perform the + various stages of compilation. The exact set of executables that may be + specified here depends on the compiler class (via the 'executables' class + attribute), but most will have: + + +--------------+------------------------------------------+ + | attribute | description | + +==============+==========================================+ + | *compiler* | the C/C++ compiler | + +--------------+------------------------------------------+ + | *linker_so* | linker used to create shared objects and | + | | libraries | + +--------------+------------------------------------------+ + | *linker_exe* | linker used to create binary executables | + +--------------+------------------------------------------+ + | *archiver* | static library creator | + +--------------+------------------------------------------+ + + On platforms with a command line (Unix, DOS/Windows), each of these is a string + that will be split into executable name and (optional) list of arguments. + (Splitting the string is done similarly to how Unix shells operate: words are + delimited by spaces, but quotes and backslashes can override this. See + :func:`packaging.util.split_quoted`.) + + The following methods invoke stages in the build process. + + + .. method:: CCompiler.compile(sources[, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None]) + + Compile one or more source files. Generates object files (e.g. transforms a + :file:`.c` file to a :file:`.o` file.) + + *sources* must be a list of filenames, most likely C/C++ files, but in reality + anything that can be handled by a particular compiler and compiler class (e.g. + an ``'msvc'`` compiler` can handle resource files in *sources*). Return a list of + object filenames, one per source filename in *sources*. Depending on the + implementation, not all source files will necessarily be compiled, but all + corresponding object filenames will be returned. + + If *output_dir* is given, object files will be put under it, while retaining + their original path component. That is, :file:`foo/bar.c` normally compiles to + :file:`foo/bar.o` (for a Unix implementation); if *output_dir* is *build*, then + it would compile to :file:`build/foo/bar.o`. + + *macros*, if given, must be a list of macro definitions. A macro definition is + either a ``(name, value)`` 2-tuple or a ``(name,)`` 1-tuple. The former defines + a macro; if the value is ``None``, the macro is defined without an explicit + value. The 1-tuple case undefines a macro. Later + definitions/redefinitions/undefinitions take precedence. + + *include_dirs*, if given, must be a list of strings, the directories to add to + the default include file search path for this compilation only. + + *debug* is a boolean; if true, the compiler will be instructed to output debug + symbols in (or alongside) the object file(s). + + *extra_preargs* and *extra_postargs* are implementation-dependent. On platforms + that have the notion of a command line (e.g. Unix, DOS/Windows), they are most + likely lists of strings: extra command-line arguments to prepend/append to the + compiler command line. On other platforms, consult the implementation class + documentation. In any event, they are intended as an escape hatch for those + occasions when the abstract compiler framework doesn't cut the mustard. + + *depends*, if given, is a list of filenames that all targets depend on. If a + source file is older than any file in depends, then the source file will be + recompiled. This supports dependency tracking, but only at a coarse + granularity. + + Raises :exc:`CompileError` on failure. + + + .. method:: CCompiler.create_static_lib(objects, output_libname[, output_dir=None, debug=0, target_lang=None]) + + Link a bunch of stuff together to create a static library file. The "bunch of + stuff" consists of the list of object files supplied as *objects*, the extra + object files supplied to :meth:`add_link_object` and/or + :meth:`set_link_objects`, the libraries supplied to :meth:`add_library` and/or + :meth:`set_libraries`, and the libraries supplied as *libraries* (if any). + + *output_libname* should be a library name, not a filename; the filename will be + inferred from the library name. *output_dir* is the directory where the library + file will be put. XXX defaults to what? + + *debug* is a boolean; if true, debugging information will be included in the + library (note that on most platforms, it is the compile step where this matters: + the *debug* flag is included here just for consistency). + + *target_lang* is the target language for which the given objects are being + compiled. This allows specific linkage time treatment of certain languages. + + Raises :exc:`LibError` on failure. + + + .. method:: CCompiler.link(target_desc, objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + + Link a bunch of stuff together to create an executable or shared library file. + + The "bunch of stuff" consists of the list of object files supplied as *objects*. + *output_filename* should be a filename. If *output_dir* is supplied, + *output_filename* is relative to it (i.e. *output_filename* can provide + directory components if needed). + + *libraries* is a list of libraries to link against. These are library names, + not filenames, since they're translated into filenames in a platform-specific + way (e.g. *foo* becomes :file:`libfoo.a` on Unix and :file:`foo.lib` on + DOS/Windows). However, they can include a directory component, which means the + linker will look in that specific directory rather than searching all the normal + locations. + + *library_dirs*, if supplied, should be a list of directories to search for + libraries that were specified as bare library names (i.e. no directory + component). These are on top of the system default and those supplied to + :meth:`add_library_dir` and/or :meth:`set_library_dirs`. *runtime_library_dirs* + is a list of directories that will be embedded into the shared library and used + to search for other shared libraries that \*it\* depends on at run-time. (This + may only be relevant on Unix.) + + *export_symbols* is a list of symbols that the shared library will export. + (This appears to be relevant only on Windows.) + + *debug* is as for :meth:`compile` and :meth:`create_static_lib`, with the + slight distinction that it actually matters on most platforms (as opposed to + :meth:`create_static_lib`, which includes a *debug* flag mostly for form's + sake). + + *extra_preargs* and *extra_postargs* are as for :meth:`compile` (except of + course that they supply command-line arguments for the particular linker being + used). + + *target_lang* is the target language for which the given objects are being + compiled. This allows specific linkage time treatment of certain languages. + + Raises :exc:`LinkError` on failure. + + + .. method:: CCompiler.link_executable(objects, output_progname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, target_lang=None]) + + Link an executable. *output_progname* is the name of the file executable, while + *objects* are a list of object filenames to link in. Other arguments are as for + the :meth:`link` method. + + + .. method:: CCompiler.link_shared_lib(objects, output_libname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + + Link a shared library. *output_libname* is the name of the output library, + while *objects* is a list of object filenames to link in. Other arguments are + as for the :meth:`link` method. + + + .. method:: CCompiler.link_shared_object(objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + + Link a shared object. *output_filename* is the name of the shared object that + will be created, while *objects* is a list of object filenames to link in. + Other arguments are as for the :meth:`link` method. + + + .. method:: CCompiler.preprocess(source[, output_file=None, macros=None, include_dirs=None, extra_preargs=None, extra_postargs=None]) + + Preprocess a single C/C++ source file, named in *source*. Output will be written + to file named *output_file*, or *stdout* if *output_file* not supplied. + *macros* is a list of macro definitions as for :meth:`compile`, which will + augment the macros set with :meth:`define_macro` and :meth:`undefine_macro`. + *include_dirs* is a list of directory names that will be added to the default + list, in the same way as :meth:`add_include_dir`. + + Raises :exc:`PreprocessError` on failure. + + The following utility methods are defined by the :class:`CCompiler` class, for + use by the various concrete subclasses. + + + .. method:: CCompiler.executable_filename(basename[, strip_dir=0, output_dir='']) + + Returns the filename of the executable for the given *basename*. Typically for + non-Windows platforms this is the same as the basename, while Windows will get + a :file:`.exe` added. + + + .. method:: CCompiler.library_filename(libname[, lib_type='static', strip_dir=0, output_dir='']) + + Returns the filename for the given library name on the current platform. On Unix + a library with *lib_type* of ``'static'`` will typically be of the form + :file:`liblibname.a`, while a *lib_type* of ``'dynamic'`` will be of the form + :file:`liblibname.so`. + + + .. method:: CCompiler.object_filenames(source_filenames[, strip_dir=0, output_dir='']) + + Returns the name of the object files for the given source files. + *source_filenames* should be a list of filenames. + + + .. method:: CCompiler.shared_object_filename(basename[, strip_dir=0, output_dir='']) + + Returns the name of a shared object file for the given file name *basename*. + + + .. method:: CCompiler.execute(func, args[, msg=None, level=1]) + + Invokes :func:`packaging.util.execute` This method invokes a Python function + *func* with the given arguments *args*, after logging and taking into account + the *dry_run* flag. XXX see also. + + + .. method:: CCompiler.spawn(cmd) + + Invokes :func:`packaging.util.spawn`. This invokes an external process to run + the given command. XXX see also. + + + .. method:: CCompiler.mkpath(name[, mode=511]) + + Invokes :func:`packaging.dir_util.mkpath`. This creates a directory and any + missing ancestor directories. XXX see also. + + + .. method:: CCompiler.move_file(src, dst) + + Invokes :meth:`packaging.file_util.move_file`. Renames *src* to *dst*. XXX see + also. + + +:mod:`packaging.compiler.extension` --- The Extension class +=========================================================== + +.. module:: packaging.compiler.extension + :synopsis: Class used to represent C/C++ extension modules. + + +This module provides the :class:`Extension` class, used to represent C/C++ +extension modules. + +.. class:: Extension + + The Extension class describes a single C or C++ extension module. It accepts + the following keyword arguments in its constructor + + +------------------------+--------------------------------+---------------------------+ + | argument name | value | type | + +========================+================================+===========================+ + | *name* | the full name of the | string | + | | extension, including any | | + | | packages --- i.e. *not* a | | + | | filename or pathname, but | | + | | Python dotted name | | + +------------------------+--------------------------------+---------------------------+ + | *sources* | list of source filenames, | string | + | | relative to the distribution | | + | | root (where the setup script | | + | | lives), in Unix form (slash- | | + | | separated) for portability. | | + | | Source files may be C, C++, | | + | | SWIG (.i), platform-specific | | + | | resource files, or whatever | | + | | else is recognized by the | | + | | :command:`build_ext` command | | + | | as source for a Python | | + | | extension. | | + +------------------------+--------------------------------+---------------------------+ + | *include_dirs* | list of directories to search | string | + | | for C/C++ header files (in | | + | | Unix form for portability) | | + +------------------------+--------------------------------+---------------------------+ + | *define_macros* | list of macros to define; each | (string, string) tuple or | + | | macro is defined using a | (name, ``None``) | + | | 2-tuple ``(name, value)``, | | + | | where *value* is | | + | | either the string to define it | | + | | to or ``None`` to define it | | + | | without a particular value | | + | | (equivalent of ``#define FOO`` | | + | | in source or :option:`-DFOO` | | + | | on Unix C compiler command | | + | | line) | | + +------------------------+--------------------------------+---------------------------+ + | *undef_macros* | list of macros to undefine | string | + | | explicitly | | + +------------------------+--------------------------------+---------------------------+ + | *library_dirs* | list of directories to search | string | + | | for C/C++ libraries at link | | + | | time | | + +------------------------+--------------------------------+---------------------------+ + | *libraries* | list of library names (not | string | + | | filenames or paths) to link | | + | | against | | + +------------------------+--------------------------------+---------------------------+ + | *runtime_library_dirs* | list of directories to search | string | + | | for C/C++ libraries at run | | + | | time (for shared extensions, | | + | | this is when the extension is | | + | | loaded) | | + +------------------------+--------------------------------+---------------------------+ + | *extra_objects* | list of extra files to link | string | + | | with (e.g. object files not | | + | | implied by 'sources', static | | + | | library that must be | | + | | explicitly specified, binary | | + | | resource files, etc.) | | + +------------------------+--------------------------------+---------------------------+ + | *extra_compile_args* | any extra platform- and | string | + | | compiler-specific information | | + | | to use when compiling the | | + | | source files in 'sources'. For | | + | | platforms and compilers where | | + | | a command line makes sense, | | + | | this is typically a list of | | + | | command-line arguments, but | | + | | for other platforms it could | | + | | be anything. | | + +------------------------+--------------------------------+---------------------------+ + | *extra_link_args* | any extra platform- and | string | + | | compiler-specific information | | + | | to use when linking object | | + | | files together to create the | | + | | extension (or to create a new | | + | | static Python interpreter). | | + | | Similar interpretation as for | | + | | 'extra_compile_args'. | | + +------------------------+--------------------------------+---------------------------+ + | *export_symbols* | list of symbols to be exported | string | + | | from a shared extension. Not | | + | | used on all platforms, and not | | + | | generally necessary for Python | | + | | extensions, which typically | | + | | export exactly one symbol: | | + | | ``init`` + extension_name. | | + +------------------------+--------------------------------+---------------------------+ + | *depends* | list of files that the | string | + | | extension depends on | | + +------------------------+--------------------------------+---------------------------+ + | *language* | extension language (i.e. | string | + | | ``'c'``, ``'c++'``, | | + | | ``'objc'``). Will be detected | | + | | from the source extensions if | | + | | not provided. | | + +------------------------+--------------------------------+---------------------------+ diff --git a/Doc/library/packaging.database.rst b/Doc/library/packaging.database.rst new file mode 100644 index 0000000..2a8c39f --- /dev/null +++ b/Doc/library/packaging.database.rst @@ -0,0 +1,324 @@ +:mod:`packaging.database` --- Database of installed distributions +================================================================= + +.. module:: packaging.database + :synopsis: Functions to query and manipulate installed distributions. + + +This module provides an implementation of :PEP:`376`. It was originally +intended to land in :mod:`pkgutil`, but with the inclusion of Packaging in the +standard library, it was thought best to include it in a submodule of +:mod:`packaging`, leaving :mod:`pkgutil` to deal with imports. + +Installed Python distributions are represented by instances of +:class:`Distribution`, or :class:`EggInfoDistribution` for legacy egg formats. +Most functions also provide an extra argument ``use_egg_info`` to take legacy +distributions into account. + + +Classes representing installed distributions +-------------------------------------------- + +.. class:: Distribution(path) + + Class representing an installed distribution. It is different from + :class:`packaging.dist.Distribution` which holds the list of files, the + metadata and options during the run of a Packaging command. + + Instantiate with the *path* to a ``.dist-info`` directory. Instances can be + compared and sorted. Other available methods are: + + .. XXX describe how comparison works + + .. method:: get_distinfo_file(path, binary=False) + + Return a read-only file object for a file located at + :file:`{project-version}.dist-info/path}`. *path* should be a + ``'/'``-separated path relative to the ``.dist-info`` directory or an + absolute path; if it is an absolute path and doesn't start with the path + to the :file:`.dist-info` directory, a :class:`PackagingError` is raised. + + If *binary* is ``True``, the file is opened in binary mode. + + .. method:: get_resource_path(relative_path) + + .. TODO + + .. method:: list_distinfo_files(local=False) + + Return an iterator over all files located in the :file:`.dist-info` + directory. If *local* is ``True``, each returned path is transformed into + a local absolute path, otherwise the raw value found in the :file:`RECORD` + file is returned. + + .. method:: list_installed_files(local=False) + + Iterate over the files installed with the distribution and registered in + the :file:`RECORD` file and yield a tuple ``(path, md5, size)`` for each + line. If *local* is ``True``, the returned path is transformed into a + local absolute path, otherwise the raw value is returned. + + A local absolute path is an absolute path in which occurrences of ``'/'`` + have been replaced by :data:`os.sep`. + + .. method:: uses(path) + + Check whether *path* was installed by this distribution (i.e. if the path + is present in the :file:`RECORD` file). *path* can be a local absolute + path or a relative ``'/'``-separated path. Returns a boolean. + + Available attributes: + + .. attribute:: metadata + + Instance of :class:`packaging.metadata.Metadata` filled with the contents + of the :file:`{project-version}.dist-info/METADATA` file. + + .. attribute:: name + + Shortcut for ``metadata['Name']``. + + .. attribute:: version + + Shortcut for ``metadata['Version']``. + + .. attribute:: requested + + Boolean indicating whether this distribution was requested by the user of + automatically installed as a dependency. + + +.. class:: EggInfoDistribution(path) + + Class representing a legacy distribution. It is compatible with distutils' + and setuptools' :file:`.egg-info` and :file:`.egg` files and directories. + + .. FIXME should be named EggDistribution + + Instantiate with the *path* to an egg file or directory. Instances can be + compared and sorted. Other available methods are: + + .. method:: list_installed_files(local=False) + + .. method:: uses(path) + + Available attributes: + + .. attribute:: metadata + + Instance of :class:`packaging.metadata.Metadata` filled with the contents + of the :file:`{project-version}.egg-info/PKG-INFO` or + :file:`{project-version}.egg` file. + + .. attribute:: name + + Shortcut for ``metadata['Name']``. + + .. attribute:: version + + Shortcut for ``metadata['Version']``. + + +Functions to work with the database +----------------------------------- + +.. function:: get_distribution(name, use_egg_info=False, paths=None) + + Return an instance of :class:`Distribution` or :class:`EggInfoDistribution` + for the first installed distribution matching *name*. Egg distributions are + considered only if *use_egg_info* is true; if both a dist-info and an egg + file are found, the dist-info prevails. The directories to be searched are + given in *paths*, which defaults to :data:`sys.path`. Return ``None`` if no + matching distribution is found. + + .. FIXME param should be named use_egg + + +.. function:: get_distributions(use_egg_info=False, paths=None) + + Return an iterator of :class:`Distribution` instances for all installed + distributions found in *paths* (defaults to :data:`sys.path`). If + *use_egg_info* is true, also return instances of :class:`EggInfoDistribution` + for legacy distributions found. + + +.. function:: get_file_users(path) + + Return an iterator over all distributions using *path*, a local absolute path + or a relative ``'/'``-separated path. + + .. XXX does this work with prefixes or full file path only? + + +.. function:: obsoletes_distribution(name, version=None, use_egg_info=False) + + Return an iterator over all distributions that declare they obsolete *name*. + *version* is an optional argument to match only specific releases (see + :mod:`packaging.version`). If *use_egg_info* is true, legacy egg + distributions will be considered as well. + + +.. function:: provides_distribution(name, version=None, use_egg_info=False) + + Return an iterator over all distributions that declare they provide *name*. + *version* is an optional argument to match only specific releases (see + :mod:`packaging.version`). If *use_egg_info* is true, legacy egg + distributions will be considered as well. + + +Utility functions +----------------- + +.. function:: distinfo_dirname(name, version) + + Escape *name* and *version* into a filename-safe form and return the + directory name built from them, for example + :file:`{safename}-{safeversion}.dist-info.` In *name*, runs of + non-alphanumeric characters are replaced with one ``'_'``; in *version*, + spaces become dots, and runs of other non-alphanumeric characters (except + dots) a replaced by one ``'-'``. + + .. XXX wth spaces in version numbers? + +For performance purposes, the list of distributions is being internally +cached. Caching is enabled by default, but you can control it with these +functions: + +.. function:: clear_cache() + + Clear the cache. + +.. function:: disable_cache() + + Disable the cache, without clearing it. + +.. function:: enable_cache() + + Enable the internal cache, without clearing it. + + +Examples +-------- + +Print all information about a distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given a path to a ``.dist-info`` distribution, we shall print out all +information that can be obtained using functions provided in this module:: + + import sys + import packaging.database + + path = input() + # first create the Distribution instance + try: + dist = packaging.database.Distribution(path) + except IOError: + sys.exit('No such distribution') + + print('Information about %r' % dist.name) + print() + + print('Files') + print('=====') + for path, md5, size in dist.list_installed_files(): + print('* Path: %s' % path) + print(' Hash %s, Size: %s bytes' % (md5, size)) + print() + + print('Metadata') + print('========') + for key, value in dist.metadata.items(): + print('%20s: %s' % (key, value)) + print() + + print('Extra') + print('=====') + if dist.requested: + print('* It was installed by user request') + else: + print('* It was installed as a dependency') + +If we save the script above as ``print_info.py``, we can use it to extract +information from a :file:`.dist-info` directory. By typing in the console: + +.. code-block:: sh + + $ echo /tmp/choxie/choxie-2.0.0.9.dist-info | python3 print_info.py + +we get the following output: + +.. code-block:: none + + Information about 'choxie' + + Files + ===== + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/truffles.py + Hash 5e052db6a478d06bad9ae033e6bc08af, Size: 111 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py + Hash ac56bf496d8d1d26f866235b95f31030, Size: 214 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py + Hash 416aab08dfa846f473129e89a7625bbc, Size: 25 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/INSTALLER + Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA + Hash 696a209967fef3c8b8f5a7bb10386385, Size: 225 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/REQUESTED + Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/RECORD + Hash None, Size: None bytes + + Metadata + ======== + Metadata-Version: 1.2 + Name: choxie + Version: 2.0.0.9 + Platform: [] + Supported-Platform: UNKNOWN + Summary: Chocolate with a kick! + Description: UNKNOWN + Keywords: [] + Home-page: UNKNOWN + Author: UNKNOWN + Author-email: UNKNOWN + Maintainer: UNKNOWN + Maintainer-email: UNKNOWN + License: UNKNOWN + Classifier: [] + Download-URL: UNKNOWN + Obsoletes-Dist: ['truffles (<=0.8,>=0.5)', 'truffles (<=0.9,>=0.6)'] + Project-URL: [] + Provides-Dist: ['truffles (1.0)'] + Requires-Dist: ['towel-stuff (0.1)'] + Requires-Python: UNKNOWN + Requires-External: [] + + Extra + ===== + * It was installed as a dependency + + +Find out obsoleted distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, we take tackle a different problem, we are interested in finding out +which distributions have been obsoleted. This can be easily done as follows:: + + import packaging.database + + # iterate over all distributions in the system + for dist in packaging.database.get_distributions(): + name, version = dist.name, dist.version + # find out which distributions obsolete this name/version combination + replacements = packaging.database.obsoletes_distribution(name, version) + if replacements: + print('%r %s is obsoleted by' % (name, version), + ', '.join(repr(r.name) for r in replacements)) + +This is how the output might look like: + +.. code-block:: none + + 'strawberry' 0.6 is obsoleted by 'choxie' + 'grammar' 1.0a4 is obsoleted by 'towel-stuff' diff --git a/Doc/library/packaging.depgraph.rst b/Doc/library/packaging.depgraph.rst new file mode 100644 index 0000000..c384788 --- /dev/null +++ b/Doc/library/packaging.depgraph.rst @@ -0,0 +1,199 @@ +:mod:`packaging.depgraph` --- Dependency graph builder +====================================================== + +.. module:: packaging.depgraph + :synopsis: Graph builder for dependencies between releases. + + +This module provides the means to analyse the dependencies between various +distributions and to create a graph representing these dependency relationships. +In this document, "distribution" refers to an instance of +:class:`packaging.database.Distribution` or +:class:`packaging.database.EggInfoDistribution`. + +.. XXX terminology problem with dist vs. release: dists are installed, but deps + use releases + +.. XXX explain how to use it with dists not installed: Distribution can only be + instantiated with a path, but this module is useful for remote dist too + +.. XXX functions should accept and return iterators, not lists + + +The :class:`DependencyGraph` class +---------------------------------- + +.. class:: DependencyGraph + + Represent a dependency graph between releases. The nodes are distribution + instances; the edge model dependencies. An edge from ``a`` to ``b`` means + that ``a`` depends on ``b``. + + .. method:: add_distribution(distribution) + + Add *distribution* to the graph. + + .. method:: add_edge(x, y, label=None) + + Add an edge from distribution *x* to distribution *y* with the given + *label* (string). + + .. method:: add_missing(distribution, requirement) + + Add a missing *requirement* (string) for the given *distribution*. + + .. method:: repr_node(dist, level=1) + + Print a subgraph starting from *dist*. *level* gives the depth of the + subgraph. + + Direct access to the graph nodes and edges is provided through these + attributes: + + .. attribute:: adjacency_list + + Dictionary mapping distributions to a list of ``(other, label)`` tuples + where ``other`` is a distribution and the edge is labeled with ``label`` + (i.e. the version specifier, if such was provided). + + .. attribute:: reverse_list + + Dictionary mapping distributions to a list of predecessors. This allows + efficient traversal. + + .. attribute:: missing + + Dictionary mapping distributions to a list of requirements that were not + provided by any distribution. + + +Auxiliary functions +------------------- + +.. function:: dependent_dists(dists, dist) + + Recursively generate a list of distributions from *dists* that are dependent + on *dist*. + + .. XXX what does member mean here: "dist is a member of *dists* for which we + are interested" + +.. function:: generate_graph(dists) + + Generate a :class:`DependencyGraph` from the given list of distributions. + + .. XXX make this alternate constructor a DepGraph classmethod or rename; + 'generate' can suggest it creates a file or an image, use 'make' + +.. function:: graph_to_dot(graph, f, skip_disconnected=True) + + Write a DOT output for the graph to the file-like object *f*. + + If *skip_disconnected* is true, all distributions that are not dependent on + any other distribution are skipped. + + .. XXX why is this not a DepGraph method? + + +Example Usage +------------- + +Depict all dependenciess in the system +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +First, we shall generate a graph of all the distributions on the system +and then create an image out of it using the tools provided by +`Graphviz <http://www.graphviz.org/>`_:: + + from packaging.database import get_distributions + from packaging.depgraph import generate_graph + + dists = list(get_distributions()) + graph = generate_graph(dists) + +It would be interesting to print out the missing requirements. This can be done +as follows:: + + for dist, reqs in graph.missing.items(): + if reqs: + reqs = ' ,'.join(repr(req) for req in reqs) + print('Missing dependencies for %r: %s' % (dist.name, reqs)) + +Example output is: + +.. code-block:: none + + Missing dependencies for 'TurboCheetah': 'Cheetah' + Missing dependencies for 'TurboGears': 'ConfigObj', 'DecoratorTools', 'RuleDispatch' + Missing dependencies for 'jockey': 'PyKDE4.kdecore', 'PyKDE4.kdeui', 'PyQt4.QtCore', 'PyQt4.QtGui' + Missing dependencies for 'TurboKid': 'kid' + Missing dependencies for 'TurboJson: 'DecoratorTools', 'RuleDispatch' + +Now, we proceed with generating a graphical representation of the graph. First +we write it to a file, and then we generate a PNG image using the +:program:`dot` command-line tool:: + + from packaging.depgraph import graph_to_dot + with open('output.dot', 'w') as f: + # only show the interesting distributions, skipping the disconnected ones + graph_to_dot(graph, f, skip_disconnected=True) + +We can create the final picture using: + +.. code-block:: sh + + $ dot -Tpng output.dot > output.png + +An example result is: + +.. figure:: depgraph-output.png + :alt: Example PNG output from packaging.depgraph and dot + +If you want to include egg distributions as well, then the code requires only +one change, namely the line:: + + dists = list(packaging.database.get_distributions()) + +has to be replaced with:: + + dists = list(packaging.database.get_distributions(use_egg_info=True)) + +On many platforms, a richer graph is obtained because at the moment most +distributions are provided in the egg rather than the new standard +``.dist-info`` format. + +.. XXX missing image + + An example of a more involved graph for illustrative reasons can be seen + here: + + .. image:: depgraph_big.png + + +List all dependent distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We will list all distributions that are dependent on some given distibution. +This time, egg distributions will be considered as well:: + + import sys + from packaging.database import get_distribution, get_distributions + from packaging.depgraph import dependent_dists + + dists = list(get_distributions(use_egg_info=True)) + dist = get_distribution('bacon', use_egg_info=True) + if dist is None: + sys.exit('No such distribution in the system') + + deps = dependent_dists(dists, dist) + deps = ', '.join(repr(x.name) for x in deps) + print('Distributions depending on %r: %s' % (dist.name, deps)) + +And this is example output: + +.. with the dependency relationships as in the previous section + (depgraph_big) + +.. code-block:: none + + Distributions depending on 'bacon': 'towel-stuff', 'choxie', 'grammar' diff --git a/Doc/library/packaging.dist.rst b/Doc/library/packaging.dist.rst new file mode 100644 index 0000000..fb05b69 --- /dev/null +++ b/Doc/library/packaging.dist.rst @@ -0,0 +1,102 @@ +:mod:`packaging.dist` --- The Distribution class +================================================ + +.. module:: packaging.dist + :synopsis: Core Distribution class. + + +This module provides the :class:`Distribution` class, which represents the +module distribution being built/packaged/distributed/installed. + +.. class:: Distribution(arguments) + + A :class:`Distribution` describes how to build, package, distribute and + install a Python project. + + The arguments accepted by the constructor are laid out in the following + table. Some of them will end up in a metadata object, the rest will become + data attributes of the :class:`Distribution` instance. + + .. TODO improve constructor to take a Metadata object + named params? + (i.e. Distribution(metadata, cmdclass, py_modules, etc) + .. TODO also remove obsolete(?) script_name, etc. parameters? see what + py2exe and other tools need + + +--------------------+--------------------------------+-------------------------------------------------------------+ + | argument name | value | type | + +====================+================================+=============================================================+ + | *name* | The name of the project | string | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *version* | The version number of the | See :mod:`packaging.version` | + | | release | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *summary* | A single line describing the | a string | + | | project | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *description* | Longer description of the | a string | + | | project | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *author* | The name of the project author | a string | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *author_email* | The email address of the | a string | + | | project author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *maintainer* | The name of the current | a string | + | | maintainer, if different from | | + | | the author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *maintainer_email* | The email address of the | | + | | current maintainer, if | | + | | different from the author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *home_page* | A URL for the proejct | a URL | + | | (homepage) | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *download_url* | A URL to download the project | a URL | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *packages* | A list of Python packages that | a list of strings | + | | packaging will manipulate | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *py_modules* | A list of Python modules that | a list of strings | + | | packaging will manipulate | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *scripts* | A list of standalone scripts | a list of strings | + | | to be built and installed | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *ext_modules* | A list of Python extensions to | A list of instances of | + | | be built | :class:`packaging.compiler.extension.Extension` | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *classifiers* | A list of categories for the | The list of available | + | | distribution | categorizations is at | + | | | http://pypi.python.org/pypi?:action=list_classifiers. | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *distclass* | the :class:`Distribution` | A subclass of | + | | class to use | :class:`packaging.dist.Distribution` | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *script_name* | The name of the setup.py | a string | + | | script - defaults to | | + | | ``sys.argv[0]`` | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *script_args* | Arguments to supply to the | a list of strings | + | | setup script | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *options* | default options for the setup | a string | + | | script | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *license* | The license for the | a string; should be used when there is no suitable License | + | | distribution | classifier, or to specify a classifier | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *keywords* | Descriptive keywords | a list of strings; used by catalogs | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *platforms* | Platforms compatible with this | a list of strings; should be used when there is no | + | | distribution | suitable Platform classifier | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *cmdclass* | A mapping of command names to | a dictionary | + | | :class:`Command` subclasses | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *data_files* | A list of data files to | a list | + | | install | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *package_dir* | A mapping of Python packages | a dictionary | + | | to directory names | | + +--------------------+--------------------------------+-------------------------------------------------------------+ diff --git a/Doc/library/packaging.fancy_getopt.rst b/Doc/library/packaging.fancy_getopt.rst new file mode 100644 index 0000000..2c69341 --- /dev/null +++ b/Doc/library/packaging.fancy_getopt.rst @@ -0,0 +1,75 @@ +:mod:`packaging.fancy_getopt` --- Wrapper around the getopt module +================================================================== + +.. module:: packaging.fancy_getopt + :synopsis: Additional getopt functionality. + + +.. warning:: + This module is deprecated and will be replaced with :mod:`optparse`. + +This module provides a wrapper around the standard :mod:`getopt` module that +provides the following additional features: + +* short and long options are tied together + +* options have help strings, so :func:`fancy_getopt` could potentially create a + complete usage summary + +* options set attributes of a passed-in object + +* boolean options can have "negative aliases" --- e.g. if :option:`--quiet` is + the "negative alias" of :option:`--verbose`, then :option:`--quiet` on the + command line sets *verbose* to false. + +.. function:: fancy_getopt(options, negative_opt, object, args) + + Wrapper function. *options* is a list of ``(long_option, short_option, + help_string)`` 3-tuples as described in the constructor for + :class:`FancyGetopt`. *negative_opt* should be a dictionary mapping option names + to option names, both the key and value should be in the *options* list. + *object* is an object which will be used to store values (see the :meth:`getopt` + method of the :class:`FancyGetopt` class). *args* is the argument list. Will use + ``sys.argv[1:]`` if you pass ``None`` as *args*. + + +.. class:: FancyGetopt([option_table=None]) + + The option_table is a list of 3-tuples: ``(long_option, short_option, + help_string)`` + + If an option takes an argument, its *long_option* should have ``'='`` appended; + *short_option* should just be a single character, no ``':'`` in any case. + *short_option* should be ``None`` if a *long_option* doesn't have a + corresponding *short_option*. All option tuples must have long options. + +The :class:`FancyGetopt` class provides the following methods: + + +.. method:: FancyGetopt.getopt([args=None, object=None]) + + Parse command-line options in args. Store as attributes on *object*. + + If *args* is ``None`` or not supplied, uses ``sys.argv[1:]``. If *object* is + ``None`` or not supplied, creates a new :class:`OptionDummy` instance, stores + option values there, and returns a tuple ``(args, object)``. If *object* is + supplied, it is modified in place and :func:`getopt` just returns *args*; in + both cases, the returned *args* is a modified copy of the passed-in *args* list, + which is left untouched. + + .. TODO and args returned are? + + +.. method:: FancyGetopt.get_option_order() + + Returns the list of ``(option, value)`` tuples processed by the previous run of + :meth:`getopt` Raises :exc:`RuntimeError` if :meth:`getopt` hasn't been called + yet. + + +.. method:: FancyGetopt.generate_help([header=None]) + + Generate help text (a list of strings, one per suggested line of output) from + the option table for this :class:`FancyGetopt` object. + + If supplied, prints the supplied *header* at the top of the help. diff --git a/Doc/library/packaging.install.rst b/Doc/library/packaging.install.rst new file mode 100644 index 0000000..b619a98 --- /dev/null +++ b/Doc/library/packaging.install.rst @@ -0,0 +1,112 @@ +:mod:`packaging.install` --- Installation tools +=============================================== + +.. module:: packaging.install + :synopsis: Download and installation building blocks + + +Packaging provides a set of tools to deal with downloads and installation of +distributions. Their role is to download the distribution from indexes, resolve +the dependencies, and provide a safe way to install distributions. An operation +that fails will cleanly roll back, not leave half-installed distributions on the +system. Here's the basic process followed: + +#. Move all distributions that will be removed to a temporary location. + +#. Install all the distributions that will be installed in a temporary location. + +#. If the installation fails, move the saved distributions back to their + location and delete the installed distributions. + +#. Otherwise, move the installed distributions to the right location and delete + the temporary locations. + +This is a higher-level module built on :mod:`packaging.database` and +:mod:`packaging.pypi`. + + +Public functions +---------------- + +.. function:: get_infos(requirements, index=None, installed=None, \ + prefer_final=True) + + Return information about what's going to be installed and upgraded. + *requirements* is a string string containing the requirements for this + project, for example ``'FooBar 1.1'`` or ``'BarBaz (<1.2)'``. + + .. XXX are requirements comma-separated? + + If you want to use another index than the main PyPI, give its URI as *index* + argument. + + *installed* is a list of already installed distributions used to find + satisfied dependencies, obsoleted distributions and eventual conflicts. + + By default, alpha, beta and candidate versions are not picked up. Set + *prefer_final* to false to accept them too. + + The results are returned in a dictionary containing all the information + needed to perform installation of the requirements with the + :func:`install_from_infos` function: + + >>> get_install_info("FooBar (<=1.2)") + {'install': [<FooBar 1.1>], 'remove': [], 'conflict': []} + + .. TODO should return tuple or named tuple, not dict + .. TODO use "predicate" or "requirement" consistently in version and here + .. FIXME "info" cannot be plural in English, s/infos/info/ + + +.. function:: install(project) + + +.. function:: install_dists(dists, path, paths=None) + + Safely install all distributions provided in *dists* into *path*. *paths* is + a list of paths where already-installed distributions will be looked for to + find satisfied dependencies and conflicts (default: :data:`sys.path`). + Returns a list of installed dists. + + .. FIXME dists are instances of what? + + +.. function:: install_from_infos(install_path=None, install=[], remove=[], \ + conflicts=[], paths=None) + + Safely install and remove given distributions. This function is designed to + work with the return value of :func:`get_infos`: *install*, *remove* and + *conflicts* should be list of distributions returned by :func:`get_infos`. + If *install* is not empty, *install_path* must be given to specify the path + where the distributions should be installed. *paths* is a list of paths + where already-installed distributions will be looked for (default: + :data:`sys.path`). + + This function is a very basic installer; if *conflicts* is not empty, the + system will be in a conflicting state after the function completes. It is a + building block for more sophisticated installers with conflict resolution + systems. + + .. TODO document typical value for install_path + .. TODO document integration with default schemes, esp. user site-packages + + +.. function:: install_local_project(path) + + Install a distribution from a source directory, which must contain either a + Packaging-compliant :file:`setup.cfg` file or a legacy Distutils + :file:`setup.py` script (in which case Distutils will be used under the hood + to perform the installation). + + +.. function:: remove(project_name, paths=None, auto_confirm=True) + + Remove one distribution from the system. + + .. FIXME this is the only function using "project" instead of dist/release + +.. + Example usage + -------------- + + Get the scheme of what's gonna be installed if we install "foobar": diff --git a/Doc/library/packaging.metadata.rst b/Doc/library/packaging.metadata.rst new file mode 100644 index 0000000..332d69d --- /dev/null +++ b/Doc/library/packaging.metadata.rst @@ -0,0 +1,122 @@ +:mod:`packaging.metadata` --- Metadata handling +=============================================== + +.. module:: packaging.metadata + :synopsis: Class holding the metadata of a release. + + +.. TODO use sphinx-autogen to generate basic doc from the docstrings + +.. class:: Metadata + + This class can read and write metadata files complying with any of the + defined versions: 1.0 (:PEP:`241`), 1.1 (:PEP:`314`) and 1.2 (:PEP:`345`). It + implements methods to parse Metadata files and write them, and a mapping + interface to its contents. + + The :PEP:`345` implementation supports the micro-language for the environment + markers, and displays warnings when versions that are supposed to be + :PEP:`386`-compliant are violating the specification. + + +Reading metadata +---------------- + +The :class:`Metadata` class can be instantiated +with the path of the metadata file, and provides a dict-like interface to the +values:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata('PKG-INFO') + >>> metadata.keys()[:5] + ('Metadata-Version', 'Name', 'Version', 'Platform', 'Supported-Platform') + >>> metadata['Name'] + 'CLVault' + >>> metadata['Version'] + '0.5' + >>> metadata['Requires-Dist'] + ["pywin32; sys.platform == 'win32'", "Sphinx"] + + +The fields that support environment markers can be automatically ignored if +the object is instantiated using the ``platform_dependent`` option. +:class:`Metadata` will interpret in this case +the markers and will automatically remove the fields that are not compliant +with the running environment. Here's an example under Mac OS X. The win32 +dependency we saw earlier is ignored:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata('PKG-INFO', platform_dependent=True) + >>> metadata['Requires-Dist'] + ['Sphinx'] + + +If you want to provide your own execution context, let's say to test the +metadata under a particular environment that is not the current environment, +you can provide your own values in the ``execution_context`` option, which +is the dict that may contain one or more keys of the context the micro-language +expects. + +Here's an example, simulating a win32 environment:: + + >>> from packaging.metadata import Metadata + >>> context = {'sys.platform': 'win32'} + >>> metadata = Metadata('PKG-INFO', platform_dependent=True, + ... execution_context=context) + ... + >>> metadata['Requires-Dist'] = ["pywin32; sys.platform == 'win32'", + ... "Sphinx"] + ... + >>> metadata['Requires-Dist'] + ['pywin32', 'Sphinx'] + + +Writing metadata +---------------- + +Writing metadata can be done using the ``write`` method:: + + >>> metadata.write('/to/my/PKG-INFO') + +The class will pick the best version for the metadata, depending on the values +provided. If all the values provided exist in all versions, the class will +use :attr:`PKG_INFO_PREFERRED_VERSION`. It is set by default to 1.0, the most +widespread version. + + +Conflict checking and best version +---------------------------------- + +Some fields in :PEP:`345` have to comply with the version number specification +defined in :PEP:`386`. When they don't comply, a warning is emitted:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata() + >>> metadata['Requires-Dist'] = ['Funky (Groovie)'] + "Funky (Groovie)" is not a valid predicate + >>> metadata['Requires-Dist'] = ['Funky (1.2)'] + +See also :mod:`packaging.version`. + + +.. TODO talk about check() + + +:mod:`packaging.markers` --- Environment markers +================================================ + +.. module:: packaging.markers + :synopsis: Micro-language for environment markers + + +This is an implementation of environment markers `as defined in PEP 345 +<http://www.python.org/dev/peps/pep-0345/#environment-markers>`_. It is used +for some metadata fields. + +.. function:: interpret(marker, execution_context=None) + + Interpret a marker and return a boolean result depending on the environment. + Example: + + >>> interpret("python_version > '1.0'") + True diff --git a/Doc/library/packaging.pypi.dist.rst b/Doc/library/packaging.pypi.dist.rst new file mode 100644 index 0000000..aaaaab7 --- /dev/null +++ b/Doc/library/packaging.pypi.dist.rst @@ -0,0 +1,114 @@ +:mod:`packaging.pypi.dist` --- Classes representing query results +================================================================= + +.. module:: packaging.pypi.dist + :synopsis: Classes representing the results of queries to indexes. + + +Information coming from the indexes is held in instances of the classes defined +in this module. + +Keep in mind that each project (eg. FooBar) can have several releases +(eg. 1.1, 1.2, 1.3), and each of these releases can be provided in multiple +distributions (eg. a source distribution, a binary one, etc). + + +ReleaseInfo +----------- + +Each release has a project name, version, metadata, and related distributions. + +This information is stored in :class:`ReleaseInfo` +objects. + +.. class:: ReleaseInfo + + +DistInfo +--------- + +:class:`DistInfo` is a simple class that contains +information related to distributions; mainly the URLs where distributions +can be found. + +.. class:: DistInfo + + +ReleasesList +------------ + +The :mod:`~packaging.pypi.dist` module provides a class which works +with lists of :class:`ReleaseInfo` classes; +used to filter and order results. + +.. class:: ReleasesList + + +Example usage +------------- + +Build a list of releases and order them +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Assuming we have a list of releases:: + + >>> from packaging.pypi.dist import ReleasesList, ReleaseInfo + >>> fb10 = ReleaseInfo("FooBar", "1.0") + >>> fb11 = ReleaseInfo("FooBar", "1.1") + >>> fb11a = ReleaseInfo("FooBar", "1.1a1") + >>> ReleasesList("FooBar", [fb11, fb11a, fb10]) + >>> releases.sort_releases() + >>> releases.get_versions() + ['1.1', '1.1a1', '1.0'] + >>> releases.add_release("1.2a1") + >>> releases.get_versions() + ['1.1', '1.1a1', '1.0', '1.2a1'] + >>> releases.sort_releases() + ['1.2a1', '1.1', '1.1a1', '1.0'] + >>> releases.sort_releases(prefer_final=True) + >>> releases.get_versions() + ['1.1', '1.0', '1.2a1', '1.1a1'] + + +Add distribution related information to releases +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's easy to add distribution information to releases:: + + >>> from packaging.pypi.dist import ReleasesList, ReleaseInfo + >>> r = ReleaseInfo("FooBar", "1.0") + >>> r.add_distribution("sdist", url="http://example.org/foobar-1.0.tar.gz") + >>> r.dists + {'sdist': FooBar 1.0 sdist} + >>> r['sdist'].url + {'url': 'http://example.org/foobar-1.0.tar.gz', 'hashname': None, 'hashval': + None, 'is_external': True} + + +Getting attributes from the dist objects +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To abstract querying information returned from the indexes, attributes and +release information can be retrieved directly from dist objects. + +For instance, if you have a release instance that does not contain the metadata +attribute, it can be fetched by using the "fetch_metadata" method:: + + >>> r = Release("FooBar", "1.1") + >>> print r.metadata + None # metadata field is actually set to "None" + >>> r.fetch_metadata() + <Metadata for FooBar 1.1> + +.. XXX add proper roles to these constructs + + +It's possible to retrieve a project's releases (`fetch_releases`), +metadata (`fetch_metadata`) and distributions (`fetch_distributions`) using +a similar work flow. + +.. XXX what is possible? + +Internally, this is possible because while retrieving information about +projects, releases or distributions, a reference to the client used is +stored which can be accessed using the objects `_index` attribute. diff --git a/Doc/library/packaging.pypi.rst b/Doc/library/packaging.pypi.rst new file mode 100644 index 0000000..93b61c9 --- /dev/null +++ b/Doc/library/packaging.pypi.rst @@ -0,0 +1,53 @@ +:mod:`packaging.pypi` --- Interface to projects indexes +======================================================= + +.. module:: packaging.pypi + :synopsis: Low-level and high-level APIs to query projects indexes. + + +Packaging queries PyPI to get information about projects or download them. The +low-level facilities used internally are also part of the public API designed to +be used by other tools. + +The :mod:`packaging.pypi` package provides those facilities, which can be +used to access information about Python projects registered at indexes, the +main one being PyPI, located ad http://pypi.python.org/. + +There is two ways to retrieve data from these indexes: a screen-scraping +interface called the "simple API", and XML-RPC. The first one uses HTML pages +located under http://pypi.python.org/simple/, the second one makes XML-RPC +requests to http://pypi.python.org/pypi/. All functions and classes also work +with other indexes such as mirrors, which typically implement only the simple +interface. + +Packaging provides a class that wraps both APIs to provide full query and +download functionality: :class:`packaging.pypi.client.ClientWrapper`. If you +want more control, you can use the underlying classes +:class:`packaging.pypi.simple.Crawler` and :class:`packaging.pypi.xmlrpc.Client` +to connect to one specific interface. + + +:mod:`packaging.pypi.client` --- High-level query API +===================================================== + +.. module:: packaging.pypi.client + :synopsis: Wrapper around :mod;`packaging.pypi.xmlrpc` and + :mod:`packaging.pypi.simple` to query indexes. + + +This module provides a high-level API to query indexes and search +for releases and distributions. The aim of this module is to choose the best +way to query the API automatically, either using XML-RPC or the simple index, +with a preference toward the latter. + +.. class:: ClientWrapper + + Instances of this class will use the simple interface or XML-RPC requests to + query indexes and return :class:`packaging.pypi.dist.ReleaseInfo` and + :class:`packaging.pypi.dist.ReleasesList` objects. + + .. method:: find_projects + + .. method:: get_release + + .. method:: get_releases diff --git a/Doc/library/packaging.pypi.simple.rst b/Doc/library/packaging.pypi.simple.rst new file mode 100644 index 0000000..ea5edca --- /dev/null +++ b/Doc/library/packaging.pypi.simple.rst @@ -0,0 +1,157 @@ +:mod:`packaging.pypi.simple` --- Crawler using the PyPI "simple" interface +========================================================================== + +.. module:: packaging.pypi.simple + :synopsis: Crawler using the screen-scraping "simple" interface to fetch info + and distributions. + + +`packaging.pypi.simple` can process Python Package Indexes and provides +useful information about distributions. It also can crawl local indexes, for +instance. + +You should use `packaging.pypi.simple` for: + + * Search distributions by name and versions. + * Process index external pages. + * Download distributions by name and versions. + +And should not be used for: + + * Things that will end up in too long index processing (like "finding all + distributions with a specific version, no matters the name") + + +API +--- + +.. class:: Crawler + + +Usage Exemples +--------------- + +To help you understand how using the `Crawler` class, here are some basic +usages. + +Request the simple index to get a specific distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Supposing you want to scan an index to get a list of distributions for +the "foobar" project. You can use the "get_releases" method for that. +The get_releases method will browse the project page, and return +:class:`ReleaseInfo` objects for each found link that rely on downloads. :: + + >>> from packaging.pypi.simple import Crawler + >>> crawler = Crawler() + >>> crawler.get_releases("FooBar") + [<ReleaseInfo "Foobar 1.1">, <ReleaseInfo "Foobar 1.2">] + + +Note that you also can request the client about specific versions, using version +specifiers (described in `PEP 345 +<http://www.python.org/dev/peps/pep-0345/#version-specifiers>`_):: + + >>> client.get_releases("FooBar < 1.2") + [<ReleaseInfo "FooBar 1.1">, ] + + +`get_releases` returns a list of :class:`ReleaseInfo`, but you also can get the +best distribution that fullfil your requirements, using "get_release":: + + >>> client.get_release("FooBar < 1.2") + <ReleaseInfo "FooBar 1.1"> + + +Download distributions +^^^^^^^^^^^^^^^^^^^^^^ + +As it can get the urls of distributions provided by PyPI, the `Crawler` +client also can download the distributions and put it for you in a temporary +destination:: + + >>> client.download("foobar") + /tmp/temp_dir/foobar-1.2.tar.gz + + +You also can specify the directory you want to download to:: + + >>> client.download("foobar", "/path/to/my/dir") + /path/to/my/dir/foobar-1.2.tar.gz + + +While downloading, the md5 of the archive will be checked, if not matches, it +will try another time, then if fails again, raise `MD5HashDoesNotMatchError`. + +Internally, that's not the Crawler which download the distributions, but the +`DistributionInfo` class. Please refer to this documentation for more details. + + +Following PyPI external links +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default behavior for packaging is to *not* follow the links provided +by HTML pages in the "simple index", to find distributions related +downloads. + +It's possible to tell the PyPIClient to follow external links by setting the +`follow_externals` attribute, on instantiation or after:: + + >>> client = Crawler(follow_externals=True) + +or :: + + >>> client = Crawler() + >>> client.follow_externals = True + + +Working with external indexes, and mirrors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default `Crawler` behavior is to rely on the Python Package index stored +on PyPI (http://pypi.python.org/simple). + +As you can need to work with a local index, or private indexes, you can specify +it using the index_url parameter:: + + >>> client = Crawler(index_url="file://filesystem/path/") + +or :: + + >>> client = Crawler(index_url="http://some.specific.url/") + + +You also can specify mirrors to fallback on in case the first index_url you +provided doesnt respond, or not correctly. The default behavior for +`Crawler` is to use the list provided by Python.org DNS records, as +described in the :PEP:`381` about mirroring infrastructure. + +If you don't want to rely on these, you could specify the list of mirrors you +want to try by specifying the `mirrors` attribute. It's a simple iterable:: + + >>> mirrors = ["http://first.mirror","http://second.mirror"] + >>> client = Crawler(mirrors=mirrors) + + +Searching in the simple index +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's possible to search for projects with specific names in the package index. +Assuming you want to find all projects containing the "distutils" keyword:: + + >>> c.search_projects("distutils") + [<Project "collective.recipe.distutils">, <Project "Distutils">, <Project + "Packaging">, <Project "distutilscross">, <Project "lpdistutils">, <Project + "taras.recipe.distutils">, <Project "zerokspot.recipe.distutils">] + + +You can also search the projects starting with a specific text, or ending with +that text, using a wildcard:: + + >>> c.search_projects("distutils*") + [<Project "Distutils">, <Project "Packaging">, <Project "distutilscross">] + + >>> c.search_projects("*distutils") + [<Project "collective.recipe.distutils">, <Project "Distutils">, <Project + "lpdistutils">, <Project "taras.recipe.distutils">, <Project + "zerokspot.recipe.distutils">] diff --git a/Doc/library/packaging.pypi.xmlrpc.rst b/Doc/library/packaging.pypi.xmlrpc.rst new file mode 100644 index 0000000..0253d68 --- /dev/null +++ b/Doc/library/packaging.pypi.xmlrpc.rst @@ -0,0 +1,143 @@ +:mod:`packaging.pypi.xmlrpc` --- Crawler using the PyPI XML-RPC interface +========================================================================= + +.. module:: packaging.pypi.xmlrpc + :synopsis: Client using XML-RPC requests to fetch info and distributions. + + +Indexes can be queried using XML-RPC calls, and Packaging provides a simple +way to interface with XML-RPC. + +You should **use** XML-RPC when: + +* Searching the index for projects **on other fields than project + names**. For instance, you can search for projects based on the + author_email field. +* Searching all the versions that have existed for a project. +* you want to retrieve METADATAs information from releases or + distributions. + + +You should **avoid using** XML-RPC method calls when: + +* Retrieving the last version of a project +* Getting the projects with a specific name and version. +* The simple index can match your needs + + +When dealing with indexes, keep in mind that the index queries will always +return you :class:`packaging.pypi.dist.ReleaseInfo` and +:class:`packaging.pypi.dist.ReleasesList` objects. + +Some methods here share common APIs with the one you can find on +:class:`packaging.pypi.simple`, internally, :class:`packaging.pypi.client` +is inherited by :class:`Client` + + +API +--- + +.. class:: Client + + +Usage examples +-------------- + +Use case described here are use case that are not common to the other clients. +If you want to see all the methods, please refer to API or to usage examples +described in :class:`packaging.pypi.client.Client` + + +Finding releases +^^^^^^^^^^^^^^^^ + +It's a common use case to search for "things" within the index. We can +basically search for projects by their name, which is the most used way for +users (eg. "give me the last version of the FooBar project"). + +This can be accomplished using the following syntax:: + + >>> client = xmlrpc.Client() + >>> client.get_release("Foobar (<= 1.3)) + <FooBar 1.2.1> + >>> client.get_releases("FooBar (<= 1.3)") + [FooBar 1.1, FooBar 1.1.1, FooBar 1.2, FooBar 1.2.1] + + +And we also can find for specific fields:: + + >>> client.search_projects(field=value) + + +You could specify the operator to use, default is "or":: + + >>> client.search_projects(field=value, operator="and") + + +The specific fields you can search are: + +* name +* version +* author +* author_email +* maintainer +* maintainer_email +* home_page +* license +* summary +* description +* keywords +* platform +* download_url + + +Getting metadata information +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +XML-RPC is a prefered way to retrieve metadata information from indexes. +It's really simple to do so:: + + >>> client = xmlrpc.Client() + >>> client.get_metadata("FooBar", "1.1") + <ReleaseInfo FooBar 1.1> + + +Assuming we already have a :class:`packaging.pypi.ReleaseInfo` object defined, +it's possible to pass it to the xmlrpc client to retrieve and complete its +metadata:: + + >>> foobar11 = ReleaseInfo("FooBar", "1.1") + >>> client = xmlrpc.Client() + >>> returned_release = client.get_metadata(release=foobar11) + >>> returned_release + <ReleaseInfo FooBar 1.1> + + +Get all the releases of a project +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To retrieve all the releases for a project, you can build them using +`get_releases`:: + + >>> client = xmlrpc.Client() + >>> client.get_releases("FooBar") + [<ReleaseInfo FooBar 0.9>, <ReleaseInfo FooBar 1.0>, <ReleaseInfo 1.1>] + + +Get information about distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Indexes have information about projects, releases **and** distributions. +If you're not familiar with those, please refer to the documentation of +:mod:`packaging.pypi.dist`. + +It's possible to retrieve information about distributions, e.g "what are the +existing distributions for this release ? How to retrieve them ?":: + + >>> client = xmlrpc.Client() + >>> release = client.get_distributions("FooBar", "1.1") + >>> release.dists + {'sdist': <FooBar 1.1 sdist>, 'bdist': <FooBar 1.1 bdist>} + +As you see, this does not return a list of distributions, but a release, +because a release can be used like a list of distributions. diff --git a/Doc/library/packaging.rst b/Doc/library/packaging.rst new file mode 100644 index 0000000..5bf2c06 --- /dev/null +++ b/Doc/library/packaging.rst @@ -0,0 +1,78 @@ +:mod:`packaging` --- Packaging support +====================================== + +.. module:: packaging + :synopsis: Packaging system and building blocks for other packaging systems. +.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>, distutils and packaging + contributors + + +The :mod:`packaging` package provides support for building, packaging, +distributing and installing additional projects into a Python installation. +Projects may include Python modules, extension modules, packages and scripts. +:mod:`packaging` also provides building blocks for other packaging systems +that are not tied to the command system. + +This manual is the reference documentation for those standalone building +blocks and for extending Packaging. If you're looking for the user-centric +guides to install a project or package your own code, head to `See also`__. + + +Building blocks +--------------- + +.. toctree:: + :maxdepth: 2 + :numbered: + + packaging-misc + packaging.version + packaging.metadata + packaging.database + packaging.depgraph + packaging.pypi + packaging.pypi.dist + packaging.pypi.simple + packaging.pypi.xmlrpc + packaging.install + + +The command machinery +--------------------- + +.. toctree:: + :maxdepth: 2 + :numbered: + + packaging.dist + packaging.command + packaging.compiler + packaging.fancy_getopt + + +Other utilities +---------------- + +.. toctree:: + :maxdepth: 2 + :numbered: + + packaging.util + packaging.tests.pypi_server + +.. XXX missing: compat config create (dir_util) run pypi.{base,mirrors} + + +.. __: + +.. seealso:: + + :ref:`packaging-index` + The manual for developers of Python projects who want to package and + distribute them. This describes how to use :mod:`packaging` to make + projects easily found and added to an existing Python installation. + + :ref:`packaging-install-index` + A user-centered manual which includes information on adding projects + into an existing Python installation. You do not need to be a Python + programmer to read this manual. diff --git a/Doc/library/packaging.tests.pypi_server.rst b/Doc/library/packaging.tests.pypi_server.rst new file mode 100644 index 0000000..f3b7720 --- /dev/null +++ b/Doc/library/packaging.tests.pypi_server.rst @@ -0,0 +1,105 @@ +:mod:`packaging.tests.pypi_server` --- PyPI mock server +======================================================= + +.. module:: packaging.tests.pypi_server + :synopsis: Mock server used to test PyPI-related modules and commands. + + +When you are testing code that works with Packaging, you might find these tools +useful. + + +The mock server +--------------- + +.. class:: PyPIServer + + PyPIServer is a class that implements an HTTP server running in a separate + thread. All it does is record the requests for further inspection. The recorded + data is available under ``requests`` attribute. The default + HTTP response can be overridden with the ``default_response_status``, + ``default_response_headers`` and ``default_response_data`` attributes. + + By default, when accessing the server with urls beginning with `/simple/`, + the server also record your requests, but will look for files under + the `/tests/pypiserver/simple/` path. + + You can tell the sever to serve static files for other paths. This could be + accomplished by using the `static_uri_paths` parameter, as below:: + + server = PyPIServer(static_uri_paths=["first_path", "second_path"]) + + + You need to create the content that will be served under the + `/tests/pypiserver/default` path. If you want to serve content from another + place, you also can specify another filesystem path (which needs to be under + `tests/pypiserver/`. This will replace the default behavior of the server, and + it will not serve content from the `default` dir :: + + server = PyPIServer(static_filesystem_paths=["path/to/your/dir"]) + + + If you just need to add some paths to the existing ones, you can do as shown, + keeping in mind that the server will always try to load paths in reverse order + (e.g here, try "another/super/path" then the default one) :: + + server = PyPIServer(test_static_path="another/super/path") + server = PyPIServer("another/super/path") + # or + server.static_filesystem_paths.append("another/super/path") + + + As a result of what, in your tests, while you need to use the PyPIServer, in + order to isolates the test cases, the best practice is to place the common files + in the `default` folder, and to create a directory for each specific test case:: + + server = PyPIServer(static_filesystem_paths = ["default", "test_pypi_server"], + static_uri_paths=["simple", "external"]) + + +Base class and decorator for tests +---------------------------------- + +.. class:: PyPIServerTestCase + + ``PyPIServerTestCase`` is a test case class with setUp and tearDown methods that + take care of a single PyPIServer instance attached as a ``pypi`` attribute on + the test class. Use it as one of the base classes in your test case:: + + + class UploadTestCase(PyPIServerTestCase): + + def test_something(self): + cmd = self.prepare_command() + cmd.ensure_finalized() + cmd.repository = self.pypi.full_address + cmd.run() + + environ, request_data = self.pypi.requests[-1] + self.assertEqual(request_data, EXPECTED_REQUEST_DATA) + + +.. decorator:: use_pypi_server + + You also can use a decorator for your tests, if you do not need the same server + instance along all you test case. So, you can specify, for each test method, + some initialisation parameters for the server. + + For this, you need to add a `server` parameter to your method, like this:: + + class SampleTestCase(TestCase): + + @use_pypi_server() + def test_something(self, server): + ... + + + The decorator will instantiate the server for you, and run and stop it just + before and after your method call. You also can pass the server initializer, + just like this:: + + class SampleTestCase(TestCase): + + @use_pypi_server("test_case_name") + def test_something(self, server): + ... diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst new file mode 100644 index 0000000..b95d5b5 --- /dev/null +++ b/Doc/library/packaging.util.rst @@ -0,0 +1,186 @@ +:mod:`packaging.util` --- Miscellaneous utility functions +========================================================= + +.. module:: packaging.util + :synopsis: Miscellaneous utility functions. + + +This module contains various helpers for the other modules. + +.. XXX a number of functions are missing, but the module may be split first + (it's ginormous right now, some things could go to compat for example) + +.. function:: get_platform() + + Return a string that identifies the current platform. This is used mainly to + distinguish platform-specific build directories and platform-specific built + distributions. Typically includes the OS name and version and the + architecture (as supplied by 'os.uname()'), although the exact information + included depends on the OS; e.g. for IRIX the architecture isn't particularly + important (IRIX only runs on SGI hardware), but for Linux the kernel version + isn't particularly important. + + Examples of returned values: + + * ``linux-i586`` + * ``linux-alpha`` + * ``solaris-2.6-sun4u`` + * ``irix-5.3`` + * ``irix64-6.2`` + + For non-POSIX platforms, currently just returns ``sys.platform``. + + For Mac OS X systems the OS version reflects the minimal version on which + binaries will run (that is, the value of ``MACOSX_DEPLOYMENT_TARGET`` + during the build of Python), not the OS version of the current system. + + For universal binary builds on Mac OS X the architecture value reflects + the univeral binary status instead of the architecture of the current + processor. For 32-bit universal binaries the architecture is ``fat``, + for 64-bit universal binaries the architecture is ``fat64``, and + for 4-way universal binaries the architecture is ``universal``. Starting + from Python 2.7 and Python 3.2 the architecture ``fat3`` is used for + a 3-way universal build (ppc, i386, x86_64) and ``intel`` is used for + a univeral build with the i386 and x86_64 architectures + + Examples of returned values on Mac OS X: + + * ``macosx-10.3-ppc`` + + * ``macosx-10.3-fat`` + + * ``macosx-10.5-universal`` + + * ``macosx-10.6-intel`` + + .. XXX reinvention of platform module? + + +.. function:: convert_path(pathname) + + Return 'pathname' as a name that will work on the native filesystem, i.e. + split it on '/' and put it back together again using the current directory + separator. Needed because filenames in the setup script are always supplied + in Unix style, and have to be converted to the local convention before we + can actually use them in the filesystem. Raises :exc:`ValueError` on + non-Unix-ish systems if *pathname* either starts or ends with a slash. + + +.. function:: change_root(new_root, pathname) + + Return *pathname* with *new_root* prepended. If *pathname* is relative, this + is equivalent to ``os.path.join(new_root,pathname)`` Otherwise, it requires + making *pathname* relative and then joining the two, which is tricky on + DOS/Windows. + + +.. function:: check_environ() + + Ensure that 'os.environ' has all the environment variables we guarantee that + users can use in config files, command-line options, etc. Currently this + includes: + + * :envvar:`HOME` - user's home directory (Unix only) + * :envvar:`PLAT` - description of the current platform, including hardware + and OS (see :func:`get_platform`) + + +.. function:: find_executable(executable, path=None) + + Search the path for a given executable name. + + +.. function:: subst_vars(s, local_vars) + + Perform shell/Perl-style variable substitution on *s*. Every occurrence of + ``$`` followed by a name is considered a variable, and variable is + substituted by the value found in the *local_vars* dictionary, or in + ``os.environ`` if it's not in *local_vars*. *os.environ* is first + checked/augmented to guarantee that it contains certain values: see + :func:`check_environ`. Raise :exc:`ValueError` for any variables not found + in either *local_vars* or ``os.environ``. + + Note that this is not a fully-fledged string interpolation function. A valid + ``$variable`` can consist only of upper and lower case letters, numbers and + an underscore. No { } or ( ) style quoting is available. + + +.. function:: split_quoted(s) + + Split a string up according to Unix shell-like rules for quotes and + backslashes. In short: words are delimited by spaces, as long as those spaces + are not escaped by a backslash, or inside a quoted string. Single and double + quotes are equivalent, and the quote characters can be backslash-escaped. + The backslash is stripped from any two-character escape sequence, leaving + only the escaped character. The quote characters are stripped from any + quoted string. Returns a list of words. + + .. TODO Should probably be moved into the standard library. + + +.. function:: execute(func, args[, msg=None, verbose=0, dry_run=0]) + + Perform some action that affects the outside world (for instance, writing to + the filesystem). Such actions are special because they are disabled by the + *dry_run* flag. This method takes care of all that bureaucracy for you; + all you have to do is supply the function to call and an argument tuple for + it (to embody the "external action" being performed), and an optional message + to print. + + +.. function:: newer(source, target) + + Return true if *source* exists and is more recently modified than *target*, + or if *source* exists and *target* doesn't. Return false if both exist and + *target* is the same age or newer than *source*. Raise + :exc:`PackagingFileError` if *source* does not exist. + + +.. function:: strtobool(val) + + Convert a string representation of truth to true (1) or false (0). + + True values are ``y``, ``yes``, ``t``, ``true``, ``on`` and ``1``; false + values are ``n``, ``no``, ``f``, ``false``, ``off`` and ``0``. Raises + :exc:`ValueError` if *val* is anything else. + +.. TODO Add :term: markup to bytecode when merging into the stdlib + +.. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) + + Byte-compile a collection of Python source files to either :file:`.pyc` or + :file:`.pyo` files in the same directory. *py_files* is a list of files to + compile; any files that don't end in :file:`.py` are silently skipped. + *optimize* must be one of the following: + + * ``0`` - don't optimize (generate :file:`.pyc`) + * ``1`` - normal optimization (like ``python -O``) + * ``2`` - extra optimization (like ``python -OO``) + + If *force* is true, all files are recompiled regardless of timestamps. + + The source filename encoded in each bytecode file defaults to the filenames + listed in *py_files*; you can modify these with *prefix* and *basedir*. + *prefix* is a string that will be stripped off of each source filename, and + *base_dir* is a directory name that will be prepended (after *prefix* is + stripped). You can supply either or both (or neither) of *prefix* and + *base_dir*, as you wish. + + If *dry_run* is true, doesn't actually do anything that would affect the + filesystem. + + Byte-compilation is either done directly in this interpreter process with the + standard :mod:`py_compile` module, or indirectly by writing a temporary + script and executing it. Normally, you should let :func:`byte_compile` + figure out to use direct compilation or not (see the source for details). + The *direct* flag is used by the script generated in indirect mode; unless + you know what you're doing, leave it set to ``None``. + + +.. function:: rfc822_escape(header) + + Return a version of *header* escaped for inclusion in an :rfc:`822` header, by + ensuring there are 8 spaces space after each newline. Note that it does no + other modification of the string. + + .. TODO this _can_ be replaced diff --git a/Doc/library/packaging.version.rst b/Doc/library/packaging.version.rst new file mode 100644 index 0000000..f36cdab --- /dev/null +++ b/Doc/library/packaging.version.rst @@ -0,0 +1,104 @@ +:mod:`packaging.version` --- Version number classes +=================================================== + +.. module:: packaging.version + :synopsis: Classes that represent project version numbers. + + +This module contains classes and functions useful to deal with version numbers. +It's an implementation of version specifiers `as defined in PEP 345 +<http://www.python.org/dev/peps/pep-0345/#version-specifiers>`_. + + +Version numbers +--------------- + +.. class:: NormalizedVersion(self, s, error_on_huge_major_num=True) + + A specific version of a distribution, as described in PEP 345. *s* is a + string object containing the version number (for example ``'1.2b1'``), + *error_on_huge_major_num* a boolean specifying whether to consider an + apparent use of a year or full date as the major version number an error. + + The rationale for the second argument is that there were projects using years + or full dates as version numbers, which could cause problems with some + packaging systems sorting. + + Instances of this class can be compared and sorted:: + + >>> NormalizedVersion('1.2b1') < NormalizedVersion('1.2') + True + + :class:`NormalizedVersion` is used internally by :class:`VersionPredicate` to + do its work. + + +.. class:: IrrationalVersionError + + Exception raised when an invalid string is given to + :class:`NormalizedVersion`. + + >>> NormalizedVersion("irrational_version_number") + ... + IrrationalVersionError: irrational_version_number + + +.. function:: suggest_normalized_version(s) + + Before standardization in PEP 386, various schemes were in use. Packaging + provides a function to try to convert any string to a valid, normalized + version:: + + >>> suggest_normalized_version('2.1-rc1') + 2.1c1 + + + If :func:`suggest_normalized_version` can't make sense of the given string, + it will return ``None``:: + + >>> print(suggest_normalized_version('not a version')) + None + + +Version predicates +------------------ + +.. class:: VersionPredicate(predicate) + + This class deals with the parsing of field values like + ``ProjectName (>=version)``. + + .. method:: match(version) + + Test if a version number matches the predicate: + + >>> version = VersionPredicate("ProjectName (<1.2, >1.0)") + >>> version.match("1.2.1") + False + >>> version.match("1.1.1") + True + + +Validation helpers +------------------ + +If you want to use :term:`LBYL`-style checks instead of instantiating the +classes and catching :class:`IrrationalVersionError` and :class:`ValueError`, +you can use these functions: + +.. function:: is_valid_version(predicate) + + Check whether the given string is a valid version number. Example of valid + strings: ``'1.2'``, ``'4.2.0.dev4'``, ``'2.5.4.post2'``. + + +.. function:: is_valid_versions(predicate) + + Check whether the given string is a valid value for specifying multiple + versions, such as in the Requires-Python field. Example: ``'2.7, >=3.2'``. + + +.. function:: is_valid_predicate(predicate) + + Check whether the given string is a valid version predicate. Examples: + ``'some.project == 4.5, <= 4.7'``, ``'speciallib (> 1.0, != 1.4.2, < 2.0)'``. diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index fc54208..3a86331 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -193,7 +193,7 @@ Example ------- To demonstrate several uses of the :func:`pprint` function and its parameters, -let's fetch information about a package from PyPI:: +let's fetch information about a project from PyPI:: >>> import json >>> import pprint @@ -201,8 +201,8 @@ let's fetch information about a package from PyPI:: >>> with urlopen('http://pypi.python.org/pypi/configparser/json') as url: ... http_info = url.info() ... raw_data = url.read().decode(http_info.get_content_charset()) - >>> package_data = json.loads(raw_data) - >>> result = {'headers': http_info.items(), 'body': package_data} + >>> project_info = json.loads(raw_data) + >>> result = {'headers': http_info.items(), 'body': project_info} In its basic form, :func:`pprint` shows the whole object:: diff --git a/Doc/library/python.rst b/Doc/library/python.rst index b67fbfc..07eadb4 100644 --- a/Doc/library/python.rst +++ b/Doc/library/python.rst @@ -25,4 +25,5 @@ overview: inspect.rst site.rst fpectl.rst + packaging.rst distutils.rst diff --git a/Doc/library/random.rst b/Doc/library/random.rst index f0c4add..3040411 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -43,6 +43,12 @@ The :mod:`random` module also provides the :class:`SystemRandom` class which uses the system function :func:`os.urandom` to generate random numbers from sources provided by the operating system. +.. warning:: + + The generators of the :mod:`random` module should not be used for security + purposes. Use :func:`ssl.RAND_bytes` if you require a cryptographically + secure pseudorandom number generator. + Bookkeeping functions: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 84f3118..606825c 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1301,24 +1301,27 @@ The text categories are specified with regular expressions. The technique is to combine those into a single master regular expression and to loop over successive matches:: - Token = collections.namedtuple('Token', 'typ value line column') + import collections + import re + + Token = collections.namedtuple('Token', ['typ', 'value', 'line', 'column']) def tokenize(s): - keywords = {'IF', 'THEN', 'FOR', 'NEXT', 'GOSUB', 'RETURN'} - tok_spec = [ - ('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number - ('ASSIGN', r':='), # Assignment operator - ('END', ';'), # Statement terminator - ('ID', r'[A-Za-z]+'), # Identifiers - ('OP', r'[+*\/\-]'), # Arithmetic operators - ('NEWLINE', r'\n'), # Line endings - ('SKIP', r'[ \t]'), # Skip over spaces and tabs + keywords = {'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'} + token_specification = [ + ('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number + ('ASSIGN', r':='), # Assignment operator + ('END', r';'), # Statement terminator + ('ID', r'[A-Za-z]+'), # Identifiers + ('OP', r'[+*\/\-]'), # Arithmetic operators + ('NEWLINE', r'\n'), # Line endings + ('SKIP', r'[ \t]'), # Skip over spaces and tabs ] - tok_re = '|'.join('(?P<%s>%s)' % pair for pair in tok_spec) - gettok = re.compile(tok_re).match + tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification) + get_token = re.compile(tok_regex).match line = 1 pos = line_start = 0 - mo = gettok(s) + mo = get_token(s) while mo is not None: typ = mo.lastgroup if typ == 'NEWLINE': @@ -1330,13 +1333,15 @@ successive matches:: typ = val yield Token(typ, val, line, mo.start()-line_start) pos = mo.end() - mo = gettok(s, pos) + mo = get_token(s, pos) if pos != len(s): raise RuntimeError('Unexpected character %r on line %d' %(s[pos], line)) - statements = '''\ - total := total + price * quantity; - tax := price * 0.05; + statements = ''' + IF quantity THEN + total := total + price * quantity; + tax := price * 0.05; + ENDIF; ''' for token in tokenize(statements): @@ -1344,17 +1349,22 @@ successive matches:: The tokenizer produces the following output:: - Token(typ='ID', value='total', line=1, column=8) - Token(typ='ASSIGN', value=':=', line=1, column=14) - Token(typ='ID', value='total', line=1, column=17) - Token(typ='OP', value='+', line=1, column=23) - Token(typ='ID', value='price', line=1, column=25) - Token(typ='OP', value='*', line=1, column=31) - Token(typ='ID', value='quantity', line=1, column=33) - Token(typ='END', value=';', line=1, column=41) - Token(typ='ID', value='tax', line=2, column=9) - Token(typ='ASSIGN', value=':=', line=2, column=13) - Token(typ='ID', value='price', line=2, column=16) - Token(typ='OP', value='*', line=2, column=22) - Token(typ='NUMBER', value='0.05', line=2, column=24) - Token(typ='END', value=';', line=2, column=28) + Token(typ='IF', value='IF', line=2, column=5) + Token(typ='ID', value='quantity', line=2, column=8) + Token(typ='THEN', value='THEN', line=2, column=17) + Token(typ='ID', value='total', line=3, column=9) + Token(typ='ASSIGN', value=':=', line=3, column=15) + Token(typ='ID', value='total', line=3, column=18) + Token(typ='OP', value='+', line=3, column=24) + Token(typ='ID', value='price', line=3, column=26) + Token(typ='OP', value='*', line=3, column=32) + Token(typ='ID', value='quantity', line=3, column=34) + Token(typ='END', value=';', line=3, column=42) + Token(typ='ID', value='tax', line=4, column=9) + Token(typ='ASSIGN', value=':=', line=4, column=13) + Token(typ='ID', value='price', line=4, column=16) + Token(typ='OP', value='*', line=4, column=22) + Token(typ='NUMBER', value='0.05', line=4, column=24) + Token(typ='END', value=';', line=4, column=28) + Token(typ='ENDIF', value='ENDIF', line=5, column=5) + Token(typ='END', value=';', line=5, column=10) diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 68a5d2c..f2a37cc 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -187,10 +187,9 @@ The :mod:`signal` module defines the following functions: Send the signal *signum* to the thread *thread_id*, another thread in the same process as the caller. The signal is asynchronously directed to thread. - *thread_id* can be read from the :attr:`~threading.Thread.ident` attribute - of :attr:`threading.Thread`. For example, - ``threading.current_thread().ident`` gives the identifier of the current - thread. + Use :func:`threading.get_ident()` or the :attr:`~threading.Thread.ident` + attribute of :attr:`threading.Thread` to get a 'thread identifier' for + *thread_id*. If *signum* is 0, then no signal is sent, but error checking is still performed; this can be used to check if a thread is still running. diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 23ce465..46bd5dc 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -129,6 +129,10 @@ empty, and the path manipulations are skipped; however the import of unless the :program:`python` interpreter was started with the :option:`-S` flag. + .. versionchanged:: 3.3 + This function used to be called unconditionnally. + + .. function:: addsitedir(sitedir, known_paths=None) Adds a directory to sys.path and processes its pth files. diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index ed547f5e..a4c9b63 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -153,8 +153,21 @@ Server Objects .. method:: BaseServer.serve_forever(poll_interval=0.5) Handle requests until an explicit :meth:`shutdown` request. Polls for - shutdown every *poll_interval* seconds. + shutdown every *poll_interval* seconds. It also calls + :meth:`service_actions` which may be used by a subclass or Mixin to provide + various cleanup actions. For e.g. ForkingMixin class uses + :meth:`service_actions` to cleanup the zombie child processes. + .. versionchanged:: 3.3 + Added service_actions call to the serve_forever method. + + +.. method:: BaseServer.service_actions() + + This is called by the serve_forever loop. This method is can be overridden + by Mixin's to add cleanup or service specific actions. + + .. versionadded:: 3.3 .. method:: BaseServer.shutdown() diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 5ece8cf..ca71d20 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -162,6 +162,35 @@ instead. Random generation ^^^^^^^^^^^^^^^^^ +.. function:: RAND_bytes(num) + + Returns *num* cryptographically strong pseudo-random bytes. Raises an + :class:`SSLError` if the PRNG has not been seeded with enough data or if the + operation is not supported by the current RAND method. :func:`RAND_status` + can be used to check the status of the PRNG and :func:`RAND_add` can be used + to seed the PRNG. + + Read the Wikipedia article, `Cryptographically secure pseudorandom number + generator (CSPRNG) + <http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator>`_, + to get the requirements of a cryptographically generator. + + .. versionadded:: 3.3 + +.. function:: RAND_pseudo_bytes(num) + + Returns (bytes, is_cryptographic): bytes are *num* pseudo-random bytes, + is_cryptographic is True if the bytes generated are cryptographically + strong. Raises an :class:`SSLError` if the operation is not supported by the + current RAND method. + + Generated pseudo-random byte sequences will be unique if they are of + sufficient length, but are not necessarily unpredictable. They can be used + for non-cryptographic purposes and for certain purposes in cryptographic + protocols, but usually not for key generation etc. + + .. versionadded:: 3.3 + .. function:: RAND_status() Returns True if the SSL pseudo-random number generator has been seeded with @@ -171,7 +200,7 @@ Random generation .. function:: RAND_egd(path) - If you are running an entropy-gathering daemon (EGD) somewhere, and ``path`` + If you are running an entropy-gathering daemon (EGD) somewhere, and *path* is the pathname of a socket connection open to it, this will read 256 bytes of randomness from the socket, and add it to the SSL pseudo-random number generator to increase the security of generated secret keys. This is @@ -182,8 +211,8 @@ Random generation .. function:: RAND_add(bytes, entropy) - Mixes the given ``bytes`` into the SSL pseudo-random number generator. The - parameter ``entropy`` (a float) is a lower bound on the entropy contained in + Mixes the given *bytes* into the SSL pseudo-random number generator. The + parameter *entropy* (a float) is a lower bound on the entropy contained in string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index df47045..504a2fb 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -48,6 +48,17 @@ This module defines the following functions and objects: returned. +.. function:: get_ident() + + Return the 'thread identifier' of the current thread. This is a nonzero + integer. Its value has no direct meaning; it is intended as a magic cookie + to be used e.g. to index a dictionary of thread-specific data. Thread + identifiers may be recycled when a thread exits and another thread is + created. + + .. versionadded:: 3.3 + + .. function:: enumerate() Return a list of all :class:`Thread` objects currently alive. The list @@ -332,10 +343,10 @@ impossible to detect the termination of alien threads. .. attribute:: ident The 'thread identifier' of this thread or ``None`` if the thread has not - been started. This is a nonzero integer. See the - :func:`thread.get_ident()` function. Thread identifiers may be recycled - when a thread exits and another thread is created. The identifier is - available even after the thread has exited. + been started. This is a nonzero integer. See the :func:`get_ident()` + function. Thread identifiers may be recycled when a thread exits and + another thread is created. The identifier is available even after the + thread has exited. .. method:: is_alive() |
