diff options
Diffstat (limited to 'Doc/reference')
| -rw-r--r-- | Doc/reference/datamodel.rst | 239 | ||||
| -rw-r--r-- | Doc/reference/expressions.rst | 33 | ||||
| -rw-r--r-- | Doc/reference/import.rst | 697 | ||||
| -rw-r--r-- | Doc/reference/index.rst | 1 | ||||
| -rw-r--r-- | Doc/reference/lexical_analysis.rst | 41 | ||||
| -rw-r--r-- | Doc/reference/simple_stmts.rst | 236 |
6 files changed, 1022 insertions, 225 deletions
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 322e8c8..d093383 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -35,12 +35,19 @@ represented by objects.) Every object has an identity, a type and a value. An object's *identity* never changes once it has been created; you may think of it as the object's address in memory. The ':keyword:`is`' operator compares the identity of two objects; the -:func:`id` function returns an integer representing its identity (currently -implemented as its address). An object's :dfn:`type` is also unchangeable. [#]_ +:func:`id` function returns an integer representing its identity. + +.. impl-detail:: + + For CPython, ``id(x)`` is the memory address where ``x`` is stored. + An object's type determines the operations that the object supports (e.g., "does it have a length?") and also defines the possible values for objects of that type. The :func:`type` function returns an object's type (which is an object -itself). The *value* of some objects can change. Objects whose value can +itself). Like its identity, an object's :dfn:`type` is also unchangeable. +[#]_ + +The *value* of some objects can change. Objects whose value can change are said to be *mutable*; objects whose value is unchangeable once they are created are called *immutable*. (The value of an immutable container object that contains a reference to a mutable object can change when the latter's value @@ -276,16 +283,16 @@ Sequences single: integer single: Unicode - The items of a string object are Unicode code units. A Unicode code - unit is represented by a string object of one item and can hold either - a 16-bit or 32-bit value representing a Unicode ordinal (the maximum - value for the ordinal is given in ``sys.maxunicode``, and depends on - how Python is configured at compile time). Surrogate pairs may be - present in the Unicode object, and will be reported as two separate - items. The built-in functions :func:`chr` and :func:`ord` convert - between code units and nonnegative integers representing the Unicode - ordinals as defined in the Unicode Standard 3.0. Conversion from and to - other encodings are possible through the string method :meth:`encode`. + A string is a sequence of values that represent Unicode codepoints. + All the codepoints in range ``U+0000 - U+10FFFF`` can be represented + in a string. Python doesn't have a :c:type:`chr` type, and + every character in the string is represented as a string object + with length ``1``. The built-in function :func:`ord` converts a + character to its codepoint (as an integer); :func:`chr` converts + an integer in range ``0 - 10FFFF`` to the corresponding character. + :meth:`str.encode` can be used to convert a :class:`str` to + :class:`bytes` using the given encoding, and :meth:`bytes.decode` can + be used to achieve the opposite. Tuples .. index:: @@ -448,6 +455,11 @@ Callable types +-------------------------+-------------------------------+-----------+ | :attr:`__name__` | The function's name | Writable | +-------------------------+-------------------------------+-----------+ + | :attr:`__qualname__` | The function's | Writable | + | | :term:`qualified name` | | + | | | | + | | .. versionadded:: 3.3 | | + +-------------------------+-------------------------------+-----------+ | :attr:`__module__` | The name of the module the | Writable | | | function was defined in, or | | | | ``None`` if unavailable. | | @@ -639,17 +651,20 @@ Modules statement: import object: module - Modules are imported by the :keyword:`import` statement (see section - :ref:`import`). A module object has a - namespace implemented by a dictionary object (this is the dictionary referenced - by the __globals__ attribute of functions defined in the module). Attribute - references are translated to lookups in this dictionary, e.g., ``m.x`` is - equivalent to ``m.__dict__["x"]``. A module object does not contain the code - object used to initialize the module (since it isn't needed once the - initialization is done). - - Attribute assignment updates the module's namespace dictionary, e.g., ``m.x = - 1`` is equivalent to ``m.__dict__["x"] = 1``. + Modules are a basic organizational unit of Python code, and are created by + the :ref:`import system <importsystem>` as invoked either by the + :keyword:`import` statement (see :keyword:`import`), or by calling + functions such as :func:`importlib.import_module` and built-in + :func:`__import__`. A module object has a namespace implemented by a + dictionary object (this is the dictionary referenced by the ``__globals__`` + attribute of functions defined in the module). Attribute references are + translated to lookups in this dictionary, e.g., ``m.x`` is equivalent to + ``m.__dict__["x"]``. A module object does not contain the code object used + to initialize the module (since it isn't needed once the initialization is + done). + + Attribute assignment updates the module's namespace dictionary, e.g., + ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``. .. index:: single: __dict__ (module attribute) @@ -671,11 +686,12 @@ Modules Predefined (writable) attributes: :attr:`__name__` is the module's name; :attr:`__doc__` is the module's documentation string, or ``None`` if - unavailable; :attr:`__file__` is the pathname of the file from which the module - was loaded, if it was loaded from a file. The :attr:`__file__` attribute is not - present for C modules that are statically linked into the interpreter; for - extension modules loaded dynamically from a shared library, it is the pathname - of the shared library file. + unavailable; :attr:`__file__` is the pathname of the file from which the + module was loaded, if it was loaded from a file. The :attr:`__file__` + attribute may be missing for certain types of modules, such as C modules + that are statically linked into the interpreter; for extension modules + loaded dynamically from a shared library, it is the pathname of the shared + library file. Custom classes Custom class types are typically created by class definitions (see section @@ -1250,10 +1266,10 @@ Basic customization immutable (if the object's hash value changes, it will be in the wrong hash bucket). - User-defined classes have :meth:`__eq__` and :meth:`__hash__` methods by default; with them, all objects compare unequal (except with themselves) - and ``x.__hash__()`` returns ``id(x)``. + and ``x.__hash__()`` returns an appropriate value such that ``x == y`` + implies both that ``x is y`` and ``hash(x) == hash(y)``. A class that overrides :meth:`__eq__` and does not define :meth:`__hash__` will have its :meth:`__hash__` implicitly set to ``None``. When the @@ -1272,7 +1288,27 @@ Basic customization a :exc:`TypeError` would be incorrectly identified as hashable by an ``isinstance(obj, collections.Hashable)`` call. - See also the :option:`-R` command-line option. + + .. note:: + + By default, the :meth:`__hash__` values of str, bytes and datetime + objects are "salted" with an unpredictable random value. Although they + remain constant within an individual Python process, they are not + predictable between repeated invocations of Python. + + This is intended to provide protection against a denial-of-service caused + by carefully-chosen inputs that exploit the worst case performance of a + dict insertion, O(n^2) complexity. See + http://www.ocert.org/advisories/ocert-2011-003.html for details. + + Changing hash values affects the iteration order of dicts, sets and + other mappings. Python has never made guarantees about this ordering + (and it typically varies between 32-bit and 64-bit builds). + + See also :envvar:`PYTHONHASHSEED`. + + .. versionchanged:: 3.3 + Hash randomization is enabled by default. .. method:: object.__bool__(self) @@ -1353,7 +1389,8 @@ access (use of, assignment to, or deletion of ``x.name``) for class instances. .. method:: object.__dir__(self) - Called when :func:`dir` is called on the object. A list must be returned. + Called when :func:`dir` is called on the object. A sequence must be + returned. :func:`dir` converts the returned sequence to a list and sorts it. .. _descriptors: @@ -1524,53 +1561,115 @@ Notes on using *__slots__* Customizing class creation -------------------------- -By default, classes are constructed using :func:`type`. A class definition is -read into a separate namespace and the value of class name is bound to the -result of ``type(name, bases, dict)``. +By default, classes are constructed using :func:`type`. The class body is +executed in a new namespace and the class name is bound locally to the +result of ``type(name, bases, namespace)``. + +The class creation process can be customised by passing the ``metaclass`` +keyword argument in the class definition line, or by inheriting from an +existing class that included such an argument. In the following example, +both ``MyClass`` and ``MySubclass`` are instances of ``Meta``:: + + class Meta(type): + pass + + class MyClass(metaclass=Meta): + pass + + class MySubclass(MyClass): + pass -When the class definition is read, if a callable ``metaclass`` keyword argument -is passed after the bases in the class definition, the callable given will be -called instead of :func:`type`. If other keyword arguments are passed, they -will also be passed to the metaclass. This allows classes or functions to be -written which monitor or alter the class creation process: +Any other keyword arguments that are specified in the class definition are +passed through to all metaclass operations described below. -* Modifying the class dictionary prior to the class being created. +When a class definition is executed, the following steps occur: -* Returning an instance of another class -- essentially performing the role of a - factory function. +* the appropriate metaclass is determined +* the class namespace is prepared +* the class body is executed +* the class object is created -These steps will have to be performed in the metaclass's :meth:`__new__` method --- :meth:`type.__new__` can then be called from this method to create a class -with different properties. This example adds a new element to the class -dictionary before creating the class:: +Determining the appropriate metaclass +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - class metacls(type): - def __new__(mcs, name, bases, dict): - dict['foo'] = 'metacls was here' - return type.__new__(mcs, name, bases, dict) +The appropriate metaclass for a class definition is determined as follows: -You can of course also override other class methods (or add new methods); for -example defining a custom :meth:`__call__` method in the metaclass allows custom -behavior when the class is called, e.g. not always creating a new instance. +* if no bases and no explicit metaclass are given, then :func:`type` is used +* if an explicit metaclass is given and it is *not* an instance of + :func:`type`, then it is used directly as the metaclass +* if an instance of :func:`type` is given as the explicit metaclass, or + bases are defined, then the most derived metaclass is used -If the metaclass has a :meth:`__prepare__` attribute (usually implemented as a -class or static method), it is called before the class body is evaluated with -the name of the class and a tuple of its bases for arguments. It should return -an object that supports the mapping interface that will be used to store the -namespace of the class. The default is a plain dictionary. This could be used, -for example, to keep track of the order that class attributes are declared in by -returning an ordered dictionary. +The most derived metaclass is selected from the explicitly specified +metaclass (if any) and the metaclasses (i.e. ``type(cls)``) of all specified +base classes. The most derived metaclass is one which is a subtype of *all* +of these candidate metaclasses. If none of the candidate metaclasses meets +that criterion, then the class definition will fail with ``TypeError``. -The appropriate metaclass is determined by the following precedence rules: -* If the ``metaclass`` keyword argument is passed with the bases, it is used. +Preparing the class namespace +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once the appropriate metaclass has been identified, then the class namespace +is prepared. If the metaclass has a ``__prepare__`` attribute, it is called +as ``namespace = metaclass.__prepare__(name, bases, **kwds)`` (where the +additional keyword arguments, if any, come from the class definition). + +If the metaclass has no ``__prepare__`` attribute, then the class namespace +is initialised as an empty :func:`dict` instance. + +.. seealso:: + + :pep:`3115` - Metaclasses in Python 3000 + Introduced the ``__prepare__`` namespace hook + + +Executing the class body +^^^^^^^^^^^^^^^^^^^^^^^^ + +The class body is executed (approximately) as +``exec(body, globals(), namespace)``. The key difference from a normal +call to :func:`exec` is that lexical scoping allows the class body (including +any methods) to reference names from the current and outer scopes when the +class definition occurs inside a function. + +However, even when the class definition occurs inside the function, methods +defined inside the class still cannot see names defined at the class scope. +Class variables must be accessed through the first parameter of instance or +class methods, and cannot be accessed at all from static methods. + + +Creating the class object +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once the class namespace has been populated by executing the class body, +the class object is created by calling +``metaclass(name, bases, namespace, **kwds)`` (the additional keywords +passed here are the same as those passed to ``__prepare__``). + +This class object is the one that will be referenced by the zero-argument +form of :func:`super`. ``__class__`` is an implicit closure reference +created by the compiler if any methods in a class body refer to either +``__class__`` or ``super``. This allows the zero argument form of +:func:`super` to correctly identify the class being defined based on +lexical scoping, while the class or instance that was used to make the +current call is identified based on the first argument passed to the method. + +After the class object is created, it is passed to the class decorators +included in the class definition (if any) and the resulting object is bound +in the local namespace as the defined class. + +.. seealso:: + + :pep:`3135` - New super + Describes the implicit ``__class__`` closure reference -* Otherwise, if there is at least one base class, its metaclass is used. -* Otherwise, the default metaclass (:class:`type`) is used. +Metaclass example +^^^^^^^^^^^^^^^^^ The potential uses for metaclasses are boundless. Some ideas that have been -explored including logging, interface checking, automatic delegation, automatic +explored include logging, interface checking, automatic delegation, automatic property creation, proxies, frameworks, and automatic resource locking/synchronization. @@ -1583,9 +1682,9 @@ to remember the order that class members were defined:: def __prepare__(metacls, name, bases, **kwds): return collections.OrderedDict() - def __new__(cls, name, bases, classdict): - result = type.__new__(cls, name, bases, dict(classdict)) - result.members = tuple(classdict) + def __new__(cls, name, bases, namespace, **kwds): + result = type.__new__(cls, name, bases, dict(namespace)) + result.members = tuple(namespace) return result class A(metaclass=OrderedClass): diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 2c6acb6..41523cb 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -318,7 +318,7 @@ Yield expressions .. productionlist:: yield_atom: "(" `yield_expression` ")" - yield_expression: "yield" [`expression_list`] + yield_expression: "yield" [`expression_list` | "from" `expression`] The :keyword:`yield` expression is only used when defining a generator function, and can only be used in the body of a function definition. Using a @@ -336,7 +336,10 @@ the internal evaluation stack. When the execution is resumed by calling one of the generator's methods, the function can proceed exactly as if the :keyword:`yield` expression was just another external call. The value of the :keyword:`yield` expression after resuming depends on the method which resumed -the execution. +the execution. If :meth:`__next__` is used (typically via either a +:keyword:`for` or the :func:`next` builtin) then the result is :const:`None`, +otherwise, if :meth:`send` is used, then the result will be the value passed +in to that method. .. index:: single: coroutine @@ -346,12 +349,32 @@ suspended. The only difference is that a generator function cannot control where should the execution continue after it yields; the control is always transferred to the generator's caller. -The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a +:keyword:`yield` expressions are allowed in the :keyword:`try` clause of a :keyword:`try` ... :keyword:`finally` construct. If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:`close` method will be called, allowing any pending :keyword:`finally` clauses to execute. +When ``yield from <expr>`` is used, it treats the supplied expression as +a subiterator. All values produced by that subiterator are passed directly +to the caller of the current generator's methods. Any values passed in with +:meth:`send` and any exceptions passed in with :meth:`throw` are passed to +the underlying iterator if it has the appropriate methods. If this is not the +case, then :meth:`send` will raise :exc:`AttributeError` or :exc:`TypeError`, +while :meth:`throw` will just raise the passed in exception immediately. + +When the underlying iterator is complete, the :attr:`~StopIteration.value` +attribute of the raised :exc:`StopIteration` instance becomes the value of +the yield expression. It can be either set explicitly when raising +:exc:`StopIteration`, or automatically when the sub-iterator is a generator +(by returning a value from the sub-generator). + + .. versionchanged:: 3.3 + Added ``yield from <expr>`` to delegate control flow to a subiterator + +The parentheses can be omitted when the :keyword:`yield` expression is the +sole expression on the right hand side of an assignment statement. + .. index:: object: generator @@ -452,6 +475,10 @@ generator functions:: The proposal to enhance the API and syntax of generators, making them usable as simple coroutines. + :pep:`0380` - Syntax for Delegating to a Subgenerator + The proposal to introduce the :token:`yield_from` syntax, making delegation + to sub-generators easy. + .. _primaries: diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst new file mode 100644 index 0000000..4688e78 --- /dev/null +++ b/Doc/reference/import.rst @@ -0,0 +1,697 @@ + +.. _importsystem: + +***************** +The import system +***************** + +.. index:: single: import machinery + +Python code in one :term:`module` gains access to the code in another module +by the process of :term:`importing` it. The :keyword:`import` statement is +the most common way of invoking the import machinery, but it is not the only +way. Functions such as :func:`importlib.import_module` and built-in +:func:`__import__` can also be used to invoke the import machinery. + +The :keyword:`import` statement combines two operations; it searches for the +named module, then it binds the results of that search to a name in the local +scope. The search operation of the :keyword:`import` statement is defined as +a call to the :func:`__import__` function, with the appropriate arguments. +The return value of :func:`__import__` is used to perform the name +binding operation of the :keyword:`import` statement. See the +:keyword:`import` statement for the exact details of that name binding +operation. + +A direct call to :func:`__import__` performs only the module search and, if +found, the module creation operation. While certain side-effects may occur, +such as the importing of parent packages, and the updating of various caches +(including :data:`sys.modules`), only the :keyword:`import` statement performs +a name binding operation. + +When calling :func:`__import__` as part of an import statement, the +import system first checks the module global namespace for a function by +that name. If it is not found, then the standard builtin :func:`__import__` +is called. Other mechanisms for invoking the import system (such as +:func:`importlib.import_module`) do not perform this check and will always +use the standard import system. + +When a module is first imported, Python searches for the module and if found, +it creates a module object [#fnmo]_, initializing it. If the named module +cannot be found, an :exc:`ImportError` is raised. Python implements various +strategies to search for the named module when the import machinery is +invoked. These strategies can be modified and extended by using various hooks +described in the sections below. + +.. versionchanged:: 3.3 + The import system has been updated to fully implement the second phase + of PEP 302. There is no longer any implicit import machinery - the full + import system is exposed through :data:`sys.meta_path`. In addition, + native namespace package support has been implemented (see PEP 420). + + +:mod:`importlib` +================ + +The :mod:`importlib` module provides a rich API for interacting with the +import system. For example :func:`importlib.import_module` provides a +recommended, simpler API than built-in :func:`__import__` for invoking the +import machinery. Refer to the :mod:`importlib` library documentation for +additional detail. + + + +Packages +======== + +.. index:: + single: package + +Python has only one type of module object, and all modules are of this type, +regardless of whether the module is implemented in Python, C, or something +else. To help organize modules and provide a naming hierarchy, Python has a +concept of :term:`packages <package>`. + +You can think of packages as the directories on a file system and modules as +files within directories, but don't take this analogy too literally since +packages and modules need not originate from the file system. For the +purposes of this documentation, we'll use this convenient analogy of +directories and files. Like file system directories, packages are organized +hierarchically, and packages may themselves contain subpackages, as well as +regular modules. + +It's important to keep in mind that all packages are modules, but not all +modules are packages. Or put another way, packages are just a special kind of +module. Specifically, any module that contains a ``__path__`` attribute is +considered a package. + +All modules have a name. Subpackage names are separated from their parent +package name by dots, akin to Python's standard attribute access syntax. Thus +you might have a module called :mod:`sys` and a package called :mod:`email`, +which in turn has a subpackage called :mod:`email.mime` and a module within +that subpackage called :mod:`email.mime.text`. + + +Regular packages +---------------- + +.. index:: + pair: package; regular + +Python defines two types of packages, :term:`regular packages <regular +package>` and :term:`namespace packages <namespace package>`. Regular +packages are traditional packages as they existed in Python 3.2 and earlier. +A regular package is typically implemented as a directory containing an +``__init__.py`` file. When a regular package is imported, this +``__init__.py`` file is implicitly executed, and the objects it defines are +bound to names in the package's namespace. The ``__init__.py`` file can +contain the same Python code that any other module can contain, and Python +will add some additional attributes to the module when it is imported. + +For example, the following file system layout defines a top level ``parent`` +package with three subpackages:: + + parent/ + __init__.py + one/ + __init__.py + two/ + __init__.py + three/ + __init__.py + +Importing ``parent.one`` will implicitly execute ``parent/__init__.py`` and +``parent/one/__init__.py``. Subsequent imports of ``parent.two`` or +``parent.three`` will execute ``parent/two/__init__.py`` and +``parent/three/__init__.py`` respectively. + + +Namespace packages +------------------ + +.. index:: + pair:: package; namespace + pair:: package; portion + +A namespace package is a composite of various :term:`portions <portion>`, +where each portion contributes a subpackage to the parent package. Portions +may reside in different locations on the file system. Portions may also be +found in zip files, on the network, or anywhere else that Python searches +during import. Namespace packages may or may not correspond directly to +objects on the file system; they may be virtual modules that have no concrete +representation. + +Namespace packages do not use an ordinary list for their ``__path__`` +attribute. They instead use a custom iterable type which will automatically +perform a new search for package portions on the next import attempt within +that package if the path of their parent package (or :data:`sys.path` for a +top level package) changes. + +With namespace packages, there is no ``parent/__init__.py`` file. In fact, +there may be multiple ``parent`` directories found during import search, where +each one is provided by a different portion. Thus ``parent/one`` may not be +physically located next to ``parent/two``. In this case, Python will create a +namespace package for the top-level ``parent`` package whenever it or one of +its subpackages is imported. + +See also :pep:`420` for the namespace package specification. + + +Searching +========= + +To begin the search, Python needs the :term:`fully qualified <qualified name>` +name of the module (or package, but for the purposes of this discussion, the +difference is immaterial) being imported. This name may come from various +arguments to the :keyword:`import` statement, or from the parameters to the +:func:`importlib.import_module` or :func:`__import__` functions. + +This name will be used in various phases of the import search, and it may be +the dotted path to a submodule, e.g. ``foo.bar.baz``. In this case, Python +first tries to import ``foo``, then ``foo.bar``, and finally ``foo.bar.baz``. +If any of the intermediate imports fail, an :exc:`ImportError` is raised. + + +The module cache +---------------- + +.. index:: + single: sys.modules + +The first place checked during import search is :data:`sys.modules`. This +mapping serves as a cache of all modules that have been previously imported, +including the intermediate paths. So if ``foo.bar.baz`` was previously +imported, :data:`sys.modules` will contain entries for ``foo``, ``foo.bar``, +and ``foo.bar.baz``. Each key will have as its value the corresponding module +object. + +During import, the module name is looked up in :data:`sys.modules` and if +present, the associated value is the module satisfying the import, and the +process completes. However, if the value is ``None``, then an +:exc:`ImportError` is raised. If the module name is missing, Python will +continue searching for the module. + +:data:`sys.modules` is writable. Deleting a key may not destroy the +associated module (as other modules may hold references to it), +but it will invalidate the cache entry for the named module, causing +Python to search anew for the named module upon its next +import. The key can also be assigned to ``None``, forcing the next import +of the module to result in an :exc:`ImportError`. + +Beware though, as if you keep a reference to the module object, +invalidate its cache entry in :data:`sys.modules`, and then re-import the +named module, the two module objects will *not* be the same. By contrast, +:func:`imp.reload` will reuse the *same* module object, and simply +reinitialise the module contents by rerunning the module's code. + + +Finders and loaders +------------------- + +.. index:: + single: finder + single: loader + +If the named module is not found in :data:`sys.modules`, then Python's import +protocol is invoked to find and load the module. This protocol consists of +two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`. +A finder's job is to determine whether it can find the named module using +whatever strategy it knows about. Objects that implement both of these +interfaces are referred to as :term:`importers <importer>` - they return +themselves when they find that they can load the requested module. + +Python includes a number of default finders and importers. One +knows how to locate frozen modules, and another knows how to locate +built-in modules. A third default finder searches an :term:`import path` +for modules. The :term:`import path` is a list of locations that may +name file system paths or zip files. It can also be extended to search +for any locatable resource, such as those identified by URLs. + +The import machinery is extensible, so new finders can be added to extend the +range and scope of module searching. + +Finders do not actually load modules. If they can find the named module, they +return a :term:`loader`, which the import machinery then invokes to load the +module and create the corresponding module object. + +The following sections describe the protocol for finders and loaders in more +detail, including how you can create and register new ones to extend the +import machinery. + + +Import hooks +------------ + +.. index:: + single: import hooks + single: meta hooks + single: path hooks + pair: hooks; import + pair: hooks; meta + pair: hooks; path + +The import machinery is designed to be extensible; the primary mechanism for +this are the *import hooks*. There are two types of import hooks: *meta +hooks* and *import path hooks*. + +Meta hooks are called at the start of import processing, before any other +import processing has occurred, other than :data:`sys.modules` cache look up. +This allows meta hooks to override :data:`sys.path` processing, frozen +modules, or even built-in modules. Meta hooks are registered by adding new +finder objects to :data:`sys.meta_path`, as described below. + +Import path hooks are called as part of :data:`sys.path` (or +``package.__path__``) processing, at the point where their associated path +item is encountered. Import path hooks are registered by adding new callables +to :data:`sys.path_hooks` as described below. + + +The meta path +------------- + +.. index:: + single: sys.meta_path + pair: finder; find_module + pair: finder; find_loader + +When the named module is not found in :data:`sys.modules`, Python next +searches :data:`sys.meta_path`, which contains a list of meta path finder +objects. These finders are queried in order to see if they know how to handle +the named module. Meta path finders must implement a method called +:meth:`find_module()` which takes two arguments, a name and an import path. +The meta path finder can use any strategy it wants to determine whether it can +handle the named module or not. + +If the meta path finder knows how to handle the named module, it returns a +loader object. If it cannot handle the named module, it returns ``None``. If +:data:`sys.meta_path` processing reaches the end of its list without returning +a loader, then an :exc:`ImportError` is raised. Any other exceptions raised +are simply propagated up, aborting the import process. + +The :meth:`find_module()` method of meta path finders is called with two +arguments. The first is the fully qualified name of the module being +imported, for example ``foo.bar.baz``. The second argument is the path +entries to use for the module search. For top-level modules, the second +argument is ``None``, but for submodules or subpackages, the second +argument is the value of the parent package's ``__path__`` attribute. If +the appropriate ``__path__`` attribute cannot be accessed, an +:exc:`ImportError` is raised. + +The meta path may be traversed multiple times for a single import request. +For example, assuming none of the modules involved has already been cached, +importing ``foo.bar.baz`` will first perform a top level import, calling +``mpf.find_module("foo", None)`` on each meta path finder (``mpf``). After +``foo`` has been imported, ``foo.bar`` will be imported by traversing the +meta path a second time, calling +``mpf.find_module("foo.bar", foo.__path__)``. Once ``foo.bar`` has been +imported, the final traversal will call +``mpf.find_module("foo.bar.baz", foo.bar.__path__)``. + +Some meta path finders only support top level imports. These importers will +always return ``None`` when anything other than ``None`` is passed as the +second argument. + +Python's default :data:`sys.meta_path` has three meta path finders, one that +knows how to import built-in modules, one that knows how to import frozen +modules, and one that knows how to import modules from an :term:`import path` +(i.e. the :term:`path based finder`). + + +Loaders +======= + +If and when a module loader is found its +:meth:`~importlib.abc.Loader.load_module` method is called, with a single +argument, the fully qualified name of the module being imported. This method +has several responsibilities, and should return the module object it has +loaded [#fnlo]_. If it cannot load the module, it should raise an +:exc:`ImportError`, although any other exception raised during +:meth:`load_module()` will be propagated. + +In many cases, the finder and loader can be the same object; in such cases the +:meth:`finder.find_module()` would just return ``self``. + +Loaders must satisfy the following requirements: + + * If there is an existing module object with the given name in + :data:`sys.modules`, the loader must use that existing module. (Otherwise, + :func:`imp.reload` will not work correctly.) If the named module does + not exist in :data:`sys.modules`, the loader must create a new module + object and add it to :data:`sys.modules`. + + Note that the module *must* exist in :data:`sys.modules` before the loader + executes the module code. This is crucial because the module code may + (directly or indirectly) import itself; adding it to :data:`sys.modules` + beforehand prevents unbounded recursion in the worst case and multiple + loading in the best. + + If loading fails, the loader must remove any modules it has inserted into + :data:`sys.modules`, but it must remove **only** the failing module, and + only if the loader itself has loaded it explicitly. Any module already in + the :data:`sys.modules` cache, and any module that was successfully loaded + as a side-effect, must remain in the cache. + + * The loader may set the ``__file__`` attribute of the module. If set, this + attribute's value must be a string. The loader may opt to leave + ``__file__`` unset if it has no semantic meaning (e.g. a module loaded from + a database). + + * The loader may set the ``__name__`` attribute of the module. While not + required, setting this attribute is highly recommended so that the + :meth:`repr()` of the module is more informative. + + * If the module is a package (either regular or namespace), the loader must + set the module object's ``__path__`` attribute. The value must be + iterable, but may be empty if ``__path__`` has no further significance + to the loader. If ``__path__`` is not empty, it must produce strings + when iterated over. More details on the semantics of ``__path__`` are + given :ref:`below <package-path-rules>`. + + * The ``__loader__`` attribute must be set to the loader object that loaded + the module. This is mostly for introspection and reloading, but can be + used for additional loader-specific functionality, for example getting + data associated with a loader. + + * The module's ``__package__`` attribute should be set. Its value must be a + string, but it can be the same value as its ``__name__``. If the attribute + is set to ``None`` or is missing, the import system will fill it in with a + more appropriate value. When the module is a package, its ``__package__`` + value should be set to its ``__name__``. When the module is not a package, + ``__package__`` should be set to the empty string for top-level modules, or + for submodules, to the parent package's name. See :pep:`366` for further + details. + + This attribute is used instead of ``__name__`` to calculate explicit + relative imports for main modules, as defined in :pep:`366`. + + * If the module is a Python module (as opposed to a built-in module or a + dynamically loaded extension), the loader should execute the module's code + in the module's global name space (``module.__dict__``). + + +Module reprs +------------ + +By default, all modules have a usable repr, however depending on the +attributes set above, and hooks in the loader, you can more explicitly control +the repr of module objects. + +Loaders may implement a :meth:`module_repr()` method which takes a single +argument, the module object. When ``repr(module)`` is called for a module +with a loader supporting this protocol, whatever is returned from +``module.__loader__.module_repr(module)`` is returned as the module's repr +without further processing. This return value must be a string. + +If the module has no ``__loader__`` attribute, or the loader has no +:meth:`module_repr()` method, then the module object implementation itself +will craft a default repr using whatever information is available. It will +try to use the ``module.__name__``, ``module.__file__``, and +``module.__loader__`` as input into the repr, with defaults for whatever +information is missing. + +Here are the exact rules used: + + * If the module has a ``__loader__`` and that loader has a + :meth:`module_repr()` method, call it with a single argument, which is the + module object. The value returned is used as the module's repr. + + * If an exception occurs in :meth:`module_repr()`, the exception is caught + and discarded, and the calculation of the module's repr continues as if + :meth:`module_repr()` did not exist. + + * If the module has a ``__file__`` attribute, this is used as part of the + module's repr. + + * If the module has no ``__file__`` but does have a ``__loader__``, then the + loader's repr is used as part of the module's repr. + + * Otherwise, just use the module's ``__name__`` in the repr. + +This example, from :pep:`420` shows how a loader can craft its own module +repr:: + + class NamespaceLoader: + @classmethod + def module_repr(cls, module): + return "<module '{}' (namespace)>".format(module.__name__) + + +.. _package-path-rules: + +module.__path__ +--------------- + +By definition, if a module has an ``__path__`` attribute, it is a package, +regardless of its value. + +A package's ``__path__`` attribute is used during imports of its subpackages. +Within the import machinery, it functions much the same as :data:`sys.path`, +i.e. providing a list of locations to search for modules during import. +However, ``__path__`` is typically much more constrained than +:data:`sys.path`. + +``__path__`` must be an iterable of strings, but it may be empty. +The same rules used for :data:`sys.path` also apply to a package's +``__path__``, and :data:`sys.path_hooks` (described below) are +consulted when traversing a package's ``__path__``. + +A package's ``__init__.py`` file may set or alter the package's ``__path__`` +attribute, and this was typically the way namespace packages were implemented +prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no +longer need to supply ``__init__.py`` files containing only ``__path__`` +manipulation code; the namespace loader automatically sets ``__path__`` +correctly for the namespace package. + + +The Path Based Finder +===================== + +.. index:: + single: path based finder + +As mentioned previously, Python comes with several default meta path finders. +One of these, called the :term:`path based finder`, searches an :term:`import +path`, which contains a list of :term:`path entries <path entry>`. Each path +entry names a location to search for modules. + +The path based finder itself doesn't know how to import anything. Instead, it +traverses the individual path entries, associating each of them with a +path entry finder that knows how to handle that particular kind of path. + +The default set of path entry finders implement all the semantics for finding +modules on the file system, handling special file types such as Python source +code (``.py`` files), Python byte code (``.pyc`` and ``.pyo`` files) and +shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport` +module in the standard library, the default path entry finders also handle +loading all of these file types (other than shared libraries) from zipfiles. + +Path entries need not be limited to file system locations. They can refer to +URLs, database queries, or any other location that can be specified as a +string. + +The path based finder provides additional hooks and protocols so that you +can extend and customize the types of searchable path entries. For example, +if you wanted to support path entries as network URLs, you could write a hook +that implements HTTP semantics to find modules on the web. This hook (a +callable) would return a :term:`path entry finder` supporting the protocol +described below, which was then used to get a loader for the module from the +web. + +A word of warning: this section and the previous both use the term *finder*, +distinguishing between them by using the terms :term:`meta path finder` and +:term:`path entry finder`. These two types of finders are very similar, +support similar protocols, and function in similar ways during the import +process, but it's important to keep in mind that they are subtly different. +In particular, meta path finders operate at the beginning of the import +process, as keyed off the :data:`sys.meta_path` traversal. + +By contrast, path entry finders are in a sense an implementation detail +of the path based finder, and in fact, if the path based finder were to be +removed from :data:`sys.meta_path`, none of the path entry finder semantics +would be invoked. + + +Path entry finders +------------------ + +.. index:: + single: sys.path + single: sys.path_hooks + single: sys.path_importer_cache + single: PYTHONPATH + +The :term:`path based finder` is responsible for finding and loading Python +modules and packages whose location is specified with a string :term:`path +entry`. Most path entries name locations in the file system, but they need +not be limited to this. + +As a meta path finder, the :term:`path based finder` implements the +:meth:`find_module()` protocol previously described, however it exposes +additional hooks that can be used to customize how modules are found and +loaded from the :term:`import path`. + +Three variables are used by the :term:`path based finder`, :data:`sys.path`, +:data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__`` +attributes on package objects are also used. These provide additional ways +that the import machinery can be customized. + +:data:`sys.path` contains a list of strings providing search locations for +modules and packages. It is initialized from the :data:`PYTHONPATH` +environment variable and various other installation- and +implementation-specific defaults. Entries in :data:`sys.path` can name +directories on the file system, zip files, and potentially other "locations" +(see the :mod:`site` module) that should be searched for modules, such as +URLs, or database queries. + +The :term:`path based finder` is a :term:`meta path finder`, so the import +machinery begins the :term:`import path` search by calling the path +based finder's :meth:`find_module()` method as described previously. When +the ``path`` argument to :meth:`find_module()` is given, it will be a +list of string paths to traverse - typically a package's ``__path__`` +attribute for an import within that package. If the ``path`` argument +is ``None``, this indicates a top level import and :data:`sys.path` is used. + +The path based finder iterates over every entry in the search path, and +for each of these, looks for an appropriate :term:`path entry finder` for the +path entry. Because this can be an expensive operation (e.g. there may be +`stat()` call overheads for this search), the path based finder maintains +a cache mapping path entries to path entry finders. This cache is maintained +in :data:`sys.path_importer_cache` (despite the name, this cache actually +stores finder objects rather than being limited to :term:`importer` objects). +In this way, the expensive search for a particular :term:`path entry` +location's :term:`path entry finder` need only be done once. User code is +free to remove cache entries from :data:`sys.path_importer_cache` forcing +the path based finder to perform the path entry search again [#fnpic]_. + +If the path entry is not present in the cache, the path based finder iterates +over every callable in :data:`sys.path_hooks`. Each of the +:term:`path entry hooks <path entry hook>` in this list is called with a +single argument, the path entry to be searched. This callable may either +return a :term:`path entry finder` that can handle the path entry, or it may +raise :exc:`ImportError`. +An :exc:`ImportError` is used by the path based finder to signal that the hook +cannot find a :term:`path entry finder` for that :term:`path entry`. The +exception is ignored and :term:`import path` iteration continues. + +If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` +being returned, then the path based finder's :meth:`find_module()` method +will store ``None`` in :data:`sys.path_importer_cache` (to indicate that +there is no finder for this path entry) and return ``None``, indicating that +this :term:`meta path finder` could not find the module. + +If a :term:`path entry finder` *is* returned by one of the :term:`path entry +hook` callables on :data:`sys.path_hooks`, then the following protocol is used +to ask the finder for a module loader, which is then used to load the module. + + +Path entry finder protocol +-------------------------- + +In order to support imports of modules and initialized packages and also to +contribute portions to namespace packages, path entry finders must implement +the :meth:`find_loader()` method. + +:meth:`find_loader()` takes one argument, the fully qualified name of the +module being imported. :meth:`find_loader()` returns a 2-tuple where the +first item is the loader and the second item is a namespace :term:`portion`. +When the first item (i.e. the loader) is ``None``, this means that while the +path entry finder does not have a loader for the named module, it knows that the +path entry contributes to a namespace portion for the named module. This will +almost always be the case where Python is asked to import a namespace package +that has no physical presence on the file system. When a path entry finder +returns ``None`` for the loader, the second item of the 2-tuple return value +must be a sequence, although it can be empty. + +If :meth:`find_loader()` returns a non-``None`` loader value, the portion is +ignored and the loader is returned from the path based finder, terminating +the search through the path entries. + +For backwards compatibility with other implementations of the import +protocol, many path entry finders also support the same, +traditional :meth:`find_module()` method that meta path finders support. +However path entry finder :meth:`find_module()` methods are never called +with a ``path`` argument (they are expected to record the appropriate +path information from the initial call to the path hook). + +The :meth:`find_module()` method on path entry finders is deprecated, +as it does not allow the path entry finder to contribute portions to +namespace packages. Instead path entry finders should implement the +:meth:`find_loader()` method as described above. If it exists on the path +entry finder, the import system will always call :meth:`find_loader()` +in preference to :meth:`find_module()`. + + +Replacing the standard import system +==================================== + +The most reliable mechanism for replacing the entire import system is to +delete the default contents of :data:`sys.meta_path`, replacing them +entirely with a custom meta path hook. + +If it is acceptable to only alter the behaviour of import statements +without affecting other APIs that access the import system, then replacing +the builtin :func:`__import__` function may be sufficient. This technique +may also be employed at the module level to only alter the behaviour of +import statements within that module. + +To selectively prevent import of some modules from a hook early on the +meta path (rather than disabling the standard import system entirely), +it is sufficient to raise :exc:`ImportError` directly from +:meth:`find_module` instead of returning ``None``. The latter indicates +that the meta path search should continue. while raising an exception +terminates it immediately. + + +Open issues +=========== + +XXX It would be really nice to have a diagram. + +XXX * (import_machinery.rst) how about a section devoted just to the +attributes of modules and packages, perhaps expanding upon or supplanting the +related entries in the data model reference page? + +XXX runpy, pkgutil, et al in the library manual should all get "See Also" +links at the top pointing to the new import system section. + + +References +========== + +The import machinery has evolved considerably since Python's early days. The +original `specification for packages +<http://www.python.org/doc/essays/packages.html>`_ is still available to read, +although some details have changed since the writing of that document. + +The original specification for :data:`sys.meta_path` was :pep:`302`, with +subsequent extension in :pep:`420`. + +:pep:`420` introduced :term:`namespace packages <namespace package>` for +Python 3.3. :pep:`420` also introduced the :meth:`find_loader` protocol as an +alternative to :meth:`find_module`. + +:pep:`366` describes the addition of the ``__package__`` attribute for +explicit relative imports in main modules. + +:pep:`328` introduced absolute and explicit relative imports and initially +proposed ``__name__`` for semantics :pep:`366` would eventually specify for +``__package__``. + +:pep:`338` defines executing modules as scripts. + + +Footnotes +========= + +.. [#fnmo] See :class:`types.ModuleType`. + +.. [#fnlo] The importlib implementation avoids using the return value + directly. Instead, it gets the module object by looking the module name up + in :data:`sys.modules`. The indirect effect of this is that an imported + module may replace itself in :data:`sys.modules`. This is + implementation-specific behavior that is not guaranteed to work in other + Python implementations. + +.. [#fnpic] In legacy code, it is possible to find instances of + :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It + is recommended that code be changed to use ``None`` instead. See + :ref:`portingpythoncode` for more details. diff --git a/Doc/reference/index.rst b/Doc/reference/index.rst index bd1a281..55f93d7 100644 --- a/Doc/reference/index.rst +++ b/Doc/reference/index.rst @@ -24,6 +24,7 @@ interfaces available to C/C++ programmers in detail. lexical_analysis.rst datamodel.rst executionmodel.rst + import.rst expressions.rst simple_stmts.rst compound_stmts.rst diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 4b49738..94f219b 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -401,7 +401,7 @@ String literals are described by the following lexical definitions: .. productionlist:: stringliteral: [`stringprefix`](`shortstring` | `longstring`) - stringprefix: "r" | "R" + stringprefix: "r" | "u" | "R" | "U" shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' longstring: "'''" `longstringitem`* "'''" | '"""' `longstringitem`* '"""' shortstringitem: `shortstringchar` | `stringescapeseq` @@ -412,7 +412,7 @@ String literals are described by the following lexical definitions: .. productionlist:: bytesliteral: `bytesprefix`(`shortbytes` | `longbytes`) - bytesprefix: "b" | "B" | "br" | "Br" | "bR" | "BR" + bytesprefix: "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB" shortbytes: "'" `shortbytesitem`* "'" | '"' `shortbytesitem`* '"' longbytes: "'''" `longbytesitem`* "'''" | '"""' `longbytesitem`* '"""' shortbytesitem: `shortbyteschar` | `bytesescapeseq` @@ -441,10 +441,24 @@ instance of the :class:`bytes` type instead of the :class:`str` type. They may only contain ASCII characters; bytes with a numeric value of 128 or greater must be expressed with escapes. +As of Python 3.3 it is possible again to prefix unicode strings with a +``u`` prefix to simplify maintenance of dual 2.x and 3.x codebases. + Both string and bytes literals may optionally be prefixed with a letter ``'r'`` or ``'R'``; such strings are called :dfn:`raw strings` and treat backslashes as literal characters. As a result, in string literals, ``'\U'`` and ``'\u'`` -escapes in raw strings are not treated specially. +escapes in raw strings are not treated specially. Given that Python 2.x's raw +unicode literals behave differently than Python 3.x's the ``'ur'`` syntax +is not supported. + + .. versionadded:: 3.3 + The ``'rb'`` prefix of raw bytes literals has been added as a synonym + of ``'br'``. + + .. versionadded:: 3.3 + Support for the unicode legacy literal (``u'value'``) was reintroduced + to simplify the maintenance of dual Python 2.x and 3.x codebases. + See :pep:`414` for more information. In triple-quoted strings, unescaped newlines and quotes are allowed (and are retained), except that three unescaped quotes in a row terminate the string. (A @@ -492,13 +506,13 @@ Escape sequences only recognized in string literals are: +-----------------+---------------------------------+-------+ | Escape Sequence | Meaning | Notes | +=================+=================================+=======+ -| ``\N{name}`` | Character named *name* in the | | +| ``\N{name}`` | Character named *name* in the | \(4) | | | Unicode database | | +-----------------+---------------------------------+-------+ -| ``\uxxxx`` | Character with 16-bit hex value | \(4) | +| ``\uxxxx`` | Character with 16-bit hex value | \(5) | | | *xxxx* | | +-----------------+---------------------------------+-------+ -| ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(5) | +| ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(6) | | | *xxxxxxxx* | | +-----------------+---------------------------------+-------+ @@ -516,13 +530,15 @@ Notes: with the given value. (4) + .. versionchanged:: 3.3 + Support for name aliases [#]_ has been added. + +(5) Individual code units which form parts of a surrogate pair can be encoded using this escape sequence. Exactly four hex digits are required. -(5) - Any Unicode character can be encoded this way, but characters outside the Basic - Multilingual Plane (BMP) will be encoded using a surrogate pair if Python is - compiled to use 16-bit code units (the default). Exactly eight hex digits +(6) + Any Unicode character can be encoded this way. Exactly eight hex digits are required. @@ -706,3 +722,8 @@ The following printing ASCII characters are not used in Python. Their occurrence outside string literals and comments is an unconditional error:: $ ? ` + + +.. rubric:: Footnotes + +.. [#] http://www.unicode.org/Public/6.1.0/ucd/NameAliases.txt diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 73183d5..81dc748 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -424,10 +424,10 @@ When :keyword:`return` passes control out of a :keyword:`try` statement with a :keyword:`finally` clause, that :keyword:`finally` clause is executed before really leaving the function. -In a generator function, the :keyword:`return` statement is not allowed to -include an :token:`expression_list`. In that context, a bare :keyword:`return` -indicates that the generator is done and will cause :exc:`StopIteration` to be -raised. +In a generator function, the :keyword:`return` statement indicates that the +generator is done and will cause :exc:`StopIteration` to be raised. The returned +value (if any) is used as an argument to construct :exc:`StopIteration` and +becomes the :attr:`StopIteration.value` attribute. .. _yield: @@ -449,6 +449,7 @@ The :keyword:`yield` statement is only used when defining a generator function, and is only used in the body of the generator function. Using a :keyword:`yield` statement in a function definition is sufficient to cause that definition to create a generator function instead of a normal function. + When a generator function is called, it returns an iterator known as a generator iterator, or more commonly, a generator. The body of the generator function is executed by calling the :func:`next` function on the generator repeatedly until @@ -468,14 +469,28 @@ resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:`close` method will be called, allowing any pending :keyword:`finally` clauses to execute. +When ``yield from <expr>`` is used, it treats the supplied expression as +a subiterator, producing values from it until the underlying iterator is +exhausted. + + .. versionchanged:: 3.3 + Added ``yield from <expr>`` to delegate control flow to a subiterator + +For full details of :keyword:`yield` semantics, refer to the :ref:`yieldexpr` +section. + .. seealso:: :pep:`0255` - Simple Generators The proposal for adding generators and the :keyword:`yield` statement to Python. :pep:`0342` - Coroutines via Enhanced Generators - The proposal that, among other generator enhancements, proposed allowing - :keyword:`yield` to appear inside a :keyword:`try` ... :keyword:`finally` block. + The proposal to enhance the API and syntax of generators, making them + usable as simple coroutines. + + :pep:`0380` - Syntax for Delegating to a Subgenerator + The proposal to introduce the :token:`yield_from` syntax, making delegation + to sub-generators easy. .. _raise: @@ -645,161 +660,98 @@ The :keyword:`import` statement relative_module: "."* `module` | "."+ name: `identifier` -Import statements are executed in two steps: (1) find a module, and initialize -it if necessary; (2) define a name or names in the local namespace (of the scope -where the :keyword:`import` statement occurs). The statement comes in two -forms differing on whether it uses the :keyword:`from` keyword. The first form -(without :keyword:`from`) repeats these steps for each identifier in the list. -The form with :keyword:`from` performs step (1) once, and then performs step -(2) repeatedly. For a reference implementation of step (1), see the -:mod:`importlib` module. +The basic import statement (no :keyword:`from` clause) is executed in two +steps: -.. index:: - single: package +#. find a module, loading and initializing it if necessary +#. define a name or names in the local namespace for the scope where + the :keyword:`import` statement occurs. -To understand how step (1) occurs, one must first understand how Python handles -hierarchical naming of modules. To help organize modules and provide a -hierarchy in naming, Python has a concept of packages. A package can contain -other packages and modules while modules cannot contain other modules or -packages. From a file system perspective, packages are directories and modules -are files. The original `specification for packages -<http://www.python.org/doc/essays/packages.html>`_ is still available to read, -although minor details have changed since the writing of that document. +When the statement contains multiple clauses (separated by +commas) the two steps are carried out separately for each clause, just +as though the clauses had been separated out into individiual import +statements. -.. index:: - single: sys.modules +The details of the first step, finding and loading modules is described in +greater detail in the section on the :ref:`import system <importsystem>`, +which also describes the various types of packages and modules that can +be imported, as well as all the hooks that can be used to customize +the import system. Note that failures in this step may indicate either +that the module could not be located, *or* that an error occurred while +initializing the module, which includes execution of the module's code. -Once the name of the module is known (unless otherwise specified, the term -"module" will refer to both packages and modules), searching -for the module or package can begin. The first place checked is -:data:`sys.modules`, the cache of all modules that have been imported -previously. If the module is found there then it is used in step (2) of import -unless ``None`` is found in :data:`sys.modules`, in which case -:exc:`ImportError` is raised. +If the requested module is retrieved successfully, it will be made +available in the local namespace in one of three ways: -.. index:: - single: sys.meta_path - single: finder - pair: finder; find_module - single: __path__ - -If the module is not found in the cache, then :data:`sys.meta_path` is searched -(the specification for :data:`sys.meta_path` can be found in :pep:`302`). -The object is a list of :term:`finder` objects which are queried in order as to -whether they know how to load the module by calling their :meth:`find_module` -method with the name of the module. If the module happens to be contained -within a package (as denoted by the existence of a dot in the name), then a -second argument to :meth:`find_module` is given as the value of the -:attr:`__path__` attribute from the parent package (everything up to the last -dot in the name of the module being imported). If a finder can find the module -it returns a :term:`loader` (discussed later) or returns ``None``. +* If the module name is followed by :keyword:`as`, then the name + following :keyword:`as` is bound directly to the imported module. +* If no other name is specified, and the module being imported is a top + level module, the module's name is bound in the local namespace as a + reference to the imported module +* If the module being imported is *not* a top level module, then the name + of the top level package that contains the module is bound in the local + namespace as a reference to the top level package. The imported module + must be accessed using its full qualified name rather than directly -.. index:: - single: sys.path_hooks - single: sys.path_importer_cache - single: sys.path - -If none of the finders on :data:`sys.meta_path` are able to find the module -then some implicitly defined finders are queried. Implementations of Python -vary in what implicit meta path finders are defined. The one they all do -define, though, is one that handles :data:`sys.path_hooks`, -:data:`sys.path_importer_cache`, and :data:`sys.path`. - -The implicit finder searches for the requested module in the "paths" specified -in one of two places ("paths" do not have to be file system paths). If the -module being imported is supposed to be contained within a package then the -second argument passed to :meth:`find_module`, :attr:`__path__` on the parent -package, is used as the source of paths. If the module is not contained in a -package then :data:`sys.path` is used as the source of paths. - -Once the source of paths is chosen it is iterated over to find a finder that -can handle that path. The dict at :data:`sys.path_importer_cache` caches -finders for paths and is checked for a finder. If the path does not have a -finder cached then :data:`sys.path_hooks` is searched by calling each object in -the list with a single argument of the path, returning a finder or raises -:exc:`ImportError`. If a finder is returned then it is cached in -:data:`sys.path_importer_cache` and then used for that path entry. If no finder -can be found but the path exists then a value of ``None`` is -stored in :data:`sys.path_importer_cache` to signify that an implicit, -file-based finder that handles modules stored as individual files should be -used for that path. If the path does not exist then a finder which always -returns ``None`` is placed in the cache for the path. .. index:: - single: loader - pair: loader; load_module - exception: ImportError - -If no finder can find the module then :exc:`ImportError` is raised. Otherwise -some finder returned a loader whose :meth:`load_module` method is called with -the name of the module to load (see :pep:`302` for the original definition of -loaders). A loader has several responsibilities to perform on a module it -loads. First, if the module already exists in :data:`sys.modules` (a -possibility if the loader is called outside of the import machinery) then it -is to use that module for initialization and not a new module. But if the -module does not exist in :data:`sys.modules` then it is to be added to that -dict before initialization begins. If an error occurs during loading of the -module and it was added to :data:`sys.modules` it is to be removed from the -dict. If an error occurs but the module was already in :data:`sys.modules` it -is left in the dict. + pair: name; binding + keyword: from + exception: ImportError -.. index:: - single: __name__ - single: __file__ - single: __path__ - single: __package__ - single: __loader__ - -The loader must set several attributes on the module. :data:`__name__` is to be -set to the name of the module. :data:`__file__` is to be the "path" to the file -unless the module is built-in (and thus listed in -:data:`sys.builtin_module_names`) in which case the attribute is not set. -If what is being imported is a package then :data:`__path__` is to be set to a -list of paths to be searched when looking for modules and packages contained -within the package being imported. :data:`__package__` is optional but should -be set to the name of package that contains the module or package (the empty -string is used for module not contained in a package). :data:`__loader__` is -also optional but should be set to the loader object that is loading the -module. +The :keyword:`from` form uses a slightly more complex process: -.. index:: - exception: ImportError +#. find the module specified in the :keyword:`from` clause loading and + initializing it if necessary; +#. for each of the identifiers specified in the :keyword:`import` clauses: -If an error occurs during loading then the loader raises :exc:`ImportError` if -some other exception is not already being propagated. Otherwise the loader -returns the module that was loaded and initialized. + #. check if the imported module has an attribute by that name + #. if not, attempt to import a submodule with that name and then + check the imported module again for that attribute + #. if the attribute is not found, :exc:`ImportError` is raised. + #. otherwise, a reference to that value is bound in the local namespace, + using the name in the :keyword:`as` clause if it is present, + otherwise using the attribute name -When step (1) finishes without raising an exception, step (2) can begin. +Examples:: -The first form of :keyword:`import` statement binds the module name in the local -namespace to the module object, and then goes on to import the next identifier, -if any. If the module name is followed by :keyword:`as`, the name following -:keyword:`as` is used as the local name for the module. + import foo # foo imported and bound locally + import foo.bar.baz # foo.bar.baz imported, foo bound locally + import foo.bar.baz as fbb # foo.bar.baz imported and bound as fbb + from foo.bar import baz # foo.bar.baz imported and bound as baz + from foo import attr # foo imported and foo.attr bound as attr -.. index:: - pair: name; binding - exception: ImportError +If the list of identifiers is replaced by a star (``'*'``), all public +names defined in the module are bound in the local namespace for the scope +where the :keyword:`import` statement occurs. -The :keyword:`from` form does not bind the module name: it goes through the list -of identifiers, looks each one of them up in the module found in step (1), and -binds the name in the local namespace to the object thus found. As with the -first form of :keyword:`import`, an alternate local name can be supplied by -specifying ":keyword:`as` localname". If a name is not found, -:exc:`ImportError` is raised. If the list of identifiers is replaced by a star -(``'*'``), all public names defined in the module are bound in the local -namespace of the :keyword:`import` statement. +.. index:: single: __all__ (optional module attribute) + +The *public names* defined by a module are determined by checking the module's +namespace for a variable named ``__all__``; if defined, it must be a sequence +of strings which are names defined or imported by that module. The names +given in ``__all__`` are all considered public and are required to exist. If +``__all__`` is not defined, the set of public names includes all names found +in the module's namespace which do not begin with an underscore character +(``'_'``). ``__all__`` should contain the entire public API. It is intended +to avoid accidentally exporting items that are not part of the API (such as +library modules which were imported and used within the module). + +The :keyword:`from` form with ``*`` may only occur in a module scope. +Attempting to use it in class or function definitions will raise a +:exc:`SyntaxError`. .. index:: single: __all__ (optional module attribute) The *public names* defined by a module are determined by checking the module's -namespace for a variable named ``__all__``; if defined, it must be a sequence of -strings which are names defined or imported by that module. The names given in -``__all__`` are all considered public and are required to exist. If ``__all__`` -is not defined, the set of public names includes all names found in the module's -namespace which do not begin with an underscore character (``'_'``). -``__all__`` should contain the entire public API. It is intended to avoid -accidentally exporting items that are not part of the API (such as library -modules which were imported and used within the module). +namespace for a variable named ``__all__``; if defined, it must be a sequence +of strings which are names defined or imported by that module. The names +given in ``__all__`` are all considered public and are required to exist. If +``__all__`` is not defined, the set of public names includes all names found +in the module's namespace which do not begin with an underscore character +(``'_'``). ``__all__`` should contain the entire public API. It is intended +to avoid accidentally exporting items that are not part of the API (such as +library modules which were imported and used within the module). The :keyword:`from` form with ``*`` may only occur in a module scope. The wild card form of import --- ``import *`` --- is only allowed at the module level. |