diff options
| -rw-r--r-- | Doc/library/importlib.rst | 218 | ||||
| -rw-r--r-- | Lib/importlib/abc.py | 270 | ||||
| -rw-r--r-- | Lib/importlib/test/source/test_abc_loader.py | 400 | ||||
| -rw-r--r-- | Misc/NEWS | 3 |
4 files changed, 764 insertions, 127 deletions
diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index e89582b..300653a 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -18,12 +18,12 @@ implementation of the :keyword:`import` statement (and thus, by extension, the :func:`__import__` function) in Python source code. This provides an implementation of :keyword:`import` which is portable to any Python interpreter. This also provides a reference implementation which is easier to -comprehend than one in a programming language other than Python. +comprehend than one implemented in a programming language other than Python. -Two, the components to implement :keyword:`import` can be exposed in this +Two, the components to implement :keyword:`import` are exposed in this package, making it easier for users to create their own custom objects (known generically as an :term:`importer`) to participate in the import process. -Details on providing custom importers can be found in :pep:`302`. +Details on custom importers can be found in :pep:`302`. .. seealso:: @@ -37,7 +37,7 @@ Details on providing custom importers can be found in :pep:`302`. The :func:`.__import__` function The built-in function for which the :keyword:`import` statement is - syntactic sugar. + syntactic sugar for. :pep:`235` Import on Case-Insensitive Platforms @@ -46,7 +46,7 @@ Details on providing custom importers can be found in :pep:`302`. Defining Python Source Code Encodings :pep:`302` - New Import Hooks. + New Import Hooks :pep:`328` Imports: Multi-Line and Absolute/Relative @@ -66,8 +66,7 @@ Functions .. function:: __import__(name, globals={}, locals={}, fromlist=list(), level=0) - An implementation of the built-in :func:`__import__` function. See the - built-in function's documentation for usage instructions. + An implementation of the built-in :func:`__import__` function. .. function:: import_module(name, package=None) @@ -213,22 +212,108 @@ are also provided to help in implementing the core ABCs. .. method:: get_filename(fullname) - An abstract method that is to return the value for :attr:`__file__` for + An abstract method that is to return the value of :attr:`__file__` for the specified module. If no path is available, :exc:`ImportError` is raised. + If source code is available, then the method should return the path to + the source file, regardless of whether a bytecode was used to load the + module. + + +.. class:: SourceLoader + + An abstract base class for implementing source (and optionally bytecode) + file loading. The class inherits from both :class:`ResourceLoader` and + :class:`ExecutionLoader`, requiring the implementation of: + + * :meth:`ResourceLoader.get_data` + * :meth:`ExecutionLoader.get_filename` + Implement to only return the path to the source file; sourceless + loading is not supported. + + The abstract methods defined by this class are to add optional bytecode + file support. Not implementing these optional methods causes the loader to + only work with source code. Implementing the methods allows the loader to + work with source *and* bytecode files; it does not allow for *sourceless* + loading where only bytecode is provided. Bytecode files are an + optimization to speed up loading by removing the parsing step of Python's + compiler, and so no bytecode-specific API is exposed. + + .. method:: path_mtime(self, path) + + Optional abstract method which returns the modification time for the + specified path. + + .. method:: set_data(self, path, data) + + Optional abstract method which writes the specified bytes to a file + path. + + .. method:: get_code(self, fullname) + + Concrete implementation of :meth:`InspectLoader.get_code`. + + .. method:: load_module(self, fullname) + + Concrete implementation of :meth:`Loader.load_module`. + + .. method:: get_source(self, fullname) + + Concrete implementation of :meth:`InspectLoader.get_source`. + + .. method:: is_package(self, fullname) + + Concrete implementation of :meth:`InspectLoader.is_package`. A module + is determined to be a package if its file path is a file named + ``__init__`` when the file extension is removed. + .. class:: PyLoader An abstract base class inheriting from - :class:`importlib.abc.ExecutionLoader` and - :class:`importlib.abc.ResourceLoader` designed to ease the loading of + :class:`ExecutionLoader` and + :class:`ResourceLoader` designed to ease the loading of Python source modules (bytecode is not handled; see - :class:`importlib.abc.PyPycLoader` for a source/bytecode ABC). A subclass + :class:`SourceLoader` for a source/bytecode ABC). A subclass implementing this ABC will only need to worry about exposing how the source code is stored; all other details for loading Python source code will be handled by the concrete implementations of key methods. + .. deprecated:: 3.2 + This class has been deprecated in favor of :class:`SourceLoader` and is + slated for removal in Python 3.4. See below for how to create a + subclass that is compatbile with Python 3.1 onwards. + + If compatibility with Python 3.1 is required, then use the following idiom + to implement a subclass that will work with Python 3.1 onwards (make sure + to implement :meth:`ExecutionLoader.get_filename`):: + + try: + from importlib.abc import SourceLoader + except ImportError: + from importlib.abc import PyLoader as SourceLoader + + + class CustomLoader(SourceLoader): + def get_filename(self, fullname): + """Return the path to the source file.""" + # Implement ... + + def source_path(self, fullname): + """Implement source_path in terms of get_filename.""" + try: + return self.get_filename(fullname) + except ImportError: + return None + + def is_package(self, fullname): + """Implement is_package by looking for an __init__ file + name as returned by get_filename.""" + filename = os.path.basename(self.get_filename(fullname)) + return os.path.splitext(filename)[0] == '__init__' + + .. method:: source_path(fullname) An abstract method that returns the path to the source code for a @@ -270,10 +355,18 @@ are also provided to help in implementing the core ABCs. .. class:: PyPycLoader - An abstract base class inheriting from :class:`importlib.abc.PyLoader`. + An abstract base class inheriting from :class:`PyLoader`. This ABC is meant to help in creating loaders that support both Python source and bytecode. + .. deprecated:: 3.2 + This class has been deprecated in favor of :class:`SourceLoader` and to + properly support :pep:`3147`. If compatibility is required with + Python 3.1, implement both :class:`SourceLoader` and :class:`PyLoader`; + instructions on how to do so are included in the documentation for + :class:`PyLoader`. Do note that this solution will not support + sourceless/bytecode-only loading; only source *and* bytecode loading. + .. method:: source_mtime(fullname) An abstract method which returns the modification time for the source @@ -292,8 +385,8 @@ are also provided to help in implementing the core ABCs. .. method:: get_filename(fullname) A concrete implementation of - :meth:`importlib.abc.ExecutionLoader.get_filename` that relies on - :meth:`importlib.abc.PyLoader.source_path` and :meth:`bytecode_path`. + :meth:`ExecutionLoader.get_filename` that relies on + :meth:`PyLoader.source_path` and :meth:`bytecode_path`. If :meth:`source_path` returns a path, then that value is returned. Else if :meth:`bytecode_path` returns a path, that path will be returned. If a path is not available from both methods, @@ -420,100 +513,3 @@ an :term:`importer`. attribute to be used at the global level of the module during initialization. - -Example -------- - -Below is an example meta path importer that uses a dict for back-end storage -for source code. While not an optimal solution -- manipulations of -:attr:`__path__` on packages does not influence import -- it does illustrate -what little is required to implement an importer. - -.. testcode:: - - """An importer where source is stored in a dict.""" - from importlib import abc - - - class DictImporter(abc.Finder, abc.PyLoader): - - """A meta path importer that stores source code in a dict. - - The keys are the module names -- packages must end in ``.__init__``. - The values must be something that can be passed to 'bytes'. - - """ - - def __init__(self, memory): - """Store the dict.""" - self.memory = memory - - def contains(self, name): - """See if a module or package is in the dict.""" - if name in self.memory: - return name - package_name = '{}.__init__'.format(name) - if package_name in self.memory: - return package_name - return False - - __contains__ = contains # Convenience. - - def find_module(self, fullname, path=None): - """Find the module in the dict.""" - if fullname in self: - return self - return None - - def source_path(self, fullname): - """Return the module name if the module is in the dict.""" - if not fullname in self: - raise ImportError - return fullname - - def get_data(self, path): - """Return the bytes for the source. - - The value found in the dict is passed through 'bytes' before being - returned. - - """ - name = self.contains(path) - if not name: - raise IOError - return bytes(self.memory[name]) - - def is_package(self, fullname): - """Tell if module is a package based on whether the dict contains the - name with ``.__init__`` appended to it.""" - if fullname not in self: - raise ImportError - if fullname in self.memory: - return False - # If name is in this importer but not as it is then it must end in - # ``__init__``. - else: - return True - -.. testcode:: - :hide: - - import importlib - import sys - - - # Build the dict; keys of name, value of __package__. - names = {'_top_level': '', '_pkg.__init__': '_pkg', '_pkg.mod': '_pkg'} - source = {name: "name = {!r}".format(name).encode() for name in names} - - # Register the meta path importer. - importer = DictImporter(source) - sys.meta_path.append(importer) - - # Sanity check. - for name in names: - module = importlib.import_module(name) - assert module.__name__ == name - assert getattr(module, 'name') == name - assert module.__loader__ is importer - assert module.__package__ == names[name] diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py index d6f4520..6a688d1 100644 --- a/Lib/importlib/abc.py +++ b/Lib/importlib/abc.py @@ -1,8 +1,16 @@ """Abstract base classes related to import.""" from . import _bootstrap from . import machinery +from . import util import abc +import imp +import io +import marshal +import os.path +import sys +import tokenize import types +import warnings class Loader(metaclass=abc.ABCMeta): @@ -58,19 +66,19 @@ class InspectLoader(Loader): def is_package(self, fullname:str) -> bool: """Abstract method which when implemented should return whether the module is a package.""" - return NotImplementedError + raise NotImplementedError @abc.abstractmethod def get_code(self, fullname:str) -> types.CodeType: """Abstract method which when implemented should return the code object for the module""" - return NotImplementedError + raise NotImplementedError @abc.abstractmethod def get_source(self, fullname:str) -> str: """Abstract method which should return the source code for the module.""" - return NotImplementedError + raise NotImplementedError InspectLoader.register(machinery.BuiltinImporter) InspectLoader.register(machinery.FrozenImporter) @@ -92,33 +100,273 @@ class ExecutionLoader(InspectLoader): raise NotImplementedError -class PyLoader(_bootstrap.PyLoader, ResourceLoader, ExecutionLoader): +class SourceLoader(ResourceLoader, ExecutionLoader): - """Abstract base class to assist in loading source code by requiring only - back-end storage methods to be implemented. + """Abstract base class for loading source code (and optionally any + corresponding bytecode). - The methods get_code, get_source, and load_module are implemented for the - user. + To support loading from source code, the abstractmethods inherited from + ResourceLoader and ExecutionLoader need to be implemented. To also support + loading from bytecode, the optional methods specified directly by this ABC + is required. + + Inherited abstractmethods not implemented in this ABC: + + * ResourceLoader.get_data + * ExecutionLoader.get_filename + + """ + + def path_mtime(self, path:str) -> int: + """Optional method that returns the modification time for the specified + path. + + Implementing this method allows the loader to read bytecode files. + + """ + raise NotImplementedError + + def set_data(self, path:str, data:bytes) -> None: + """Optional method which writes data to a file path. + + Implementing this method allows for the writing of bytecode files. + + """ + raise NotImplementedError + + def is_package(self, fullname): + """Concrete implementation of InspectLoader.is_package by checking if + the path returned by get_filename has a filename of '__init__.py'.""" + filename = os.path.basename(self.get_filename(fullname)) + return os.path.splitext(filename)[0] == '__init__' + + def get_source(self, fullname): + """Concrete implementation of InspectLoader.get_source.""" + path = self.get_filename(fullname) + try: + source_bytes = self.get_data(path) + except IOError: + raise ImportError("source not available through get_data()") + encoding = tokenize.detect_encoding(io.BytesIO(source_bytes).readline) + return source_bytes.decode(encoding[0]) + + def get_code(self, fullname): + """Concrete implementation of InspectLoader.get_code. + + Reading of bytecode requires path_mtime to be implemented. To write + bytecode, set_data must also be implemented. + + """ + source_path = self.get_filename(fullname) + bytecode_path = imp.cache_from_source(source_path) + source_mtime = None + if bytecode_path is not None: + try: + source_mtime = self.path_mtime(source_path) + except NotImplementedError: + pass + else: + try: + data = self.get_data(bytecode_path) + except IOError: + pass + else: + magic = data[:4] + raw_timestamp = data[4:8] + if (len(magic) == 4 and len(raw_timestamp) == 4 and + magic == imp.get_magic() and + marshal._r_long(raw_timestamp) == source_mtime): + return marshal.loads(data[8:]) + source_bytes = self.get_data(source_path) + code_object = compile(source_bytes, source_path, 'exec', + dont_inherit=True) + if (not sys.dont_write_bytecode and bytecode_path is not None and + source_mtime is not None): + # If e.g. Jython ever implements imp.cache_from_source to have + # their own cached file format, this block of code will most likely + # throw an exception. + data = bytearray(imp.get_magic()) + data.extend(marshal._w_long(source_mtime)) + data.extend(marshal.dumps(code_object)) + try: + self.set_data(bytecode_path, data) + except (NotImplementedError, IOError): + pass + return code_object + + @util.module_for_loader + def load_module(self, module): + """Concrete implementation of Loader.load_module. + + Requires ExecutionLoader.get_filename and ResourceLoader.get_data to be + implemented to load source code. Use of bytecode is dictated by whether + get_code uses/writes bytecode. + + """ + name = module.__name__ + code_object = self.get_code(name) + module.__file__ = self.get_filename(name) + module.__cached__ = imp.cache_from_source(module.__file__) + module.__package__ = name + is_package = self.is_package(name) + if is_package: + module.__path__ = [os.path.dirname(module.__file__)] + else: + module.__package__ = module.__package__.rpartition('.')[0] + module.__loader__ = self + exec(code_object, module.__dict__) + return module + + +class PyLoader(SourceLoader): + + """Implement the deprecated PyLoader ABC in terms of SourceLoader. + + This class has been deprecated! It is slated for removal in Python 3.4. + If compatibility with Python 3.1 is not needed then implement the + SourceLoader ABC instead of this class. If Python 3.1 compatibility is + needed, then use the following idiom to have a single class that is + compatible with Python 3.1 onwards:: + + try: + from importlib.abc import SourceLoader + except ImportError: + from importlib.abc import PyLoader as SourceLoader + + + class CustomLoader(SourceLoader): + def get_filename(self, fullname): + # Implement ... + + def source_path(self, fullname): + '''Implement source_path in terms of get_filename.''' + try: + return self.get_filename(fullname) + except ImportError: + return None + + def is_package(self, fullname): + filename = os.path.basename(self.get_filename(fullname)) + return os.path.splitext(filename)[0] == '__init__' """ @abc.abstractmethod + def is_package(self, fullname): + raise NotImplementedError + + @abc.abstractmethod def source_path(self, fullname:str) -> object: """Abstract method which when implemented should return the path to the - sourced code for the module.""" + source code for the module.""" raise NotImplementedError + def get_filename(self, fullname): + """Implement get_filename in terms of source_path. + + As get_filename should only return a source file path there is no + chance of the path not existing but loading still being possible, so + ImportError should propagate instead of being turned into returning + None. -class PyPycLoader(_bootstrap.PyPycLoader, PyLoader): + """ + warnings.warn("importlib.abc.PyLoader is deprecated and is " + "slated for removal in Python 3.4; " + "use SourceLoader instead. " + "See the importlib documentation on how to be " + "compatible with Python 3.1 onwards.", + PendingDeprecationWarning) + path = self.source_path(fullname) + if path is None: + raise ImportError + else: + return path + +PyLoader.register(_bootstrap.PyLoader) + + +class PyPycLoader(PyLoader): """Abstract base class to assist in loading source and bytecode by requiring only back-end storage methods to be implemented. + This class has been deprecated! Removal is slated for Python 3.4. Implement + the SourceLoader ABC instead. If Python 3.1 compatibility is needed, see + PyLoader. + The methods get_code, get_source, and load_module are implemented for the user. """ + def get_filename(self, fullname): + """Return the source or bytecode file path.""" + path = self.source_path(fullname) + if path is not None: + return path + path = self.bytecode_path(fullname) + if path is not None: + return path + raise ImportError("no source or bytecode path available for " + "{0!r}".format(fullname)) + + def get_code(self, fullname): + """Get a code object from source or bytecode.""" + warnings.warn("importlib.abc.PyPycLoader is deprecated and slated for " + "removal in Python 3.4; use SourceLoader instead. " + "If Python 3.1 compatibility is required, see the " + "latest documentation for PyLoader.", + PendingDeprecationWarning) + source_timestamp = self.source_mtime(fullname) + # Try to use bytecode if it is available. + bytecode_path = self.bytecode_path(fullname) + if bytecode_path: + data = self.get_data(bytecode_path) + try: + magic = data[:4] + if len(magic) < 4: + raise ImportError("bad magic number in {}".format(fullname)) + raw_timestamp = data[4:8] + if len(raw_timestamp) < 4: + raise EOFError("bad timestamp in {}".format(fullname)) + pyc_timestamp = marshal._r_long(raw_timestamp) + bytecode = data[8:] + # Verify that the magic number is valid. + if imp.get_magic() != magic: + raise ImportError("bad magic number in {}".format(fullname)) + # Verify that the bytecode is not stale (only matters when + # there is source to fall back on. + if source_timestamp: + if pyc_timestamp < source_timestamp: + raise ImportError("bytecode is stale") + except (ImportError, EOFError): + # If source is available give it a shot. + if source_timestamp is not None: + pass + else: + raise + else: + # Bytecode seems fine, so try to use it. + return marshal.loads(bytecode) + elif source_timestamp is None: + raise ImportError("no source or bytecode available to create code " + "object for {0!r}".format(fullname)) + # Use the source. + source_path = self.source_path(fullname) + if source_path is None: + message = "a source path must exist to load {0}".format(fullname) + raise ImportError(message) + source = self.get_data(source_path) + code_object = compile(source, source_path, 'exec', dont_inherit=True) + # Generate bytecode and write it out. + if not sys.dont_write_bytecode: + data = bytearray(imp.get_magic()) + data.extend(marshal._w_long(source_timestamp)) + data.extend(marshal.dumps(code_object)) + self.write_bytecode(fullname, data) + return code_object + + @abc.abstractmethod def source_mtime(self, fullname:str) -> int: """Abstract method which when implemented should return the @@ -137,3 +385,5 @@ class PyPycLoader(_bootstrap.PyPycLoader, PyLoader): bytecode for the module, returning a boolean representing whether the bytecode was written or not.""" raise NotImplementedError + +PyPycLoader.register(_bootstrap.PyPycLoader) diff --git a/Lib/importlib/test/source/test_abc_loader.py b/Lib/importlib/test/source/test_abc_loader.py index 8c69cfd..69cc9fd 100644 --- a/Lib/importlib/test/source/test_abc_loader.py +++ b/Lib/importlib/test/source/test_abc_loader.py @@ -1,14 +1,67 @@ import importlib from importlib import abc + from .. import abc as testing_abc from .. import util from . import util as source_util + import imp +import inspect import marshal import os import sys import types import unittest +import warnings + + +class SourceOnlyLoaderMock(abc.SourceLoader): + + # Globals that should be defined for all modules. + source = (b"_ = '::'.join([__name__, __file__, __cached__, __package__, " + b"repr(__loader__)])") + + def __init__(self, path): + self.path = path + + def get_data(self, path): + assert self.path == path + return self.source + + def get_filename(self, fullname): + return self.path + + +class SourceLoaderMock(SourceOnlyLoaderMock): + + source_mtime = 1 + + def __init__(self, path, magic=imp.get_magic()): + super().__init__(path) + self.bytecode_path = imp.cache_from_source(self.path) + data = bytearray(magic) + data.extend(marshal._w_long(self.source_mtime)) + code_object = compile(self.source, self.path, 'exec', + dont_inherit=True) + data.extend(marshal.dumps(code_object)) + self.bytecode = bytes(data) + self.written = {} + + def get_data(self, path): + if path == self.path: + return super().get_data(path) + elif path == self.bytecode_path: + return self.bytecode + else: + raise IOError + + def path_mtime(self, path): + assert path == self.path + return self.source_mtime + + def set_data(self, path, data): + self.written[path] = bytes(data) + return path == self.bytecode_path class PyLoaderMock(abc.PyLoader): @@ -33,17 +86,42 @@ class PyLoaderMock(abc.PyLoader): return self.source def is_package(self, name): + filename = os.path.basename(self.get_filename(name)) + return os.path.splitext(filename)[0] == '__init__' + + def source_path(self, name): try: - return '__init__' in self.module_paths[name] + return self.module_paths[name] except KeyError: raise ImportError - def source_path(self, name): + def get_filename(self, name): + """Silence deprecation warning.""" + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + path = super().get_filename(name) + assert len(w) == 1 + assert issubclass(w[0].category, PendingDeprecationWarning) + return path + + +class PyLoaderCompatMock(PyLoaderMock): + + """Mock that matches what is suggested to have a loader that is compatible + from Python 3.1 onwards.""" + + def get_filename(self, fullname): try: - return self.module_paths[name] + return self.module_paths[fullname] except KeyError: raise ImportError + def source_path(self, fullname): + try: + return self.get_filename(fullname) + except ImportError: + return None + class PyPycLoaderMock(abc.PyPycLoader, PyLoaderMock): @@ -114,6 +192,13 @@ class PyPycLoaderMock(abc.PyPycLoader, PyLoaderMock): except TypeError: return '__init__' in self.bytecode_to_path[name] + def get_code(self, name): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + code_object = super().get_code(name) + assert len(w) == 1 + assert issubclass(w[0].category, PendingDeprecationWarning) + return code_object class PyLoaderTests(testing_abc.LoaderTests): @@ -200,6 +285,14 @@ class PyLoaderTests(testing_abc.LoaderTests): return mock +class PyLoaderCompatTests(PyLoaderTests): + + """Test that the suggested code to make a loader that is compatible from + Python 3.1 forward works.""" + + mocker = PyLoaderCompatMock + + class PyLoaderInterfaceTests(unittest.TestCase): """Tests for importlib.abc.PyLoader to make sure that when source_path() @@ -413,7 +506,7 @@ class BadBytecodeFailureTests(unittest.TestCase): def test_bad_bytecode(self): # Malformed code object bytecode should lead to a ValueError. name = 'mod' - bc = {name: {'path': os.path.join('path', 'to', 'mod'), 'bc': b'XXX'}} + bc = {name: {'path': os.path.join('path', 'to', 'mod'), 'bc': b'NNN'}} mock = PyPycLoaderMock({name: None}, bc) with util.uncache(name), self.assertRaises(ValueError): mock.load_module(name) @@ -465,12 +558,307 @@ class MissingPathsTests(unittest.TestCase): mock.load_module(name) +class SourceLoaderTestHarness(unittest.TestCase): + + def setUp(self, *, is_package=True, **kwargs): + self.package = 'pkg' + if is_package: + self.path = os.path.join(self.package, '__init__.py') + self.name = self.package + else: + module_name = 'mod' + self.path = os.path.join(self.package, '.'.join(['mod', 'py'])) + self.name = '.'.join([self.package, module_name]) + self.cached = imp.cache_from_source(self.path) + self.loader = self.loader_mock(self.path, **kwargs) + + def verify_module(self, module): + self.assertEqual(module.__name__, self.name) + self.assertEqual(module.__file__, self.path) + self.assertEqual(module.__cached__, self.cached) + self.assertEqual(module.__package__, self.package) + self.assertEqual(module.__loader__, self.loader) + values = module._.split('::') + self.assertEqual(values[0], self.name) + self.assertEqual(values[1], self.path) + self.assertEqual(values[2], self.cached) + self.assertEqual(values[3], self.package) + self.assertEqual(values[4], repr(self.loader)) + + def verify_code(self, code_object): + module = imp.new_module(self.name) + module.__file__ = self.path + module.__cached__ = self.cached + module.__package__ = self.package + module.__loader__ = self.loader + module.__path__ = [] + exec(code_object, module.__dict__) + self.verify_module(module) + + +class SourceOnlyLoaderTests(SourceLoaderTestHarness): + + """Test importlib.abc.SourceLoader for source-only loading. + + Reload testing is subsumed by the tests for + importlib.util.module_for_loader. + + """ + + loader_mock = SourceOnlyLoaderMock + + def test_get_source(self): + # Verify the source code is returned as a string. + # If an IOError is raised by get_data then raise ImportError. + expected_source = self.loader.source.decode('utf-8') + self.assertEqual(self.loader.get_source(self.name), expected_source) + def raise_IOError(path): + raise IOError + self.loader.get_data = raise_IOError + with self.assertRaises(ImportError): + self.loader.get_source(self.name) + + def test_is_package(self): + # Properly detect when loading a package. + self.setUp(is_package=True) + self.assertTrue(self.loader.is_package(self.name)) + self.setUp(is_package=False) + self.assertFalse(self.loader.is_package(self.name)) + + def test_get_code(self): + # Verify the code object is created. + code_object = self.loader.get_code(self.name) + self.verify_code(code_object) + + def test_load_module(self): + # Loading a module should set __name__, __loader__, __package__, + # __path__ (for packages), __file__, and __cached__. + # The module should also be put into sys.modules. + with util.uncache(self.name): + module = self.loader.load_module(self.name) + self.verify_module(module) + self.assertEqual(module.__path__, [os.path.dirname(self.path)]) + self.assertTrue(self.name in sys.modules) + + def test_package_settings(self): + # __package__ needs to be set, while __path__ is set on if the module + # is a package. + # Testing the values for a package are covered by test_load_module. + self.setUp(is_package=False) + with util.uncache(self.name): + module = self.loader.load_module(self.name) + self.verify_module(module) + self.assertTrue(not hasattr(module, '__path__')) + + def test_get_source_encoding(self): + # Source is considered encoded in UTF-8 by default unless otherwise + # specified by an encoding line. + source = "_ = 'ü'" + self.loader.source = source.encode('utf-8') + returned_source = self.loader.get_source(self.name) + self.assertEqual(returned_source, source) + source = "# coding: latin-1\n_ = ü" + self.loader.source = source.encode('latin-1') + returned_source = self.loader.get_source(self.name) + self.assertEqual(returned_source, source) + + +@unittest.skipIf(sys.dont_write_bytecode, "sys.dont_write_bytecode is true") +class SourceLoaderBytecodeTests(SourceLoaderTestHarness): + + """Test importlib.abc.SourceLoader's use of bytecode. + + Source-only testing handled by SourceOnlyLoaderTests. + + """ + + loader_mock = SourceLoaderMock + + def verify_code(self, code_object, *, bytecode_written=False): + super().verify_code(code_object) + if bytecode_written: + self.assertIn(self.cached, self.loader.written) + data = bytearray(imp.get_magic()) + data.extend(marshal._w_long(self.loader.source_mtime)) + data.extend(marshal.dumps(code_object)) + self.assertEqual(self.loader.written[self.cached], bytes(data)) + + def test_code_with_everything(self): + # When everything should work. + code_object = self.loader.get_code(self.name) + self.verify_code(code_object) + + def test_no_bytecode(self): + # If no bytecode exists then move on to the source. + self.loader.bytecode_path = "<does not exist>" + # Sanity check + with self.assertRaises(IOError): + bytecode_path = imp.cache_from_source(self.path) + self.loader.get_data(bytecode_path) + code_object = self.loader.get_code(self.name) + self.verify_code(code_object, bytecode_written=True) + + def test_code_bad_timestamp(self): + # Bytecode is only used when the timestamp matches the source EXACTLY. + for source_mtime in (0, 2): + assert source_mtime != self.loader.source_mtime + original = self.loader.source_mtime + self.loader.source_mtime = source_mtime + # If bytecode is used then EOFError would be raised by marshal. + self.loader.bytecode = self.loader.bytecode[8:] + code_object = self.loader.get_code(self.name) + self.verify_code(code_object, bytecode_written=True) + self.loader.source_mtime = original + + def test_code_bad_magic(self): + # Skip over bytecode with a bad magic number. + self.setUp(magic=b'0000') + # If bytecode is used then EOFError would be raised by marshal. + self.loader.bytecode = self.loader.bytecode[8:] + code_object = self.loader.get_code(self.name) + self.verify_code(code_object, bytecode_written=True) + + def test_dont_write_bytecode(self): + # Bytecode is not written if sys.dont_write_bytecode is true. + # Can assume it is false already thanks to the skipIf class decorator. + try: + sys.dont_write_bytecode = True + self.loader.bytecode_path = "<does not exist>" + code_object = self.loader.get_code(self.name) + self.assertNotIn(self.cached, self.loader.written) + finally: + sys.dont_write_bytecode = False + + def test_no_set_data(self): + # If set_data is not defined, one can still read bytecode. + self.setUp(magic=b'0000') + original_set_data = self.loader.__class__.set_data + try: + del self.loader.__class__.set_data + code_object = self.loader.get_code(self.name) + self.verify_code(code_object) + finally: + self.loader.__class__.set_data = original_set_data + + def test_set_data_raises_exceptions(self): + # Raising NotImplementedError or IOError is okay for set_data. + def raise_exception(exc): + def closure(*args, **kwargs): + raise exc + return closure + + self.setUp(magic=b'0000') + for exc in (NotImplementedError, IOError): + self.loader.set_data = raise_exception(exc) + code_object = self.loader.get_code(self.name) + self.verify_code(code_object) + +class AbstractMethodImplTests(unittest.TestCase): + + """Test the concrete abstractmethod implementations.""" + + class Loader(abc.Loader): + def load_module(self, fullname): + super().load_module(fullname) + + class Finder(abc.Finder): + def find_module(self, _): + super().find_module(_) + + class ResourceLoader(Loader, abc.ResourceLoader): + def get_data(self, _): + super().get_data(_) + + class InspectLoader(Loader, abc.InspectLoader): + def is_package(self, _): + super().is_package(_) + + def get_code(self, _): + super().get_code(_) + + def get_source(self, _): + super().get_source(_) + + class ExecutionLoader(InspectLoader, abc.ExecutionLoader): + def get_filename(self, _): + super().get_filename(_) + + class SourceLoader(ResourceLoader, ExecutionLoader, abc.SourceLoader): + pass + + class PyLoader(ResourceLoader, InspectLoader, abc.PyLoader): + def source_path(self, _): + super().source_path(_) + + class PyPycLoader(PyLoader, abc.PyPycLoader): + def bytecode_path(self, _): + super().bytecode_path(_) + + def source_mtime(self, _): + super().source_mtime(_) + + def write_bytecode(self, _, _2): + super().write_bytecode(_, _2) + + def raises_NotImplementedError(self, ins, *args): + for method_name in args: + method = getattr(ins, method_name) + arg_count = len(inspect.getfullargspec(method)[0]) - 1 + args = [''] * arg_count + try: + method(*args) + except NotImplementedError: + pass + else: + msg = "{}.{} did not raise NotImplementedError" + self.fail(msg.format(ins.__class__.__name__, method_name)) + + def test_Loader(self): + self.raises_NotImplementedError(self.Loader(), 'load_module') + + def test_Finder(self): + self.raises_NotImplementedError(self.Finder(), 'find_module') + + def test_ResourceLoader(self): + self.raises_NotImplementedError(self.ResourceLoader(), 'load_module', + 'get_data') + + def test_InspectLoader(self): + self.raises_NotImplementedError(self.InspectLoader(), 'load_module', + 'is_package', 'get_code', 'get_source') + + def test_ExecutionLoader(self): + self.raises_NotImplementedError(self.ExecutionLoader(), 'load_module', + 'is_package', 'get_code', 'get_source', + 'get_filename') + + def test_SourceLoader(self): + ins = self.SourceLoader() + # Required abstractmethods. + self.raises_NotImplementedError(ins, 'get_filename', 'get_data') + # Optional abstractmethods. + self.raises_NotImplementedError(ins,'path_mtime', 'set_data') + + def test_PyLoader(self): + self.raises_NotImplementedError(self.PyLoader(), 'source_path', + 'get_data', 'is_package') + + def test_PyPycLoader(self): + self.raises_NotImplementedError(self.PyPycLoader(), 'source_path', + 'source_mtime', 'bytecode_path', + 'write_bytecode') + + def test_main(): from test.support import run_unittest - run_unittest(PyLoaderTests, PyLoaderInterfaceTests, PyLoaderGetSourceTests, + run_unittest(PyLoaderTests, PyLoaderCompatTests, + PyLoaderInterfaceTests, PyLoaderGetSourceTests, PyPycLoaderTests, PyPycLoaderInterfaceTests, SkipWritingBytecodeTests, RegeneratedBytecodeTests, - BadBytecodeFailureTests, MissingPathsTests) + BadBytecodeFailureTests, MissingPathsTests, + SourceOnlyLoaderTests, + SourceLoaderBytecodeTests, + AbstractMethodImplTests) if __name__ == '__main__': @@ -456,6 +456,9 @@ C-API Library ------- +- Implement importlib.abc.SourceLoader and deprecate PyLoader and PyPycLoader + for removal in Python 3.4. + - Issue #9064: pdb's "up" and "down" commands now accept an optional argument. - Issue #9018: os.path.normcase() now raises a TypeError if the argument is |