From 17bd792cd3efa13c3fd53bc5c8b81768efa748a9 Mon Sep 17 00:00:00 2001 From: Antoine Pitrou Date: Sat, 3 Dec 2011 22:56:02 +0100 Subject: Reformulate answer, and remove off-topic discussion of bytecode in a question about compiling to C. --- Doc/faq/design.rst | 49 +++++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 32 deletions(-) diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 02417b1..9e6bebc 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -380,11 +380,24 @@ is exactly the same type of object that a lambda form yields) is assigned! Can Python be compiled to machine code, C or some other language? ----------------------------------------------------------------- -Not easily. Python's high level data types, dynamic typing of objects and +Practical answer: + +`Cython `_ and `Pyrex `_ +compile a modified version of Python with optional annotations into C +extensions. `Weave `_ makes it easy to +intermingle Python and C code in various ways to increase performance. +`Nuitka `_ is an up-and-coming compiler of Python +into C++ code, aiming to support the full Python language. + +Theoretical answer: + + .. XXX not sure what to make of this + +Not trivially. Python's high level data types, dynamic typing of objects and run-time invocation of the interpreter (using :func:`eval` or :func:`exec`) -together mean that a "compiled" Python program would probably consist mostly of -calls into the Python run-time system, even for seemingly simple operations like -``x+1``. +together mean that a naïvely "compiled" Python program would probably consist +mostly of calls into the Python run-time system, even for seemingly simple +operations like ``x+1``. Several projects described in the Python newsgroup or at past `Python conferences `_ have shown that this @@ -395,34 +408,6 @@ speedups of 1000x are feasible for small demo programs. See the proceedings from the `1997 Python conference `_ for more information.) -Internally, Python source code is always translated into a bytecode -representation, and this bytecode is then executed by the Python virtual -machine. In order to avoid the overhead of repeatedly parsing and translating -modules that rarely change, this byte code is written into a file whose name -ends in ".pyc" whenever a module is parsed. When the corresponding .py file is -changed, it is parsed and translated again and the .pyc file is rewritten. - -There is no performance difference once the .pyc file has been loaded, as the -bytecode read from the .pyc file is exactly the same as the bytecode created by -direct translation. The only difference is that loading code from a .pyc file -is faster than parsing and translating a .py file, so the presence of -precompiled .pyc files improves the start-up time of Python scripts. If -desired, the Lib/compileall.py module can be used to create valid .pyc files for -a given set of modules. - -Note that the main script executed by Python, even if its filename ends in .py, -is not compiled to a .pyc file. It is compiled to bytecode, but the bytecode is -not saved to a file. Usually main scripts are quite short, so this doesn't cost -much speed. - -.. XXX check which of these projects are still alive - -There are also several programs which make it easier to intermingle Python and C -code in various ways to increase performance. See, for example, `Cython -`_, `Pyrex -`_ and `Weave -`_. - How does Python manage memory? ------------------------------ -- cgit v0.12 From c561a9adac579ed88446385775e635da3e55cf83 Mon Sep 17 00:00:00 2001 From: Antoine Pitrou Date: Sat, 3 Dec 2011 23:06:50 +0100 Subject: Break down and refine memory management question --- Doc/faq/design.rst | 75 +++++++++++++++++++++++++----------------------------- 1 file changed, 34 insertions(+), 41 deletions(-) diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 9e6bebc..d215ab1 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -413,66 +413,59 @@ How does Python manage memory? ------------------------------ The details of Python memory management depend on the implementation. The -standard C implementation of Python uses reference counting to detect -inaccessible objects, and another mechanism to collect reference cycles, +standard implementation of Python, :term:`CPython`, uses reference counting to +detect inaccessible objects, and another mechanism to collect reference cycles, periodically executing a cycle detection algorithm which looks for inaccessible cycles and deletes the objects involved. The :mod:`gc` module provides functions to perform a garbage collection, obtain debugging statistics, and tune the collector's parameters. -Jython relies on the Java runtime so the JVM's garbage collector is used. This -difference can cause some subtle porting problems if your Python code depends on -the behavior of the reference counting implementation. +Other implementations (such as `Jython `_ or +`PyPy `_), however, can rely on a different mechanism +such as a full-blown garbage collector. This difference can cause some +subtle porting problems if your Python code depends on the behavior of the +reference counting implementation. -.. XXX relevant for Python 3? +In some Python implementations, the following code (which is fine in CPython) +will probably run out of file descriptors:: - Sometimes objects get stuck in traceback temporarily and hence are not - deallocated when you might expect. Clear the traceback with:: + for file in very_long_list_of_files: + f = open(file) + c = f.read(1) + +Indeed, using CPython's reference counting and destructor scheme, each new +assignment to *f* closes the previous file. With a traditional GC, however, +those file objects will only get collected (and closed) at varying and possibly +long intervals. + +If you want to write code that will work with any Python implementation, +you should explicitly close the file or use the :keyword:`with` statement; +this will work regardless of memory management scheme:: - import sys - sys.last_traceback = None + for file in very_long_list_of_files: + with open(file) as f: + c = f.read(1) - Tracebacks are used for reporting errors, implementing debuggers and related - things. They contain a portion of the program state extracted during the - handling of an exception (usually the most recent exception). -In the absence of circularities, Python programs do not need to manage memory -explicitly. +Why doesn't CPython use a more traditional garbage collection scheme? +--------------------------------------------------------------------- -Why doesn't Python use a more traditional garbage collection scheme? For one -thing, this is not a C standard feature and hence it's not portable. (Yes, we -know about the Boehm GC library. It has bits of assembler code for *most* -common platforms, not for all of them, and although it is mostly transparent, it -isn't completely transparent; patches are required to get Python to work with -it.) +For one thing, this is not a C standard feature and hence it's not portable. +(Yes, we know about the Boehm GC library. It has bits of assembler code for +*most* common platforms, not for all of them, and although it is mostly +transparent, it isn't completely transparent; patches are required to get +Python to work with it.) Traditional GC also becomes a problem when Python is embedded into other applications. While in a standalone Python it's fine to replace the standard malloc() and free() with versions provided by the GC library, an application embedding Python may want to have its *own* substitute for malloc() and free(), -and may not want Python's. Right now, Python works with anything that +and may not want Python's. Right now, CPython works with anything that implements malloc() and free() properly. -In Jython, the following code (which is fine in CPython) will probably run out -of file descriptors long before it runs out of memory:: - - for file in very_long_list_of_files: - f = open(file) - c = f.read(1) - -Using the current reference counting and destructor scheme, each new assignment -to f closes the previous file. Using GC, this is not guaranteed. If you want -to write code that will work with any Python implementation, you should -explicitly close the file or use the :keyword:`with` statement; this will work -regardless of GC:: - - for file in very_long_list_of_files: - with open(file) as f: - c = f.read(1) - -Why isn't all memory freed when Python exits? ---------------------------------------------- +Why isn't all memory freed when CPython exits? +---------------------------------------------- Objects referenced from the global namespaces of Python modules are not always deallocated when Python exits. This may happen if there are circular -- cgit v0.12 From dec0f21efcdd931bd10ccb8f41809de2e9284cee Mon Sep 17 00:00:00 2001 From: Antoine Pitrou Date: Sat, 3 Dec 2011 23:08:57 +0100 Subject: Streamline mention of sorted() --- Doc/faq/design.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index d215ab1..e45aaaa 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -625,10 +625,10 @@ order to remind you of that fact, it does not return the sorted list. This way, you won't be fooled into accidentally overwriting a list when you need a sorted copy but also need to keep the unsorted version around. -In Python 2.4 a new built-in function -- :func:`sorted` -- has been added. -This function creates a new list from a provided iterable, sorts it and returns -it. For example, here's how to iterate over the keys of a dictionary in sorted -order:: +If you want to return a new list, use the built-in :func:`sorted` function +instead. This function creates a new list from a provided iterable, sorts +it and returns it. For example, here's how to iterate over the keys of a +dictionary in sorted order:: for key in sorted(mydict): ... # do whatever with mydict[key]... -- cgit v0.12