[3.8] bpo-38234: Backport init path config changes from master (GH-16423)

* bpo-38234: Py_SetPath() uses the program full path (GH-16357)

Py_SetPath() now sets sys.executable to the program full path
(Py_GetProgramFullPath()), rather than to the program name
(Py_GetProgramName()).

Fix also memory leaks in pathconfig_set_from_config().

(cherry picked from commit 1ce152a42eaa917d7763bce93f1e1ca72530d7ca)

* bpo-38234: Add tests for Python init path config (GH-16358)


(cherry picked from commit bb6bf7d342b4503a6227fd209fac934905b6a1aa)

* bpo-38234: test_embed: test pyvenv.cfg and pybuilddir.txt (GH-16366)

Add test_init_pybuilddir() and test_init_pyvenv_cfg() to test_embed
to test pyvenv.cfg and pybuilddir.txt configuration files.

Fix sysconfig._generate_posix_vars(): pybuilddir.txt uses UTF-8
encoding, not ASCII.

(cherry picked from commit 52ad33abbfb6637d74932617c7013bae0ccf6e32)

* bpo-38234: Cleanup getpath.c (GH-16367)

* search_for_prefix() directly calls reduce() if found is greater
  than 0.
* Add calculate_pybuilddir() subfunction.
* search_for_prefix(): add path string buffer for readability.
* Fix some error handling code paths: release resources on error.
* calculate_read_pyenv(): rename tmpbuffer to filename.
* test.pythoninfo now also logs windows.dll_path

(cherry picked from commit 221fd84703c545408bbb4a6e0b58459651331f5c)

* bpo-38234: Fix test_embed pathconfig tests (GH-16390)

bpo-38234: On macOS and FreeBSD, the temporary directory can be
symbolic link. For example, /tmp can be a symbolic link to /var/tmp.
Call realpath() to resolve all symbolic links.

(cherry picked from commit 00508a7407d7d300b487532e2271534b20e378a7)

* bpo-38234: Add test_init_setpath_config() to test_embed (GH-16402)

* Add test_embed.test_init_setpath_config(): test Py_SetPath()
  with PyConfig.
* test_init_setpath() and test_init_setpythonhome() no longer call
  Py_SetProgramName(), but use the default program name.
* _PyPathConfig: isolated, site_import  and base_executable
  fields are now only available on Windows.
* If executable is set explicitly in the configuration, ignore
  calculated base_executable: _PyConfig_InitPathConfig() copies
  executable to base_executable.
* Complete path config documentation.

(cherry picked from commit 8bf39b606ef7b02c0279a80789f3c4824b0da5e9)

* bpo-38234: Complete init config documentation (GH-16404)


(cherry picked from commit 88feaecd46a8f427e30ef7ad8cfcddfe392a2402)

* bpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406)

Explicitly preinitializes with a Python preconfiguration to avoid
Py_SetPath() implicit preinitialization with a compat
preconfiguration.

Fix also test_init_setpath() and test_init_setpythonhome() on macOS:
use self.test_exe as the executable (and base_executable), rather
than shutil.which('python3').

(cherry picked from commit 49d99f01e6e51acec5ca57a02e857f0796bc418b)

* bpo-38234: Py_Initialize() sets global path configuration (GH-16421)

* Py_InitializeFromConfig() now writes PyConfig path configuration to
  the global path configuration (_Py_path_config).
* Add test_embed.test_get_pathconfig().
* Fix typo in _PyWideStringList_Join().

(cherry picked from commit 12f2f177fc483723406d7917194e7f655a20631b)

2 files changed, 57 insertions, 30 deletions

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 0f8ff3b..0b7a84d 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -472,8 +472,8 @@ Process-wide parameters
    dependent delimiter character, which is ``':'`` on Unix and Mac OS X, ``';'``
    on Windows.
 
-   This also causes :data:`sys.executable` to be set only to the raw program
-   name (see :c:func:`Py_SetProgramName`) and for :data:`sys.prefix` and
+   This also causes :data:`sys.executable` to be set to the program
+   full path (see :c:func:`Py_GetProgramFullPath`) and for :data:`sys.prefix` and
    :data:`sys.exec_prefix` to be empty.  It is up to the caller to modify these
    if required after calling :c:func:`Py_Initialize`.
 
@@ -483,6 +483,10 @@ Process-wide parameters
    The path argument is copied internally, so the caller may free it after the
    call completes.
 
+   .. versionchanged:: 3.8
+      The program full path is now used for :data:`sys.executable`, instead
+      of the program name.
+
 
 .. c:function:: const char* Py_GetVersion()
 
diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index bc24fa0..0c3c725 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -241,6 +241,7 @@ PyPreConfig
       locale to decide if it should be coerced.
 
    .. c:member:: int coerce_c_locale_warn
+
       If non-zero, emit a warning if the C locale is coerced.
 
    .. c:member:: int dev_mode
@@ -300,7 +301,7 @@ For :ref:`Python Configuration <init-python-config>`
 (:c:func:`PyPreConfig_InitPythonConfig`), if Python is initialized with
 command line arguments, the command line arguments must also be passed to
 preinitialize Python, since they have an effect on the pre-configuration
-like encodings. For example, the :option:`-X` ``utf8`` command line option
+like encodings. For example, the :option:`-X utf8 <-X>` command line option
 enables the UTF-8 Mode.
 
 ``PyMem_SetAllocator()`` can be called after :c:func:`Py_PreInitialize` and
@@ -464,7 +465,7 @@ PyConfig
 
    .. c:member:: int dev_mode
 
-      Development mode: see :option:`-X` ``dev``.
+      Development mode: see :option:`-X dev <-X>`.
 
    .. c:member:: int dump_refs
 
@@ -482,7 +483,7 @@ PyConfig
 
    .. c:member:: int faulthandler
 
-      If non-zero, call :func:`faulthandler.enable`.
+      If non-zero, call :func:`faulthandler.enable` at startup.
 
    .. c:member:: wchar_t* filesystem_encoding
 
@@ -504,6 +505,9 @@ PyConfig
 
       Python home directory.
 
+      Initialized from :envvar:`PYTHONHOME` environment variable value by
+      default.
+
    .. c:member:: int import_time
 
       If non-zero, profile import time.
@@ -561,7 +565,7 @@ PyConfig
 
       :data:`sys.path`. If :c:member:`~PyConfig.module_search_paths_set` is
       equal to 0, the :c:member:`~PyConfig.module_search_paths` is overridden
-      by the function computing the :ref:`Path Configuration
+      by the function calculating the :ref:`Path Configuration
       <init-path-config>`.
 
    .. c:member:: int optimization_level
@@ -586,9 +590,9 @@ PyConfig
 
    .. c:member:: int pathconfig_warnings
 
-      If equal to 0, suppress warnings when computing the path configuration
-      (Unix only, Windows does not log any warning). Otherwise, warnings are
-      written into ``stderr``.
+      If equal to 0, suppress warnings when calculating the :ref:`Path
+      Configuration <init-path-config>` (Unix only, Windows does not log any
+      warning). Otherwise, warnings are written into ``stderr``.
 
    .. c:member:: wchar_t* prefix
 
@@ -596,39 +600,46 @@ PyConfig
 
    .. c:member:: wchar_t* program_name
 
-      Program name.
+      Program name. Used to initialize :c:member:`~PyConfig.executable`, and in
+      early error messages.
 
    .. c:member:: wchar_t* pycache_prefix
 
-      ``.pyc`` cache prefix.
+      :data:`sys.pycache_prefix`: ``.pyc`` cache prefix.
+
+      If NULL, :data:`sys.pycache_prefix` is set to ``None``.
 
    .. c:member:: int quiet
 
       Quiet mode. For example, don't display the copyright and version messages
-      even in interactive mode.
+      in interactive mode.
 
    .. c:member:: wchar_t* run_command
 
-      ``python3 -c COMMAND`` argument.
+      ``python3 -c COMMAND`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: wchar_t* run_filename
 
-      ``python3 FILENAME`` argument.
+      ``python3 FILENAME`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: wchar_t* run_module
 
-      ``python3 -m MODULE`` argument.
+      ``python3 -m MODULE`` argument. Used by :c:func:`Py_RunMain`.
 
    .. c:member:: int show_alloc_count
 
       Show allocation counts at exit?
 
+      Set to 1 by :option:`-X showalloccount <-X>` command line option.
+
       Need a special Python build with ``COUNT_ALLOCS`` macro defined.
 
    .. c:member:: int show_ref_count
 
       Show total reference count at exit?
 
+      Set to 1 by :option:`-X showrefcount <-X>` command line option.
+
       Need a debug build of Python (``Py_REF_DEBUG`` macro must be defined).
 
    .. c:member:: int site_import
@@ -647,7 +658,7 @@ PyConfig
 
    .. c:member:: int tracemalloc
 
-      If non-zero, call :func:`tracemalloc.start`.
+      If non-zero, call :func:`tracemalloc.start` at startup.
 
    .. c:member:: int use_environment
 
@@ -669,6 +680,9 @@ PyConfig
 
       If non-zero, write ``.pyc`` files.
 
+      :data:`sys.dont_write_bytecode` is initialized to the inverted value of
+      :c:member:`~PyConfig.write_bytecode`.
+
    .. c:member:: PyWideStringList xoptions
 
       :data:`sys._xoptions`.
@@ -694,8 +708,8 @@ Function to initialize Python:
 The caller is responsible to handle exceptions (error or exit) using
 :c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
 
-``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
-``PyImport_ExtendInittab()`` is used: they must be set or called after Python
+If ``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
+``PyImport_ExtendInittab()`` are used, they must be set or called after Python
 preinitialization and before the Python initialization.
 
 Example setting the program name::
@@ -760,7 +774,7 @@ configuration, and then override some parameters::
 
         /* Append our custom search path to sys.path */
         status = PyWideStringList_Append(&config.module_search_paths,
-                                      L"/path/to/more/modules");
+                                         L"/path/to/more/modules");
         if (PyStatus_Exception(status)) {
             goto done;
         }
@@ -791,9 +805,9 @@ isolate Python from the system. For example, to embed Python into an
 application.
 
 This configuration ignores global configuration variables, environments
-variables and command line arguments (:c:member:`PyConfig.argv` is not parsed).
-The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left
-unchanged by default.
+variables, command line arguments (:c:member:`PyConfig.argv` is not parsed)
+and user site directory. The C standard streams (ex: ``stdout``) and the
+LC_CTYPE locale are left unchanged. Signal handlers are not installed.
 
 Configuration files are still used with this configuration. Set the
 :ref:`Path Configuration <init-path-config>` ("output fields") to ignore these
@@ -864,29 +878,38 @@ Path Configuration
 
 :c:type:`PyConfig` contains multiple fields for the path configuration:
 
-* Path configuration input fields:
+* Path configuration inputs:
 
   * :c:member:`PyConfig.home`
   * :c:member:`PyConfig.pathconfig_warnings`
   * :c:member:`PyConfig.program_name`
   * :c:member:`PyConfig.pythonpath_env`
+  * current working directory: to get absolute paths
+  * ``PATH`` environment variable to get the program full path
+    (from :c:member:`PyConfig.program_name`)
+  * ``__PYVENV_LAUNCHER__`` environment variable
+  * (Windows only) Application paths in the registry under
+    "Software\Python\PythonCore\X.Y\PythonPath" of HKEY_CURRENT_USER and
+    HKEY_LOCAL_MACHINE (where X.Y is the Python version).
 
 * Path configuration output fields:
 
+  * :c:member:`PyConfig.base_exec_prefix`
   * :c:member:`PyConfig.base_executable`
+  * :c:member:`PyConfig.base_prefix`
   * :c:member:`PyConfig.exec_prefix`
   * :c:member:`PyConfig.executable`
-  * :c:member:`PyConfig.prefix`
   * :c:member:`PyConfig.module_search_paths_set`,
     :c:member:`PyConfig.module_search_paths`
+  * :c:member:`PyConfig.prefix`
 
-If at least one "output field" is not set, Python computes the path
+If at least one "output field" is not set, Python calculates the path
 configuration to fill unset fields. If
 :c:member:`~PyConfig.module_search_paths_set` is equal to 0,
 :c:member:`~PyConfig.module_search_paths` is overridden and
 :c:member:`~PyConfig.module_search_paths_set` is set to 1.
 
-It is possible to completely ignore the function computing the default
+It is possible to completely ignore the function calculating the default
 path configuration by setting explicitly all path configuration output
 fields listed above. A string is considered as set even if it is non-empty.
 ``module_search_paths`` is considered as set if
@@ -894,7 +917,7 @@ fields listed above. A string is considered as set even if it is non-empty.
 configuration input fields are ignored as well.
 
 Set :c:member:`~PyConfig.pathconfig_warnings` to 0 to suppress warnings when
-computing the path configuration (Unix only, Windows does not log any warning).
+calculating the path configuration (Unix only, Windows does not log any warning).
 
 If :c:member:`~PyConfig.base_prefix` or :c:member:`~PyConfig.base_exec_prefix`
 fields are not set, they inherit their value from :c:member:`~PyConfig.prefix`
@@ -961,7 +984,7 @@ initialization, the core feature of the :pep:`432`:
   * Builtin exceptions;
   * Builtin and frozen modules;
   * The :mod:`sys` module is only partially initialized
-    (ex: :data:`sys.path` doesn't exist yet);
+    (ex: :data:`sys.path` doesn't exist yet).
 
 * "Main" initialization phase, Python is fully initialized:
 
@@ -987,9 +1010,9 @@ No module is imported during the "Core" phase and the ``importlib`` module is
 not configured: the :ref:`Path Configuration <init-path-config>` is only
 applied during the "Main" phase. It may allow to customize Python in Python to
 override or tune the :ref:`Path Configuration <init-path-config>`, maybe
-install a custom sys.meta_path importer or an import hook, etc.
+install a custom :data:`sys.meta_path` importer or an import hook, etc.
 
-It may become possible to compute the :ref:`Path Configuration
+It may become possible to calculatin the :ref:`Path Configuration
 <init-path-config>` in Python, after the Core phase and before the Main phase,
 which is one of the :pep:`432` motivation.