Changes pyvenv.cfg trick into an actual sys.path file.

5 files changed, 154 insertions, 99 deletions

diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
index b703f0a..6836d63 100644
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -29,13 +29,13 @@ Supported Versions
 
 As specified in :pep:`11`, a Python release only supports a Windows platform
 while Microsoft considers the platform under extended support. This means that
-Python 3.5 supports Windows Vista and newer. If you require Windows XP support
+Python 3.6 supports Windows Vista and newer. If you require Windows XP support
 then please install Python 3.4.
 
 Installation Steps
 ------------------
 
-Four Python 3.5 installers are available for download - two each for the 32-bit
+Four Python 3.6 installers are available for download - two each for the 32-bit
 and 64-bit versions of the interpreter. The *web installer* is a small initial
 download, and it will automatically download the required components as
 necessary. The *offline installer* includes the components necessary for a
@@ -193,13 +193,13 @@ of available options is shown below.
 For example, to silently install a default, system-wide Python installation,
 you could use the following command (from an elevated command prompt)::
 
-    python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0
+    python-3.6.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0
 
 To allow users to easily install a personal copy of Python without the test
 suite, you could provide a shortcut with the following command. This will
 display a simplified initial page and disallow customization::
 
-    python-3.5.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0
+    python-3.6.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0
         SimpleInstall=1 SimpleInstallDescription="Just for me, no test suite."
 
 (Note that omitting the launcher also omits file associations, and is only
@@ -234,13 +234,13 @@ where a large number of installations are going to be performed it is very
 useful to have a locally cached copy.
 
 Execute the following command from Command Prompt to download all possible
-required files.  Remember to substitute ``python-3.5.0.exe`` for the actual
+required files.  Remember to substitute ``python-3.6.0.exe`` for the actual
 name of your installer, and to create layouts in their own directories to
 avoid collisions between files with the same name.
 
 ::
 
-    python-3.5.0.exe /layout [optional target directory]
+    python-3.6.0.exe /layout [optional target directory]
 
 You may also specify the ``/quiet`` option to hide the progress display.
 
@@ -345,7 +345,7 @@ User level and the System level, or temporarily in a command prompt.
 To temporarily set environment variables, open Command Prompt and use the
 :command:`set` command::
 
-    C:\>set PATH=C:\Program Files\Python 3.5;%PATH%
+    C:\>set PATH=C:\Program Files\Python 3.6;%PATH%
     C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib
     C:\>python
 
@@ -401,10 +401,10 @@ Finding the Python executable
 
 Besides using the automatically created start menu entry for the Python
 interpreter, you might want to start Python in the command prompt. The
-installer for Python 3.5 and later has an option to set that up for you.
+installer for Python 3.6 has an option to set that up for you.
 
-On the first page of the installer, an option labelled "Add Python 3.5 to
-PATH" can be selected to have the installer add the install location into the
+On the first page of the installer, an option labelled "Add Python to PATH"
+may be selected to have the installer add the install location into the
 :envvar:`PATH`.  The location of the :file:`Scripts\\` folder is also added.
 This allows you to type :command:`python` to run the interpreter, and
 :command:`pip` for the package installer. Thus, you can also execute your
@@ -418,7 +418,7 @@ of your Python installation, delimited by a semicolon from other entries.  An
 example variable could look like this (assuming the first two entries already
 existed)::
 
-    C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5
+    C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.6
 
 .. _launcher:
 
@@ -720,7 +720,15 @@ installation directory.  So, if you had installed Python to
 :file:`C:\\Python\\Lib\\` and third-party modules should be stored in
 :file:`C:\\Python\\Lib\\site-packages\\`.
 
-This is how :data:`sys.path` is populated on Windows:
+To completely override :data:`sys.path`, create a text file named ``'sys.path'``
+containing a list of paths alongside the Python executable. This will ignore all
+registry settings and environment variables, enable isolated mode, disable
+importing :mod:`site`, and fill :data:`sys.path` with exactly the paths listed
+in the file. Paths may be absolute or relative to the directory containing the
+file.
+
+When the ``'sys.path'`` file is missing, this is how :data:`sys.path` is
+populated on Windows:
 
 * An empty entry is added at the start, which corresponds to the current
   directory.
@@ -755,10 +763,6 @@ directory one level above the executable, the following variations apply:
   path is used instead of the path to the main executable when deducing the
   home location.
 
-* If ``applocal`` is set to true, the ``home`` property or the main executable
-  is always used as the home path, and all environment variables or registry
-  values affecting the path are ignored. The landmark file is not checked.
-
 The end result of all this is:
 
 * When running :file:`python.exe`, or any other .exe in the main Python
@@ -777,13 +781,11 @@ The end result of all this is:
 For those who want to bundle Python into their application or distribution, the
 following advice will prevent conflicts with other installations:
 
-* Include a ``pyvenv.cfg`` file alongside your executable containing
-  ``applocal = true``. This will ensure that your own directory will be used to
-  resolve paths even if you have included the standard library in a ZIP file.
-  It will also ignore user site-packages and other paths listed in the
-  registry.
+* Include a ``sys.path`` file alongside your executable containing the
+  directories to include. This will ignore user site-packages and other paths
+  listed in the registry or in environment variables.
 
-* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own
+* If you are loading :file:`python3.dll` or :file:`python36.dll` in your own
   executable, explicitly call :c:func:`Py_SetPath` or (at least)
   :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`.
 
@@ -801,6 +803,11 @@ Otherwise, your users may experience problems using your application. Note that
 the first suggestion is the best, as the other may still be susceptible to
 non-standard paths in the registry and user site-packages.
 
+.. versionchanged:: 3.6
+
+   Adds ``sys.path`` file support and removes ``applocal`` option from
+   ``pyvenv.cfg``.
+
 Additional modules
 ==================
 
@@ -900,7 +907,7 @@ directly accessed by end-users.
 When extracted, the embedded distribution is (almost) fully isolated from the
 user's system, including environment variables, system registry settings, and
 installed packages. The standard library is included as pre-compiled and
-optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python35.dll``,
+optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python36.dll``,
 ``python.exe`` and ``pythonw.exe`` are all provided. Tcl/tk (including all
 dependants, such as Idle), pip and the Python documentation are not included.
 
diff --git a/Lib/site.py b/Lib/site.py
index a536ef1..5250266 100644
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -463,12 +463,6 @@ def venv(known_paths):
                         system_site = value.lower()
                     elif key == 'home':
                         sys._home = value
-                    elif key == 'applocal' and value.lower() == 'true':
-                        # App-local installs use the exe_dir as prefix,
-                        # not one level higher, and do not use system
-                        # site packages.
-                        site_prefix = exe_dir
-                        system_site = 'false'
 
         sys.prefix = sys.exec_prefix = site_prefix
 
diff --git a/PC/getpathp.c b/PC/getpathp.c
index cb9f1db..5df9d7d 100644
--- a/PC/getpathp.c
+++ b/PC/getpathp.c
@@ -6,7 +6,8 @@
    PATH RULES FOR WINDOWS:
    This describes how sys.path is formed on Windows.  It describes the
    functionality, not the implementation (ie, the order in which these
-   are actually fetched is different)
+   are actually fetched is different). The presence of a sys.path file
+   alongside the program overrides these rules - see below.
 
    * Python always adds an empty entry at the start, which corresponds
      to the current directory.
@@ -36,6 +37,12 @@
      used (eg. .\Lib;.\plat-win, etc)
 
 
+   If a sys.path file exists adjacent to python.exe, it must contain a
+   list of paths to add to sys.path, one per line (like a .pth file but without
+   the ability to execute arbitrary code). Each path is relative to the
+   directory containing the file. No other paths are added to the search path,
+   and the registry finder is not enabled.
+
   The end result of all this is:
   * When running python.exe, or any other .exe in the main Python directory
     (either an installed version, or directly from the PCbuild directory),
@@ -52,7 +59,10 @@
     some default, but relative, paths.
 
   * An embedding application can use Py_SetPath() to override all of
-    these authomatic path computations.
+    these automatic path computations.
+
+  * An isolation install of Python can disable all implicit paths by
+    providing a sys.path file.
 
    ---------------------------------------------------------------- */
 
@@ -61,10 +71,13 @@
 #include "osdefs.h"
 #include <wchar.h>
 
-#ifdef MS_WINDOWS
-#include <windows.h>
+#ifndef MS_WINDOWS
+#error getpathp.c should only be built on Windows
 #endif
 
+#include <windows.h>
+#include <Shlwapi.h>
+
 #ifdef HAVE_SYS_TYPES_H
 #include <sys/types.h>
 #endif /* HAVE_SYS_TYPES_H */
@@ -163,24 +176,30 @@ ismodule(wchar_t *filename, int update_filename) /* Is module -- check for .pyc/
    than MAXPATHLEN characters at exit.  If stuff is too long, only as much of
    stuff as fits will be appended.
 */
+
+static int _PathCchCombineEx_Initialized = 0;
+typedef HRESULT(__stdcall *PPathCchCombineEx)(PWSTR pszPathOut, size_t cchPathOut, PCWSTR pszPathIn, PCWSTR pszMore, unsigned long dwFlags);
+static PPathCchCombineEx _PathCchCombineEx;
+
 static void
 join(wchar_t *buffer, const wchar_t *stuff)
 {
-    size_t n;
-    if (is_sep(stuff[0]) ||
-        (wcsnlen_s(stuff, 4) >= 3 && stuff[1] == ':' && is_sep(stuff[2]))) {
-        if (wcscpy_s(buffer, MAXPATHLEN+1, stuff) != 0)
-            Py_FatalError("buffer overflow in getpathp.c's join()");
-        return;
+    if (_PathCchCombineEx_Initialized == 0) {
+        HMODULE pathapi = LoadLibraryW(L"api-ms-win-core-path-l1-1-0.dll");
+        if (pathapi)
+            _PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi, "PathCchCombineEx");
+        else
+            _PathCchCombineEx = NULL;
+        _PathCchCombineEx_Initialized = 1;
     }
 
-    n = wcsnlen_s(buffer, MAXPATHLEN+1);
-    if (n > 0 && !is_sep(buffer[n - 1]) && n < MAXPATHLEN) {
-        buffer[n] = SEP;
-        buffer[n + 1] = '\0';
+    if (_PathCchCombineEx) {
+        if (FAILED(_PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0)))
+            Py_FatalError("buffer overflow in getpathp.c's join()");
+    } else {
+        if (!PathCombineW(buffer, NULL, stuff))
+            Py_FatalError("buffer overflow in getpathp.c's join()");
     }
-    if (wcscat_s(buffer, MAXPATHLEN+1, stuff) != 0)
-        Py_FatalError("buffer overflow in getpathp.c's join()");
 }
 
 /* gotlandmark only called by search_for_prefix, which ensures
@@ -214,7 +233,6 @@ search_for_prefix(wchar_t *argv0_path, wchar_t *landmark)
     return 0;
 }
 
-#ifdef MS_WINDOWS
 #ifdef Py_ENABLE_SHARED
 
 /* a string loaded from the DLL at startup.*/
@@ -369,7 +387,6 @@ done:
     return retval;
 }
 #endif /* Py_ENABLE_SHARED */
-#endif /* MS_WINDOWS */
 
 static void
 get_progpath(void)
@@ -378,7 +395,6 @@ get_progpath(void)
     wchar_t *path = _wgetenv(L"PATH");
     wchar_t *prog = Py_GetProgramName();
 
-#ifdef MS_WINDOWS
 #ifdef Py_ENABLE_SHARED
     extern HANDLE PyWin_DLLhModule;
     /* static init of progpath ensures final char remains \0 */
@@ -390,7 +406,6 @@ get_progpath(void)
 #endif
     if (GetModuleFileNameW(NULL, progpath, MAXPATHLEN))
         return;
-#endif
     if (prog == NULL || *prog == '\0')
         prog = L"python";
 
@@ -483,6 +498,67 @@ find_env_config_value(FILE * env_file, const wchar_t * key, wchar_t * value)
     return result;
 }
 
+static int
+read_sys_path_file(const wchar_t *path, const wchar_t *prefix)
+{
+    FILE *sp_file = _Py_wfopen(path, L"r");
+    if (sp_file == NULL)
+        return -1;
+
+    size_t bufsiz = MAXPATHLEN;
+    size_t prefixlen = wcslen(prefix);
+
+    wchar_t *buf = (wchar_t*)PyMem_RawMalloc(bufsiz * sizeof(wchar_t));
+    buf[0] = '\0';
+
+    while (!feof(sp_file)) {
+        char line[MAXPATHLEN + 1];
+        char *p = fgets(line, MAXPATHLEN + 1, sp_file);
+        if (!p)
+            break;
+
+        DWORD n = strlen(line);
+        if (n == 0 || p[n - 1] != '\n')
+            break;
+        if (n > 2 && p[n - 1] == '\r')
+            --n;
+
+        DWORD wn = MultiByteToWideChar(CP_UTF8, 0, line, n - 1, NULL, 0);
+        wchar_t *wline = (wchar_t*)PyMem_RawMalloc((wn + 1) * sizeof(wchar_t));
+        wn = MultiByteToWideChar(CP_UTF8, 0, line, n - 1, wline, wn);
+        wline[wn] = '\0';
+
+        while (wn + prefixlen + 4 > bufsiz) {
+            bufsiz += MAXPATHLEN;
+            buf = (wchar_t*)PyMem_RawRealloc(buf, (bufsiz + 1) * sizeof(wchar_t));
+            if (!buf) {
+                PyMem_RawFree(wline);
+                goto error;
+            }
+        }
+
+        if (buf[0])
+            wcscat_s(buf, bufsiz, L";");
+        wchar_t *b = &buf[wcslen(buf)];
+        
+        wcscat_s(buf, bufsiz, prefix);
+        join(b, wline);
+
+        PyMem_RawFree(wline);
+    }
+
+    module_search_path = buf;
+
+    fclose(sp_file);
+    return 0;
+
+error:
+    PyMem_RawFree(buf);
+    fclose(sp_file);
+    return -1;
+}
+
+
 static void
 calculate_path(void)
 {
@@ -492,32 +568,34 @@ calculate_path(void)
     wchar_t *pythonhome = Py_GetPythonHome();
     wchar_t *envpath = NULL;
 
-#ifdef MS_WINDOWS
     int skiphome, skipdefault;
     wchar_t *machinepath = NULL;
     wchar_t *userpath = NULL;
     wchar_t zip_path[MAXPATHLEN+1];
-    int applocal = 0;
 
     if (!Py_IgnoreEnvironmentFlag) {
         envpath = _wgetenv(L"PYTHONPATH");
     }
-#else
-    char *_envpath = Py_GETENV("PYTHONPATH");
-    wchar_t wenvpath[MAXPATHLEN+1];
-    if (_envpath) {
-        size_t r = mbstowcs(wenvpath, _envpath, MAXPATHLEN+1);
-        envpath = wenvpath;
-        if (r == (size_t)-1 || r >= MAXPATHLEN)
-            envpath = NULL;
-    }
-#endif
 
     get_progpath();
     /* progpath guaranteed \0 terminated in MAXPATH+1 bytes. */
     wcscpy_s(argv0_path, MAXPATHLEN+1, progpath);
     reduce(argv0_path);
 
+    /* Search for a sys.path file */
+    {
+        wchar_t spbuffer[MAXPATHLEN+1];
+
+        wcscpy_s(spbuffer, MAXPATHLEN+1, argv0_path);
+        join(spbuffer, L"sys.path");
+        if (exists(spbuffer) && read_sys_path_file(spbuffer, argv0_path) == 0) {
+            wcscpy_s(prefix, MAXPATHLEN + 1, argv0_path);
+            Py_IsolatedFlag = 1;
+            Py_NoSiteFlag = 1;
+            return;
+        }
+    }
+
     /* Search for an environment configuration file, first in the
        executable's directory and then in the parent directory.
        If found, open it for use when searching for prefixes.
@@ -543,17 +621,6 @@ calculate_path(void)
             }
         }
         if (env_file != NULL) {
-            /* Look for an 'applocal' variable and, if true, ignore all registry
-             * keys and environment variables, but retain the default paths
-             * (DLLs, Lib) and the zip file. Setting pythonhome here suppresses
-             * the search for LANDMARK below and overrides %PYTHONHOME%.
-             */
-            if (find_env_config_value(env_file, L"applocal", tmpbuffer) &&
-                (applocal = (wcsicmp(tmpbuffer, L"true") == 0))) {
-                envpath = NULL;
-                pythonhome = argv0_path;
-            }
-
             /* Look for a 'home' variable and set argv0_path to it, if found */
             if (find_env_config_value(env_file, L"home", tmpbuffer)) {
                 wcscpy_s(argv0_path, MAXPATHLEN+1, tmpbuffer);
@@ -576,7 +643,6 @@ calculate_path(void)
         envpath = NULL;
 
 
-#ifdef MS_WINDOWS
     /* Calculate zip archive path from DLL or exe path */
     if (wcscpy_s(zip_path, MAXPATHLEN+1, dllpath[0] ? dllpath : progpath))
         /* exceeded buffer length - ignore zip_path */
@@ -590,16 +656,13 @@ calculate_path(void)
 
     skiphome = pythonhome==NULL ? 0 : 1;
 #ifdef Py_ENABLE_SHARED
-    if (!applocal) {
-        machinepath = getpythonregpath(HKEY_LOCAL_MACHINE, skiphome);
-        userpath = getpythonregpath(HKEY_CURRENT_USER, skiphome);
-    }
+    machinepath = getpythonregpath(HKEY_LOCAL_MACHINE, skiphome);
+    userpath = getpythonregpath(HKEY_CURRENT_USER, skiphome);
 #endif
     /* We only use the default relative PYTHONPATH if we havent
        anything better to use! */
     skipdefault = envpath!=NULL || pythonhome!=NULL || \
                   machinepath!=NULL || userpath!=NULL;
-#endif
 
     /* We need to construct a path from the following parts.
        (1) the PYTHONPATH environment variable, if set;
@@ -612,7 +675,6 @@ calculate_path(void)
        Extra rules:
        - If PYTHONHOME is set (in any way) item (3) is ignored.
        - If registry values are used, (4) and (5) are ignored.
-       - If applocal is set, (1), (3), and registry values are ignored
     */
 
     /* Calculate size of return buffer */
@@ -629,13 +691,11 @@ calculate_path(void)
         bufsz = 0;
     bufsz += wcslen(PYTHONPATH) + 1;
     bufsz += wcslen(argv0_path) + 1;
-#ifdef MS_WINDOWS
-    if (!applocal && userpath)
+    if (userpath)
         bufsz += wcslen(userpath) + 1;
-    if (!applocal && machinepath)
+    if (machinepath)
         bufsz += wcslen(machinepath) + 1;
     bufsz += wcslen(zip_path) + 1;
-#endif
     if (envpath != NULL)
         bufsz += wcslen(envpath) + 1;
 
@@ -651,10 +711,8 @@ calculate_path(void)
             fprintf(stderr, "Using default static path.\n");
             module_search_path = PYTHONPATH;
         }
-#ifdef MS_WINDOWS
         PyMem_RawFree(machinepath);
         PyMem_RawFree(userpath);
-#endif /* MS_WINDOWS */
         return;
     }
 
@@ -664,7 +722,6 @@ calculate_path(void)
         buf = wcschr(buf, L'\0');
         *buf++ = DELIM;
     }
-#ifdef MS_WINDOWS
     if (zip_path[0]) {
         if (wcscpy_s(buf, bufsz - (buf - module_search_path), zip_path))
             Py_FatalError("buffer overflow in getpathp.c's calculate_path()");
@@ -692,15 +749,7 @@ calculate_path(void)
             buf = wcschr(buf, L'\0');
             *buf++ = DELIM;
         }
-    }
-#else
-    if (pythonhome == NULL) {
-        wcscpy(buf, PYTHONPATH);
-        buf = wcschr(buf, L'\0');
-        *buf++ = DELIM;
-    }
-#endif /* MS_WINDOWS */
-    else {
+    } else {
         wchar_t *p = PYTHONPATH;
         wchar_t *q;
         size_t n;
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
index 69bd519..cad747e 100644
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -69,7 +69,7 @@
       <PreprocessorDefinitions>_USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;MS_DLL_ID="$(SysWinVer)";%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
     <Link>
-      <AdditionalDependencies>ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalDependencies>shlwapi.lib;ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <BaseAddress>0x1e000000</BaseAddress>
     </Link>
   </ItemDefinitionGroup>
diff --git a/Tools/msi/make_zip.py b/Tools/msi/make_zip.py
index 0e8a4a6..6c43256 100644
--- a/Tools/msi/make_zip.py
+++ b/Tools/msi/make_zip.py
@@ -46,6 +46,10 @@ EXCLUDE_FILE_FROM_LIBS = {
     'python3stub',
 }
 
+EXCLUDED_FILES = {
+    'pyshellext',
+}
+
 def is_not_debug(p):
     if DEBUG_RE.search(p.name):
         return False
@@ -53,7 +57,7 @@ def is_not_debug(p):
     if TKTCL_RE.search(p.name):
         return False
 
-    return p.stem.lower() not in DEBUG_FILES
+    return p.stem.lower() not in DEBUG_FILES and p.stem.lower() not in EXCLUDED_FILES
 
 def is_not_debug_or_python(p):
     return is_not_debug(p) and not PYTHON_DLL_RE.search(p.name)
@@ -209,8 +213,9 @@ def main():
             copied = copy_to_layout(temp / t.rstrip('/'), rglob(s, p, c))
             print('Copied {} files'.format(copied))
 
-        with open(str(temp / 'pyvenv.cfg'), 'w') as f:
-            print('applocal = true', file=f)
+        with open(str(temp / 'sys.path'), 'w') as f:
+            print('python{0.major}{0.minor}.zip'.format(sys.version_info), file=f)
+            print('.', file=f)
 
         if out:
             total = copy_to_layout(out, rglob(temp, '**/*', None))