.. _history: =============== Project History =============== Work on PyOxidizer started in November 2018 by Gregory Szorc. Blog Posts ========== * `C Extension Support in PyOxidizer `_ (2019-06-30) * `Building Standalone Python Applications with PyOxidizer `_ (2019-06-24) * `PyOxidizer Support for Windows `_ (2019-01-06) * `Faster In-Memory Python Module Importing `_ (2018-12-28) * `Distributing Standalone Python Applications `_ (2018-12-18) .. _version_history: Version History =============== .. _version_0_6_0: 0.6.0 ----- Released February 12, 2020. Backwards Compatibility Notes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * The ``default_python_distribution()`` Starlark function now accepts a ``flavor`` argument denoting the distribution flavor. * The ``pyembed`` crate no longer includes the auto-generated default configuration file. Instead, it is consumed by the application that instantiates a Python interpreter. * Rust projects for the main executable now utilize and require a Cargo build script so metadata can be passed from ``pyembed`` to the project that is consuming it. * The ``pyembed`` crate is no longer added to created Rust projects. Instead, the generated ``Cargo.toml`` will reference a version of the ``pyembed`` crate identical to the ``PyOxidizer`` version currently running. Or if ``pyoxidizer`` is running from a Git checkout of the canonical ``PyOxidizer`` Git repository, a local filesystem path will be used. * The fields of ``EmbeddedPythonConfig`` and ``pyembed::PythonConfig`` have been renamed and reordered to align with Python 3.8's config API naming. This was done for the Starlark type in version 0.5. We have made similar changes to 0.6 so naming is consistent across the various types. Bug Fixes ^^^^^^^^^ * Module names without a ``.`` are now properly recognized when scanning the filesystem for Python resources and a package allow list is used (#223). Previously, if filtering scanned resources through an explicit list of allowed packages, the top-level module/package without a dot in its full name would not be passed through the filter. New Features ^^^^^^^^^^^^ * The ``PythonDistribution()`` Starlark function now accepts a ``flavor`` argument to denote the distribution type. This allows construction of alternate distribution types. * The ``default_python_distribution()`` Starlark function now accepts a ``flavor`` argument which can be set to ``windows_embeddable`` to return a distribution based on the zip file distributions published by the official CPython project. * The ``pyembed`` crate and generated Rust projects now have various ``build-mode-*`` feature flags to control how build artifacts are built. See :ref:`rust_projects` for more. * The ``pyembed`` crate can now be built standalone, without being bound to a specific ``PyOxidizer`` configuration. * The ``register_target()`` Starlark function now accepts an optional ``default_build_script`` argument to define the default target when evaluating in *Rust build script* mode. * The ``pyembed`` crate now builds against published ``cpython`` and ``python3-sys`` crates instead of a a specific Git commit. * Embedded Python interpreters can now be configured to run a file specified by a filename. See the ``run_file`` argument of :ref:`config_python_interpreter_config`. Other Relevant Changes ^^^^^^^^^^^^^^^^^^^^^^ * Rust internals have been overhauled to use traits to represent various types, namely Python distributions. The goal is to allow different Python distribution flavors to implement different logic for building binaries. * The ``pyembed`` crate's ``build.rs`` has been tweaked so it can support calling out to ``pyoxidizer``. It also no longer has a build dependency on ``pyoxidizer``. .. _version_0_5_1: 0.5.1 ----- Released January 26, 2020. Bug Fixes ^^^^^^^^^ * Fixed bad Starlark example for building ``black`` in docs. * Remove resources attached to packages that don't exist. (This was a regression in 0.5.) * Warn on failure to annotate a package. (This was a regression in 0.5.) * Building embedded Python resources now emits warnings when ``__file__`` is seen. (This was a regression in 0.5.) * Missing parent packages are now automatically added when constructing embedded resources. (This was a regression in 0.5.) .. _version_0_5_0: 0.5.0 ----- Released January 26, 2020. General Notes ^^^^^^^^^^^^^ This release of PyOxidizer is significant rewrite of the previous version. The impetus for the rewrite is to transition from TOML to Starlark configuration files. The new configuration file format should allow vastly greater flexibility for building applications and will unlock a world of new possibilities. The transition to Starlark configuration files represented a shift from static configuration to something more dynamic. This required refactoring a ton of code. As part of refactoring code, we took the opportunity to shore up lots of the code base. PyOxidizer was the project author's first real Rust project and a lot of bad practices (such as use of `.unwrap()`/panics) were prevalent. The code mostly now has proper error handling. Another new addition to the code is unit tests. While coverage still isn't great, we now have tests performing meaningful packaging activities. So regressions should hopefully be less common going forward. Because of the scale of the rewritten code in this release, it is expected that there are tons of bugs of regressions. This will likely be a transitional release with a more robust release to follow. Backwards Compatibility Notes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Support for building distributions/installers has been temporarily dropped. * Support for installing license files has been temporarily dropped. * Python interpreter configuration setting names have been changed to reflect names from Python 3.8's interpreter initialization API. * ``.egg-info`` directories are now ignored when scanning for Python resources on the filesystem (matching the behavior for ``.dist-info`` directories). * The ``pyoxidizer init`` sub-command has been renamed to ``init-rust-project``. * The ``pyoxidizer app-path`` sub-command has been removed. * Support for building distributions has been removed. * The minimum Rust version to build has been increased from 1.31 to 1.36. This is mainly due to requirements from the ``starlark`` crate. We could potentially reduce the minimum version requirements again with minimal changes to 3rd party crates. * PyOxidizer configuration files are now `Starlark `_ instead of TOML files. The default file name is ``pyoxidizer.bzl`` instead of ``pyoxidizer.toml``. All existing configuration files will need to be ported to the new format. Bug Fixes ^^^^^^^^^ * The ``repl`` run mode now properly exits with a non-zero exit code if an error occurs. * Compiled C extensions now properly honor the ``ext_package`` argument passed to ``setup()``, resulting in extensions which properly have the package name in their extension name (#26). New Features ^^^^^^^^^^^^ * A :ref:`config_glob` function has been added to config files to allow referencing existing files on the filesystem. * The in-memory ``MetaPathFinder`` now implements ``find_module()``. * A ``pyoxidizer init-config-file`` command has been implemented to create just a ``pyoxidizer.bzl`` configuration file. * A ``pyoxidizer python-distribution-info`` command has been implemented to print information about a Python distribution archive. * The ``EmbeddedPythonConfig()`` config function now accepts a ``legacy_windows_stdio`` argument to control the value of ``Py_LegacyWindowsStdioFlag`` (#190). * The ``EmbeddedPythonConfig()`` config function now accepts a ``legacy_windows_fs_encoding`` argument to control the value of ``Py_LegacyWindowsFSEncodingFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts an ``isolated`` argument to control the value of ``Py_IsolatedFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts a ``use_hash_seed`` argument to control the value of ``Py_HashRandomizationFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts an ``inspect`` argument to control the value of ``Py_InspectFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts an ``interactive`` argument to control the value of ``Py_InteractiveFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts a ``quiet`` argument to control the value of ``Py_QuietFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts a ``verbose`` argument to control the value of ``Py_VerboseFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts a ``parser_debug`` argument to control the value of ``Py_DebugFlag``. * The ``EmbeddedPythonConfig()`` config function now accepts a ``bytes_warning`` argument to control the value of ``Py_BytesWarningFlag``. * The ``Stdlib()`` packaging rule now now accepts an optional ``excludes`` list of modules to ignore. This is useful for removing unnecessary Python packages such as ``distutils``, ``pip``, and ``ensurepip``. * The ``PipRequirementsFile()`` and ``PipInstallSimple()`` packaging rules now accept an optional ``extra_env`` dict of extra environment variables to set when invoking ``pip install``. * The ``PipRequirementsFile()`` packaging rule now accepts an optional ``extra_args`` list of extra command line arguments to pass to ``pip install``. Other Relevant Changes ^^^^^^^^^^^^^^^^^^^^^^ * PyOxidizer no longer requires a forked version of the ``rust-cpython`` project (the ``python3-sys`` and ``cpython`` crates. All changes required by PyOxidizer are now present in the official project. .. _version_0_4_0: 0.4.0 ----- Released October 27, 2019. Backwards Compatibility Notes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * The ``setup-py-install`` packaging rule now has its ``package_path`` evaluated relative to the PyOxidizer config file path rather than the current working directory. Bug Fixes ^^^^^^^^^ * Windows now explicitly requires dynamic linking against ``msvcrt``. Previously, this wasn't explicit. And sometimes linking the final executable would result in unresolved symbol errors because the Windows Python distributions used external linkage of CRT symbols and for some reason Cargo wasn't dynamically linking the CRT. * Read-only files in Python distributions are now made writable to avoid future permissions errors (#123). * In-memory ``InspectLoader.get_source()`` implementation no longer errors due to passing a ``memoryview`` to a function that can't handle it (#134). * In-memory ``ResourceReader`` now properly handles multiple resources (#128). New Features ^^^^^^^^^^^^ * Added an ``app-path`` command that prints the path to a packaged application. This command can be useful for tools calling PyOxidizer, as it will emit the path containing the packaged files without forcing the caller to parse command output. * The ``setup-py-install`` packaging rule now has an ``excludes`` option that allows ignoring specific packages or modules. * ``.py`` files installed into app-relative locations now have corresponding ``.pyc`` bytecode files written. * The ``setup-py-install`` packaging rule now has an ``extra_global_arguments`` option to allow passing additional command line arguments to the ``setup.py`` invocation. * Packaging rules that invoke ``pip`` or ``setup.py`` will now set a ``PYOXIDIZER=1`` environment variable so Python code knows at packaging time whether it is running in the context of PyOxidizer. * The ``setup-py-install`` packaging rule now has an ``extra_env`` option to allow passing additional environment variables to ``setup.py`` invocations. * ``[[embedded_python_config]]`` now supports a ``sys_frozen`` flag to control setting ``sys.frozen = True``. * ``[[embedded_python_config]]`` now supports a ``sys_meipass`` flag to control setting ``sys._MEIPASS = ``. * Default Python distribution upgraded to 3.7.5 (from 3.7.4). Various dependency packages also upgraded to latest versions. All Other Relevant Changes ^^^^^^^^^^^^^^^^^^^^^^^^^^ * Built extension modules marked as app-relative are now embedded in the finaly binary rather than being ignored. .. _version_0_3_0: 0.3.0 ----- Released on August 16, 2019. Backwards Compatibility Notes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * The ``pyembed::PythonConfig`` struct now has an additional ``extra_extension_modules`` field. * The default musl Python distribution now uses LibreSSL instead of OpenSSL. This should hopefully be an invisible change. * Default Python distributions now use CPython 3.7.4 instead of 3.7.3. * Applications are now built into directories named ``apps///`` rather than ``apps//``. This enables builds for multiple targets to coexist in an application's output directory. * The ``program_name`` field from the ``[[embedded_python_config]]`` config section has been removed. At run-time, the current executable's path is always used when calling ``Py_SetProgramName()``. * The format of embedded Python module data has changed. The ``pyembed`` crate and ``pyoxidizer`` versions must match exactly or else the ``pyembed`` crate will likely crash at run-time when parsing module data. Bug Fixes ^^^^^^^^^ * The ``libedit`` extension variant for the ``readline`` extension should now link on Linux. Before, attempting to link a binary using this extension variant would result in missing symbol errors. * The ``setup-py-install`` ``[[packaging_rule]]`` now performs actions to appease ``setuptools``, thus allowing installation of packages using ``setuptools`` to (hopefully) work without issue (#70). * The ``virtualenv`` ``[[packaging_rule]]`` now properly finds the ``site-packages`` directory on Windows (#83). * The ``filter-include`` ``[[packaging_rule]]`` no longer requires both ``files`` and ``glob_files`` be defined (#88). * ``import ctypes`` now works on Windows (#61). * The in-memory module importer now implements ``get_resource_reader()`` instead of ``get_resource_loader()``. (The CPython documentation steered us in the wrong direction - https://bugs.python.org/issue37459.) * The in-memory module importer now correctly populates ``__package__`` in more cases than it did previously. Before, whether a module was a package was derived from the presence of a ``foo.bar`` module. Now, a module will be identified as a package if the file providing it is named ``__init__``. This more closely matches the behavior of Python's filesystem based importer. (#53) New Features ^^^^^^^^^^^^ * The default Python distributions have been updated. Archives are generally about half the size from before. Tcl/tk is included in the Linux and macOS distributions (but PyOxidizer doesn't yet package the Tcl files). * Extra extension modules can now be registered with ``PythonConfig`` instances. This can be useful for having the application embedding Python provide its own extension modules without having to go through Python build mechanisms to integrate those extension modules into the Python executable parts. * Built applications now have the ability to detect and use ``terminfo`` databases on the execution machine. This allows applications to interact with terminals properly. (e.g. the backspace key will now work in interactive ``pdb`` sessions). By default, applications on non-Windows platforms will look for ``terminfo`` databases at well-known locations and attempt to load them. * Default Python distributions now use CPython 3.7.4 instead of 3.7.3. * A warning is now emitted when a Python source file contains ``__file__``. This should help trace down modules using ``__file__``. * Added 32-bit Windows distribution. * New ``pyoxidizer distribution`` command for producing distributable artifacts of applications. Currently supports building tar archives and ``.msi`` and ``.exe`` installers using the WiX Toolset. * Libraries required by C extensions are now passed into the linker as library dependencies. This should allow C extensions linked against libraries to be embedded into produced executables. * ``pyoxidizer --verbose`` will now pass verbose to invoked ``pip`` and ``setup.py`` scripts. This can help debug what Python packaging tools are doing. All Other Relevant Changes ^^^^^^^^^^^^^^^^^^^^^^^^^^ * The list of modules being added by the Python standard library is no longer printed during rule execution unless ``--verbose`` is used. The output was excessive and usually not very informative. .. _version_0_2_0: 0.2.0 ----- Released on June 30, 2019. Backwards Compatibility Notes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Applications are now built into an ``apps//(debug|release)`` directory instead of ``apps/``. This allows debug and release builds to exist side-by-side. Bug Fixes ^^^^^^^^^ * Extracted ``.egg`` directories in Python package directories should now have their resources detected properly and not as Python packages with the name ``*.egg``. * ``site-packages`` directories are now recognized as Python resource package roots and no longer have their contents packaged under a ``site-packages`` Python package. New Features ^^^^^^^^^^^^ * Support for building and embedding C extensions on Windows, Linux, and macOS in many circumstances. See :ref:`status_extension_modules` for support status. * ``pyoxidizer init`` now accepts a ``--pip-install`` option to pre-configure generated ``pyoxidizer.toml`` files with packages to install via ``pip``. Combined with the ``--python-code`` option, it is now possible to create ``pyoxidizer.toml`` files for a ready-to-use Python application! * ``pyoxidizer`` now accepts a ``--verbose`` flag to make operations more verbose. Various low-level output is no longer printed by default and requires ``--verbose`` to see. All Other Relevant Changes ^^^^^^^^^^^^^^^^^^^^^^^^^^ * Packaging now automatically creates empty modules for missing parent packages. This prevents a module from being packaged without its parent. This could occur with *namespace packages*, for example. * ``pip-install-simple`` rule now passes ``--no-binary :all:`` to pip. * Cargo packages updated to latest versions. 0.1.3 ----- Released on June 29, 2019. Bug Fixes ^^^^^^^^^ * Fix Python refcounting bug involving call to ``PyImport_AddModule()`` when ``mode = module`` evaluation mode is used. The bug would likely lead to a segfault when destroying the Python interpreter. (#31) * Various functionality will no longer fail when running ``pyoxidizer`` from a Git repository that isn't the canonical ``PyOxidizer`` repository. (#34) New Features ^^^^^^^^^^^^ * ``pyoxidizer init`` now accepts a ``--python-code`` option to control which Python code is evaluated in the produced executable. This can be used to create applications that do not run a Python REPL by default. * ``pip-install-simple`` packaging rule now supports ``excludes`` for excluding resources from packaging. (#21) * ``pip-install-simple`` packaging rule now supports ``extra_args`` for adding parameters to the pip install command. (#42) All Relevant Changes ^^^^^^^^^^^^^^^^^^^^ * Minimum Rust version decreased to 1.31 (the first Rust 2018 release). (#24) * Added CI powered by Azure Pipelines. (#45) * Comments in auto-generated ``pyoxidizer.toml`` have been tweaked to improve understanding. (#29) 0.1.2 ----- Released on June 25, 2019. Bug Fixes ^^^^^^^^^ * Honor ``HTTP_PROXY`` and ``HTTPS_PROXY`` environment variables when downloading Python distributions. (#15) * Handle BOM when compiling Python source files to bytecode. (#13) All Relevant Changes ^^^^^^^^^^^^^^^^^^^^ * ``pyoxidizer`` now verifies the minimum Rust version meets requirements before building. 0.1.1 ----- Released on June 24, 2019. Bug Fixes ^^^^^^^^^ * ``pyoxidizer`` binaries built from crates should now properly refer to an appropriate commit/tag in PyOxidizer's canonical Git repository in auto-generated ``Cargo.toml`` files. (#11) 0.1 --- Released on June 24, 2019. This is the initial formal release of PyOxidizer. The first ``pyoxidizer`` crate was published to ``crates.io``. New Features ^^^^^^^^^^^^ * Support for building standalone, single file executables embedding Python for 64-bit Windows, macOS, and Linux. * Support for importing Python modules from memory using zero-copy. * Basic Python packaging support. * Support for jemalloc as Python's memory allocator. * ``pyoxidizer`` CLI command with basic support for managing project lifecycle.