Project History¶
Work on PyOxidizer started in November 2018 by Gregory Szorc.
Blog Posts¶
- PyOxidizer 0.7 (2020-04-09)
- C Extension Support in PyOxidizer (2019-06-30)
- Building Standalone Python Applications with PyOxidizer (2019-06-24)
- PyOxidizer Support for Windows (2019-01-06)
- Faster In-Memory Python Module Importing (2018-12-28)
- Distributing Standalone Python Applications (2018-12-18)
Version History¶
0.7.0¶
Released April 9, 2020.
Backwards Compatibility Notes¶
- Packages imported from memory using PyOxidizer now set
__path__
with a value formed by joining the current executable’s path with the package name. This mimics the behavior ofzipimport
. - Resolved Python resource names have changed behavior. See the note in the bug fixes section below.
- The
PythonDistribution.to_python_executable()
Starlark method has added aresources_policy
named argument as its 2nd argument / 1st named argument. If you were affected by this, you should add argument names to all arguments passed to this method. - The default Rust project for built executables now builds executables such that dynamic symbols are exported from the executable. This change is necessary in order to support executables loading Python extension modules, which are shared libraries which need access to Python symbols defined in executables.
- The
PythonResourceData
Starlark type has been renamed toPythonPackageResource
. - The
PythonDistribution.resources_data()
Starlark method has been renamed toPythonDistribution.package_resources()
. - The
PythonExecutable.to_embedded_data()
Starlark method has been renamed toPythonExecutable.to_embedded_resources()
. - The
PythonEmbeddedData
Starlark type has been renamed toPythonEmbeddedResources
. - The format of Python resource data embedded in binaries has been completely rewritten. The separate modules and resource data structures have been merged into a single data structure. Embedded resources data can now express more primitives such as package distribution metadata and different bytecode optimization levels.
- The pyembed crate now has a dev dependency on the pyoxidizer crate in order to run tests.
Bug Fixes¶
- PyOxidizer’s importer now always sets
__path__
on imported packages in accordance with Python’s stated behavior (#51). - The mechanism for resolving Python resource files from the filesystem has
been rewritten. Before, it was possible for files like
package/resources/foo.txt
to be normalized to a (package, resource_name) tuple of (package, resources.foo.txt), which was weird and not compatible with Python’s resource loading mechanism. Resources in sub-directories should no longer encounter munging of directory separators to.
. In the above example, the resource path will now be expressed as(package, resources/foo.txt)
. - Certain packaging actions are only performed once during building instead of twice. The user-visible impact of this change is that some duplicate log messages no longer appear.
- Added a missing ) for add_python_resources() in auto-generated pyoxidizer.bzl files.
New Features¶
- Python resource scanning now recognizes
*.dist-info
and*.egg-info
directories as package distribution metadata. Files within these directories are exposed to Starlark as PythonPackageDistributionResource instances. These resources can be added to the embedded resources payload and made available for loading from memory or the filesystem, just like any other resource. The custom Python importer implementsget_distributions()
and returns objects that expose package distribution files. However, functionality of the returned distribution objects is not yet complete. See importlib.metadata Compatibility for details. - The custom Python importer now implements
get_data(path)
, allowing loading of resources from filesystem paths (#139). - The
PythonDistribution.to_python_executable()
Starlark method now accepts aresources_policy
argument to control a policy and default behavior for resources on the produced executable. Using this argument, it is possible to control how resources should be materialized. For example, you can specify that resources should be loaded from memory if supported and from the filesystem if not. The argument can also be used to materialize the Python standard library on the filesystem, like how Python distributions typically work. - Python resources can now be installed next to built binaries using the new
Starlark functions
PythonExecutable.add_filesystem_relative_module_source()
,PythonExecutable.add_filesystem_relative_module_bytecode()
,PythonExecutable.add_filesystem_relative_package_resource()
,PythonExecutable.add_filesystem_relative_extension_module()
,PythonExecutable.add_filesystem_relative_python_resource()
,PythonExecutable.add_filesystem_relative_package_distribution_resource()
, andPythonExecutable.add_filesystem_relative_python_resources()
. Unlike adding Python resources toFileManifest
instances, Python resources added this way have their metadata serialized into the built executable. This allows the special Python module importer present in built binaries to service theimport
request without going through Python’s default filesystem-based importer. Because metadata for the file-based Python resources is frozen into the application, Python has to do far less work at run-time to load resources, making operations faster. Resources loaded from the filesystem in this manner have attributes like__file__
,__cached__
, and__path__
set, emulating behavior of the default Python importer. The custom import now also implements theimportlib.abc.ExecutionLoader
interface. - Windows binaries can now import extension modules defined as shared libraries
(e.g.
.pyd
files) from memory. PyOxidizer will detect.pyd
files during packaging and embed them into the binary as resources. When the module is imported, the extension module/shared library is loaded from memory and initialized. This feature enables PyOxidizer to package pre-built extension modules (e.g. from Windows binary wheels published on PyPI) while still maintaining the property of a (mostly) self-contained executable. - Multiple bytecode optimization levels can now be embedded in binaries. Previously, it was only possible to embed bytecode for a given module at a single optimization level.
- The
default_python_distribution()
Starlark function now accepts valuesstandalone_static
andstandalone_dynamic
to specify a standalone distribution that is either statically or dynamically linked. - Support for parsing version 4 of the
PYTHON.json
distribution descriptor present in standalone Python distribution archives. - Default Python distributions upgraded to CPython 3.7.7.
Other Relevant Changes¶
- The directory for downloaded Python distributions in the build directory now uses a truncated SHA-256 hash instead of the full hash to help avoid path length limit issues (#224).
- The documentation for the
pyembed
crate has been moved out of the Sphinx documentation and into the Rust crate itself. Rendered docs can be seen by following the Documentation link at https://crates.io/crates/pyembed or by runningcargo doc
from a source checkout.
0.6.0¶
Released February 12, 2020.
Backwards Compatibility Notes¶
- The
default_python_distribution()
Starlark function now accepts aflavor
argument denoting the distribution flavor. - The
pyembed
crate no longer includes the auto-generated default configuration file. Instead, it is consumed by the application that instantiates a Python interpreter. - Rust projects for the main executable now utilize and require a Cargo build script
so metadata can be passed from
pyembed
to the project that is consuming it. - The
pyembed
crate is no longer added to created Rust projects. Instead, the generatedCargo.toml
will reference a version of thepyembed
crate identical to thePyOxidizer
version currently running. Or ifpyoxidizer
is running from a Git checkout of the canonicalPyOxidizer
Git repository, a local filesystem path will be used. - The fields of
EmbeddedPythonConfig
andpyembed::PythonConfig
have been renamed and reordered to align with Python 3.8’s config API naming. This was done for the Starlark type in version 0.5. We have made similar changes to 0.6 so naming is consistent across the various types.
Bug Fixes¶
- Module names without a
.
are now properly recognized when scanning the filesystem for Python resources and a package allow list is used (#223). Previously, if filtering scanned resources through an explicit list of allowed packages, the top-level module/package without a dot in its full name would not be passed through the filter.
New Features¶
- The
PythonDistribution()
Starlark function now accepts aflavor
argument to denote the distribution type. This allows construction of alternate distribution types. - The
default_python_distribution()
Starlark function now accepts aflavor
argument which can be set towindows_embeddable
to return a distribution based on the zip file distributions published by the official CPython project. - The
pyembed
crate and generated Rust projects now have variousbuild-mode-*
feature flags to control how build artifacts are built. See Rust Projects for more. - The
pyembed
crate can now be built standalone, without being bound to a specificPyOxidizer
configuration. - The
register_target()
Starlark function now accepts an optionaldefault_build_script
argument to define the default target when evaluating in Rust build script mode. - The
pyembed
crate now builds against publishedcpython
andpython3-sys
crates instead of a a specific Git commit. - Embedded Python interpreters can now be configured to run a file specified
by a filename. See the
run_file
argument of PythonInterpreterConfig(...).
Other Relevant Changes¶
- Rust internals have been overhauled to use traits to represent various types, namely Python distributions. The goal is to allow different Python distribution flavors to implement different logic for building binaries.
- The
pyembed
crate’sbuild.rs
has been tweaked so it can support calling out topyoxidizer
. It also no longer has a build dependency onpyoxidizer
.
0.5.1¶
Released January 26, 2020.
Bug Fixes¶
- Fixed bad Starlark example for building
black
in docs. - Remove resources attached to packages that don’t exist. (This was a regression in 0.5.)
- Warn on failure to annotate a package. (This was a regression in 0.5.)
- Building embedded Python resources now emits warnings when
__file__
is seen. (This was a regression in 0.5.) - Missing parent packages are now automatically added when constructing embedded resources. (This was a regression in 0.5.)
0.5.0¶
Released January 26, 2020.
General Notes¶
This release of PyOxidizer is significant rewrite of the previous version. The impetus for the rewrite is to transition from TOML to Starlark configuration files. The new configuration file format should allow vastly greater flexibility for building applications and will unlock a world of new possibilities.
The transition to Starlark configuration files represented a shift from static configuration to something more dynamic. This required refactoring a ton of code.
As part of refactoring code, we took the opportunity to shore up lots of the code base. PyOxidizer was the project author’s first real Rust project and a lot of bad practices (such as use of .unwrap()/panics) were prevalent. The code mostly now has proper error handling. Another new addition to the code is unit tests. While coverage still isn’t great, we now have tests performing meaningful packaging activities. So regressions should hopefully be less common going forward.
Because of the scale of the rewritten code in this release, it is expected that there are tons of bugs of regressions. This will likely be a transitional release with a more robust release to follow.
Backwards Compatibility Notes¶
- Support for building distributions/installers has been temporarily dropped.
- Support for installing license files has been temporarily dropped.
- Python interpreter configuration setting names have been changed to reflect names from Python 3.8’s interpreter initialization API.
.egg-info
directories are now ignored when scanning for Python resources on the filesystem (matching the behavior for.dist-info
directories).- The
pyoxidizer init
sub-command has been renamed toinit-rust-project
. - The
pyoxidizer app-path
sub-command has been removed. - Support for building distributions has been removed.
- The minimum Rust version to build has been increased from 1.31 to
1.36. This is mainly due to requirements from the
starlark
crate. We could potentially reduce the minimum version requirements again with minimal changes to 3rd party crates. - PyOxidizer configuration files are now
Starlark instead of TOML
files. The default file name is
pyoxidizer.bzl
instead ofpyoxidizer.toml
. All existing configuration files will need to be ported to the new format.
Bug Fixes¶
- The
repl
run mode now properly exits with a non-zero exit code if an error occurs. - Compiled C extensions now properly honor the
ext_package
argument passed tosetup()
, resulting in extensions which properly have the package name in their extension name (#26).
New Features¶
- A glob(include, exclude=None, strip_prefix=None) function has been added to config files to allow referencing existing files on the filesystem.
- The in-memory
MetaPathFinder
now implementsfind_module()
. - A
pyoxidizer init-config-file
command has been implemented to create just apyoxidizer.bzl
configuration file. - A
pyoxidizer python-distribution-info
command has been implemented to print information about a Python distribution archive. - The
EmbeddedPythonConfig()
config function now accepts alegacy_windows_stdio
argument to control the value ofPy_LegacyWindowsStdioFlag
(#190). - The
EmbeddedPythonConfig()
config function now accepts alegacy_windows_fs_encoding
argument to control the value ofPy_LegacyWindowsFSEncodingFlag
. - The
EmbeddedPythonConfig()
config function now accepts anisolated
argument to control the value ofPy_IsolatedFlag
. - The
EmbeddedPythonConfig()
config function now accepts ause_hash_seed
argument to control the value ofPy_HashRandomizationFlag
. - The
EmbeddedPythonConfig()
config function now accepts aninspect
argument to control the value ofPy_InspectFlag
. - The
EmbeddedPythonConfig()
config function now accepts aninteractive
argument to control the value ofPy_InteractiveFlag
. - The
EmbeddedPythonConfig()
config function now accepts aquiet
argument to control the value ofPy_QuietFlag
. - The
EmbeddedPythonConfig()
config function now accepts averbose
argument to control the value ofPy_VerboseFlag
. - The
EmbeddedPythonConfig()
config function now accepts aparser_debug
argument to control the value ofPy_DebugFlag
. - The
EmbeddedPythonConfig()
config function now accepts abytes_warning
argument to control the value ofPy_BytesWarningFlag
. - The
Stdlib()
packaging rule now now accepts an optionalexcludes
list of modules to ignore. This is useful for removing unnecessary Python packages such asdistutils
,pip
, andensurepip
. - The
PipRequirementsFile()
andPipInstallSimple()
packaging rules now accept an optionalextra_env
dict of extra environment variables to set when invokingpip install
. - The
PipRequirementsFile()
packaging rule now accepts an optionalextra_args
list of extra command line arguments to pass topip install
.
Other Relevant Changes¶
- PyOxidizer no longer requires a forked version of the
rust-cpython
project (thepython3-sys
andcpython
crates. All changes required by PyOxidizer are now present in the official project.
0.4.0¶
Released October 27, 2019.
Backwards Compatibility Notes¶
- The
setup-py-install
packaging rule now has itspackage_path
evaluated relative to the PyOxidizer config file path rather than the current working directory.
Bug Fixes¶
- Windows now explicitly requires dynamic linking against
msvcrt
. Previously, this wasn’t explicit. And sometimes linking the final executable would result in unresolved symbol errors because the Windows Python distributions used external linkage of CRT symbols and for some reason Cargo wasn’t dynamically linking the CRT. - Read-only files in Python distributions are now made writable to avoid future permissions errors (#123).
- In-memory
InspectLoader.get_source()
implementation no longer errors due to passing amemoryview
to a function that can’t handle it (#134). - In-memory
ResourceReader
now properly handles multiple resources (#128).
New Features¶
- Added an
app-path
command that prints the path to a packaged application. This command can be useful for tools calling PyOxidizer, as it will emit the path containing the packaged files without forcing the caller to parse command output. - The
setup-py-install
packaging rule now has anexcludes
option that allows ignoring specific packages or modules. .py
files installed into app-relative locations now have corresponding.pyc
bytecode files written.- The
setup-py-install
packaging rule now has anextra_global_arguments
option to allow passing additional command line arguments to thesetup.py
invocation. - Packaging rules that invoke
pip
orsetup.py
will now set aPYOXIDIZER=1
environment variable so Python code knows at packaging time whether it is running in the context of PyOxidizer. - The
setup-py-install
packaging rule now has anextra_env
option to allow passing additional environment variables tosetup.py
invocations. [[embedded_python_config]]
now supports asys_frozen
flag to control settingsys.frozen = True
.[[embedded_python_config]]
now supports asys_meipass
flag to control settingsys._MEIPASS = <exe directory>
.- Default Python distribution upgraded to 3.7.5 (from 3.7.4). Various dependency packages also upgraded to latest versions.
All Other Relevant Changes¶
- Built extension modules marked as app-relative are now embedded in the finaly binary rather than being ignored.
0.3.0¶
Released on August 16, 2019.
Backwards Compatibility Notes¶
- The
pyembed::PythonConfig
struct now has an additionalextra_extension_modules
field. - The default musl Python distribution now uses LibreSSL instead of OpenSSL. This should hopefully be an invisible change.
- Default Python distributions now use CPython 3.7.4 instead of 3.7.3.
- Applications are now built into directories named
apps/<app_name>/<target>/<build_type>
rather thanapps/<app_name>/<build_type>
. This enables builds for multiple targets to coexist in an application’s output directory. - The
program_name
field from the[[embedded_python_config]]
config section has been removed. At run-time, the current executable’s path is always used when callingPy_SetProgramName()
. - The format of embedded Python module data has changed. The
pyembed
crate andpyoxidizer
versions must match exactly or else thepyembed
crate will likely crash at run-time when parsing module data.
Bug Fixes¶
- The
libedit
extension variant for thereadline
extension should now link on Linux. Before, attempting to link a binary using this extension variant would result in missing symbol errors. - The
setup-py-install
[[packaging_rule]]
now performs actions to appeasesetuptools
, thus allowing installation of packages usingsetuptools
to (hopefully) work without issue (#70). - The
virtualenv
[[packaging_rule]]
now properly finds thesite-packages
directory on Windows (#83). - The
filter-include
[[packaging_rule]]
no longer requires bothfiles
andglob_files
be defined (#88). import ctypes
now works on Windows (#61).- The in-memory module importer now implements
get_resource_reader()
instead ofget_resource_loader()
. (The CPython documentation steered us in the wrong direction - https://bugs.python.org/issue37459.) - The in-memory module importer now correctly populates
__package__
in more cases than it did previously. Before, whether a module was a package was derived from the presence of afoo.bar
module. Now, a module will be identified as a package if the file providing it is named__init__
. This more closely matches the behavior of Python’s filesystem based importer. (#53)
New Features¶
- The default Python distributions have been updated. Archives are generally about half the size from before. Tcl/tk is included in the Linux and macOS distributions (but PyOxidizer doesn’t yet package the Tcl files).
- Extra extension modules can now be registered with
PythonConfig
instances. This can be useful for having the application embedding Python provide its own extension modules without having to go through Python build mechanisms to integrate those extension modules into the Python executable parts. - Built applications now have the ability to detect and use
terminfo
databases on the execution machine. This allows applications to interact with terminals properly. (e.g. the backspace key will now work in interactivepdb
sessions). By default, applications on non-Windows platforms will look forterminfo
databases at well-known locations and attempt to load them. - Default Python distributions now use CPython 3.7.4 instead of 3.7.3.
- A warning is now emitted when a Python source file contains
__file__
. This should help trace down modules using__file__
. - Added 32-bit Windows distribution.
- New
pyoxidizer distribution
command for producing distributable artifacts of applications. Currently supports building tar archives and.msi
and.exe
installers using the WiX Toolset. - Libraries required by C extensions are now passed into the linker as library dependencies. This should allow C extensions linked against libraries to be embedded into produced executables.
pyoxidizer --verbose
will now pass verbose to invokedpip
andsetup.py
scripts. This can help debug what Python packaging tools are doing.
All Other Relevant Changes¶
- The list of modules being added by the Python standard library is
no longer printed during rule execution unless
--verbose
is used. The output was excessive and usually not very informative.
0.2.0¶
Released on June 30, 2019.
Backwards Compatibility Notes¶
- Applications are now built into an
apps/<appname>/(debug|release)
directory instead ofapps/<appname>
. This allows debug and release builds to exist side-by-side.
Bug Fixes¶
- Extracted
.egg
directories in Python package directories should now have their resources detected properly and not as Python packages with the name*.egg
. site-packages
directories are now recognized as Python resource package roots and no longer have their contents packaged under asite-packages
Python package.
New Features¶
- Support for building and embedding C extensions on Windows, Linux, and macOS in many circumstances. See Native Extension Modules for support status.
pyoxidizer init
now accepts a--pip-install
option to pre-configure generatedpyoxidizer.toml
files with packages to install viapip
. Combined with the--python-code
option, it is now possible to createpyoxidizer.toml
files for a ready-to-use Python application!pyoxidizer
now accepts a--verbose
flag to make operations more verbose. Various low-level output is no longer printed by default and requires--verbose
to see.
All Other Relevant Changes¶
- Packaging now automatically creates empty modules for missing parent packages. This prevents a module from being packaged without its parent. This could occur with namespace packages, for example.
pip-install-simple
rule now passes--no-binary :all:
to pip.- Cargo packages updated to latest versions.
0.1.3¶
Released on June 29, 2019.
Bug Fixes¶
- Fix Python refcounting bug involving call to
PyImport_AddModule()
whenmode = module
evaluation mode is used. The bug would likely lead to a segfault when destroying the Python interpreter. (#31) - Various functionality will no longer fail when running
pyoxidizer
from a Git repository that isn’t the canonicalPyOxidizer
repository. (#34)
New Features¶
pyoxidizer init
now accepts a--python-code
option to control which Python code is evaluated in the produced executable. This can be used to create applications that do not run a Python REPL by default.pip-install-simple
packaging rule now supportsexcludes
for excluding resources from packaging. (#21)pip-install-simple
packaging rule now supportsextra_args
for adding parameters to the pip install command. (#42)
All Relevant Changes¶
- Minimum Rust version decreased to 1.31 (the first Rust 2018 release). (#24)
- Added CI powered by Azure Pipelines. (#45)
- Comments in auto-generated
pyoxidizer.toml
have been tweaked to improve understanding. (#29)
0.1.2¶
Released on June 25, 2019.
Bug Fixes¶
- Honor
HTTP_PROXY
andHTTPS_PROXY
environment variables when downloading Python distributions. (#15) - Handle BOM when compiling Python source files to bytecode. (#13)
All Relevant Changes¶
pyoxidizer
now verifies the minimum Rust version meets requirements before building.
0.1.1¶
Released on June 24, 2019.
Bug Fixes¶
pyoxidizer
binaries built from crates should now properly refer to an appropriate commit/tag in PyOxidizer’s canonical Git repository in auto-generatedCargo.toml
files. (#11)
0.1¶
Released on June 24, 2019. This is the initial formal release of PyOxidizer.
The first pyoxidizer
crate was published to crates.io
.
New Features¶
- Support for building standalone, single file executables embedding Python for 64-bit Windows, macOS, and Linux.
- Support for importing Python modules from memory using zero-copy.
- Basic Python packaging support.
- Support for jemalloc as Python’s memory allocator.
pyoxidizer
CLI command with basic support for managing project lifecycle.