Project History¶
Work on PyOxidizer started in November 2018 by Gregory Szorc.
Blog Posts¶
- C Extension Support in PyOxidizer (2019-06-30)
- Building Standalone Python Applications with PyOxidizer (2019-06-24)
- PyOxidizer Support for Windows (2019-01-06)
- Faster In-Memory Python Module Importing (2018-12-28)
- Distributing Standalone Python Applications (2018-12-18)
Version History¶
0.6.0¶
Released February 12, 2020.
Backwards Compatibility Notes¶
- The
default_python_distribution()
Starlark function now accepts aflavor
argument denoting the distribution flavor. - The
pyembed
crate no longer includes the auto-generated default configuration file. Instead, it is consumed by the application that instantiates a Python interpreter. - Rust projects for the main executable now utilize and require a Cargo build script
so metadata can be passed from
pyembed
to the project that is consuming it. - The
pyembed
crate is no longer added to created Rust projects. Instead, the generatedCargo.toml
will reference a version of thepyembed
crate identical to thePyOxidizer
version currently running. Or ifpyoxidizer
is running from a Git checkout of the canonicalPyOxidizer
Git repository, a local filesystem path will be used. - The fields of
EmbeddedPythonConfig
andpyembed::PythonConfig
have been renamed and reordered to align with Python 3.8’s config API naming. This was done for the Starlark type in version 0.5. We have made similar changes to 0.6 so naming is consistent across the various types.
Bug Fixes¶
- Module names without a
.
are now properly recognized when scanning the filesystem for Python resources and a package allow list is used (#223). Previously, if filtering scanned resources through an explicit list of allowed packages, the top-level module/package without a dot in its full name would not be passed through the filter.
New Features¶
- The
PythonDistribution()
Starlark function now accepts aflavor
argument to denote the distribution type. This allows construction of alternate distribution types. - The
default_python_distribution()
Starlark function now accepts aflavor
argument which can be set towindows_embeddable
to return a distribution based on the zip file distributions published by the official CPython project. - The
pyembed
crate and generated Rust projects now have variousbuild-mode-*
feature flags to control how build artifacts are built. See Rust Projects for more. - The
pyembed
crate can now be built standalone, without being bound to a specificPyOxidizer
configuration. - The
register_target()
Starlark function now accepts an optionaldefault_build_script
argument to define the default target when evaluating in Rust build script mode. - The
pyembed
crate now builds against publishedcpython
andpython3-sys
crates instead of a a specific Git commit. - Embedded Python interpreters can now be configured to run a file specified
by a filename. See the
run_file
argument of PythonInterpreterConfig(...)`.
Other Relevant Changes¶
- Rust internals have been overhauled to use traits to represent various types, namely Python distributions. The goal is to allow different Python distribution flavors to implement different logic for building binaries.
- The
pyembed
crate’sbuild.rs
has been tweaked so it can support calling out topyoxidizer
. It also no longer has a build dependency onpyoxidizer
.
0.5.1¶
Released January 26, 2020.
Bug Fixes¶
- Fixed bad Starlark example for building
black
in docs. - Remove resources attached to packages that don’t exist. (This was a regression in 0.5.)
- Warn on failure to annotate a package. (This was a regression in 0.5.)
- Building embedded Python resources now emits warnings when
__file__
is seen. (This was a regression in 0.5.) - Missing parent packages are now automatically added when constructing embedded resources. (This was a regression in 0.5.)
0.5.0¶
Released January 26, 2020.
General Notes¶
This release of PyOxidizer is significant rewrite of the previous version. The impetus for the rewrite is to transition from TOML to Starlark configuration files. The new configuration file format should allow vastly greater flexibility for building applications and will unlock a world of new possibilities.
The transition to Starlark configuration files represented a shift from static configuration to something more dynamic. This required refactoring a ton of code.
As part of refactoring code, we took the opportunity to shore up lots of the code base. PyOxidizer was the project author’s first real Rust project and a lot of bad practices (such as use of .unwrap()/panics) were prevalent. The code mostly now has proper error handling. Another new addition to the code is unit tests. While coverage still isn’t great, we now have tests performing meaningful packaging activities. So regressions should hopefully be less common going forward.
Because of the scale of the rewritten code in this release, it is expected that there are tons of bugs of regressions. This will likely be a transitional release with a more robust release to follow.
Backwards Compatibility Notes¶
- Support for building distributions/installers has been temporarily dropped.
- Support for installing license files has been temporarily dropped.
- Python interpreter configuration setting names have been changed to reflect names from Python 3.8’s interpreter initialization API.
.egg-info
directories are now ignored when scanning for Python resources on the filesystem (matching the behavior for.dist-info
directories).- The
pyoxidizer init
sub-command has been renamed toinit-rust-project
. - The
pyoxidizer app-path
sub-command has been removed. - Support for building distributions has been removed.
- The minimum Rust version to build has been increased from 1.31 to
1.36. This is mainly due to requirements from the
starlark
crate. We could potentially reduce the minimum version requirements again with minimal changes to 3rd party crates. - PyOxidizer configuration files are now
Starlark instead of TOML
files. The default file name is
pyoxidizer.bzl
instead ofpyoxidizer.toml
. All existing configuration files will need to be ported to the new format.
Bug Fixes¶
- The
repl
run mode now properly exits with a non-zero exit code if an error occurs. - Compiled C extensions now properly honor the
ext_package
argument passed tosetup()
, resulting in extensions which properly have the package name in their extension name (#26).
New Features¶
- A glob(include, exclude=None, strip_prefix=None) function has been added to config files to allow referencing existing files on the filesystem.
- The in-memory
MetaPathFinder
now implementsfind_module()
. - A
pyoxidizer init-config-file
command has been implemented to create just apyoxidizer.bzl
configuration file. - A
pyoxidizer python-distribution-info
command has been implemented to print information about a Python distribution archive. - The
EmbeddedPythonConfig()
config function now accepts alegacy_windows_stdio
argument to control the value ofPy_LegacyWindowsStdioFlag
(#190). - The
EmbeddedPythonConfig()
config function now accepts alegacy_windows_fs_encoding
argument to control the value ofPy_LegacyWindowsFSEncodingFlag
. - The
EmbeddedPythonConfig()
config function now accepts anisolated
argument to control the value ofPy_IsolatedFlag
. - The
EmbeddedPythonConfig()
config function now accepts ause_hash_seed
argument to control the value ofPy_HashRandomizationFlag
. - The
EmbeddedPythonConfig()
config function now accepts aninspect
argument to control the value ofPy_InspectFlag
. - The
EmbeddedPythonConfig()
config function now accepts aninteractive
argument to control the value ofPy_InteractiveFlag
. - The
EmbeddedPythonConfig()
config function now accepts aquiet
argument to control the value ofPy_QuietFlag
. - The
EmbeddedPythonConfig()
config function now accepts averbose
argument to control the value ofPy_VerboseFlag
. - The
EmbeddedPythonConfig()
config function now accepts aparser_debug
argument to control the value ofPy_DebugFlag
. - The
EmbeddedPythonConfig()
config function now accepts abytes_warning
argument to control the value ofPy_BytesWarningFlag
. - The
Stdlib()
packaging rule now now accepts an optionalexcludes
list of modules to ignore. This is useful for removing unnecessary Python packages such asdistutils
,pip
, andensurepip
. - The
PipRequirementsFile()
andPipInstallSimple()
packaging rules now accept an optionalextra_env
dict of extra environment variables to set when invokingpip install
. - The
PipRequirementsFile()
packaging rule now accepts an optionalextra_args
list of extra command line arguments to pass topip install
.
Other Relevant Changes¶
- PyOxidizer no longer requires a forked version of the
rust-cpython
project (thepython3-sys
andcpython
crates. All changes required by PyOxidizer are now present in the official project.
0.4.0¶
Released October 27, 2019.
Backwards Compatibility Notes¶
- The
setup-py-install
packaging rule now has itspackage_path
evaluated relative to the PyOxidizer config file path rather than the current working directory.
Bug Fixes¶
- Windows now explicitly requires dynamic linking against
msvcrt
. Previously, this wasn’t explicit. And sometimes linking the final executable would result in unresolved symbol errors because the Windows Python distributions used external linkage of CRT symbols and for some reason Cargo wasn’t dynamically linking the CRT. - Read-only files in Python distributions are now made writable to avoid future permissions errors (#123).
- In-memory
InspectLoader.get_source()
implementation no longer errors due to passing amemoryview
to a function that can’t handle it (#134). - In-memory
ResourceReader
now properly handles multiple resources (#128).
New Features¶
- Added an
app-path
command that prints the path to a packaged application. This command can be useful for tools calling PyOxidizer, as it will emit the path containing the packaged files without forcing the caller to parse command output. - The
setup-py-install
packaging rule now has anexcludes
option that allows ignoring specific packages or modules. .py
files installed into app-relative locations now have corresponding.pyc
bytecode files written.- The
setup-py-install
packaging rule now has anextra_global_arguments
option to allow passing additional command line arguments to thesetup.py
invocation. - Packaging rules that invoke
pip
orsetup.py
will now set aPYOXIDIZER=1
environment variable so Python code knows at packaging time whether it is running in the context of PyOxidizer. - The
setup-py-install
packaging rule now has anextra_env
option to allow passing additional environment variables tosetup.py
invocations. [[embedded_python_config]]
now supports asys_frozen
flag to control settingsys.frozen = True
.[[embedded_python_config]]
now supports asys_meipass
flag to control settingsys._MEIPASS = <exe directory>
.- Default Python distribution upgraded to 3.7.5 (from 3.7.4). Various dependency packages also upgraded to latest versions.
All Other Relevant Changes¶
- Built extension modules marked as app-relative are now embedded in the finaly binary rather than being ignored.
0.3.0¶
Released on August 16, 2019.
Backwards Compatibility Notes¶
- The
pyembed::PythonConfig
struct now has an additionalextra_extension_modules
field. - The default musl Python distribution now uses LibreSSL instead of OpenSSL. This should hopefully be an invisible change.
- Default Python distributions now use CPython 3.7.4 instead of 3.7.3.
- Applications are now built into directories named
apps/<app_name>/<target>/<build_type>
rather thanapps/<app_name>/<build_type>
. This enables builds for multiple targets to coexist in an application’s output directory. - The
program_name
field from the[[embedded_python_config]]
config section has been removed. At run-time, the current executable’s path is always used when callingPy_SetProgramName()
. - The format of embedded Python module data has changed. The
pyembed
crate andpyoxidizer
versions must match exactly or else thepyembed
crate will likely crash at run-time when parsing module data.
Bug Fixes¶
- The
libedit
extension variant for thereadline
extension should now link on Linux. Before, attempting to link a binary using this extension variant would result in missing symbol errors. - The
setup-py-install
[[packaging_rule]]
now performs actions to appeasesetuptools
, thus allowing installation of packages usingsetuptools
to (hopefully) work without issue (#70). - The
virtualenv
[[packaging_rule]]
now properly finds thesite-packages
directory on Windows (#83). - The
filter-include
[[packaging_rule]]
no longer requires bothfiles
andglob_files
be defined (#88). import ctypes
now works on Windows (#61).- The in-memory module importer now implements
get_resource_reader()
instead ofget_resource_loader()
. (The CPython documentation steered us in the wrong direction - https://bugs.python.org/issue37459.) - The in-memory module importer now correctly populates
__package__
in more cases than it did previously. Before, whether a module was a package was derived from the presence of afoo.bar
module. Now, a module will be identified as a package if the file providing it is named__init__
. This more closely matches the behavior of Python’s filesystem based importer. (#53)
New Features¶
- The default Python distributions have been updated. Archives are generally about half the size from before. Tcl/tk is included in the Linux and macOS distributions (but PyOxidizer doesn’t yet package the Tcl files).
- Extra extension modules can now be registered with
PythonConfig
instances. This can be useful for having the application embedding Python provide its own extension modules without having to go through Python build mechanisms to integrate those extension modules into the Python executable parts. - Built applications now have the ability to detect and use
terminfo
databases on the execution machine. This allows applications to interact with terminals properly. (e.g. the backspace key will now work in interactivepdb
sessions). By default, applications on non-Windows platforms will look forterminfo
databases at well-known locations and attempt to load them. - Default Python distributions now use CPython 3.7.4 instead of 3.7.3.
- A warning is now emitted when a Python source file contains
__file__
. This should help trace down modules using__file__
. - Added 32-bit Windows distribution.
- New
pyoxidizer distribution
command for producing distributable artifacts of applications. Currently supports building tar archives and.msi
and.exe
installers using the WiX Toolset. - Libraries required by C extensions are now passed into the linker as library dependencies. This should allow C extensions linked against libraries to be embedded into produced executables.
pyoxidizer --verbose
will now pass verbose to invokedpip
andsetup.py
scripts. This can help debug what Python packaging tools are doing.
All Other Relevant Changes¶
- The list of modules being added by the Python standard library is
no longer printed during rule execution unless
--verbose
is used. The output was excessive and usually not very informative.
0.2.0¶
Released on June 30, 2019.
Backwards Compatibility Notes¶
- Applications are now built into an
apps/<appname>/(debug|release)
directory instead ofapps/<appname>
. This allows debug and release builds to exist side-by-side.
Bug Fixes¶
- Extracted
.egg
directories in Python package directories should now have their resources detected properly and not as Python packages with the name*.egg
. site-packages
directories are now recognized as Python resource package roots and no longer have their contents packaged under asite-packages
Python package.
New Features¶
- Support for building and embedding C extensions on Windows, Linux, and macOS in many circumstances. See Native Extension Modules for support status.
pyoxidizer init
now accepts a--pip-install
option to pre-configure generatedpyoxidizer.toml
files with packages to install viapip
. Combined with the--python-code
option, it is now possible to createpyoxidizer.toml
files for a ready-to-use Python application!pyoxidizer
now accepts a--verbose
flag to make operations more verbose. Various low-level output is no longer printed by default and requires--verbose
to see.
All Other Relevant Changes¶
- Packaging now automatically creates empty modules for missing parent packages. This prevents a module from being packaged without its parent. This could occur with namespace packages, for example.
pip-install-simple
rule now passes--no-binary :all:
to pip.- Cargo packages updated to latest versions.
0.1.3¶
Released on June 29, 2019.
Bug Fixes¶
- Fix Python refcounting bug involving call to
PyImport_AddModule()
whenmode = module
evaluation mode is used. The bug would likely lead to a segfault when destroying the Python interpreter. (#31) - Various functionality will no longer fail when running
pyoxidizer
from a Git repository that isn’t the canonicalPyOxidizer
repository. (#34)
New Features¶
pyoxidizer init
now accepts a--python-code
option to control which Python code is evaluated in the produced executable. This can be used to create applications that do not run a Python REPL by default.pip-install-simple
packaging rule now supportsexcludes
for excluding resources from packaging. (#21)pip-install-simple
packaging rule now supportsextra_args
for adding parameters to the pip install command. (#42)
All Relevant Changes¶
- Minimum Rust version decreased to 1.31 (the first Rust 2018 release). (#24)
- Added CI powered by Azure Pipelines. (#45)
- Comments in auto-generated
pyoxidizer.toml
have been tweaked to improve understanding. (#29)
0.1.2¶
Released on June 25, 2019.
Bug Fixes¶
- Honor
HTTP_PROXY
andHTTPS_PROXY
environment variables when downloading Python distributions. (#15) - Handle BOM when compiling Python source files to bytecode. (#13)
All Relevant Changes¶
pyoxidizer
now verifies the minimum Rust version meets requirements before building.
0.1.1¶
Released on June 24, 2019.
Bug Fixes¶
pyoxidizer
binaries built from crates should now properly refer to an appropriate commit/tag in PyOxidizer’s canonical Git repository in auto-generatedCargo.toml
files. (#11)
0.1¶
Released on June 24, 2019. This is the initial formal release of PyOxidizer.
The first pyoxidizer
crate was published to crates.io
.
New Features¶
- Support for building standalone, single file executables embedding Python for 64-bit Windows, macOS, and Linux.
- Support for importing Python modules from memory using zero-copy.
- Basic Python packaging support.
- Support for jemalloc as Python’s memory allocator.
pyoxidizer
CLI command with basic support for managing project lifecycle.