Packaging Pitfalls

While PyOxidizer is capable of building fully self-contained binaries containing a Python application, many Python packages and applications make assumptions that don’t hold inside PyOxidizer. This section talks about all the things that can go wrong when attempting to package a Python application.

C and Other Native Extension Modules

Many Python packages compile extension modules to native code. (Typically C is used to implement extension modules.)

The way this typically works is some build system (often distutils via a setup.py script) produces a shared library file containing the extension. On Linux and macOS, the file extension is typically .so. On Windows, it is .pyd. When an import is requested, Python’s importing mechanism looks for these files in addition to normal .py and .pyc files. If an extension module is found, Python will dlopen() the file and load the shared library into the process. It will then call into an initialization function exported by that shared library to obtain a Python module instance.

Python packaging has defined various conventions for distributing pre-compiled extension modules in wheels. If you see an e.g. <package>-<version>-cp38-cp38-win_amd64.whl, <package>-<version>-cp38-cp38-manylinux2014_x86_64.whl, or <package>-<version>-cp38-cp38-macosx_10_9_x86_64.whl file, you are installing a Python package with a pre-compiled extension module. Inside the wheel is a shared library providing the extension module. And that shared library is configured to work with a Python distribution (typically CPython) built in a specific way. e.g. with a libpythonXY shared library exporting Python symbols.

PyOxidizer currently has some support for extension modules. The way this works depends on the platform and Python distribution.

Dynamically Linked Python Distributions on Windows

When using a dynamically linked Python distribution on Windows (e.g. via the flavor="standalone_dynamic" argument to default_python_distribution(flavor="standalone", build_target=None), PyOxidizer:

  • Supports importing shared library extension modules (e.g. .pyd files) from memory.
  • Automatically detects and uses .pyd files from pre-built binary packages installed as part of packaging.
  • Automatically detects and uses .pyd files produced during package building.

However, there are caveats to this support!

PyOxidizer doesn’t currently support resolving additional library dependencies from .pyd extension modules / shared libraries when importing from memory. If an extension module depends on another shared library (almost certainly a .dll) outside the normal set of libraries (namely the C Runtime and other common Windows system DLLs), you will need to manually package this library next to the application .exe. Failure to do this could result in a failure at import time.

PyOxidizer does support loading shared library extension modules from .pyd files on the filesystem like a typical Python program. So if you cannot make in-memory extension module importing work, you can fall back to packaging a .pyd file in a directory registered on sys.path, as set through the PythonInterpreterConfig(...) Starlark primitive.

Extension Modules Everywhere Else

If PyOxidizer is not able to easily reuse a Python extension module built or distributed in a traditional manner, it will attempt to compile the extension module from source in a way that is compatible with the PyOxidizer distribution and application configuration.

The way PyOxidizer achieves this is a bit crude, but effective.

When PyOxidizer invokes pip or setup.py to build a package, it installs a modified version of distutils into the invoked Python’s sys.path. This modified distutils changes the behavior of some key build steps (notably how C extensions are built) such that the build emits artifacts that PyOxidizer can use to integrate the extension module into a custom binary. For example, on Linux, PyOxidizer copies the intermediate object files produced by the build and links them into the same binary containing Python: PyOxidizer completely ignores the shared library that is or would typically be produced.

If setup.py scripts are following the traditional pattern of using distutils.core.Extension to define extension modules, things tend to just work (assuming extension modules are supported by PyOxidizer for the target platform). However, if setup.py scripts are doing their own monkeypatching of distutils, rely on custom build steps or types to compile extension modules, or invoke separate Python processes to interact with distutils, things may break.

If you run into an extension module packaging problem that isn’t recorded here or on the static page, please file an issue so it may be tracked.

Identifying PyOxidizer

Python code may want to know whether it is running in the context of PyOxidizer.

At packaging time, pip and setup.py invocations made by PyOxidizer should set a PYOXIDIZER=1 environment variable. setup.py scripts, etc can look for this environment variable to determine if they are being packaged by PyOxidizer.

At run-time, PyOxidizer will always set a sys.oxidized attribute with value True. So, Python code can test whether it is running in PyOxidizer like so:

import sys

if getattr(sys, 'oxidized', False):
    print('running in PyOxidizer!')