PyOxidizer Rust Projects

PyOxidizer uses Rust projects to build binaries embedding Python. This documentation describes how they work. If you are only interested in embedding Python in a Rust application without using PyOxidizer as part of the regular development workflow, see Generic Python Embedding in Rust Applications for instructions.

If you just have a standalone configuration file (such as when running pyoxidizer init-config-file), a temporary Rust project will be created as part of building binaries. That project will be built, its build artifacts copied, and the temporary project will be deleted.

If you use pyoxidizer init-rust-project to initialize a PyOxidizer application, the Rust project exists side-by-side with the PyOxidizer configuration file and can be modified like any other Rust project.

Layout

Generated Rust projects all have a similar layout:

$ find pyapp -type f | grep -v .git
.cargo/config
Cargo.toml
Cargo.lock
build.rs
pyapp.exe.manifest
pyapp-manifest.rc
pyoxidizer.bzl
src/main.rs

The Cargo.toml file is the configuration file for the Rust project. Read more in the official Cargo documentation. The magic lines in this file to enable PyOxidizer are the following:

[package]
build = "build.rs"

[dependencies]
pyembed = ...

These lines declare a dependency on the pyembed package, which holds the smarts for running an embedded Python interpreter.

In addition, the build = "build.rs" helps to dynamically configure the crate.

Next let’s look at src/main.rs. If you aren’t familiar with Rust projects, the src/main.rs file is the default location for the source file implementing an executable. If we open that file, we see a fn main() { line, which declares the main function for our executable. The file is relatively straightforward. We import some symbols from the pyembed crate. We then construct a config object, use that to construct a Python interpreter, then we run the interpreter and pass its exit code to exit(). Succinctly, we instantiate and run an embedded Python interpreter. That’s our executable.

The pyoxidizer.bzl is our auto-generated PyOxidizer configuration file.

Crate Features

The auto-generated Rust project defines a number of features to control behavior. These are documented in the sections below.

build-mode-standalone

This is the default build mode. It is enabled by default.

This build mode uses default Python linking behavior and feature detection as implemented by the pyo3. It will attempt to find a python in PATH or from the PYO3_PYTHON environment variable and link against it.

This is the default mode for convenience, as it enables the pyembed crate to build in the most environments. However, the built binaries will have a dependency against a foreign libpython and likely aren’t suitable for distribution.

This mode does not attempt to invoke pyoxidizer or find artifacts it would have built. It is possible to build the pyembed crate in this mode if the pyo3 crate can find a Python interpreter. But, the pyembed crate may not be usable or work in the way you want it to.

This mode is intended to be used for performing quick testing on the pyembed crate. It is quite possible that linking errors will occur in this mode unless you take additional actions to point Cargo at appropriate libraries.

pyembed has a dependency on Python 3.8+. If an older Python is detected, it can result in build errors, including unresolved symbol errors.

build-mode-pyoxidizer-exe

A pyoxidizer executable will be run to generate build artifacts.

The path to this executable can be defined via the PYOXIDIZER_EXE environment variable. Otherwise PATH will be used.

At build time, pyoxidizer run-build-script will be run. A PyOxidizer configuration file will be discovered using PyOxidizer’s heuristics for doing so. OUT_DIR will be set if running from cargo, so a pyoxidizer.bzl next to the main Rust project being built should be found and used.

pyoxidizer run-build-script will resolve the default build script target by default. To override which target should be resolved, specify the target name via the PYOXIDIZER_BUILD_TARGET environment variable. e.g.:

$ PYOXIDIZER_BUILD_TARGET=build-artifacts cargo build

build-mode-prebuilt-artifacts

This mode tells the build script to reuse artifacts that were already built. (Perhaps you called pyoxidizer build or pyoxidizer run-build-script outside the context of a normal cargo build.)

In this mode, the build script will look for artifacts in the directory specified by PYOXIDIZER_ARTIFACT_DIR if set, falling back to OUT_DIR.

global-allocator-jemalloc

This feature will configure the Rust global allocator to use jemalloc.

global-allocator-mimalloc

This feature will configure the Rust global allocator to use mimalloc.

global-allocator-snmalloc

This feature will configure the Rust global allocator to use snmalloc.

allocator-jemalloc

This configures the pyembed crate with support for having the Python interpreter use the jemalloc allocator.

allocator-mimalloc

This configures the pyembed crate with support for having the Python interpreter use the mimalloc allocator.

allocator-snmalloc

This configures the pyembed crate with support for having the Python interpreter use the snmalloc allocator.

Using Cargo With Generated Rust Projects

Building a PyOxidizer-enabled Rust project with cargo is not as turn-key as it is with pyoxidizer. That’s because PyOxidizer has to do some non-conventional things to get Rust projects to build in very specific ways. Commands like pyoxidizer build abstract away all of this complexity for you.

If you do want to use cargo directly, the following sections will give you some tips.

Linking Against Python

Autogenerated Rust projects need to link against Python. The link settings are ultimately derived from the pyo3-build-config crate via the dependency on pyo3 in the pyembed crate. (pyembed is part of the PyOxidizer project.)

See Building for documentation on how to configure the Python linking settings of the pyembed crate.

Important

If you don’t set environment variables to point pyembed/pyo3 at a custom Python, Python won’t be linked into your binary the way that pyoxidizer build would link it.

For best results, you’ll want to use a Python library built the same way that PyOxidizer builds it. The pyoxidizer generate-python-embedding-artifacts command can be used to produce such a library along with a PyO3 configuration file for linking it. See Generic Python Embedding in Rust Applications for details.

Cargo Configuration

Linking a custom libpython into the final Rust binary can be finicky, especially when statically linking on Windows.

The auto-generated .cargo/config file defines some custom compiler settings to enable things to work. However, this only works for some configurations. The file contains some commented out settings that may need to be set for some configurations (e.g. the standalone_static Windows distributions).

Please consult this file if running into build errors when not building through pyoxidizer.

Also consider porting these linker settings to your own crate.

Building with Cargo and PyOxidizer

It is possible to use cargo to drive builds but still invoke pyoxidizer as part of the build. This is an advanced workflow that hasn’t been optimized for ergonomics and it requires setting many environment variables to get things to play together nicely.

This is essentially a 2 step process:

1. Generate build artifacts consumed by the pyembed and pyo3 crates.

2. Build with cargo.

Starting from a project freshly created with pyoxidizer init-rust-project sample, you’ll first need to generate required build artifacts:

$ CARGO_MANIFEST_DIR=. \
  TARGET=x86_64-unknown-linux-gnu \
  PROFILE=debug \
  OUT_DIR=target/out \
  pyoxidizer run-build-script build.rs

This command will evaluate your PyOxidizer configuration file and write output files. The environment variables simulate the Cargo environment from which this command is usually called.

If all works correctly, build artifacts will be written to target/out.

Then you can run cargo to build your crate, consuming the built artifacts:

$ PYOXIDIZER_ARTIFACT_DIR=$(pwd)/target/out \
  PYO3_CONFIG_FILE=$(pwd)/target/out/pyo3-build-config-file.txt \
  cargo build \
    --no-default-features \
    --features "build-mode-prebuilt-artifacts global-allocator-jemalloc allocator-jemalloc"

After building, you should find an executable in target/debug/.

Note

On Windows, you should remove the features referencing jemalloc, as this feature isn’t available on Windows.

Important

When building through cargo, additional files are not copied into place next to the built crate. This can include required shared libraries, extension modules, and even the Python standard library. This can result in the embedded Python interpreter not working correctly.

You may need to manually copy additional files for the built binary to work as expected. The easiest way to do this is to build your project with pyoxidizer build and copy the files from its output.