The pyoxidizer Command Line Tool

The pyoxidizer command line tool is a frontend to the various functionality of PyOxidizer. See Components for more on the various components of PyOxidizer.

Settings

Cache Directory

pyoxidizer may need to download resources such as Python distributions and Rust toolchains from the Internet. These resources are cached in a per-user directory.

PyOxidizer chooses the first available directory from the following list to use as the cache:

  • The value of the environment variable PYOXIDIZER_CACHE_DIR.

  • $XDG_CACHE_HOME/pyoxidizer on Linux if XDG_CACHE_HOME is set.

  • $HOME/.cache/pyoxidizer on Linux if HOME is set.

  • $HOME/Library/Caches/pyoxidizer on macOS if HOME is set.

  • {FOLDERID_LocalAppData}/pyoxidizer on Windows.

  • ~/.pyoxidizer/cache

The pyoxidizer cache-clear command can be used to delete the contents of the cache.

Managed Rust Toolchain

PyOxidizer leverages the Rust programming language and its tooling for building binaries embedding Python.

By default, PyOxidizer will automatically download and use Rust toolchains (the Rust compiler, standard library, and Cargo) when their functionality is needed. PyOxidizer will store these Rust toolchains in the configured cache.

If you already have Rust installed on your machine and want PyOxidizer to use the existing Rust installation, either pass the --system-rust flag to pyoxidizer invocations or define the PYOXIDIZER_SYSTEM_RUST environment variable to any value. When the system Rust is being used, pyoxidizer will automatically use the cargo executable found on the current search path (typically the PATH environment variable).

Creating New Projects with init-config-file

The pyoxidizer init-config-file command will create a new pyoxidizer.bzl configuration file in the target directory:

$ pyoxidizer init-config-file pyapp

This should have printed out details on what happened and what to do next.

Creating New Rust Projects with init-rust-project

The pyoxidizer init-rust-project command creates a minimal Rust project configured to build an application that runs an embedded Python interpreter from a configuration defined in a pyoxidizer.bzl configuration file. Run it by specifying the directory to contain the new project:

$ pyoxidizer init-rust-project pyapp

This should have printed out details on what happened and what to do next.

The explicit creation of Rust projects to use PyOxidizer is not required. If your produced binaries only need to perform actions configurable via PyOxidizer configuration files (like running some Python code), an explicit Rust project isn’t required, as PyOxidizer can auto-generate a temporary Rust project at build time.

But if you want to supplement the behavior of the binaries built with Rust, an explicit and persisted Rust project can facilitate that. For example, you may want to run custom Rust code before, during, and after a Python interpreter runs in the process.

See PyOxidizer Rust Projects for more on the composition of Rust projects.

Building PyObject Projects with build

The pyoxidizer build command is probably the most important and used pyoxidizer command. This command evaluates a pyoxidizer.bzl configuration file by resolving targets in it.

By default, the default target in the configuration file is resolved. However, callers can specify a list of explicit targets to resolve. e.g.:

# Resolve the default target.
$ pyoxidizer build

# Resolve the "exe" and "install" targets, in that order.
$ pyoxidizer build exe install

PyOxidizer configuration files are effectively defining a build system, hence the name build for the command to resolve targets within.

Running the Result of Building with run

Target functions in PyOxidizer configuration files return objects that may be runnable. For example, a PythonExecutable returned by a target function that defines a Python executable binary can be run by executing a new process.

The pyoxidizer run command is used to attempt to run an object returned by a build target. It is effectively pyoxidizer build followed by running the returned object. e.g.:

# Run the default target.
$ pyoxidizer run

# Run the "install" target.
$ pyoxidizer run --target install

Analyzing Produced Binaries with analyze

The pyoxidizer analyze command is a generic command for analyzing the contents of executables and libraries. While it is generic, its output is specifically tailored for PyOxidizer.

Run the command with the path to an executable. For example:

$ pyoxidizer analyze build/apps/myapp/x86_64-unknown-linux-gnu/debug/myapp

Behavior is dependent on the format of the file being analyzed. But the general theme is that the command attempts to identify the run-time requirements for that binary. For example, for ELF binaries it will list all shared library dependencies and analyze glibc symbol versions and print out which Linux distributions it thinks the binary is compatible with.

Note

pyoxidizer analyze is not yet implemented for all executable file types that PyOxidizer supports.

Inspecting Python Distributions

PyOxidizer uses special pre-built Python distributions to build binaries containing Python.

These Python distributions are zstandard compressed tar files. Zstandard is a modern compression format that is really, really, really good. (PyOxidizer’s maintainer also maintains Python bindings to zstandard and has written about the benefits of zstandard on his blog. You should read that blog post so you are enlightened on how amazing zstandard is.) But because zstandard is relatively new, not all systems have utilities for decompressing that format yet. So, the pyoxidizer python-distribution-extract command can be used to extract the zstandard compressed tar archive to a local filesystem path.

Python distributions contain software governed by a number of licenses. This of course has implications for application distribution. See Licensing Considerations for more.

The pyoxidizer python-distribution-licenses command can be used to inspect a Python distribution archive for information about its licenses. The command will print information about the licensing of the Python distribution itself along with a per-extension breakdown of which libraries are used by which extensions and which licenses apply to what. This command can be super useful to audit for license usage and only allow extensions with licenses that you are legally comfortable with.

For example, the entry for the readline extension shows that the extension links against the ncurses and readline libraries, which are governed by the X11, and GPL-3.0 licenses:

readline
--------

Dependency: ncurses
Link Type: library

Dependency: readline
Link Type: library

Licenses: GPL-3.0, X11
License Info: https://spdx.org/licenses/GPL-3.0.html
License Info: https://spdx.org/licenses/X11.html

Note

The license annotations in Python distributions are best effort and can be wrong. They do not constitute a legal promise. Paranoid individuals may want to double check the license annotations by verifying with source code distributions, for example.

Debugging Resource Scanning and Identification with find-resources

The pyoxidizer find-resources command can be used to scan for resources in a given source and then print information on what’s found.

PyOxidizer’s packaging functionality scans directories and files and classifies them as Python resources which can be operated on. See Resource Types. PyOxidizer’s run-time importer/loader (oxidized_importer Python Extension) works by reading a pre-built index of known resources. This all works in contrast to how Python typically works, which is to put a bunch of files in directories and let the built-in importer/loader figure it out by dynamically probing for various files.

Because PyOxidizer has introduced structure where it doesn’t exist in Python and because there are many subtle nuances with how files are classified, there can be bugs in PyOxidizer’s resource scanning code.

The pyoxidizer find-resources command exists to facilitate debugging PyOxidizer’s resource scanning code.

Simply give the command a path to a directory or Python wheel archive and it will tell you what it discovers. e.g.:

$ pyoxidizer find-resources dist/oxidized_importer-0.1-cp38-cp38-manylinux1_x86_64.whl
parsing dist/oxidized_importer-0.1-cp38-cp38-manylinux1_x86_64.whl as a wheel archive
PythonExtensionModule { name: oxidized_importer }
PythonPackageDistributionResource { package: oxidized-importer, version: 0.1, name: LICENSE }
PythonPackageDistributionResource { package: oxidized-importer, version: 0.1, name: WHEEL }
PythonPackageDistributionResource { package: oxidized-importer, version: 0.1, name: top_level.txt }
PythonPackageDistributionResource { package: oxidized-importer, version: 0.1, name: METADATA }
PythonPackageDistributionResource { package: oxidized-importer, version: 0.1, name: RECORD }

Or give it the path to a site-packages directory:

$ pyoxidizer find-resources ~/.pyenv/versions/3.8.6/lib/python3.8/site-packages
...

This command needs to use a Python distribution so it knows what file extensions correspond to Python extensions, etc. By default, it will download one of the built-in distributions that is compatible with the current machine and use that. You can specify a --distributions-dir to use to cache downloaded distributions:

$ pyoxidizer find-resources --distributions-dir distributions /usr/lib/python3.8
...

Defining Extra Variables in Starlark Environment

Various pyoxidizer commands (like build and run) accept arguments to define extra variables in the Starlark environment in the VARS global dict. This feature can be used to parameterize and conditionalize the evaluation of configuration files.

Note

While we could inject global variables into the Starlark environment, since it is illegal to access an undefined symbol (there’s not even a way to test if a symbol is defined) and since we have no hook point to inject variables after the symbol has been defined, we resort to populating a global VARS dict with variables.

For example, let’s make the name of the built executable dynamic:

DEFAULT_APP_NAME = "default"

def make_exe(dist):
    dist = default_python_distribution()
    return dist.to_python_executable(name = VARS.get("app_name", DEFAULT_APP_NAME))

register_target("exe", make_exe)

resolve_targets()

Then let’s build it:

# Uses `default` as the application name.
$ pyoxidizer build

# Uses `my_app` as the application name.
$ pyoxidizer build --var app_name my_app

# Uses `env_name` as the application name via an environment variable.
$ APP_NAME=env_name pyoxidizer build --var-env app_name APP_NAME