.. py:currentmodule:: starlark_pyoxidizer .. _pyoxidizer_rust_generic_embedding: ============================================= Generic Python Embedding in Rust Applications ============================================= PyOxidizer can be used to produce artifacts facilitating the embedding of Python in a Rust application. This enables Rust developers to leverage PyOxidizer's technology for linking an embedded Python and managing the Python interpreter at run-time without a build-time dependency on PyOxidizer. This can greatly simplify development workflows at the cost of not being able to utilize the full power of PyOxidizer during builds. If you would like to use PyOxidizer as a build dependency, see :ref:`rust_projects` instead. Producing Embedding Artifacts ============================= The ``pyoxidizer generate-python-embedding-artifacts`` command can be used to write Python embedding artifacts into an output directory. e.g.:: $ pyoxidizer generate-python-embedding-artifacts artifacts $ ls artifacts default_python_config.rs libpython3.a packed-resources pyo3-build-config-file.txt stdlib tcl This command essentially runs ``pyoxidizer run-build-script`` with a default configuration file that produces artifacts suitable for generic Python embedding scenarios. The Written Artifacts ===================== ``pyoxidizer generate-python-embedding-artifacts`` will write the following files. A Linkable Python Library ------------------------- On UNIX platforms, this will likely be named ``libpython3.a``. On Windows, ``python3.dll`` and a ``pythonXY.dll`` (where ``XY`` is the major-minor Python version, e.g. ``39``). The library can be linked to provide an embedded Python interpreter. A Rust Source File Containing a Python Interpreter Config --------------------------------------------------------- The ``default_python_config.rs`` file contains the definition of a ``pyembed::OxidizedPythonInterpreterConfig`` Rust struct for defining an embedded Python interpreter. The config should *just work* with the other files produced. You can ``include!(...)`` this file in your Rust program if you want. Or you can ignore it and write your own configuration. Packed Resources for the Standard Library ----------------------------------------- A file containing the :ref:`python_packed_resources` for the Python standard library will be written. This file can be used by :ref:`oxidized_importer` to import the standard library efficiently. PyO3 Build Configuration ------------------------ A ``pyo3-build-config-file.txt`` file will be written defining a configuration for the ``pyo3-build-config`` crate which will link the ``libpython`` produced by this command. To use this configuration, set the ``PYO3_CONFIG_FILE`` environment variable to its **absolute** path and Python should get linked the way PyOxidizer would link it. Python Standard Library ----------------------- The ``stdlib`` directory will contain a copy of the Python standard library as it existed in the source distribution. .. note:: ``.pyc`` files are often not present and PyOxidizer doesn't yet provide a turnkey way to produce these files. Tcl/tk Support Files -------------------- The ``tcl`` directory will contain tcl/tk support files to support the ``tkinter`` Python module. Exporting Python Symbols ======================== The binary embedding libpython **must** export libpython's public symbols in order to support loading external extension modules / shared libraries (which need to be able to resolve libpython's public symbols). If libpython is loaded as a library, its symbols should be exported automatically. However, if libpython is embedded in an executable (non-library) binary, the default linker behavior is likely to not export symbols. To ensure a binary embedding libpython is exporting the proper symbols, you may need to define custom linker arguments. On Linux, you typically pass ``-export-dynamic`` to the linker, often via ``-Wl,-export-dynamic``. On macOS, you typically pass ``-rdynamic``. From a Cargo build script, you can add these extra arguments by printing a ``cargo:rustc-link-arg={}`` line. e.g. .. code-block:: rust let target_os = std::env::var("CARGO_CFG_TARGET_OS").expect("CARGO_CFG_TARGET_OS not defined"); match target_os.as_str() { "linux" => { println!("cargo:rustc-link-arg=-Wl,-export-dynamic"); } "macos" => { println!("cargo:rustc-link-arg=-rdynamic"); } _ => {} } Example Workflows ================= Embed Python With ``pyo3`` -------------------------- In this example, we will produce a Rust executable that uses the ``pyo3`` crate for interfacing with an embedded Python interpreter. We will not use PyOxidizer's ``pyembed`` crate or the ``oxidized_importer`` extension module for enhancing functionality of Python. First, create a new Rust project:: $ cargo init --bin pyapp Then edit its ``Cargo.toml`` to add the ``pyo3`` dependency. e.g. .. code-block:: toml [package] name = "pyapp" version = "0.1.0" edition = "2021" [dependencies] pyo3 = "0.14" And define a ``src/main.rs``: .. code-block:: rust use pyo3::prelude::*; fn main() -> PyResult<()> { unsafe { pyo3::with_embedded_python_interpreter(|py| { py.run("print('hello, world')", None, None) }) } } Now use ``pyoxidizer`` to generate the Python embedding artifacts:: $ pyoxidizer generate-python-embedding-artifacts pyembedded And finally build the Rust project using the PyO3 configuration file to tell PyO3 how to link the Python library we just generated:: $ PYO3_CONFIG_FILE=$(pwd)/pyembedded/pyo3-build-config-file.txt cargo run If you are doing this on a UNIX-like platform like Linux or macOS, chances are this fails with an error similar to the following:: Could not find platform independent libraries Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] Python path configuration: PYTHONHOME = (not set) PYTHONPATH = (not set) program name = 'python3' isolated = 0 environment = 1 user site = 1 import site = 1 sys._base_executable = '/usr/bin/python3' sys.base_prefix = '/install' sys.base_exec_prefix = '/install' sys.platlibdir = 'lib' sys.executable = '/usr/bin/python3' sys.prefix = '/install' sys.exec_prefix = '/install' sys.path = [ '/install/lib/python39.zip', '/install/lib/python3.9', '/install/lib/lib-dynload', ] Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Python runtime state: core initialized ModuleNotFoundError: No module named 'encodings' Current thread 0x00007ffa5abd9c80 (most recent call first): This is because the embedded Python library doesn't know how to locate the Python standard library. Essentially, the compiled Python library has some hard-coded defaults for where the Python standard library is located and its default logic is to search in those paths. The references to ``/install`` are referring to the build environment for the Python distributions. The quick fix for this is to define the ``PYTHONPATH`` environment variable to the location of the Python standard library. e.g.:: $ PYO3_CONFIG_FILE=$(pwd)/pyembedded/pyo3-build-config-file.txt PYTHONPATH=pyembedded/stdlib cargo run Could not find platform independent libraries Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] hello, world We still get some warnings. But our embedded Python interpreter does work! To make these config changes more permanent and to silence the remaining warnings, you'll need to customize the initialization of the Python interpreter using C APIs like the `Python Initialization Configuration `_ APIs. This requires a fair bit of ``unsafe`` code. Abstracting away the complexities of initializing the embedded Python interpreter is one of the reasons the :ref:`pyembed ` Rust crate exists. So if you want a simpler approach, consider using ``pyembed`` for controlling the Python interpreter. Embed Python with ``pyembed`` ----------------------------- In this example we'll use the :ref:`pyembed ` crate (part of the PyOxidizer project) for managing the embedded Python interpreter. First, create a new Rust project:: $ cargo init --bin pyapp Then edit its ``Cargo.toml`` to add the ``pyembed`` dependency. e.g. .. code-block:: toml [package] name = "pyapp" version = "0.1.0" edition = "2021" [dependencies] # Check for the latest version in case these docs are out of date. pyembed = "0.18" And define a ``src/main.rs``: .. code-block:: rust include!("../pyembedded/default_python_config.rs"); fn main() { // Get config from default_python_config.rs. let config = default_python_config(); let interp = pyembed::MainPythonInterpreter::new(config).unwrap(); // `py` is a `pyo3::Python` instance. interp.with_gil(|py| { py.run("print('hello, world')", None, None).unwrap(); }); } Now use ``pyoxidizer`` to generate the Python embedding artifacts:: $ pyoxidizer generate-python-embedding-artifacts pyembedded And finally build the Rust project using the PyO3 configuration file to tell PyO3 how to link the Python library we just generated:: $ PYO3_CONFIG_FILE=$(pwd)/pyembedded/pyo3-build-config-file.txt cargo run ... Finished dev [unoptimized + debuginfo] target(s) in 3.87s Running `target/debug/pyapp` hello, world If all goes as expected, this should *just work*!