.. _rust_porting: ==================================== Porting a Python Application to Rust ==================================== PyOxidizer can be used to gradually port a Python application to Rust. What we mean by this is that Python code in an application would slowly be rewritten in Rust. Overview ======== When porting a Python application to Rust, the goal is to port Python code - and possibly Python C extension code - to Rust. Parts of the Rust code will presumably need to call into Python code and vice-versa. When porting code to Rust, there are essentially two *flavors* of Rust code that will be written and executed: 1. *Vanilla* Rust code 2. *Python-flavored* Rust code *Vanilla* Rust code is standard Rust code. It is what you would write if authoring a Rust-only project. *Python-flavored* Rust code is Rust code that interacts with the Python C API. It is regular Rust code, of course, but it is littered with references to `PyObject` and function calls into the Python C API (although these function calls may be abstracted so you don't have to use ``unsafe``). These different *flavors* of Rust code dictate different approaches to porting. Both *flavors*/approaches can be used simultaneously when porting an application to Rust. *Vanilla* Rust code will supplement the boilerplate Rust code that PyOxidizer uses to define and build a standalone executable embedded Python. See :ref:`extending_rust_projects` for more. *Python-flavored* Rust code typically involves writing Python extension modules in Rust. In this approach, you create a Python extension modules implemented in Rust and then make them available to the Python interpreter, which is managed by a Rust project. .. _extending_rust_projects: Extending Rust Projects ======================= When building an application from a standalone ``pyoxidizer.bzl`` file, PyOxidizer creates and builds a temporary, boilerplate Rust project behind the scenes. This Rust project has just enough code to initialize and run an embedded Python interpreter. That's the extent of the Rust code. PyOxidizer also supports persistent Rust projects. In this mode, you have full control over the Rust project and can add custom Rust code to it as you desire. In this mode, you can run Rust code independent of the Python interpreter. Supplementing the Rust code contained in your executable gives you the power to run arbitrary Rust code however you see fit. Here are some common scenarios this can enable: * Implementing argument parsing in Rust instead of Python. This could allow you to parse out the sub-command being invoked and dispatch to pure Rust code paths if possible, falling back to running Python code only if necessary. * Running a *forking* server, which doesn't start a Python interpreter until an event occurs. * Starting a thread with a high-performance application component implemented in Rust. For example, you could run a thread servicing a high-performance logging subsystem or HTTP server implemented in Rust and have that thread interact with a Python interpreter via a pipe or some other handle. Getting Started --------------- To extend a Rust project with custom Rust code, you'll first want to materialize the boilerplate Rust project used by PyOxidizer:: $ pyoxidizer init-rust-project myapp See :ref:`rust_projects` for details on the files materialized by this command. If you are using version control, now would be a good time to add the created files to version control. e.g.:: $ git add myapp $ git commit -m 'create boilerplate PyOxidizer project' From here, your next steps are to modify the Rust project to do something new and different. The auto-generated ``src/main.rs`` file contains the ``main()`` function used as the entrypoint for the Rust executable. The default file will simply instantiate a Python interpreter from a configuration, run that interpreter, then exit the process. To extend your application with custom Rust code, simply add custom code to ``main()``. e.g. .. code-block:: rust fn main() { println!("hello from Rust!") // Code auto-generated by ``pyoxidizer init-rust-project`` goes here. // ... } That is literally all there is to it! To build your custom Rust project, ``pyoxidizer build`` is the most robust way to do that. But it is also possible to use ``cargo build``. What Can Go Wrong ----------------- ``pyoxidizer`` Not Found or Rust Code Version Mismatch ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When using ``cargo build``, the ``pyoxidizer`` executable will be invoked behind the scenes. This requires that executable to be on ``PATH`` and for the version to be compatible with the Rust code you are trying to build. (The Rust APIs do change from time to time.) If the ``pyoxidizer`` executable is not on ``PATH`` or its version doesn't match the Rust code, you can forcefully tell the Rust build system which ``pyoxidizer`` executable to use:: $ PYOXIDIZER_EXE=/path/to/pyoxidizer cargo build ``thread 'main' panicked at 'jemalloc is not available in this build configuration'`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you see this error, the problem is that the Python interpreter configuration says to use *jemalloc* as the memory allocator but the Rust project was built without *jemalloc* support. This is likely because the default Rust project features in ``Cargo.toml`` don't include *jemalloc* by default. You can resolve this issue by either disabling jemalloc in the Python configuration or by enabling jemalloc in Rust. To disable jemalloc, open your ``pyoxidizer.bzl`` file and find the definition of ``raw_allocator``. You can set it to ``raw_allocator="system"`` so Python uses the system memory allocator instead of jemalloc. To enable jemalloc, you have a few options. First, you could build the Rust project with jemalloc support:: $ cargo build --features jemalloc Or, you modify ``Cargo.toml`` so the *jemalloc* feature is enabled by default:: .. code-block:: toml [features] default = ["build-mode-pyoxidizer-exe", "jemalloc"] *jemalloc* is typically a faster allocator than the system allocator. So if you care about performance, you may want to use it. Implementing Python Extension Modules in Rust ============================================= If you want to port a Python application to Rust, chances are that you will need to have Rust and Python code interact with each other. A common way to do this is to implement Python extensions in Rust so that Rust code will be invoked as a Python interpreter is running. There are two ways Rust-implemented Python extension modules can be consumed by PyOxidizer: 1. Define them via Python packaging tools (e.g. via a ``setup.py`` file for your Python package). 2. Define them in Rust code and register them as a *built-in* extension module. Python Built Rust Extension Modules ----------------------------------- If you've defined a Rust Python extension module via a Python package build tool (e.g. inside a ``setup.py``), PyOxidizer should automatically detect said extension module as part of packaging the corresponding Python package: there is no need to take special action to tell PyOxidizer it is a Rust extension, as this is all handled by Python packaging tools invoked as part of processing your ``pyoxidizer.bzl`` file. See :ref:`packaging` for more. The topic of authoring Python extension modules implemented in Rust is arguably outside the scope of this documentation. A search engine search for ``Rust Python extension`` should set you on the right track. Built-in Rust Extension Modules ------------------------------- A Python extension module is defined as a ``PyInit__`` function which is called to initialize an extension module. Typically, Python extension modules are compiled as standalone shared libraries, which are then loaded into a process, after which their ``PyInit__`` function is called. But Python has an additional mechanism for defining extension modules: *built-ins*. A *built-in* extension module is simply an extension module whose ``PyInit__`` function is already present in the process address space. Typically, these are extensions that are part of the Python distribution itself and are compiled directly into ``libpython``. When you instantiate a Python interpreter, you give it a list of the available *built-in* Python extension modules. And PyOxidizer's ``pyembed`` crate allows you to supplement the default list with custom extensions. To use *built-in* extension modules implemented in Rust, you'll need to implement said extension module in Rust, either as part of your application's Rust crate or as part of a different crate. Either way, you'll need to extend the boilerplate Rust project code (see :ref:`extending_rust_projects`) and tell it about additional *built-in* extension modules. See :ref:`rust_python_extension_modules` for instructions on how to do this. The tricky part here is implementing your Rust extension module. You probably want to use the `cpython `_ or `PyO3 `_ Rust crates for interfacing with the CPython API, as these provide an interface that is more ergonomic and doesn't require use of ``unsafe { }``. Use of these crates is beyond the scope of the PyOxidizer documentation. If you attempt to use the ``cpython`` or ``PyO3`` macros for defining a Python extension module, you'll likely run into problems because these assume that extension modules are standalone shared libraries, which isn't the case for *built-in* extension modules! If you attempt to use a separate Rust crate to define your extension module, you may run into Python symbol issues at link time because the build system for the ``cpython`` and ``PyO3`` crates will use their own logic for locating a Python interpreter and that interpreter may not have a configuration that is compatible with the one embedded in your PyOxidizer binary! At the end of the day, all you need to register a *built-in* extension module with PyOxidizer is an ``extern "C" fn () -> *mut python3_sys::PyObject``. Here is the boilerplate for defining a Python extension module in Rust (this uses the ``cpython`` crate). .. code-block:: rust use python3_sys as pyffi; use cpython::{PyErr, PyModule, PyObject}; static mut MODULE_DEF: pyffi::PyModuleDef = pyffi::PyModuleDef { m_base: pyffi::PyModuleDef_HEAD_INIT, m_name: std::ptr::null(), m_doc: std::ptr::null(), m_size: std::mem::size_of::() as isize, m_methods: 0 as *mut _, m_slots: 0 as *mut _, m_traverse: None, m_clear: None, m_free: None, }; #[allow(non_snake_case)] pub extern "C" fn PyInit_my_module() -> *mut pyffi::PyObject { let py = unsafe { cpython::Python::assume_gil_acquired() }; unsafe { if MODULE_DEF.m_name.is_null() { MODULE_DEF.m_name = "my_module".as_ptr() as *const _; MODULE_DEF.m_doc = "usage docs".as_ptr() as *const _; } } let module = unsafe { pyffi::PyModule_Create(&mut MODULE_DEF) }; if module.is_null() { return module; } let module = match unsafe { pyffi::from_owned_ptr(py, module).cast_into::(py) } { Ok(m) => m, Err(e) => { PyErr::from(e).restore(py); return std::ptr::null_mut(); } }; match module_init(py, &module) { Ok(()) => module.into_object().steal_ptr(), Err(e) => { e.restore(py); std::ptr::null_mut() } } } If you want a concrete example of what this looks like and how to do things like define Python types and have Python functions implemented in Rust, do a search for ``PyInit_oxidized_importer`` in the source code of the ``pyembed`` crate (which is part of the PyOxidizer repository) and go from there. The documentation for authoring Python extension modules and using the Python C API is well beyond the scope of this document. A good place to start is the `official documentation `_.