Packaging Python Files Next to the Built Application

By default PyOxidizer will embed Python resources such as modules into the compiled executable. This is the ideal method to produce distributable Python applications because it can keep the entire application self-contained to a single executable and can result in performance wins.

But sometimes embedded resources into the binary isn’t desired or doesn’t work. Fear not: PyOxidizer has you covered!

Let’s give an example of this by attempting to package black, a Python code formatter.

We start by creating a new project:

$ pyoxidizer init-config-file black

Then edit the pyoxidizer.bzl file to have the following:

def make_exe():
    dist = default_python_distribution()

    config = dist.make_python_interpreter_config()
    config.run_mode = "module:black"

    exe = dist.to_python_executable(
        name="black",
    )

    for resource in exe.pip_install(["black==19.3b0"]):
        exe.add_location = "in-memory"
        exe.add_python_resource(resource)

    return exe

Then let’s attempt to build the application:

$ pyoxidizer build --path black
processing config file /home/gps/src/black/pyoxidizer.bzl
resolving Python distribution...
...

Looking good so far!

Now let’s try to run it:

$ pyoxidizer run --path black
Traceback (most recent call last):
  File "black", line 46, in <module>
  File "blib2to3.pygram", line 15, in <module>
NameError: name '__file__' is not defined
SystemError

Uh oh - that’s didn’t work as expected.

As the error message shows, the blib2to3.pygram module is trying to access __file__, which is not defined. As explained by __file__ and __cached__ Module Attributes, PyOxidizer doesn’t set __file__ for modules loaded from memory. This is perfectly legal as Python doesn’t mandate that __file__ be defined. So black (and every other Python file assuming the existence of __file__) is arguably buggy.

Let’s assume we can’t easily change the offending source code to work around the issue.

To fix this problem, we change the configuration file to install black relative to the built application. This requires changing our approach a little. Before, we ran exe.pip_install() from make_exe() to collect Python resources and added them to a PythonEmbeddedResources instance. This meant those resources were embedded in the self-contained PythonExecutable instance returned from make_exe().

Our auto-generated pyoxidizer.bzl file also contains an install target defined by the make_install() function. This target produces an FileManifest, which represents a collection of relative files and their content. When this type is resolved, those files are manifested on the filesystem. To package black’s Python resources next to our executable instead of embedded within it, we need to move the pip_install() invocation from make_exe() to make_install().

Change your configuration file to look like the following:

def make_python_dist():
    return default_python_distribution()

def make_exe(dist):
    let policy = dist.make_python_packaging_policy()
    policy.extension_module_filter = "all"
    policy.include_distribution_sources = True
    policy.include_distribution_resources = False
    policy.include_test = False

    python_config = dist.make_python_interpreter_config()
    python_config.run_mode = "module:black"
    python_config.module_search_paths = ["$ORIGIN/lib"]

    return dist.to_python_executable(
        name="black",
        packaging_policy=policy,
        config=python_config,
    )


def make_install(exe):
    files = FileManifest()

    files.add_python_resource(".", exe)

    files.add_python_resources("lib", exe.pip_install(["black==19.3b0"]))

    return files

register_target("python_dist", make_python_dist)
register_target("exe", make_exe, depends=["python_dist"])
register_target("install", make_install, depends=["exe"], default=True)

resolve_targets()

There are a few changes here.

We added a new make_dist() function and python_dist target to represent obtaining the Python distribution. This isn’t strictly required, but it helps avoid redundant work during execution.

We obtain a PythonInterpreterConfig via a method on the distribution. We then set module_search_paths = ["$ORIGIN/lib"] to adjust sys.path at run-time to include the lib directory next to the executable file. It allows the Python interpreter to import Python files on the filesystem using the standard importer.

The make_install() function/target has also gained a call to files.add_python_resources(). This method call takes the Python resources collected from running pip install black==19.3b0 and adds them to the FileManifest instance under the lib directory. When the FileManifest is resolved, those Python resources will be manifested as files on the filesystem (e.g. as .py and .pyc files).

With the new configuration in place, let’s re-build the application:

$ pyoxidizer build --path black install
...
packaging application into /home/gps/src/black/build/apps/black/x86_64-unknown-linux-gnu/debug
purging /home/gps/src/black/build/apps/black/x86_64-unknown-linux-gnu/debug
copying /home/gps/src/black/build/target/x86_64-unknown-linux-gnu/debug/black to /home/gps/src/black/build/apps/black/x86_64-unknown-linux-gnu/debug/black
resolving packaging state...
installing resources into 1 app-relative directories
installing 46 app-relative Python source modules to /home/gps/src/black/build/apps/black/x86_64-unknown-linux-gnu/debug/lib
...
black packaged into /home/gps/src/black/build/apps/black/x86_64-unknown-linux-gnu/debug

If you examine the output, you’ll see that various Python modules files were written to the output directory, just as our configuration file requested!

Let’s try to run the application:

$ pyoxidizer run --path black --target install
No paths given. Nothing to do 😴

Success!