PythonInterpreterConfig

class starlark_pyoxidizer.PythonInterpreterConfig

This type configures the default behavior of the embedded Python interpreter.

Embedded Python interpreters are configured and instantiated using a Rust pyembed::OxidizedPythonInterpreterConfig data structure. The pyembed crate defines a default instance of this data structure with parameters defined by the settings in this type.

Note

If you are writing custom Rust code and constructing a custom pyembed::OxidizedPythonInterpreterConfig instance and don’t use the default instance, this config type is not relevant to you and can be omitted from your config file.

Danger

Some of the settings exposed by Python’s initialization APIs are extremely low level and brittle. Various combinations can cause the process to crash/exit ungracefully. Be very cautious when setting these low-level settings.

Instances are constructed by calling PythonDistribution.make_python_interpreter_config().

Instance state is managed via attributes.

There are a ton of attributes and most attributes are not relevant to most applications. The bulk of the attributes exist to give full control over Python interpreter initialization.

The following attributes control features provided by the pyembed Rust crate, which manages the embedded Python interpreter in generated executables. These attributes provide features and level of control over embedded Python interpreters beyond what is possible with Python’s initialization C API.

The following attributes correspond to fields of the PyPreConfig C struct used to initialize the Python interpreter.

The following attributes correspond to fields of the PyConfig C struct used to initialize the Python interpreter.

allocator_backend

(string)

Configures a custom memory allocator to be used by Python.

Accepted values are:

default

Let Python choose how to configure the allocator.

This will likely use the malloc(), free(), etc functions linked to the binary.

jemalloc

Use the jemalloc allocator.

(Not available on Windows.)

mimalloc
Use the mimalloc allocator (https://github.com/microsoft/mimalloc).
rust
Use Rust’s global allocator (whatever that may be).
snmalloc
Use the snmalloc allocator (https://github.com/microsoft/snmalloc).

The jemalloc, mimalloc, and snmalloc allocators require the presence of additional Rust crates. A run-time error will occur if these allocators are configured but the binary was built without these crates. (This should not occur when using pyoxidizer to build the binary.)

When a custom allocator is configured, the autogenerated Rust crate used to build the binary will configure the Rust global allocator (#[global_allocator] attribute) to use the specified allocator.

Important

The rust allocator is not recommended because it introduces performance overhead. But it may help with debugging in some situations.

Note

Both mimalloc and snmalloc require the cmake build tool to compile code as part of their build process. If this tool is not available in the build environment, you will encounter a build error with a message similar to failed to execute command: The system cannot find the file specified. (os error 2) is `cmake` not installed?.

The workaround is to install cmake or use a different allocator.

Note

snmalloc only supports targeting to macOS 10.14 or newer. You will likely see build errors when building a binary targeting macOS 10.13 or older.

Default is jemalloc on non-Windows targets and default on Windows. (The jemalloc-sys crate doesn’t work on Windows MSVC targets.)

allocator_raw

(bool)

Controls whether to install a custom allocator (defined by allocator_backend) into Python’s raw allocator domain (PYMEM_DOMAIN_RAW in Python C API speak).

Setting this to True will replace the system allocator (e.g. malloc(), free()) for this domain.

A value of True only has an effect if allocator_backend is some value other than default.

Defaults to True.

allocator_mem

(bool)

Controls whether to install a custom allocator (defined by allocator_backend) into Python’s mem allocator domain (PYMEM_DOMAIN_MEM in Python C API speak).

Setting this to True will replace pymalloc as the allocator for this domain.

A value of True only has an effect if allocator_backend is some value other than default.

Defaults to False.

allocator_obj

(bool)

Controls whether to install a custom allocator (defined by allocator_backend) into Python’s obj allocator domain (PYMEM_DOMAIN_OBJ in Python C API speak).

Setting this to True will replace pymalloc as the allocator for this domain.

A value of True only has an effect if allocator_backend is some value other than default.

Defaults to False.

allocator_pymalloc_arena

(bool)

Controls whether to install a custom allocator (defined by allocator_backend) into Python’s pymalloc to be used as its arena allocator.

The pymalloc allocator is used by Python by default and will use the system’s allocator functions (malloc(), VirtualAlloc(), etc) by default.

Setting this to True will have no effect if pymalloc is not being used (the allocator_mem and allocator_obj settings are True and have replaced pymalloc as the allocator backend for these domains).

A value of True only has an effect if allocator_backend is some value other than default.

Defaults to False.

allocator_debug

(bool)

Whether to enable debug hooks for Python’s memory allocators.

Enabling debug hooks enables debugging of memory-related issues in the Python interpreter. This setting effectively controls whether to call PyMem_SetupDebugHooks() during interpreter initialization. See the linked documentation for more.

Defaults to False.

oxidized_importer

(bool)

Whether to install the oxidized_importer meta path importer (oxidized_importer Python Extension) on sys.meta_path and sys.path_hooks during interpreter initialization. If installed, we will always occupy the first element in these lists.

Defaults to True.

filesystem_importer

(bool)

Whether to install the standard library path-based importer for loading Python modules from the filesystem.

If disabled, sys.meta_path and sys.path_hooks will not have entries provided by the standard library’s path-based importer.

Due to quirks in how the Python interpreter is initialized, the standard library’s path-based importer will be registered on sys.meta_path and sys.path_hooks for a brief moment when the interpreter is initialized. If sys.path contains valid entries that would be serviced by this importer and oxidized_importer isn’t able to service imports, it is possible for the path-based importer to be used to import some Python modules needed to initialize the Python interpreter. In many cases, this behavior is harmless. In all cases, the path-based importer is disabled after Python interpreter initialization, so future imports won’t be serviced by the path-based importer if it is disabled by this flag.

The filesystem importer is enabled automatically if PythonInterpreterConfig.module_search_paths is non-empty.

argvb

(bool)

Whether to expose a sys.argvb attribute containing bytes versions of process arguments.

On platforms where the process receives char * arguments, Python normalizes these values to unicode and makes them available via sys.argv. On platforms where the process receives wchar_t * arguments, Python may interpret the bytes as a certain encoding. This encoding normalization can be lossy.

Enabling this feature will give Python applications access to the raw bytes values of arguments that are actually used. The single or double width bytes nature of the data is preserved.

Unlike sys.argv which may chomp off leading arguments depending on the Python execution mode, sys.argvb has all the arguments used to initialize the process. The first argument is always the executable.

multiprocessing_auto_dispatch

(bool)

Controls whether the main execution routine of the binary will detect when the process is supposed to act as a multiprocessing worker process and will dispatch to multiprocessing automatically, instead of any other configured code.

Values of True have the same effect as calling multiprocessing.freeze_support() in your application code’s __main__ and replace the need to do so.

Default value is True.

See Automatic Detection and Dispatch of multiprocessing Processes for more.

multiprocessing_start_method

(str)

Controls how to call multiprocessing.set_start_method() upon the import of the multiprocessing module.

Accepted values are:

none

Do not call multiprocessing.set_start_method() automatically.

This mode is what Python programs do by default.

auto
Call multiprocessing.set_start_method() with the appropriate value for the environment. This likely maps to spawn on Windows and fork on non-Windows.
fork
Call with the value fork.
forkserver
Call with the value forkserver.
spawn
Call with the value spawn.

The default value is auto.

When set to a value that is not none, when oxidized_importer.OxidizedFinder services an import of the multiprocessing module, it will automatically call multiprocessing.set_start_method() to configure how worker processes are created.

If the multiprocessing module is not imported by oxidized_importer.OxidizedFinder, this setting has no effect.

sys_frozen

(bool)

Controls whether to set the sys.frozen attribute to True. If false, sys.frozen is not set.

Default is True.

sys_meipass

(bool)

Controls whether to set the sys._MEIPASS attribute to the path of the executable.

Setting this and sys_frozen to True will emulate the behavior of PyInstaller and could possibly help self-contained applications that are aware of PyInstaller also work with PyOxidizer.

Default is False.

terminfo_resolution

(string)

Defines how the terminal information database (terminfo) should be configured.

See Terminfo Database for more about terminal databases.

Accepted values are:

dynamic

Looks at the currently running operating system and attempts to do something reasonable.

For example, on Debian based distributions, it will look for the terminfo database in /etc/terminfo, /lib/terminfo, and /usr/share/terminfo, which is how Debian configures ncurses to behave normally. Similar behavior exists for other recognized operating systems.

If the operating system is unknown, PyOxidizer falls back to looking for the terminfo database in well-known directories that often contain the database (like /usr/share/terminfo).

none
The value none indicates that no configuration of the terminfo database path should be performed. This is useful for applications that don’t interact with terminals. Using none can prevent some filesystem I/O at application startup.
static:<path>

Indicates that a static path should be used for the path to the terminfo database.

This values consists of a : delimited list of filesystem paths that ncurses should be configured to use. This value will be used to populate the TERMINFO_DIRS environment variable at application run time.

terminfo is not used on Windows and this setting is ignored on that platform.

write_modules_directory_env

(string or None)

Environment variable that defines a directory where modules-<UUID> files containing a \n delimited list of loaded Python modules (from sys.modules) will be written upon interpreter shutdown.

If this setting is not defined or if the environment variable specified by its value is not present at run-time, no special behavior will occur. Otherwise, the environment variable’s value is interpreted as a directory, that directory and any of its parents will be created, and a modules-<UUID> file will be written to the directory.

This setting is useful for determining which Python modules are loaded when running Python code.

config_profile

(string)

This attribute controls which set of default values to use for attributes that aren’t explicitly defined. It effectively controls which C API to use to initialize the PyPreConfig instance.

Accepted values are:

isolated

Use the isolated configuration.

This configuration is appropriate for applications existing in isolation and not behaving like python executables.

python

Use the Python configuration.

This configuration is appropriate for applications attempting to behave like a python executable would.

allocator

(string or None)

Controls the value of PyPreConfig.allocator.

Accepted values are:

None
Use the default.
not-set
PYMEM_ALLOCATOR_NOT_SET
default
PYMEM_ALLOCATOR_DEFAULT
debug
PYMEM_ALLOCATOR_DEBUG
malloc
PYMEM_ALLOCATOR_MALLOC
malloc-debug
PYMEM_ALLOCATOR_MALLOC_DEBUG
py-malloc
PYMEM_ALLOCATOR_PYMALLOC
py-malloc-debug
PYMEM_ALLOCATOR_PYMALLOC_DEBUG
configure_locale

(bool or None)

Controls the value of PyPreConfig.configure_locale.

coerce_c_locale

(string or None)

Controls the value of PyPreConfig.coerce_c_locale.

Accepted values are:

LC_CTYPE
Read LC_CTYPE
C
Coerce the C locale.
coerce_c_locale_warn

(bool or None)

Controls the value of PyPreConfig.coerce_c_locale_warn.

development_mode

(bool or None)

Controls the value of PyPreConfig.development_mode.

isolated

(bool or None)

Controls the value of PyPreConfig.isolated.

legacy_windows_fs_encoding

(bool or None)

Controls the value of PyPreConfig.legacy_windows_fs_encoding.

parse_argv

(bool or None)

Controls the value of PyPreConfig.parse_argv.

use_environment

(bool or None)

Controls the value of PyPreConfig.use_environment.

utf8_mode

(bool or None)

Controls the value of PyPreConfig.utf8_mode.

base_exec_prefix

(string or None)

Controls the value of PyConfig.base_exec_prefix.

base_executable

(string or None)

Controls the value of PyConfig.base_exectuable.

base_prefix

(string or None)

Controls the value of PyConfig.base_prefix.

buffered_stdio

(bool or None)

Controls the value of PyConfig.buffered_stdio.

bytes_warning

(string or None)

Controls the value of PyConfig.bytes_warning.

Accepted values are:

  • None
  • none
  • warn
  • raise
check_hash_pycs_mode

(string or None)

Controls the value of PyConfig.check_hash_pycs_mode.

Accepted values are:

  • None
  • always
  • never
  • default
configure_c_stdio

(bool or None)

Controls the value of PyConfig.configure_c_stdio.

dump_refs

(bool or None)

Controls the value of PyConfig.dump_refs.

exec_prefix

(string or None)

Controls the value of PyConfig.exec_prefix.

executable

(string or None)

Controls the value of PyConfig.executable.

fault_handler

(bool or None)

Controls the value of PyConfig.fault_handler.

filesystem_encoding

(string or None)

Controls the value of PyConfig.filesystem_encoding.

filesystem_errors

(string or None)

Controls the value of PyConfig.filesystem_errors.

hash_seed

(int or None)

Controls the value of PyConfig.hash_seed.

PyConfig.use_hash_seed will automatically be set if this attribute is defined.

home

(string or None)

Controls the value of PyConfig.home.

import_time

Controls the value of PyConfig.import_time.

inspect

(bool or None)

Controls the value of PyConfig.inspect.

install_signal_handlers

(bool or None)

Controls the value of PyConfig.install_signal_handlers.

interactive

(bool or None)

Controls the value of PyConfig.interactive.

legacy_windows_stdio

(bool or None)

Controls the value of PyConfig.legacy_windows_stdio.

malloc_stats

(bool or None)

Controls the value of PyConfig.malloc_stats.

module_search_paths

(list[string] or None)

Controls the value of PyConfig.module_search_paths.

This value effectively controls the initial value of sys.path.

The special string $ORIGIN in values will be expanded to the absolute path of the directory of the executable at run-time. For example, if the executable is /opt/my-application/pyapp, $ORIGIN will expand to /opt/my-application and the value $ORIGIN/lib will expand to /opt/my-application/lib.

Setting this to a non-empty value also has the side-effect of setting filesystem_importer = True

optimization_level

(int or None)

Controls the value of PyConfig.optimization_level.

Allowed values are:

  • None
  • 0
  • 1
  • 2

This setting is only relevant if write_bytecode is True and Python modules are being imported from the filesystem using Python’s standard filesystem importer.

parser_debug

(bool or None)

Controls the value of PyConfig.parser_debug.

pathconfig_warnings

(bool or None)

Controls the value of PyConfig.pathconfig_warnings.

prefix

(string or None)

Controls the value of PyConfig.prefix.

program_name

(string or None)

Controls the value of PyConfig.program_name.

pycache_prefix

(string or None)

Controls the value of PyConfig.pycache_prefix.

python_path_env

(string or None)

Controls the value of PyConfig.pythonpath_env.

quiet

(bool or None)

Controls the value of PyConfig.quiet.

run_command

(string or None)

Controls the value of PyConfig.run_command.

run_filename

(string or None)

Controls the value of PyConfig.run_filename.

run_module

(string or None)

Controls the value of PyConfig.run_module.

show_ref_count

(bool or None)

Controls the value of PyConfig.show_ref_count.

site_import

(bool or None)

Controls the value of PyConfig.site_import.

The site module is typically not needed for standalone/isolated Python applications.

skip_first_source_line

(bool or None)

Controls the value of PyConfig.skip_first_source_line.

stdio_encoding

(string or None)

Controls the value of PyConfig.stdio_encoding.

stdio_errors

(string or None)

Controls the value of PyConfig.stdio_errors.

tracemalloc

(bool or None)

Controls the value of PyConfig.tracemalloc.

user_site_directory

(bool or None)

Controls the value of PyConfig.user_site_directory.

verbose

(bool or None)

Controls the value of PyConfig.verbose.

warn_options

(list[string] or None)

Controls the value of PyConfig.warn_options.

write_bytecode

(bool or None)

Controls the value of PyConfig.write_bytecode.

This only influences the behavior of Python standard path-based importer (controlled via filesystem_importer).

x_options

(list[string] or None)

Controls the value of PyConfig.xoptions.

Starlark Caveats

The PythonInterpreterConfig Starlark type is backed by a Rust data structure. And when attributes are retrieved, a copy of the underlying Rust struct field is returned.

This means that if you attempt to mutate a Starlark value (as opposed to assigning an attribute), the mutation won’t be reflected on the underlying Rust data structure.

For example:

config = dist.make_python_interpreter_config()

# assigns vec!["foo", "bar"].
config.module_search_paths = ["foo", "bar"]

# Creates a copy of the underlying list and appends to that copy.
# The stored value of `module_search_paths` is still `["foo", "bar"]`.
config.module_search_paths.append("baz")

To append to a list, do something like the following:

value = config.module_search_paths
value.append("baz")
config.module_search_paths = value