RFC-0129: Python Support in Fuchsia

Summary

This RFC defines the Python versioning and script requirements for Python sources within the Fuchsia project.

Motivation

On the host, not all Python scripts in the tree are executed in a deterministic manner. Some scripts assume a vendored prebuilt interpreter, others are left to the system Python. Further, there is no consistency with respect to the version of Python supported on a per-script basis.

The lack of a deterministic Python environment causes scripts to occasionally break when environmental changes are made (for example, removing the /usr/bin/python symlink). This requires the user to either restore the local environment to the expected state, or make local modifications to the script(s) in question to be compatible with their environment.

Lastly, as of January 1st, 2020, Python 2.x has been officially sunset by python.org.

Design

This design replaces the user's local host environment with a vendored prebuilt Python interpreter for all Fuchsia Project Python sources. This vendored Python SHALL be the supported Python language revision.

The current vendored Python version SHALL be kept within a current python.org support window. As the version of Python is updated, a reasonable transition period will be granted to allow sources to be made compatible with the new language version.

For Fuchsia Project Python sources, this RFC terminates support for Python language versions prior to 3.8

• Python 2.7 is henceforth deprecated.
• Backwards compatibility prior to 3.8 is not required.

Python Sources as Scripts

A Python source file may be invoked as a script by (for example) invoking it on the command line or supplying it as an argument to an invocation of the Python interpreter. Scripts are denoted by those runtime environments whose module __name__ is set equal to "__main__".

Executable Python sources are REQUIRED to defer to the vendored Python for execution. This may be accomplished using different means:

• Through documentation, instruct users to directly invoke the vendored interpreter supplying the candidate script as an interpreter argument.
• Defer to the vendored Python using a wrapping script or other mechanism which contains logic sufficient to invoke the interpreter (e.g. using something like //scripts/hermetic-env).
• Assuming a script interpreter source directive (i.e. shebang) which ultimately causes the script to be executed using the vendored interpreter.

Python Shebang

Python scripts intended to be directly invoked MUST contain a shebang which ultimately references the vendored Python. The appropriate shebang line to use is:

#!/usr/bin/env fuchsia-vendored-python

As the use of a shebang implies a dependency on the host environment, this shebang is intentionally selected to meet the following hermetic-driven goals:

• Be less-likely to collide with an existing tool, alias, or macro in the user's host environment.
• Be agnostic to the ordering by which the user added //.jiri_root/bin to their $PATH.

Hermetic Python Environments

Tools such as venv and vpython serve to assemble an ephemeral and hermetic Python environment. Packages installed are local to these hermetic environments and do not affect the local system installation base. The use of hermetic Python tools is permitted with the stipulation that they are derived from the vendored Python.

Python VirtualEnv (venv)

A venv may be instantiated by invoking the vendored Python interpreter with the args of -m venv my_venv. This will assemble a venv in the local directory my_venv. The venv may then be activated and interacted with per typical venv usage. Installing venv-local packages is permitted.

Chromium vpython

The vpython utility is another form of hermetic Python environment manager. It sources its interpreter from the $PATH and assumes the first interpreter whose --version is compatible with the input vpython.Spec protobuf. To use vpython, ensure the vendored Python is configured in the $PATH at the time vpython is invoked. Further, the vpython.Spec protobuf must reflect the current supported language version. Use of vpython wheels is permitted to the extent they comply with the current supported language version.

Implementation

All Python sources in Fuchsia Project repositories will be updated to language version 3.8 by end of Q3 2021. Those sources that are meant to be directly invoked will have their shebang point to the vendored Python according to the semantics below.

To facilitate a vendored Python interpreter as the shebang target, the following will be modified:

1. To //.jiri_root/bin, a new fuchsia-vendored-python symlink will be added which links to //scripts/fuchsia-vendored-python.
2. To //scripts, a new fuchsia-vendored-python symlink will be added which links to //scripts/fuchsia-vendored-python3.8.
3. The fuchsia-vendored-python3.8 script will be added to //scripts and be implemented as:

#!/bin/bash
hermetic-env python3.8 "$@"

Assuming //.jiri_root/bin is among the host environment's PATH (an assumed environmental configuration), this will dispatch to the prebuilt interpreter regardless of what interpreter version(s) are installed on the host.

If //.jiri_root/bin is not part of the host environment's path, it will be necessary to symlink //.jiri_root/bin/fuchsia-vendored-python to an appropriate location (e.g. ~/bin or similar).

It is expected that new Python versions will occasionally need to be rolled in via //integration. The procedure for doing this is straightforward:

1. Via //integration, download and install the prebuilt binaries to their appropriate locations.
2. Add a new bootstrapping script (e.g. fuchsia-vendored-python3.9) to //scripts.
3. Move the //scripts/fuchsia-vendored-python symlink to the new bootstrapping script.
4. (optionally) delete the old bootstrapping script(s).

Performance

N/A

Ergonomics

Ergonomically, a deterministic Python language version, and vendored prebuilt interpreter will much more comfortable of an experience. Users will not be bogged down by the class of problems caused by relying on their host environment.

Backwards Compatibility

Backwards compatibility for Python language version 2.x is no longer required, and this RFC terminates support for Python language version 2.x. The intent of this RFC is to eliminate the maintenance burden from Python 2 backwards compatibility.

Security considerations

python.org has sunset Python 2.x as end-of-life, and ceased all maintenance, including security fixes. The Fuchsia project no longer supports Python language version 2.x.

Security is improved by migrating to a maintained Python distribution (version 3.8+) and the more strict typing between byte and character array types present in the newer Python language versions.

Additionally, having assumed an exact prebuilt version rather than arbitrary host system installation will reduce maintenance burden and developer frustration due to version skew and local installation variations.

Privacy considerations

N/A

Testing

N/A

Documentation

The following documentation will need to be updated to reflect the policy here:

• //docs/development/build/build_system/policies.md
• //docs/development/languages/python/python_style.md
• //docs/get-started/get_fuchsia_source.md

Drawbacks, alternatives, and unknowns

Sun-setting a programming language revision in a backwards-incompatible manner can be problematic during migration. Not all users will anticipate this change, and some may not have themselves moved on from Python 2.x.

However, the industry has been aware of the need to migrate from 2.7 to Python 3.x for multiple years now, and this change shouldn't be unexpected.

Prior art and references

• Sunsetting Python 2
• Python VirtualEnv
• Chromium vpython