What is Fuchsia CTF?
Please see the CTF overview for an explanation of what CTF is.
What is the CTF release schedule?
CTF has multiple releases with separate cadences:
| Release | Schedule |
|---|---|
| Canary | ~4 hrs |
| Milestone | ~6 weeks |
The canary release is created when new canary releases of the Fuchsia platform are created. Likewise, milestone releases are created when new milestone releases of the Fuchsia platform are created.
Milestone branches (e.g. releases/f7) often receive cherry picks. When this happens, a new CTF for that milestone is generated and automatically rolled into CI/CQ.
When will my CTF test start running on CQ?
The tip-of-tree version of your test will immediately begin running on CI/CQ. This version of the test does not guarantee backward compatibility.
When the next CTF release is cut, it will contain a snapshot of your test from tip-of-tree which will begin running as soon as that CTF release rolls into CI/CQ. This version of your test guarantees backward compatibility.
See this section above for the release schedule.
What test environment does CTF use in CQ?
See go/fuchsia-builder-viz. Look for builders whose names end in "-cts".
At minimum, all CTF tests run on the core.x64 image in the Fuchsia emulator.
How can I tell which version of a CTF test is failing?
CQ may run several versions of the same CTF test at a time: The version from tip-of-tree, from the latest canary release, and from a previous milestone release.
The tip-of-tree version of the test has a package name without a release version. For example:
fuchsia-pkg://fuchsia.com/memfs-test-package#meta/memfs-component.cmx
The canary or milestone versions of the test include the release version. For example:
fuchsia-pkg://fuchsia.com/memfs-test-package_6.20211109.1.3166058#meta/memfs-component.cmx
How do I reproduce a CTF test failure locally?
This depends on the version of the test you'd like to run.
To run the tip-of-tree version locally, you can do:
fx set //sdk/ctf/tests
fx test TEST_NAME
For example:
fx set //sdk/ctf/tests
fx test memfs-test-package
To run the release version locally, you can do:
fx set //sdk/ctf/release:tests
fx test TEST_NAME
For example:
fx set //sdk/ctf/release:tests
fx test memfs-test-package_6.20211109.1.3166058
Please see Run Fuchsia Tests for more information about how to run tests.
What do I do if a CTF test is blocking my CL?
This is a sign that your CL is breaking a part of the platform surface area. Please verify that there are no projects in downstream repositories that rely on the APIs and ABIs modified by your CL. If so, you will need to make a soft transition. The general worklow is as follows:
- Submit a CL that introduces the new behavior in your change and verify that the tip-of-tree version of the CTF test passes.
- Notify any downstream SDK Users of the upcoming breaking change, ask them to migrate and depend on the new behavior.
- Wait for the next CTF release to roll into CI/CQ.
- Submit a CL to remove the old behavior.
Are there any examples of CTF tests?
Yes! See //sdk/ctf/examples for some examples, or peruse the complete set of tests under //sdk/ctf/tests.
When and why should I write a CTF test?
You should write a CTF test if:
- Your software is part of the public or partner SDKs.
- You want CQ to prevent backward-incompatible changes to your software across multiple releases of the Fuchsia platform.
How do I retire a CTF test?
A CTF test should stop guarding against breaking changes once the SDK element it covers is removed (deprecated and no longer supported by the platform, even for legacy clients). This process is called test "retirement" and allows Fuchsia contributors to remove things from the SDK.
To retire an entire CTF test, delete the test at HEAD before the upcoming milestone release. The version of the test from the previous CTF release will continue running in CQ until the next release is cut.
To retire a few test cases, follow the same procedure: Delete the test cases at HEAD and wait for the next milestone release.
If you must immediately make changes to a previously released version of a test, you'll need to get approval from the Release Team to have the change cherry picked onto the appropriate release branch.
To verify that your change will succeed, you should sync your local Fuchsia checkout to the release branch and test the change yourself, first. After verifying, submit the CL and file a bug against the Release Team.
How do I temporarily disable a CTF test?
You can disable a test by adding the test's archive name to the list of
disabled_tests on the appropriate compatibility_test_suite target in
//sdk/ctf/release/BUILD.gn.
For example:
compatibility_test_suite("canary") {
disabled_tests = [ "my_test" ]
}
To find the archive name, check the //prebuilt/cts/$version>/$platform/cts/test_manifest.json file for the archive name corresponding to the test.
Please include a comment with a bug ID as a reminder to enable the test again. Tests should be enabled again within 72 hours.
Additional questions
For questions and clarification on this document, please reach out to this directory's owners or file a bug in the CTF bug component.