Testing Third Party Changes


This doc is mostly focused on testing changes to Scudo, GWP-ASan and LLVM-libc, as such the doc uses these as examples but the concepts will transfer to any third_party/ project. It is intended both for Fuchsia developers and for developers of upstream projects who may not have a Fuchsia checkout or be familiar with the build.

Getting the checkout

For those who don't already have a Fuchsia checkout, you can follow the steps here.

Running the tests

Further documentation on building and running tests can be found in Configure and build Fuchsia and Run Fuchsia Tests respectively. For a quick summary see below.

All methods will refer back to how the tests are actually run. All third_party/ tests will be run differently, if they are run at all. A good place to start is to grep around *.gn{,i} files to see where in the build these projects are being referred to. In the case of Scudo, it is part of libc's build and most of the build logic concerning it and its tests can be found in //zircon/system/ulib/c/scudo/BUILD.gn. For this particular target it is depended on by //zircon/system/ulib/c:tests.

First run fx set core.qemu-x64 --with //zircon/system/ulib/c:tests, see the build docs on more details, but this configuration is a good default. This only needs to be run once.

Then run ffx emu start --headless. This starts the emulator and will build anything it needs. Also run fx serve either in the background or foreground but create a new terminal session in the case of latter. Both of these need only be run once, but need to be run again if they die or your machine is restarted. Note: if you jiri update it is good practice to rebuild and restsart the emulator and package serve.

To run the tests you can look to see which test target ends up depending on the test you are interested in. Scudo tests are rolled into libc tests which are libc-unittests-pkg, some GWP-ASan tests are there as well as gwp-asan-test-pkg. Run fx test -v libc-unittests-pkg gwp-asan-test-pkg to run just those tests. When in doubt, fx test -v can be run without arguments to run all tests, which if you only included //zircon/system/ulib/c:tests won't take too long.

Testing a change locally

To test a change locally, simply apply the patch to the respective //third_party directory. Usually this will be //third_party/${proj}/src, though in the case of GWP-ASan it is in //third_party/scudo/gwp-asan. After the diff has been applied run the tests as normal and targets will rebuild against your new changes. This is a great way to do rapid iteration on a change you are submitting upstream or testing someone elses change before they submit.

Testing a failed roll

All of scudo, gwp-asan and llvm-libc, are rolled in by auto-rollers which will fail if they break the build or if tests stop passing, either theirs or other in tree tests. To test this locally, apply the diff created by the auto-roller to //integration, this will be just a one line change bumping up the revision on that project. To modify the commit hash manually to test at another revision, either change the file by hand or run jiri edit -project=${proj}=${hash} ${manifest}, where manifest will likely be //integration/fuchsia/third_party/flower. Then run jiri update -local-manifest this will pull in the new changes specified in your //integration directory. Here, jiri will likely give an error that there are uncommitted changes in //integration, this won't actually stop jiri from correctly updating the third party project. If this bothers you, create a new branch in //integration and commit your changes, this will make jiri simply warn that //integration is not on JIRI_HEAD. To ensure your project was correctly updated, run cat ${path_to_proj}/src/.git/HEAD. Run tests as normal.

[Googlers Only] Leveraging bots

Only Google employees can make push changes to //integration and view tqr/

Some failures will only happen on certain devices and without having access to them locally, it is impossible to reproduce locally. These cases should be exceedingly rare, and as such the workflow is not particularly well refined. First, find the project you want to test, in the case of Scudo it looks like: <project name="scudo" gitsubmoduleof="fuchsia" path="third_party/scudo/src" remote="https://llvm.googlesource.com/scudo" revision="b0c7c7b80b6dfa2cd0b5dae98cb1ea33d31d2497"/> Make a git clone of the remote, here that is https://llvm.googlesource.com/scudo, and push to an accessible git host. See go/gob/users/user-repository, go/gob/users/team-repository and go/gob/users/new-host. Change the project's remote to point to your host repos address. Make changes to your clone and change the revision to the revision you want to test at. After that, push your change, go to the review and add the specific bots you wish to test.

Example: tqr/634497