Google is committed to advancing racial equity for Black communities. See how.

Sanitizers

Motivation

Sanitizers are tools for detecting certain classes of bugs in code. The operating principle varies, though sanitizers commonly (but not always) rely on some form of compile-time instrumentation to change code in ways that expose bugs at runtime. Fuchsia uses a variety of sanitizers to discover and diagnose dangerous bugs that are otherwise difficult to find.

Sanitizers are enabled by build-time flags. Sanitizer builds are continuously exercised on Fuchsia's Continuous Integration (CI) and Commit Queue (CQ), and serve Fuchsia C/C++ and Rust developers.

Developers typically benefit from sanitizers with no special action required, and only need to pay attention to sanitizers when they detect a bug. However certain limitations apply. Continue reading to learn what sanitizers are supported and how to use them.

Supported sanitizers

Fuchsia currently supports the following sanitizers:

The following sanitizer behavior is available in the Zircon kernel:

  • Physical Memory Manager (PMM) checker (pmm_checker) detects use-after-free bugs and stray DMAs.
  • Kernel AddressSanitizer (KASan) extends AddressSanitizer to kernel code in collaboration with the PMM.
  • Lockdep is a runtime lock validator that detects lock hazards such as deadlocks.

The following C/C++ compilation options are added by default to detect or prevent bugs at runtime:

  • -ftrivial-auto-var-init=pattern (see RFC) initializes automatic variables to a non-zero pattern to expose bugs related to reads from uninitialized memory.
  • ShadowCallStack and SafeStack harden the generated code against stack overflows.

Lastly, Fuchsia uses libFuzzer and syzkaller to perform coverage-directed fuzz testing. Fuzzers are similar to sanitizers in that they attempt to expose bugs in the code at runtime, and they are usually used in conjunction. Fuzzers are different from sanitizers in that fuzzers attempt to force the execution of production code into paths that may expose a bug.

Supported configurations

Sanitizers are currently supported in local builds and in CI/CQ under the following configurations:

  • bringup.x64
  • bringup.arm64
  • core.x64
  • core.arm64
  • zbi_tests.x64
  • zbi_tests.arm64

In addition, sanitizers apply to host tools.

Tests for all of the above are exercised on CI/CQ on qemu and Intel NUC.

Additional tryjobs for configurations defined under //vendor may be shown in Gerrit and in CI consoles for certain signed-in users. Look for configurations with -asan in their name.

The sanitizers listed above are applied to C/C++ code. In addition, LSan is applied to Rust code for detecting Rust memory leaks.

Troubleshooting sanitizer issues

Build

To reproduce a sanitizer build, specify the sanitizer variants:

fx set product --variant asan-ubsan --variant host_asan-ubsan

Alternatively you may select to only instrument certain binaries:

fx set product --variant asan-ubsan/executable_name

Specifically to detect use-after-free bugs in kernel code you will need to enable the kernel PMM checker.

Test

Test as you normally would, in your local workflow or on CQ. If a sanitizer detects an issue during test runtime then the test failure will include a descriptive error message indicating the nature of the problem and a stack trace pointing to the root cause.

Issues detected by sanitizers typically have similar root causes. You may be able to find references for prior work by searching Fuchsia bugs for a bug with some of the same keywords that you're seeing in the sanitizer output.

See also: UBSan issues on Open Projects.

Best practices

Ensure that your code is exercised by tests

Sanitizers expose bugs at runtime. Unless your code runs, such as within a test or generally on Fuchsia in such a way that's exercised by CI/CQ, sanitizers won't be able to expose bugs in your code.

The best way to ensure sanitizer coverage for your code is to ensure test coverage under the same configuration. Consult the guide on test coverage.

Don't suppress sanitizers in your code

Sanitizers may be suppressed for certain build targets. Most commonly this is used for issues that predate the introduction of sanitizer support, especially for issues in third party code that the Fuchsia project doesn't own.

Suppressed sanitizers should be considered tech debt, as they not only hide old bugs but keep you from discovering new bugs as they're introduced to your code. Ideally new suppressions should not be added, and existing suppressions should be removed and the underlying bugs fixed.

Suppressing sanitizers may be done by editing the BUILD.gn file that defines your executable target as follows:

executable("please_fix_the_bugs") {
  ...
  # TODO(fxbug.dev/12345): delete the below and fix the memory bug.
  deps += [ "//build/config/sanitizers:suppress-asan-stack-use-after-return" ]
  # TODO(fxbug.dev/12345): delete the below and fix the memory bug.
  deps += [ "//build/config/sanitizers:suppress-asan-container-overflow" ]
  # TODO(fxbug.dev/12345): delete the below and fix the memory leak.
  deps += [ "//build/config/sanitizers:suppress-lsan.DO-NOT-USE-THIS" ]
  # TODO(fxbug.dev/12345): delete the below and fix undefined behavior.
  configs += [ "//build/config:temporarily_disable_ubsan_do_not_use" ]
}

The example above demonstrates suppressing all sanitizers. However you should at most suppress sanitizers that are causing failures. Please track suppressions by filing a bug and referencing it in the comment as shown above.

Furthermore the example above suppresses at the granularity of an entire executable. For finer-grained suppressions you may detect the presence of sanitizers in code. See:

Test for flakiness

Sanitizer errors may be flaky if the code under test's behavior is non-deterministic. For instance a memory leak may only happen under certain race conditions. If sanitizer errors appear flaky, consult the guide on testing for flakiness in CQ.

File good bugs

When encountering a sanitizer issue, file a bug containing all the troubleshooting information that's available.

Example: Issue 73214: ASAN use-after-scope in blobfs

The bug report contains:

  • The error provided by the sanitizer (ASan in this case).
  • Instructions on how to build & test to reproduce the error.
  • Details of the investigation that followed, with specific code pointers as needed.
  • References to relevant changes, such as in this case the change to fix the root cause for the bug.

Roadmap

Ongoing work:

Areas for future work:

  • ThreadSanitizer (TSan): detecting data races.
  • Kernel support for detecting concurrency bugs.
  • Extending sanitizer support for Rust, such as detecting memory safety bugs in Rust unsafe {} code blocks or across FFI calls, or detecting undefined behavior bugs.
  • MemorySanitizer (MSan): detecting reads from uninitialized memory.

See also: sanitizers in the 2021 roadmap.