Sanitizers

Motivation

Sanitizers are tools for detecting certain classes of bugs in code. The operating principle varies, though sanitizers commonly (but not always) rely on some form of compile-time instrumentation to change code in ways that expose bugs at runtime. Fuchsia uses a variety of sanitizers to discover and diagnose dangerous bugs that are otherwise difficult to find.

Sanitizers are enabled by build-time flags. Sanitizer builds are continuously exercised on Fuchsia's Continuous Integration (CI) and Commit Queue (CQ), and serve Fuchsia C/C++ and Rust developers.

Developers typically benefit from sanitizers with no special action required, and only need to pay attention to sanitizers when they detect a bug. However certain limitations apply. Continue reading to learn what sanitizers are supported and how to use them.

Supported sanitizers

Fuchsia currently supports the following sanitizers:

  • AddressSanitizer (ASan) detects instances of out-of-bounds access, use after free / return / scope, and double free.
  • LeakSanitizer (LSan) detects memory leaks. LeakSanitizer works like a conservative garbage collector in checking for leaks. Any allocations that are not reachable from reference roots (stacks of threads, registers of threads, global variables and thread-local variables) are treated as leaked.
  • ThreadSanitizer (TSan) detects data races (host-only).
  • UndefinedBehaviorSanitizer (UBSan) detects specific issues of relying on undefined program behavior.

The following sanitizer behavior is available in the Zircon kernel:

  • Physical Memory Manager (PMM) checker (pmm_checker) detects use-after-free bugs and stray DMAs.
  • Kernel AddressSanitizer (KASan) extends AddressSanitizer to kernel code in collaboration with the PMM.
  • Lockdep is a runtime lock validator that detects lock hazards such as deadlocks.

The following C/C++ compilation options are added by default to detect or prevent bugs at runtime:

  • -ftrivial-auto-var-init=pattern (see RFC) initializes automatic variables to a non-zero pattern to expose bugs related to reads from uninitialized memory.
  • ShadowCallStack and SafeStack harden the generated code against stack overflows.

Lastly, Fuchsia uses libFuzzer and syzkaller to perform coverage-directed fuzz testing. Fuzzers are similar to sanitizers in that they attempt to expose bugs in the code at runtime, and they are usually used in conjunction. Fuzzers are different from sanitizers in that fuzzers attempt to force the execution of production code into paths that may expose a bug.

Supported configurations

Sanitizers are currently supported in local builds and in CI/CQ under the following configurations:

  • bringup.x64
  • bringup.arm64
  • core.x64
  • core.arm64
  • zbi_tests.x64
  • zbi_tests.arm64

In addition, sanitizers apply to host tools.

Tests for all of the above are exercised on CI/CQ on qemu and Intel NUC. Other platforms are not tested with sanitizers due to resources and capacity issues, but you may test on these platforms locally using the build workflow below.

Additional tryjobs for configurations defined under //vendor may be shown in Gerrit and in CI consoles for certain signed-in users. Look for configurations with -asan in their name.

The sanitizers listed above are applied to C/C++ code. In addition, LSan is applied to Rust code for detecting Rust memory leaks.

Troubleshooting sanitizer issues

Build

Fuchsia platform build (in-tree)

To reproduce a sanitizer build, you can use build variants to enable the sanitizer:

fx set product --variant asan-ubsan --variant host_asan-ubsan

Alternatively you may select to only instrument certain binaries:

fx set product --variant asan-ubsan/executable_name

The selective instrumentation workflow is useful for testing locally on hardware where a fully instrumented build does not fit on the device.

Specifically to detect use-after-free bugs in kernel code you will need to enable the kernel PMM checker.

Out-of-tree build

When compiling with the Fuchsia toolchain it is sufficient to pass the -fsanitize= flag to indicate which sanitizers to use. See the compiler documentation.

When creating a Fuchsia package with instrumented components, you need to ensure that your package contains all runtime dependencies including the sanitizer runtime, which is distributed as part of the Clang toolchain, and instrumented C library, which is distributed as part of the Fuchsia SDK under sysroot.

Test

Test as you normally would, in your local workflow or on a CQ builder which has sanitizers enabled (the tryjob has asan in the name). If a sanitizer detects an issue then messages will be printed to the logs containing one of the following strings:

  • ERROR: AddressSanitizer
  • ERROR: LeakSanitizer
  • SUMMARY: UndefinedBehaviorSanitizer
  • WARNING: ThreadSanitizer

Following these messages you will find stack traces that identify the nature of the problem and point to the root cause. You can find these messages in fx log.

Note that the test that triggered the sanitizer may still appear as passing. Sanitizer issues don't manifest as test failures.

Issues detected by sanitizers typically have similar root causes. You may be able to find references for prior work by searching Fuchsia bugs for a bug with some of the same keywords that you're seeing in the sanitizer output.

Known issues

#[should_panic]

Fuchsia's Rust builds abort on panic!. This significantly reduces binary size. An unfortunate side effect is that tests that use the #[should_panic] attribute may falsely detect memory leaks. These tests emit an expected panic and then exit without unwinding, which means they don't free their heap allocations. To LeakSanitizer this is indistinguishable from a real memory leak.

If this issue affects your test then you can disable it in sanitizer builds by following this example.

See: Issue 88496: Rust tests that should_panic trigger leaksanitizer

Best practices

Ensure that your code is exercised by tests

Sanitizers expose bugs at runtime. Unless your code runs, such as within a test or generally on Fuchsia in such a way that's exercised by CI/CQ, sanitizers won't be able to expose bugs in your code.

The best way to ensure sanitizer coverage for your code is to ensure test coverage under the same configuration. Consult the guide on test coverage.

Don't suppress sanitizers in your code

Sanitizers may be suppressed for certain build targets. Most commonly this is used for issues that predate the introduction of sanitizer support, especially for issues in third party code that the Fuchsia project doesn't own.

Suppressed sanitizers should be considered tech debt, as they not only hide old bugs but keep you from discovering new bugs as they're introduced to your code. Ideally new suppressions should not be added, and existing suppressions should be removed and the underlying bugs fixed.

Suppressing sanitizers may be done by editing the BUILD.gn file that defines your executable target as follows:

executable("please_fix_the_bugs") {
  ...
  # TODO(https://fxbug.dev/42074368): delete the below and fix the memory bug.
  deps += [ "//build/config/sanitizers:suppress-asan-stack-use-after-return" ]
  # TODO(https://fxbug.dev/42074368): delete the below and fix the memory bug.
  deps += [ "//build/config/sanitizers:suppress-asan-container-overflow" ]
  # TODO(https://fxbug.dev/42074368): delete the below and fix the memory leak.
  deps += [ "//build/config/sanitizers:suppress-lsan.DO-NOT-USE-THIS" ]
}

The examples above demonstrate suppressing all sanitizers. However you should at most suppress sanitizers that are causing failures. Please track suppressions by filing a bug and referencing it in the comment as shown above.

Another common approach for disabling sanitizers works as follows:

executable("too_slow_when_built_with_asan") {
  ...
  exclude_toolchain_tags = [ "asan" ]
}

Both examples above suppress at the granularity of an entire executable. For finer-grained suppressions you may detect the presence of sanitizers in code. This is useful for instance for suppressing sanitizers in a particular test case, but not more broadly. For instance this is used by tests that intentionally introduce memory errors and test the sanitizer runtime itself.

For C/C++ see:

For Rust, you can follow this pattern:

#[cfg(test)]
mod tests {
    #[test]
    // TODO(https://fxbug.dev/42074368): delete the below and fix the leak
    #[cfg_attr(feature = "variant_asan", ignore)]
    fn test_that_leaks() {
        // ...
    }
}

Test for flakiness

Sanitizer errors may be flaky if the code under test's behavior is non-deterministic. For instance a memory leak may only happen under certain race conditions. If sanitizer errors appear flaky, consult the guide on testing for flakiness in CQ.

File good bugs

When encountering a sanitizer issue, file a bug containing all the troubleshooting information that's available.

Example: Issue 73214: ASAN use-after-scope in blobfs

The bug report contains:

  • The error provided by the sanitizer (ASan in this case).
  • Instructions on how to build & test to reproduce the error.
  • Details of the investigation that followed, with specific code pointers as needed.
  • References to relevant changes, such as in this case the change to fix the root cause for the bug.

Roadmap

Ongoing work:

Areas for future work:

  • ThreadSanitizer (TSan): detecting data races.
  • Kernel support for detecting concurrency bugs.
  • Extending sanitizer support for Rust, such as detecting memory safety bugs in Rust unsafe {} code blocks or across FFI calls, or detecting undefined behavior bugs.
  • MemorySanitizer (MSan): detecting reads from uninitialized memory.

See also: sanitizers in the 2021 roadmap.