When your fuzzer runs, it searches for inputs that crash the program or violate checked conditions. When the fuzzer finds and reports such a test input, it is evidence of a bug that needs to be resolved.
Typically, you might use
fx fuzz when first developing your fuzzer. This can often produce results
immediately. After a fuzzer has been submitted it will be run at scale by
ClusterFuzz, and any results it finds will be filed as bugs.
Handle results from
After running a fuzzer with
fx fuzz, the tool can be used to
report any artifacts found by running:
fx fuzz check package/fuzzer
Additionally, output logs and any results are stored to the output directory. By default, this is:
A different location can be set passing the
--output option to
The most recent fuzzer run is symbolically linked to:
Crashes and other artifacts will have file name like
type-of-finding.SHA1-hash-of-input. The file
contents will be the input bytes themselves.
For example, examining a crash produced by a toy example might look like the following:
hd /crash-2fda3f03bb699c8a2151724b64b6e36c3b986aea00000000 2a 48 49 21 2a 00 08 00 00 00 2a 48 49 00 0a 66 |*HI!*.....*HI..f| 00000010 4a 33 00 0a f9 |J3...| 00000015
Reproduce a result
You can execute the fuzzer with this input again using
fx fuzz repro. For example:
fx fuzz repro examples/toy crash-2fda3f03bb699c8a2151724b64b6e36c3b986aea
If the result is reproducible, this will produce a symbolized log including a stack trace. The top of this stack trace is likely to be the error handling by libFuzzer and/or the sanitizer, and the bottom will likely be the fuzzer engine itself.
For example, the relevant function where the
panic occurred in the following stack trace is
#0 0x000023c56780a61e in _$LT$std..sys_common..backtrace.._print..DisplayBacktrace$u20$as$u20$core..fmt..Display$GT$::fmt::h510ae2e0fe71c88c <>+0x19161e #1 0x000023c56783399c in core::fmt::write::hb61ef49191e76a74 <>+0x1ba99c #2 0x000023c5678009b1 in std::io::Write::write_fmt::h41df81fb2b8460af <>+0x1879b1 #3 0x000023c56780eb92 in std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h4e9a8e3c4f33b3f4 <>+0x195b92 #4 0x000023c56780e87c in std::panicking::default_hook::hd85edcd963c04eae <>+0x19587c #5 0x000023c56780f271 in std::panicking::rust_panic_with_hook::h8960558cc7e69505 <>+0x196271 #6 0x000023c5677973d5 in std::panicking::begin_panic::h97c6d4cd722282c5 /b/s/w/ir/k/rust/src/libstd/panicking.rs:397 <>+0x11e3d5 #7 0x000023c56777f2d0 in _toy_example_arbitrary_lib_rustc_static::toy_example::h573322211ba71c22 ../../out/default/../../examples/fuzzers/rust/src/lib.rs:22 <>+0x1062d0 #8 0x000023c567780a03 in _toy_example_arbitrary_lib_rustc_static::_::toy_example_arbitrary::hc02c288d17b25ac2 ../../out/default/../../examples/fuzzers/rust/src/lib.rs:35 <>+0x107a03 #9 0x000023c56778136c in LLVMFuzzerTestOneInput ../../out/default/../../examples/fuzzers/rust/src/lib.rs:33 <>+0x10836c #10 0x000023c56772ab86 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) <>+0xb1b86 #11 0x000023c567716ae5 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) ../recipe_cleanup/clangshYTOG/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:296 <>+0x9dae5 #12 0x000023c56771c535 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) <>+0xa3535 #13 0x000023c5677457e3 in main ../recipe_cleanup/clangshYTOG/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19 <>+0xcc7e3 #14 0x000041fad9a9243b in start_main ./../../zircon/third_party/ulib/musl/src/env/__libc_start_main.c:112
For unreproducible results, you can still examine symbolized log from the original fuzzer execution for clues.
Attach a debugger
You may also want to attach a debugger when reproducing fuzzer results. By default, libFuzzer on Fuchsia creates a debug exception channel attached to the fuzzing thread in order to detect and handle crashes during fuzzing. Only one process may do this per thread, so debuggers are prevented from attaching.
libfuzzer from creating a debug exception channel, use the
--debug option with
For example, to use zxdb while reproducing a specific test case:
ffx debug connect[zxdb] attach noop-fuzzer [zxdb] break LLVMFuzzerTestOneInput
Now, in a separate terminal, start the fuzzer with your test case:
fx fuzz repro --debug zircon_fuzzers/noop-fuzzer testcase_input_file
File fuzzing bugs
It may be tempting to immediately fix the bug related to the fuzzer result, especially if the bug is obvious. No matter how trivial the bug is, please file a bug report!
To file a bug, please use the Fuzzing Bug template. This ensures you include
certain labels, such as
Sec-TriageMe. This in turn helps the
security team see where fuzzers are being used and stay aware of any critical issues they are
If you encounter problems or shortcomings in the fuzzing framework itself, open bugs or
feature requests in the
As with all potential security issues, you do not need to wait for triage to begin fixing the bug! Once fixed, reference the bug number in the commit message.
Handle bugs from ClusterFuzz
ClusterFuzz will file bugs automatically when it finds reproducible fuzzer results. If you are assigned such a bug, look for the following:
The Detailed Report will contain details about the result, including:
- What type of result it is.
- Whether it has security implications.
- What revisions exhibited the behavior.
- What stack frames appear to uniquely identify the crash.
The Reproducer Testcase will link to a fuzzer artifact. You can download this artifact and then use it to reproduce the fuzzer result as described above.
When you submit a fix so that a fuzzer stops producing an artifact from the input, ClusterFuzz will automatically close the bug.