Tutorial: Debug minidumps using zxdb

A minidump is a memory dump of a process at the time of a crash, which can be useful to debug the cause of a crash.

This tutorial walks through a debugging workflow that loads a minidump file into zxdb and then walks through debugging steps to understand the failure captured by the minidump.

Capturing a minidump file

Minidumps can be captured from any debugging session in zxdb with the savedump command.

For example, if you are using zxdb on the cobalt.cm component:

ffx component debug cobalt.cm

You should see an output like:

Waiting for process matching "job 20999".
Type "filter" to see the current filters.
👉 To get started, try "status" or "help".
Attached Process 1 state=Running koid=21246 name=cobalt.cm component=cobalt.cm
Loading 15 modules for cobalt.cm ...Done.

If you wanted to capture a minidump, you would first pause the debugger:

pause

You should see an output like:

   508                 const zx_port_packet_t* packet))
   509
 ▶ 510 BLOCKING_SYSCALL(port_wait, zx_status_t, /* no attributes */, 3, (handle, deadline, packet),
   511                  (_ZX_SYSCALL_ANNO(use_handle("Fuchsia")) zx_handle_t handle, zx_time_t deadline,
   512                   zx_port_packet_t* packet))
🛑 thread 1 $elf(SYSCALL_zx_port_wait) + 0x7 • syscalls.inc:510

Use savedump to save a minidump:

savedump cobalt.dump

You should see an output like:

Saving minidump...
Minidump written to cobalt.dump

Once you have saved the minidump, you can exit the debugger:

quit

Your minidump is now saved in your working directory. You are now ready to load the minidump file and debug it.

Loading and debugging a minidump file

Loading and debugging a minidump file in zxdb is the same regardless of the language of the component that you are trying to debug. However, there are some differences about things to look out for in each language.

Rust

To load the minidump file into zxdb, you can use ffx debug core. For example, to load a minidump file named archivist_unittest.dump:

ffx debug core archivist_unittest.dump

You should see an output like:

Opening dump file...
Dump loaded successfully.
👉 To get started, try "status" or "help".
Attached Process 1 state=Running koid=14446435 name=<_>
Loading 6 modules for <_> Downloading symbols...
Done.
🛑  (no location information)
Attached Process 2 state=Running koid=14446435 name=<_>
Attaching to previously connected processes:
14446435: <_>
Loading 6 modules for <_> Done.
Symbol downloading complete. 5 succeeded, 0 failed.

List the stack frames:

frame

You should see an output like:

▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
  1 core::future::future::«impl»::poll<…>(…) • future/future.rs:123
  2 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:27
  3 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:122
  4 fuchsia_async::atomic_future::«impl»::poll<…>(…) • atomic_future.rs:72
  5 fuchsia_async::atomic_future::AtomicFuture::try_poll(…) • atomic_future.rs:230
  6 fuchsia_async::runtime::fuchsia::executor::common::Executor::try_poll(…) • executor/common.rs:554
  7 fuchsia_async::runtime::fuchsia::executor::common::Executor::poll_ready_tasks(…) • executor/common.rs:133
  8 fuchsia_async::runtime::fuchsia::executor::common::Executor::worker_lifecycle<…>(…) • executor/common.rs:429
  9 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run<…>(…) • executor/local.rs:104
  10 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run_singlethreaded<…>(…) • executor/local.rs:70
  11 fuchsia_async::test_support::«impl»::run_singlethreaded::λ() • test_support.rs:120
  12 fuchsia_async::test_support::Config::in_parallel(…) • test_support.rs:215
  13 fuchsia_async::test_support::«impl»::run_singlethreaded(…) • test_support.rs:117
  14 fuchsia_async::test_support::run_singlethreaded_test<…>(…) • test_support.rs:227
  15 fuchsia::test_singlethreaded<…>(…) • fuchsia/src/lib.rs:195
  16 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log() • archivist.rs:524
  17 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::λ(…) • archivist.rs:525
  18 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
  19 core::ops::function::FnOnce::call_once<…>(…) • library/core/src/ops/function.rs:250 (inline)
  20 test::__rust_begin_short_backtrace<…>(…) • library/test/src/lib.rs:625
  21 test::run_test_in_spawned_subprocess(…) • library/test/src/lib.rs:753
  22 test::test_main_static_abort(…) • library/test/src/lib.rs:199
  23 archivist_lib_lib_test::main() • archivist/src/lib.rs:1
  24 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
  25 std::sys::backtrace::__rust_begin_short_backtrace<…>(…) • fuchsia-third_party-rust/library/std/src/sys/backtrace.rs:155
  26 std::rt::lang_start::λ() • fuchsia-third_party-rust/library/std/src/rt.rs:159
  27 core::ops::function::impls::«impl»::call_once<…>(…) • library/core/src/ops/function.rs:284 (inline)
  28 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
  29 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
  30 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
  31 std::rt::lang_start_internal::λ() • library/std/src/rt.rs:141 (inline)
  32 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
  33 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
  34 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
  35 std::rt::lang_start_internal(…) • library/std/src/rt.rs:141
  36 std::rt::lang_start<…>(…) • fuchsia-third_party-rust/library/std/src/rt.rs:158
  37 $elf(main) + 0x21
  38…40 «libc startup» (-r expands)

You can now use shortcuts to quickly perform actions across all threads.

Note: The wildcard operator (*) performs an action for all threads. You can use both nouns and verbs with this operator. Commands from the Step section of the help menu are not available because this isn't a real process.

List the frames of each thread:
```
thread * frame
```
You should see an output like:
```
Thread 1 state="Core Dump" koid=14450503 name=""
▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
```

List the frames of each thread, but with more detailed information, you can combine thread with the backtrace verb:

thread * backtrace

This command helps you see the local variables for each stack frame. Keep in mind that only the stack memory is captured in the minidump to improve the capturing time and size. Pointers to heap variables mostly do not resolve to anything useful.

You should see an output like:

Thread 1 state="Core Dump" koid=14450503 name=""
▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
      (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
  1 core::future::future::«impl»::poll<…>(…) • future/future.rs:123
      self = Pin<&mut core::pin::Pin<alloc…>{__pointer: (*)0xa2987a4580}
      cx = (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
  2 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:27
      (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
  3 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:122
      (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
  4 fuchsia_async::atomic_future::«impl»::poll<…>(…) • atomic_future.rs:72
      self = (*)0xa2987a4520
      cx = (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
  5 fuchsia_async::atomic_future::AtomicFuture::try_poll(…) • atomic_future.rs:230
      self = (*)0x9d587a4118
      cx = (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
  6 fuchsia_async::runtime::fuchsia::executor::common::Executor::try_poll(…) • executor/common.rs:554
      self = (*)0x9b187a9300
      task = (*)0x5c62690588 ➔ (*)0x9d587a40f0
  7 fuchsia_async::runtime::fuchsia::executor::common::Executor::poll_ready_tasks(…) • executor/common.rs:133
      self = (*)0x9b187a9300
      local_collector = (*)0x5c62690600 ➔ LocalCollector{collector: (*)0x9b187a9438, last_ticks: 1050997948718604, polls: 1, tasks_pending_max: 10}
  8 fuchsia_async::runtime::fuchsia::executor::common::Executor::worker_lifecycle<…>(…) • executor/common.rs:429
      self = (*)0xa4187a90a0
  9 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run<…>(…) • executor/local.rs:104
      self = (*)0x5c62690878 ➔ LocalExecutor{ehandle: EHandle{…}}
      main_future = AtomicFuture{state: AtomicUsize{1}, future: UnsafeCell{$(alloc::boxed::Box<dyn fuchsia_async::atomic_future::FutureOrResultAccess, alloc::alloc::Global>){…}}}
  10 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run_singlethreaded<…>(…) • executor/local.rs:70
      self = (*)0x5c62690878 ➔ LocalExecutor{ehandle: EHandle{…}}
      main_future = Unresumed{run_stream: (*)0xa3187a86a0, test: λ{…}}
  11 fuchsia_async::test_support::«impl»::run_singlethreaded::λ() • test_support.rs:120
  12 fuchsia_async::test_support::Config::in_parallel(…) • test_support.rs:215
      self = (*)0x5c62690ae8 ➔ Config{repeat_count: 1, max_concurrency: 0, max_threads: 0, timeout: …}
      f = <The value of type '*const alloc::sync::ArcInner<(dyn core::ops::function::Fn<(), Output=()> + core::marker::Send + core::marker::Sync)>' is the incorrect size (expecting 8, got 16). Please file a bug.>
  13 fuchsia_async::test_support::«impl»::run_singlethreaded(…) • test_support.rs:117
      test = <The value of type '*const alloc::sync::ArcInner<(dyn core::ops::function::Fn<(usize), Output=core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output=()>, alloc::alloc::Global>>> + core::marker::Send + core::marker::Sync)>' is the incorrect size (expecting 8, got 16). Please file a bug.>
      cfg = Config{repeat_count: 1, max_concurrency: 0, max_threads: 0, timeout: None<Duration>}
  14 fuchsia_async::test_support::run_singlethreaded_test<…>(…) • test_support.rs:227
      test = <Unavailable>
  15 fuchsia::test_singlethreaded<…>(…) • fuchsia/src/lib.rs:195
      f = <Register rdi not available.>
  16 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log() • archivist.rs:524
  17 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::λ(…) • archivist.rs:525
      (*)0x5c62690bde ➔ λ
  18 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
      λ
      <Value has no data.>
  19 core::ops::function::FnOnce::call_once<…>(…) • library/core/src/ops/function.rs:250 (inline)
      <Register rsi not available.>
      <Optimized out>
  20 test::__rust_begin_short_backtrace<…>(…) • library/test/src/lib.rs:625
      f = <Register rsi not available.>
  21 test::run_test_in_spawned_subprocess(…) • library/test/src/lib.rs:753
      desc = <Register rdi not available.>
      runnable_test = Static(&core::ops::function::FnOnce::call_once<archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::{closure_env#0}, ()>)
  22 test::test_main_static_abort(…) • library/test/src/lib.rs:199
      tests = <Unavailable>
  23 archivist_lib_lib_test::main() • archivist/src/lib.rs:1
  24 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
      &archivist_lib_lib_test::main
      <Value has no data.>
  25 std::sys::backtrace::__rust_begin_short_backtrace<…>(…) • fuchsia-third_party-rust/library/std/src/sys/backtrace.rs:155
      f = &archivist_lib_lib_test::main
  26 std::rt::lang_start::λ() • fuchsia-third_party-rust/library/std/src/rt.rs:159
  27 core::ops::function::impls::«impl»::call_once<…>(…) • library/core/src/ops/function.rs:284 (inline)
      self = $(&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)){pointer: (*)0x5c62690f50 ➔ $((dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)), vtable = <Invalid data offset 8 in object of size 8.>}
      args = <Optimized out>
  28 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
      data = <Optimized out>
  29 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
  30 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
  31 std::rt::lang_start_internal::λ() • library/std/src/rt.rs:141 (inline)
  32 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
      data = <Optimized out>
  33 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
  34 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
  35 std::rt::lang_start_internal(…) • library/std/src/rt.rs:141
      main = $(&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)){pointer: (*)0x5c62690f50 ➔ $((dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)), vtable: (*)0x13a66000958}
      argc = <Register rdx not available.>
      argv = <Register rcx not available.>
      sigpipe = <Register r8 not available.>
  36 std::rt::lang_start<…>(…) • fuchsia-third_party-rust/library/std/src/rt.rs:158
      main = &archivist_lib_lib_test::main
      argc = 2
      argv = (*)0xecf9d19fb0
      sigpipe = 0
  37 $elf(main) + 0x21
  38…40 «libc startup» (-r expands)

List the current threads:

Note: You can also use any of the regular zxdb thread and frame commands.
```
thread
```
You should see an output like:
```
  # state      koid name
▶ 1 Core Dump 14450503
```
List a specific thread. For example, thread 1:
```
thread 1
```
You should see an output like:
```
Thread 1 state="Core Dump" koid=14450503 name=""
```

You can then use frame to see this specific stack frame from thread 1:

frame

You should see an output like:

▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
  1 core::future::future::«impl»::poll<…>(…) • future/future.rs:123
  2 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:27
  3 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:122
  4 fuchsia_async::atomic_future::«impl»::poll<…>(…) • atomic_future.rs:72
  5 fuchsia_async::atomic_future::AtomicFuture::try_poll(…) • atomic_future.rs:230
  6 fuchsia_async::runtime::fuchsia::executor::common::Executor::try_poll(…) • executor/common.rs:554
  7 fuchsia_async::runtime::fuchsia::executor::common::Executor::poll_ready_tasks(…) • executor/common.rs:133
  8 fuchsia_async::runtime::fuchsia::executor::common::Executor::worker_lifecycle<…>(…) • executor/common.rs:429
  9 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run<…>(…) • executor/local.rs:104
  10 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run_singlethreaded<…>(…) • executor/local.rs:70
  11 fuchsia_async::test_support::«impl»::run_singlethreaded::λ() • test_support.rs:120
  12 fuchsia_async::test_support::Config::in_parallel(…) • test_support.rs:215
  13 fuchsia_async::test_support::«impl»::run_singlethreaded(…) • test_support.rs:117
  14 fuchsia_async::test_support::run_singlethreaded_test<…>(…) • test_support.rs:227
  15 fuchsia::test_singlethreaded<…>(…) • fuchsia/src/lib.rs:195
  16 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log() • archivist.rs:524
  17 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::λ(…) • archivist.rs:525
  18 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
  19 core::ops::function::FnOnce::call_once<…>(…) • library/core/src/ops/function.rs:250 (inline)
  20 test::__rust_begin_short_backtrace<…>(…) • library/test/src/lib.rs:625
  21 test::run_test_in_spawned_subprocess(…) • library/test/src/lib.rs:753
  22 test::test_main_static_abort(…) • library/test/src/lib.rs:199
  23 archivist_lib_lib_test::main() • archivist/src/lib.rs:1
  24 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
  25 std::sys::backtrace::__rust_begin_short_backtrace<…>(…) • fuchsia-third_party-rust/library/std/src/sys/backtrace.rs:155
  26 std::rt::lang_start::λ() • fuchsia-third_party-rust/library/std/src/rt.rs:159
  27 core::ops::function::impls::«impl»::call_once<…>(…) • library/core/src/ops/function.rs:284 (inline)
  28 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
  29 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
  30 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
  31 std::rt::lang_start_internal::λ() • library/std/src/rt.rs:141 (inline)
  32 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
  33 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
  34 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
  35 std::rt::lang_start_internal(…) • library/std/src/rt.rs:141
  36 std::rt::lang_start<…>(…) • fuchsia-third_party-rust/library/std/src/rt.rs:158
  37 $elf(main) + 0x21
  38…40 «libc startup» (-r expands)

Based on the output from the previous step, you may want to dive deeper into frame 0:

frame 0

You should see an output like:

archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547

To see additional lines of code from the current frame, use list:

list

You should see an output like:

  542
  543         let mut expected = vec!["my msg1".to_owned(), "my msg2".to_owned()];
  544         expected.sort();
  545
  546         let mut actual = vec![recv_logs.next().await.unwrap(), recv_logs.next().await.unwrap()];
▶ 547         actual.sort();
  548
  549         assert_eq!(expected, actual);
  550
  551         // can log after killing log sink proxy
  552         log_helper.kill_log_sink();
  553         log_helper.write_log("my msg1");
  554         log_helper.write_log("my msg2");
  555
  556         assert_eq!(
  557             expected,

Since this frame actually contains some archivist code, you can use locals to see the local variables in this stack frame.

To see all local variables in the current stack frame, use locals:

locals

You should see an output like:

_task_context = (*)0x5c626904e0 ➔ Context{
  waker: (*)0x5c626904c8
  local_waker: (*)0x5c626904c8
  ext: AssertUnwindSafe<core::task::wake::ExtData>(None(<Value has no data.>))
  _marker: PhantomData<fn(&())->&()>
  _marker2: PhantomData<*mut()>
}
actual = <Invalid pointer 0x9c187a04e8>
directory = <Invalid pointer 0x9c187a0460>
expected = <Invalid pointer 0x9c187a04d0>
log_helper = <Invalid pointer 0x9c187a0470>
log_helper2 = <Invalid pointer 0x9c187a04c0>
recv_logs = <Invalid pointer 0x9c187a0468>

You can then see more information about a specific variable with print. For example:

print _task_context

You should see an output like:

(*)0x5c626904e0 ➔ Context{
  waker: (*)0x5c626904c8
  local_waker: (*)0x5c626904c8
  ext: AssertUnwindSafe<core::task::wake::ExtData>(None(<Value has no data.>))
  _marker: PhantomData<fn(&())->&()>
  _marker2: PhantomData<*mut()>
}

You have successfully loaded and analyzed a minidump for Rust code.

C++

To load the minidump file into zxdb, you can use ffx debug core. For example, to load a minidump file named cobalt_minidump.dump:

ffx debug core cobalt_minidump.dump

You should see an output like:

Opening dump file...
Dump loaded successfully.
👉 To get started, try "status" or "help".
Attached Process 1 state=Running koid=15650768 name=<_>
Loading 15 modules for <_> ...Done.
Attached Process 2 state=Running koid=15650768 name=<_>
Attaching to previously connected processes:
15650768: <_>
Loading 15 modules for <_> Done.

Now that you have loaded the minidump file, you can now inspect the stacks for all threads present that were present when the minidump was captured.

List the stack frames:

frame

You should see an output like:

▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
  5 main(…) • cobalt_main.cc:349
  6…8 «libc startup» (-r expands)

You can now use shortcuts to quickly perform actions across all threads:

List the frames of each thread:

thread * frame

You should see an output like:

Thread 1 state="Core Dump" koid=19601 name=""
▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
  5 main(…) • cobalt_main.cc:349
  6…8 «libc startup» (-r expands)
Thread 2 state="Core Dump" koid=26944 name=""
▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
  1 _zx_futex_wait(…) • syscalls.inc:210
  2 __timedwait_assign_owner(…) • __timedwait.c:23
  3 __timedwait(…) • threads_impl.h:322 (inline)
  4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
  5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
  6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
  7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
  8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
  9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
  10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
  11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
  12 std::__2::condition_variable_any::wait_for<…>(…) • condition_variable:260
  13 cobalt::local_aggregation::DelayedLocalAggregateStorage::Run(…) • delayed_local_aggregate_storage.cc:200
  14 λ(…) • delayed_local_aggregate_storage.cc:45
  15 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
  16 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
  17 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
  18…19 «pthread startup» (-r expands)
Thread 3 state="Core Dump" koid=26975 name=""
▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
  1 _zx_futex_wait(…) • syscalls.inc:210
  2 __timedwait_assign_owner(…) • __timedwait.c:23
  3 __timedwait(…) • threads_impl.h:322 (inline)
  4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
  5 std::__2::__libcpp_condvar_wait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:122 (inline)
  6 std::__2::condition_variable::wait(…) • condition_variable.cpp:30
  7 std::__2::condition_variable_any::wait<…>(…) • condition_variable:225
  8 std::__2::condition_variable_any::wait<…>(…) • condition_variable:231
  9 cobalt::uploader::ShippingManager::Run(…) • shipping_manager.cc:217
  10 λ(…) • shipping_manager.cc:76
  11 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
  12 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
  13 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
  14…15 «pthread startup» (-r expands)
Thread 4 state="Core Dump" koid=26983 name=""
▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
  1 _zx_futex_wait(…) • syscalls.inc:210
  2 __timedwait_assign_owner(…) • __timedwait.c:23
  3 __timedwait(…) • threads_impl.h:322 (inline)
  4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
  5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
  6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
  7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
  8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
  9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
  10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
  11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
  12 cobalt::local_aggregation::ObservationGenerator::Run(…) • observation_generator.cc:92
  13 λ(…) • observation_generator.cc:65
  14 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
  15 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
  16 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
  17…18 «pthread startup» (-r expands)

To list the frames of each thread, but with more detailed information, you can combine thread with the backtrace verb:

thread * backtrace

You should see an output like:

Thread 1 state="Core Dump" koid=19601 name=""
▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
  5 main(…) • cobalt_main.cc:349
      argc = 2
      argv = (*)0x26866821fc0
  6…8 «libc startup» (-r expands)
Thread 2 state="Core Dump" koid=26944 name=""
▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
  1 _zx_futex_wait(…) • syscalls.inc:210
      value_ptr = (*)0x285f3fd5904
      current_value = 2
      new_futex_owner = 0
      deadline = 612921722073211
  2 __timedwait_assign_owner(…) • __timedwait.c:23
      futex = <Register rdi not available.>
      val = <Register rsi not available.>
      clk = <Register rdx not available.>
      at = <Register rcx not available.>
      new_owner = <Register r8 not available.>
  3 __timedwait(…) • threads_impl.h:322 (inline)
      futex = (*)0x285f3fd5904
      val = 2
      clk = 0
      at = (*)0x285f3fd5918
  4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
      c = (*)0x370ff24ae20
      m = (*)0x3707f2b76c8
      ts = (*)0x285f3fd5918
  5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
      __cv = <Register rdi not available.>
      __m = <Register rsi not available.>
      __ts = <Optimized out>
  6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
      this = <Register rdi not available.>
      lk = <Register rsi not available.>
      tp = <Register rdx not available.>
  7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
      this = (*)0x370ff24ae20
      __lk = <Invalid pointer 0x285f3fd5a18>
      __d = <Invalid pointer 0x285f3fd59a8>
  8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
      this = (*)0x370ff24ae20
      __lk = <Invalid pointer 0x285f3fd5a18>
      __tp = <Invalid pointer 0x285f3fd59b0>
  9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
      this = (*)0x370ff24ae20
      __lk = <Invalid pointer 0x285f3fd5a18>
      __t = <Invalid pointer 0x285f3fd5a60>
  10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
      this = (*)0x370ff24ae20
      __lock = <Invalid pointer 0x285f3fd5ab0>
      __t = <Invalid pointer 0x285f3fd5a60>
  11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
      this = (*)0x370ff24ae20
      __lock = <Invalid pointer 0x285f3fd5ab0>
      __t = <Invalid pointer 0x285f3fd5a60>
      __pred = <Invalid pointer 0x285f3fd5a40>
  12 std::__2::condition_variable_any::wait_for<…>(…) • condition_variable:260
      this = (*)0x370ff24ae20
      __lock = <Invalid pointer 0x285f3fd5ab0>
      __d = <Invalid pointer 0x370ff24ae00>
      __pred = {locked_state = <Invalid pointer 0x285f3fd5ab0>}
  13 cobalt::local_aggregation::DelayedLocalAggregateStorage::Run(…) • delayed_local_aggregate_storage.cc:200
      this = (*)0x370ff24acf0
  14 λ(…) • delayed_local_aggregate_storage.cc:45
      this = (*)0x36bff27b228
  15 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
      __f = <Invalid pointer 0x36bff27b228>
  16 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
      __t = <Invalid pointer 0x36bff27b220>
      {}
  17 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
      __vp = (*)0x36bff27b220
  18…19 «pthread startup» (-r expands)
Thread 3 state="Core Dump" koid=26975 name=""
▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
  1 _zx_futex_wait(…) • syscalls.inc:210
      value_ptr = (*)0x23b6ad77434
      current_value = 2
      new_futex_owner = 0
      deadline = 9223372036854775807
  2 __timedwait_assign_owner(…) • __timedwait.c:23
      futex = <Register rdi not available.>
      val = <Register rsi not available.>
      clk = <Register rdx not available.>
      at = <Register rcx not available.>
      new_owner = <Register r8 not available.>
  3 __timedwait(…) • threads_impl.h:322 (inline)
      futex = (*)0x23b6ad77434
      val = 2
      clk = 0
      at = (*)0x0
  4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
      c = (*)0x3673f24a610
      m = (*)0x3707f2b7998
      ts = (*)0x0
  5 std::__2::__libcpp_condvar_wait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:122 (inline)
      __cv = <Register rdi not available.>
      __m = <Register rsi not available.>
  6 std::__2::condition_variable::wait(…) • condition_variable.cpp:30
      this = <Register rdi not available.>
      lk = <Register rsi not available.>
  7 std::__2::condition_variable_any::wait<…>(…) • condition_variable:225
      this = (*)0x3673f24a610
      __lock = <Invalid pointer 0x23b6ad77608>
  8 std::__2::condition_variable_any::wait<…>(…) • condition_variable:231
      this = (*)0x3673f24a610
      __lock = <Invalid pointer 0x23b6ad77608>
      __pred = <Invalid pointer 0x23b6ad77480>
  9 cobalt::uploader::ShippingManager::Run(…) • shipping_manager.cc:217
      this = (*)0x3673f24a530
  10 λ(…) • shipping_manager.cc:76
      this = (*)0x36bff27a7a8
  11 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
      __f = <Invalid pointer 0x36bff27a7a8>
  12 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
      __t = <Invalid pointer 0x36bff27a7a0>
      {}
  13 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
      __vp = (*)0x36bff27a7a0
  14…15 «pthread startup» (-r expands)
Thread 4 state="Core Dump" koid=26983 name=""
▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
  1 _zx_futex_wait(…) • syscalls.inc:210
      value_ptr = (*)0x149c84b2bc4
      current_value = 2
      new_futex_owner = 0
      deadline = 615784347537787
  2 __timedwait_assign_owner(…) • __timedwait.c:23
      futex = <Register rdi not available.>
      val = <Register rsi not available.>
      clk = <Register rdx not available.>
      at = <Register rcx not available.>
      new_owner = <Register r8 not available.>
  3 __timedwait(…) • threads_impl.h:322 (inline)
      futex = (*)0x149c84b2bc4
      val = 2
      clk = 0
      at = (*)0x149c84b2bd8
  4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
      c = (*)0x3673f24b128
      m = (*)0x3707f2b6548
      ts = (*)0x149c84b2bd8
  5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
      __cv = <Register rdi not available.>
      __m = <Register rsi not available.>
      __ts = <Optimized out>
  6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
      this = <Register rdi not available.>
      lk = <Register rsi not available.>
      tp = <Register rdx not available.>
  7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
      this = (*)0x3673f24b128
      __lk = <Invalid pointer 0x149c84b2cd8>
      __d = <Invalid pointer 0x149c84b2c68>
  8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
      this = (*)0x3673f24b128
      __lk = <Invalid pointer 0x149c84b2cd8>
      __tp = <Invalid pointer 0x149c84b2c70>
  9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
      this = (*)0x3673f24b128
      __lk = <Invalid pointer 0x149c84b2cd8>
      __t = <Invalid pointer 0x3673f24b0d8>
  10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
      this = (*)0x3673f24b128
      __lock = <Invalid pointer 0x149c84b2d20>
      __t = <Invalid pointer 0x3673f24b0d8>
  11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
      this = (*)0x3673f24b128
      __lock = <Invalid pointer 0x149c84b2d20>
      __t = <Invalid pointer 0x3673f24b0d8>
      __pred = <Invalid pointer 0x149c84b2d00>
  12 cobalt::local_aggregation::ObservationGenerator::Run(…) • observation_generator.cc:92
      this = (*)0x3673f24b0a0
      clock = (*)0x36bff27aaa0
  13 λ(…) • observation_generator.cc:65
      this = (*)0x36bff24cf68
  14 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
      __f = <Invalid pointer 0x36bff24cf68>
  15 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
      __t = <Invalid pointer 0x36bff24cf60>
      {}
  16 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
      __vp = (*)0x36bff24cf60
  17…18 «pthread startup» (-r expands)

List the current threads:

Note: You can also use any of the regular zxdb thread and frame commands.
```
thread
```
You should see an output like:
```
  # state      koid name
▶ 1 Core Dump 19601
  2 Core Dump 26944
  3 Core Dump 26975
  4 Core Dump 26983
```
List a specific thread. For example, thread 2:
```
thread 2
```
You should see an output like:
```
Thread 2 state="Core Dump" koid=26944 name=""
```

You can then use frame to see this specific stack frame from thread 2:

frame

You should see an output like:

▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
  1 _zx_futex_wait(…) • syscalls.inc:210
  2 __timedwait_assign_owner(…) • __timedwait.c:23
  3 __timedwait(…) • threads_impl.h:322 (inline)
  4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
  5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
  6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
  7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
  8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
  9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
  10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
  11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
  12 std::__2::condition_variable_any::wait_for<…>(…) • condition_variable:260
  13 cobalt::local_aggregation::DelayedLocalAggregateStorage::Run(…) • delayed_local_aggregate_storage.cc:200
  14 λ(…) • delayed_local_aggregate_storage.cc:45
  15 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
  16 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
  17 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
  18…19 «pthread startup» (-r expands)

List another specific thread. For example, thread 1:

Note: This example is on a different thread to show the difference when you follow the thread noun with frame.
```
thread 1
```
You should see an output like:
```
Thread 1 state="Core Dump" koid=19601 name=""
```

You can then use frame to see this specific stack frame from thread 1:

frame

You should see an output like:

▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
  5 main(…) • cobalt_main.cc:349
  6…8 «libc startup» (-r expands)

Based on the output from the previous step, you may want to dive deeper into frame 5:
```
frame 5
```
You should see an output like:
```
main(…) • cobalt_main.cc:349
```

To see additional lines of code from the current frame, use list:

list

You should see an output like:

  344
  345   if (!app.ok()) {
  346     FX_LOGS(FATAL) << "Failed to construct the cobalt app: " << app.status();
  347   }
  348   inspector.Health().Ok();
▶ 349   loop.Run();
  350   FX_LOGS(INFO) << "Cobalt will now shut down.";
  351   return 0;
  352 }

Since this frame actually contains some code, you can use locals to see the local variables in this stack frame.

To see all local variables in the current stack frame, use locals:

locals

You should see an output like:

argc = 2
argv = (*)0x26866821fc0
command_line = <Invalid pointer 0x26866821418>
context = <Invalid pointer 0x26866821100>
event_aggregator_backfill_days = 2
flag_value = <Invalid pointer 0x268668211d8>
initial_interval = <Invalid pointer 0x26866821138>
inspector = <Invalid pointer 0x268668212f0>
loop = <Invalid pointer 0x26866821108>
max_bytes_per_observation_store = 215040
min_interval = <Invalid pointer 0x26866821118>
require_lifecycle_service = true
schedule_interval = <Invalid pointer 0x26866821140>
start_event_aggregator_worker = true
status = 0 (ZX_OK)
storage_quotas = {
  per_project_reserved_bytes = 1024
  total_capacity_bytes = 731136
}
test_dont_backfill_empty_reports = false
upload_jitter = 0.2
upload_schedule = {
  target_interval = {__rep_ = 3600}
  min_interval = {__rep_ = 10}
  initial_interval = {__rep_ = 60}
  jitter = 0.2
}
use_fake_clock = false
use_memory_observation_store = false

You can then see more information about a specific variable with print. For example:

print upload_schedule

You should see an output like:

{
  target_interval = {__rep_ = 3600}
  min_interval = {__rep_ = 10}
  initial_interval = {__rep_ = 60}
  jitter = 0.2
}

You have successfully loaded and analyzed a minidump for C++ code.