Tutorial: Debug minidumps using zxdb

A minidump is a memory dump of a process at the time of a crash, which can be useful to debug the cause of a crash.

This tutorial walks through a debugging workflow that loads a minidump file into zxdb and then walks through debugging steps to understand the failure captured by the minidump.

Capturing a minidump file

Minidumps can be captured from any debugging session in zxdb with the savedump command.

For example, if you are using zxdb on the cobalt.cm component:

ffx component debug cobalt.cm

You should see an output like:

Waiting for process matching "job 20999".
Type "filter" to see the current filters.
👉 To get started, try "status" or "help".
Attached Process 1 state=Running koid=21246 name=cobalt.cm component=cobalt.cm
Loading 15 modules for cobalt.cm ...Done.

If you wanted to capture a minidump, you would first pause the debugger:

pause

You should see an output like:

   508                 const zx_port_packet_t* packet))
   509
 â–¶ 510 BLOCKING_SYSCALL(port_wait, zx_status_t, /* no attributes */, 3, (handle, deadline, packet),
   511                  (_ZX_SYSCALL_ANNO(use_handle("Fuchsia")) zx_handle_t handle, zx_time_t deadline,
   512                   zx_port_packet_t* packet))
🛑 thread 1 $elf(SYSCALL_zx_port_wait) + 0x7 • syscalls.inc:510

Use savedump to save a minidump:

savedump cobalt.dump

You should see an output like:

Saving minidump...
Minidump written to cobalt.dump

Once you have saved the minidump, you can exit the debugger:

quit

Your minidump is now saved in your working directory. You are now ready to load the minidump file and debug it.

Loading and debugging a minidump file

Loading and debugging a minidump file in zxdb is the same regardless of the language of the component that you are trying to debug. However, there are some differences about things to look out for in each language.

Rust

  1. To load the minidump file into zxdb, you can use ffx debug core. For example, to load a minidump file named archivist_unittest.dump:

    ffx debug core archivist_unittest.dump

    You should see an output like:

    Opening dump file...
    Dump loaded successfully.
    👉 To get started, try "status" or "help".
    Attached Process 1 state=Running koid=14446435 name=<_>
    Loading 6 modules for <_> Downloading symbols...
    Done.
    🛑  (no location information)
    Attached Process 2 state=Running koid=14446435 name=<_>
    Attaching to previously connected processes:
    14446435: <_>
    Loading 6 modules for <_> Done.
    Symbol downloading complete. 5 succeeded, 0 failed.
    
  2. List the stack frames:

    frame
    

    You should see an output like:

    ▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
      1 core::future::future::«impl»::poll<…>(…) • future/future.rs:123
      2 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:27
      3 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:122
      4 fuchsia_async::atomic_future::«impl»::poll<…>(…) • atomic_future.rs:72
      5 fuchsia_async::atomic_future::AtomicFuture::try_poll(…) • atomic_future.rs:230
      6 fuchsia_async::runtime::fuchsia::executor::common::Executor::try_poll(…) • executor/common.rs:554
      7 fuchsia_async::runtime::fuchsia::executor::common::Executor::poll_ready_tasks(…) • executor/common.rs:133
      8 fuchsia_async::runtime::fuchsia::executor::common::Executor::worker_lifecycle<…>(…) • executor/common.rs:429
      9 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run<…>(…) • executor/local.rs:104
      10 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run_singlethreaded<…>(…) • executor/local.rs:70
      11 fuchsia_async::test_support::«impl»::run_singlethreaded::λ() • test_support.rs:120
      12 fuchsia_async::test_support::Config::in_parallel(…) • test_support.rs:215
      13 fuchsia_async::test_support::«impl»::run_singlethreaded(…) • test_support.rs:117
      14 fuchsia_async::test_support::run_singlethreaded_test<…>(…) • test_support.rs:227
      15 fuchsia::test_singlethreaded<…>(…) • fuchsia/src/lib.rs:195
      16 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log() • archivist.rs:524
      17 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::λ(…) • archivist.rs:525
      18 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
      19 core::ops::function::FnOnce::call_once<…>(…) • library/core/src/ops/function.rs:250 (inline)
      20 test::__rust_begin_short_backtrace<…>(…) • library/test/src/lib.rs:625
      21 test::run_test_in_spawned_subprocess(…) • library/test/src/lib.rs:753
      22 test::test_main_static_abort(…) • library/test/src/lib.rs:199
      23 archivist_lib_lib_test::main() • archivist/src/lib.rs:1
      24 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
      25 std::sys::backtrace::__rust_begin_short_backtrace<…>(…) • fuchsia-third_party-rust/library/std/src/sys/backtrace.rs:155
      26 std::rt::lang_start::λ() • fuchsia-third_party-rust/library/std/src/rt.rs:159
      27 core::ops::function::impls::«impl»::call_once<…>(…) • library/core/src/ops/function.rs:284 (inline)
      28 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
      29 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
      30 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
      31 std::rt::lang_start_internal::λ() • library/std/src/rt.rs:141 (inline)
      32 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
      33 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
      34 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
      35 std::rt::lang_start_internal(…) • library/std/src/rt.rs:141
      36 std::rt::lang_start<…>(…) • fuchsia-third_party-rust/library/std/src/rt.rs:158
      37 $elf(main) + 0x21
      38…40 «libc startup» (-r expands)
    
  3. You can now use shortcuts to quickly perform actions across all threads.

    List the frames of each thread:

    thread * frame
    

    You should see an output like:

    Thread 1 state="Core Dump" koid=14450503 name=""
    ▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
    
  4. List the frames of each thread, but with more detailed information, you can combine thread with the backtrace verb:

    thread * backtrace
    

    This command helps you see the local variables for each stack frame. Keep in mind that only the stack memory is captured in the minidump to improve the capturing time and size. Pointers to heap variables mostly do not resolve to anything useful.

    You should see an output like:

    Thread 1 state="Core Dump" koid=14450503 name=""
    ▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
          (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
      1 core::future::future::«impl»::poll<…>(…) • future/future.rs:123
          self = Pin<&mut core::pin::Pin<alloc…>{__pointer: (*)0xa2987a4580}
          cx = (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
      2 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:27
          (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
      3 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:122
          (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
      4 fuchsia_async::atomic_future::«impl»::poll<…>(…) • atomic_future.rs:72
          self = (*)0xa2987a4520
          cx = (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
      5 fuchsia_async::atomic_future::AtomicFuture::try_poll(…) • atomic_future.rs:230
          self = (*)0x9d587a4118
          cx = (*)0x5c626904e0 ➔ Context{waker: (*)0x5c626904c8, local_waker: (*)0x5c626904c8, ext: AssertUnwindSafe<core::task::wake::ExtData>(…), _marker: PhantomData<fn(&())->&()>, _marker2: PhantomData<*mut()>}
      6 fuchsia_async::runtime::fuchsia::executor::common::Executor::try_poll(…) • executor/common.rs:554
          self = (*)0x9b187a9300
          task = (*)0x5c62690588 âž” (*)0x9d587a40f0
      7 fuchsia_async::runtime::fuchsia::executor::common::Executor::poll_ready_tasks(…) • executor/common.rs:133
          self = (*)0x9b187a9300
          local_collector = (*)0x5c62690600 âž” LocalCollector{collector: (*)0x9b187a9438, last_ticks: 1050997948718604, polls: 1, tasks_pending_max: 10}
      8 fuchsia_async::runtime::fuchsia::executor::common::Executor::worker_lifecycle<…>(…) • executor/common.rs:429
          self = (*)0xa4187a90a0
      9 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run<…>(…) • executor/local.rs:104
          self = (*)0x5c62690878 ➔ LocalExecutor{ehandle: EHandle{…}}
          main_future = AtomicFuture{state: AtomicUsize{1}, future: UnsafeCell{$(alloc::boxed::Box<dyn fuchsia_async::atomic_future::FutureOrResultAccess, alloc::alloc::Global>){…}}}
      10 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run_singlethreaded<…>(…) • executor/local.rs:70
          self = (*)0x5c62690878 ➔ LocalExecutor{ehandle: EHandle{…}}
          main_future = Unresumed{run_stream: (*)0xa3187a86a0, test: λ{…}}
      11 fuchsia_async::test_support::«impl»::run_singlethreaded::λ() • test_support.rs:120
      12 fuchsia_async::test_support::Config::in_parallel(…) • test_support.rs:215
          self = (*)0x5c62690ae8 ➔ Config{repeat_count: 1, max_concurrency: 0, max_threads: 0, timeout: …}
          f = <The value of type '*const alloc::sync::ArcInner<(dyn core::ops::function::Fn<(), Output=()> + core::marker::Send + core::marker::Sync)>' is the incorrect size (expecting 8, got 16). Please file a bug.>
      13 fuchsia_async::test_support::«impl»::run_singlethreaded(…) • test_support.rs:117
          test = <The value of type '*const alloc::sync::ArcInner<(dyn core::ops::function::Fn<(usize), Output=core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output=()>, alloc::alloc::Global>>> + core::marker::Send + core::marker::Sync)>' is the incorrect size (expecting 8, got 16). Please file a bug.>
          cfg = Config{repeat_count: 1, max_concurrency: 0, max_threads: 0, timeout: None<Duration>}
      14 fuchsia_async::test_support::run_singlethreaded_test<…>(…) • test_support.rs:227
          test = <Unavailable>
      15 fuchsia::test_singlethreaded<…>(…) • fuchsia/src/lib.rs:195
          f = <Register rdi not available.>
      16 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log() • archivist.rs:524
      17 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::λ(…) • archivist.rs:525
          (*)0x5c62690bde ➔ λ
      18 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
          λ
          <Value has no data.>
      19 core::ops::function::FnOnce::call_once<…>(…) • library/core/src/ops/function.rs:250 (inline)
          <Register rsi not available.>
          <Optimized out>
      20 test::__rust_begin_short_backtrace<…>(…) • library/test/src/lib.rs:625
          f = <Register rsi not available.>
      21 test::run_test_in_spawned_subprocess(…) • library/test/src/lib.rs:753
          desc = <Register rdi not available.>
          runnable_test = Static(&core::ops::function::FnOnce::call_once<archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::{closure_env#0}, ()>)
      22 test::test_main_static_abort(…) • library/test/src/lib.rs:199
          tests = <Unavailable>
      23 archivist_lib_lib_test::main() • archivist/src/lib.rs:1
      24 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
          &archivist_lib_lib_test::main
          <Value has no data.>
      25 std::sys::backtrace::__rust_begin_short_backtrace<…>(…) • fuchsia-third_party-rust/library/std/src/sys/backtrace.rs:155
          f = &archivist_lib_lib_test::main
      26 std::rt::lang_start::λ() • fuchsia-third_party-rust/library/std/src/rt.rs:159
      27 core::ops::function::impls::«impl»::call_once<…>(…) • library/core/src/ops/function.rs:284 (inline)
          self = $(&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)){pointer: (*)0x5c62690f50 âž” $((dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)), vtable = <Invalid data offset 8 in object of size 8.>}
          args = <Optimized out>
      28 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
          data = <Optimized out>
      29 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
      30 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
      31 std::rt::lang_start_internal::λ() • library/std/src/rt.rs:141 (inline)
      32 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
          data = <Optimized out>
      33 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
      34 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
      35 std::rt::lang_start_internal(…) • library/std/src/rt.rs:141
          main = $(&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)){pointer: (*)0x5c62690f50 âž” $((dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)), vtable: (*)0x13a66000958}
          argc = <Register rdx not available.>
          argv = <Register rcx not available.>
          sigpipe = <Register r8 not available.>
      36 std::rt::lang_start<…>(…) • fuchsia-third_party-rust/library/std/src/rt.rs:158
          main = &archivist_lib_lib_test::main
          argc = 2
          argv = (*)0xecf9d19fb0
          sigpipe = 0
      37 $elf(main) + 0x21
      38…40 «libc startup» (-r expands)
    
  5. List the current threads:

    thread
    

    You should see an output like:

      # state      koid name
    â–¶ 1 Core Dump 14450503
    
  6. List a specific thread. For example, thread 1:

    thread 1
    

    You should see an output like:

    Thread 1 state="Core Dump" koid=14450503 name=""
    
  7. You can then use frame to see this specific stack frame from thread 1:

    frame
    

    You should see an output like:

    ▶ 0 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
      1 core::future::future::«impl»::poll<…>(…) • future/future.rs:123
      2 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:27
      3 fuchsia_async::test_support::«impl»::run_singlethreaded::λ::λ(…) • test_support.rs:122
      4 fuchsia_async::atomic_future::«impl»::poll<…>(…) • atomic_future.rs:72
      5 fuchsia_async::atomic_future::AtomicFuture::try_poll(…) • atomic_future.rs:230
      6 fuchsia_async::runtime::fuchsia::executor::common::Executor::try_poll(…) • executor/common.rs:554
      7 fuchsia_async::runtime::fuchsia::executor::common::Executor::poll_ready_tasks(…) • executor/common.rs:133
      8 fuchsia_async::runtime::fuchsia::executor::common::Executor::worker_lifecycle<…>(…) • executor/common.rs:429
      9 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run<…>(…) • executor/local.rs:104
      10 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::run_singlethreaded<…>(…) • executor/local.rs:70
      11 fuchsia_async::test_support::«impl»::run_singlethreaded::λ() • test_support.rs:120
      12 fuchsia_async::test_support::Config::in_parallel(…) • test_support.rs:215
      13 fuchsia_async::test_support::«impl»::run_singlethreaded(…) • test_support.rs:117
      14 fuchsia_async::test_support::run_singlethreaded_test<…>(…) • test_support.rs:227
      15 fuchsia::test_singlethreaded<…>(…) • fuchsia/src/lib.rs:195
      16 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log() • archivist.rs:524
      17 archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::λ(…) • archivist.rs:525
      18 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
      19 core::ops::function::FnOnce::call_once<…>(…) • library/core/src/ops/function.rs:250 (inline)
      20 test::__rust_begin_short_backtrace<…>(…) • library/test/src/lib.rs:625
      21 test::run_test_in_spawned_subprocess(…) • library/test/src/lib.rs:753
      22 test::test_main_static_abort(…) • library/test/src/lib.rs:199
      23 archivist_lib_lib_test::main() • archivist/src/lib.rs:1
      24 core::ops::function::FnOnce::call_once<…>(…) • fuchsia-third_party-rust/library/core/src/ops/function.rs:250
      25 std::sys::backtrace::__rust_begin_short_backtrace<…>(…) • fuchsia-third_party-rust/library/std/src/sys/backtrace.rs:155
      26 std::rt::lang_start::λ() • fuchsia-third_party-rust/library/std/src/rt.rs:159
      27 core::ops::function::impls::«impl»::call_once<…>(…) • library/core/src/ops/function.rs:284 (inline)
      28 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
      29 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
      30 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
      31 std::rt::lang_start_internal::λ() • library/std/src/rt.rs:141 (inline)
      32 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:553 (inline)
      33 std::panicking::try<…>() • library/std/src/panicking.rs:517 (inline)
      34 std::panic::catch_unwind<…>() • library/std/src/panic.rs:350 (inline)
      35 std::rt::lang_start_internal(…) • library/std/src/rt.rs:141
      36 std::rt::lang_start<…>(…) • fuchsia-third_party-rust/library/std/src/rt.rs:158
      37 $elf(main) + 0x21
      38…40 «libc startup» (-r expands)
    
  8. Based on the output from the previous step, you may want to dive deeper into frame 0:

    frame 0
    

    You should see an output like:

    archivist_lib_lib_test::archivist::tests::can_log_and_retrive_log::test_entry_point::λ(…) • archivist.rs:547
    
  9. To see additional lines of code from the current frame, use list:

    list
    

    You should see an output like:

      542
      543         let mut expected = vec!["my msg1".to_owned(), "my msg2".to_owned()];
      544         expected.sort();
      545
      546         let mut actual = vec![recv_logs.next().await.unwrap(), recv_logs.next().await.unwrap()];
    â–¶ 547         actual.sort();
      548
      549         assert_eq!(expected, actual);
      550
      551         // can log after killing log sink proxy
      552         log_helper.kill_log_sink();
      553         log_helper.write_log("my msg1");
      554         log_helper.write_log("my msg2");
      555
      556         assert_eq!(
      557             expected,
    

    Since this frame actually contains some archivist code, you can use locals to see the local variables in this stack frame.

  10. To see all local variables in the current stack frame, use locals:

    locals
    

    You should see an output like:

    _task_context = (*)0x5c626904e0 âž” Context{
      waker: (*)0x5c626904c8
      local_waker: (*)0x5c626904c8
      ext: AssertUnwindSafe<core::task::wake::ExtData>(None(<Value has no data.>))
      _marker: PhantomData<fn(&())->&()>
      _marker2: PhantomData<*mut()>
    }
    actual = <Invalid pointer 0x9c187a04e8>
    directory = <Invalid pointer 0x9c187a0460>
    expected = <Invalid pointer 0x9c187a04d0>
    log_helper = <Invalid pointer 0x9c187a0470>
    log_helper2 = <Invalid pointer 0x9c187a04c0>
    recv_logs = <Invalid pointer 0x9c187a0468>
    

    You can then see more information about a specific variable with print. For example:

    print _task_context
    

    You should see an output like:

    (*)0x5c626904e0 âž” Context{
      waker: (*)0x5c626904c8
      local_waker: (*)0x5c626904c8
      ext: AssertUnwindSafe<core::task::wake::ExtData>(None(<Value has no data.>))
      _marker: PhantomData<fn(&())->&()>
      _marker2: PhantomData<*mut()>
    }
    

You have successfully loaded and analyzed a minidump for Rust code.

C++

  1. To load the minidump file into zxdb, you can use ffx debug core. For example, to load a minidump file named cobalt_minidump.dump:

    ffx debug core cobalt_minidump.dump

    You should see an output like:

    Opening dump file...
    Dump loaded successfully.
    👉 To get started, try "status" or "help".
    Attached Process 1 state=Running koid=15650768 name=<_>
    Loading 15 modules for <_> ...Done.
    Attached Process 2 state=Running koid=15650768 name=<_>
    Attaching to previously connected processes:
    15650768: <_>
    Loading 15 modules for <_> Done.
    

    Now that you have loaded the minidump file, you can now inspect the stacks for all threads present that were present when the minidump was captured.

  2. List the stack frames:

    frame
    

    You should see an output like:

    ▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
      5 main(…) • cobalt_main.cc:349
      6…8 «libc startup» (-r expands)
    
  3. You can now use shortcuts to quickly perform actions across all threads:

    List the frames of each thread:

    thread * frame
    

    You should see an output like:

    Thread 1 state="Core Dump" koid=19601 name=""
    ▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
      5 main(…) • cobalt_main.cc:349
      6…8 «libc startup» (-r expands)
    Thread 2 state="Core Dump" koid=26944 name=""
    ▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
      1 _zx_futex_wait(…) • syscalls.inc:210
      2 __timedwait_assign_owner(…) • __timedwait.c:23
      3 __timedwait(…) • threads_impl.h:322 (inline)
      4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
      5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
      6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
      7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
      8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
      9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
      10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
      11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
      12 std::__2::condition_variable_any::wait_for<…>(…) • condition_variable:260
      13 cobalt::local_aggregation::DelayedLocalAggregateStorage::Run(…) • delayed_local_aggregate_storage.cc:200
      14 λ(…) • delayed_local_aggregate_storage.cc:45
      15 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
      16 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
      17 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
      18…19 «pthread startup» (-r expands)
    Thread 3 state="Core Dump" koid=26975 name=""
    ▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
      1 _zx_futex_wait(…) • syscalls.inc:210
      2 __timedwait_assign_owner(…) • __timedwait.c:23
      3 __timedwait(…) • threads_impl.h:322 (inline)
      4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
      5 std::__2::__libcpp_condvar_wait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:122 (inline)
      6 std::__2::condition_variable::wait(…) • condition_variable.cpp:30
      7 std::__2::condition_variable_any::wait<…>(…) • condition_variable:225
      8 std::__2::condition_variable_any::wait<…>(…) • condition_variable:231
      9 cobalt::uploader::ShippingManager::Run(…) • shipping_manager.cc:217
      10 λ(…) • shipping_manager.cc:76
      11 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
      12 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
      13 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
      14…15 «pthread startup» (-r expands)
    Thread 4 state="Core Dump" koid=26983 name=""
    ▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
      1 _zx_futex_wait(…) • syscalls.inc:210
      2 __timedwait_assign_owner(…) • __timedwait.c:23
      3 __timedwait(…) • threads_impl.h:322 (inline)
      4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
      5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
      6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
      7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
      8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
      9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
      10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
      11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
      12 cobalt::local_aggregation::ObservationGenerator::Run(…) • observation_generator.cc:92
      13 λ(…) • observation_generator.cc:65
      14 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
      15 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
      16 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
      17…18 «pthread startup» (-r expands)
    
  4. To list the frames of each thread, but with more detailed information, you can combine thread with the backtrace verb:

    thread * backtrace
    

    This command helps you see the local variables for each stack frame. Keep in mind that only the stack memory is captured in the minidump to improve the capturing time and size. Pointers to heap variables mostly do not resolve to anything useful, these have a format like <Invalid pointer 0x285f3fd5a18>.

    You should see an output like:

    Thread 1 state="Core Dump" koid=19601 name=""
    ▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
      5 main(…) • cobalt_main.cc:349
          argc = 2
          argv = (*)0x26866821fc0
      6…8 «libc startup» (-r expands)
    Thread 2 state="Core Dump" koid=26944 name=""
    ▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
      1 _zx_futex_wait(…) • syscalls.inc:210
          value_ptr = (*)0x285f3fd5904
          current_value = 2
          new_futex_owner = 0
          deadline = 612921722073211
      2 __timedwait_assign_owner(…) • __timedwait.c:23
          futex = <Register rdi not available.>
          val = <Register rsi not available.>
          clk = <Register rdx not available.>
          at = <Register rcx not available.>
          new_owner = <Register r8 not available.>
      3 __timedwait(…) • threads_impl.h:322 (inline)
          futex = (*)0x285f3fd5904
          val = 2
          clk = 0
          at = (*)0x285f3fd5918
      4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
          c = (*)0x370ff24ae20
          m = (*)0x3707f2b76c8
          ts = (*)0x285f3fd5918
      5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
          __cv = <Register rdi not available.>
          __m = <Register rsi not available.>
          __ts = <Optimized out>
      6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
          this = <Register rdi not available.>
          lk = <Register rsi not available.>
          tp = <Register rdx not available.>
      7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
          this = (*)0x370ff24ae20
          __lk = <Invalid pointer 0x285f3fd5a18>
          __d = <Invalid pointer 0x285f3fd59a8>
      8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
          this = (*)0x370ff24ae20
          __lk = <Invalid pointer 0x285f3fd5a18>
          __tp = <Invalid pointer 0x285f3fd59b0>
      9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
          this = (*)0x370ff24ae20
          __lk = <Invalid pointer 0x285f3fd5a18>
          __t = <Invalid pointer 0x285f3fd5a60>
      10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
          this = (*)0x370ff24ae20
          __lock = <Invalid pointer 0x285f3fd5ab0>
          __t = <Invalid pointer 0x285f3fd5a60>
      11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
          this = (*)0x370ff24ae20
          __lock = <Invalid pointer 0x285f3fd5ab0>
          __t = <Invalid pointer 0x285f3fd5a60>
          __pred = <Invalid pointer 0x285f3fd5a40>
      12 std::__2::condition_variable_any::wait_for<…>(…) • condition_variable:260
          this = (*)0x370ff24ae20
          __lock = <Invalid pointer 0x285f3fd5ab0>
          __d = <Invalid pointer 0x370ff24ae00>
          __pred = {locked_state = <Invalid pointer 0x285f3fd5ab0>}
      13 cobalt::local_aggregation::DelayedLocalAggregateStorage::Run(…) • delayed_local_aggregate_storage.cc:200
          this = (*)0x370ff24acf0
      14 λ(…) • delayed_local_aggregate_storage.cc:45
          this = (*)0x36bff27b228
      15 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
          __f = <Invalid pointer 0x36bff27b228>
      16 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
          __t = <Invalid pointer 0x36bff27b220>
          {}
      17 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
          __vp = (*)0x36bff27b220
      18…19 «pthread startup» (-r expands)
    Thread 3 state="Core Dump" koid=26975 name=""
    ▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
      1 _zx_futex_wait(…) • syscalls.inc:210
          value_ptr = (*)0x23b6ad77434
          current_value = 2
          new_futex_owner = 0
          deadline = 9223372036854775807
      2 __timedwait_assign_owner(…) • __timedwait.c:23
          futex = <Register rdi not available.>
          val = <Register rsi not available.>
          clk = <Register rdx not available.>
          at = <Register rcx not available.>
          new_owner = <Register r8 not available.>
      3 __timedwait(…) • threads_impl.h:322 (inline)
          futex = (*)0x23b6ad77434
          val = 2
          clk = 0
          at = (*)0x0
      4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
          c = (*)0x3673f24a610
          m = (*)0x3707f2b7998
          ts = (*)0x0
      5 std::__2::__libcpp_condvar_wait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:122 (inline)
          __cv = <Register rdi not available.>
          __m = <Register rsi not available.>
      6 std::__2::condition_variable::wait(…) • condition_variable.cpp:30
          this = <Register rdi not available.>
          lk = <Register rsi not available.>
      7 std::__2::condition_variable_any::wait<…>(…) • condition_variable:225
          this = (*)0x3673f24a610
          __lock = <Invalid pointer 0x23b6ad77608>
      8 std::__2::condition_variable_any::wait<…>(…) • condition_variable:231
          this = (*)0x3673f24a610
          __lock = <Invalid pointer 0x23b6ad77608>
          __pred = <Invalid pointer 0x23b6ad77480>
      9 cobalt::uploader::ShippingManager::Run(…) • shipping_manager.cc:217
          this = (*)0x3673f24a530
      10 λ(…) • shipping_manager.cc:76
          this = (*)0x36bff27a7a8
      11 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
          __f = <Invalid pointer 0x36bff27a7a8>
      12 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
          __t = <Invalid pointer 0x36bff27a7a0>
          {}
      13 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
          __vp = (*)0x36bff27a7a0
      14…15 «pthread startup» (-r expands)
    Thread 4 state="Core Dump" koid=26983 name=""
    ▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
      1 _zx_futex_wait(…) • syscalls.inc:210
          value_ptr = (*)0x149c84b2bc4
          current_value = 2
          new_futex_owner = 0
          deadline = 615784347537787
      2 __timedwait_assign_owner(…) • __timedwait.c:23
          futex = <Register rdi not available.>
          val = <Register rsi not available.>
          clk = <Register rdx not available.>
          at = <Register rcx not available.>
          new_owner = <Register r8 not available.>
      3 __timedwait(…) • threads_impl.h:322 (inline)
          futex = (*)0x149c84b2bc4
          val = 2
          clk = 0
          at = (*)0x149c84b2bd8
      4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
          c = (*)0x3673f24b128
          m = (*)0x3707f2b6548
          ts = (*)0x149c84b2bd8
      5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
          __cv = <Register rdi not available.>
          __m = <Register rsi not available.>
          __ts = <Optimized out>
      6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
          this = <Register rdi not available.>
          lk = <Register rsi not available.>
          tp = <Register rdx not available.>
      7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
          this = (*)0x3673f24b128
          __lk = <Invalid pointer 0x149c84b2cd8>
          __d = <Invalid pointer 0x149c84b2c68>
      8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
          this = (*)0x3673f24b128
          __lk = <Invalid pointer 0x149c84b2cd8>
          __tp = <Invalid pointer 0x149c84b2c70>
      9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
          this = (*)0x3673f24b128
          __lk = <Invalid pointer 0x149c84b2cd8>
          __t = <Invalid pointer 0x3673f24b0d8>
      10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
          this = (*)0x3673f24b128
          __lock = <Invalid pointer 0x149c84b2d20>
          __t = <Invalid pointer 0x3673f24b0d8>
      11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
          this = (*)0x3673f24b128
          __lock = <Invalid pointer 0x149c84b2d20>
          __t = <Invalid pointer 0x3673f24b0d8>
          __pred = <Invalid pointer 0x149c84b2d00>
      12 cobalt::local_aggregation::ObservationGenerator::Run(…) • observation_generator.cc:92
          this = (*)0x3673f24b0a0
          clock = (*)0x36bff27aaa0
      13 λ(…) • observation_generator.cc:65
          this = (*)0x36bff24cf68
      14 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
          __f = <Invalid pointer 0x36bff24cf68>
      15 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
          __t = <Invalid pointer 0x36bff24cf60>
          {}
      16 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
          __vp = (*)0x36bff24cf60
      17…18 «pthread startup» (-r expands)
    
  5. List the current threads:

    thread
    

    You should see an output like:

      # state      koid name
    â–¶ 1 Core Dump 19601
      2 Core Dump 26944
      3 Core Dump 26975
      4 Core Dump 26983
    
  6. List a specific thread. For example, thread 2:

    thread 2
    

    You should see an output like:

    Thread 2 state="Core Dump" koid=26944 name=""
    
  7. You can then use frame to see this specific stack frame from thread 2:

    frame
    

    You should see an output like:

    ▶ 0 $elf(CODE_SYSCALL_zx_futex_wait) + 0xa • syscalls.inc:210
      1 _zx_futex_wait(…) • syscalls.inc:210
      2 __timedwait_assign_owner(…) • __timedwait.c:23
      3 __timedwait(…) • threads_impl.h:322 (inline)
      4 pthread_cond_timedwait(…) • pthread_cond_timedwait.c:78
      5 std::__2::__libcpp_condvar_timedwait(…) • stage2-bins/include/c++/v1/__thread/support/pthread.h:127 (inline)
      6 std::__2::condition_variable::__do_timed_wait(…) • condition_variable.cpp:54
      7 std::__2::condition_variable::wait_for<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:196
      8 std::__2::condition_variable::__do_timed_wait<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:235
      9 std::__2::condition_variable::wait_until<…>(…) • linux-x64/include/c++/v1/__condition_variable/condition_variable.h:161
      10 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:240
      11 std::__2::condition_variable_any::wait_until<…>(…) • condition_variable:247
      12 std::__2::condition_variable_any::wait_for<…>(…) • condition_variable:260
      13 cobalt::local_aggregation::DelayedLocalAggregateStorage::Run(…) • delayed_local_aggregate_storage.cc:200
      14 λ(…) • delayed_local_aggregate_storage.cc:45
      15 std::__2::__invoke<…>(…) • linux-x64/include/c++/v1/__type_traits/invoke.h:150
      16 std::__2::__thread_execute<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:192
      17 std::__2::__thread_proxy<…>(…) • linux-x64/include/c++/v1/__thread/thread.h:201
      18…19 «pthread startup» (-r expands)
    
  8. List another specific thread. For example, thread 1:

    thread 1
    

    You should see an output like:

    Thread 1 state="Core Dump" koid=19601 name=""
    
  9. You can then use frame to see this specific stack frame from thread 1:

    frame
    

    You should see an output like:

    ▶ 0…4 «Waiting for event in async::Loop::Run()» (-r expands)
      5 main(…) • cobalt_main.cc:349
      6…8 «libc startup» (-r expands)
    
  10. Based on the output from the previous step, you may want to dive deeper into frame 5:

    frame 5
    

    You should see an output like:

    main(…) • cobalt_main.cc:349
    
  11. To see additional lines of code from the current frame, use list:

    list
    

    You should see an output like:

      344
      345   if (!app.ok()) {
      346     FX_LOGS(FATAL) << "Failed to construct the cobalt app: " << app.status();
      347   }
      348   inspector.Health().Ok();
    â–¶ 349   loop.Run();
      350   FX_LOGS(INFO) << "Cobalt will now shut down.";
      351   return 0;
      352 }
    

    Since this frame actually contains some code, you can use locals to see the local variables in this stack frame.

  12. To see all local variables in the current stack frame, use locals:

    locals
    

    You should see an output like:

    argc = 2
    argv = (*)0x26866821fc0
    command_line = <Invalid pointer 0x26866821418>
    context = <Invalid pointer 0x26866821100>
    event_aggregator_backfill_days = 2
    flag_value = <Invalid pointer 0x268668211d8>
    initial_interval = <Invalid pointer 0x26866821138>
    inspector = <Invalid pointer 0x268668212f0>
    loop = <Invalid pointer 0x26866821108>
    max_bytes_per_observation_store = 215040
    min_interval = <Invalid pointer 0x26866821118>
    require_lifecycle_service = true
    schedule_interval = <Invalid pointer 0x26866821140>
    start_event_aggregator_worker = true
    status = 0 (ZX_OK)
    storage_quotas = {
      per_project_reserved_bytes = 1024
      total_capacity_bytes = 731136
    }
    test_dont_backfill_empty_reports = false
    upload_jitter = 0.2
    upload_schedule = {
      target_interval = {__rep_ = 3600}
      min_interval = {__rep_ = 10}
      initial_interval = {__rep_ = 60}
      jitter = 0.2
    }
    use_fake_clock = false
    use_memory_observation_store = false
    

    You can then see more information about a specific variable with print. For example:

    print upload_schedule
    

    You should see an output like:

    {
      target_interval = {__rep_ = 3600}
      min_interval = {__rep_ = 10}
      initial_interval = {__rep_ = 60}
      jitter = 0.2
    }
    

You have successfully loaded and analyzed a minidump for C++ code.