Run an end-to-end test

This guide provides instructions on how to run an end-to-end test for testing a Fuchsia product.

This guide uses the Fuchsia emulator (FEMU) to emulate a device that runs Fuchsia. As for the end-to-end test, the guide uses the screen_is_not_black end-to-end test.

The screen_is_not_black_no_basemgr_test end-to-end test reboots a device under test (DUT), waits 100 seconds, and takes a snapshot of the device’s screen. If the snapshot image is not a black screen, the test concludes that Fuchsia is successfully up and running on the device after reboot.

To run this end-to-end test, the steps are:

Build a Fuchsia image to include the end-to-end test.
Start the emulator with the Fuchsia image.
Run the end-to-end test.

Also, to run any end-to-end test, see the General instructions section.

Prerequisites

Verify the following requirements:

Build a Fuchsia image to include the end-to-end test

Before you can run the screen_is_not_black_no_basemgr_test end-to-end test, you first need to build your Fuchsia image to include the test in the build artifacts:

To add the end-to-end test, run the fx set command with the following --with option:
```
fx set workstation.x64 --with //src/tests/end_to_end/screen_is_not_black:no_basemgr_test
```
//src/tests/end_to_end/screen_is_not_black is a test directory in the Fuchsia source tree. The BUILD.gn file in this directory defines the test target to include the screen_is_not_black_no_basemgr_test end-to-end test in the build artifacts.
Build your Fuchsia image:
```
fx build
```
When the fx build command completes, the build artifacts now include the screen_is_not_black_no_basemgr_test end-to-end test, which you can run from your host machine.

Start the emulator with the Fuchsia image

Start the emulator with your Fuchsia image and run a package repository server:

Configure an IPv6 network for the emulator (you only need to do this once):
```
sudo ip tuntap add dev qemu mode tap user $USER && sudo ifconfig qemu up
```
In a new terminal, start the emulator:
```
fx emu -N
```

Set the emulator to be your device:

fx set-device

If you have multiple devices, select fuchsia-5254-0063-5e7a (the emulator’s default device name), for example:

$ fx set-device

Multiple devices found, please pick one from the list:
1) fuchsia-4407-0bb4-d0eb
2) fuchsia-5254-0063-5e7a
#? 2
New default device: fuchsia-5254-0063-5e7a

In another new terminal, start a package repository server:
```
fx serve
```
Keep the fx serve command running as a package server for your device.

Run the end-to-end test

Run the screen_is_not_black_no_basemgr_test end-to-end test:

fx test --e2e screen_is_not_black_no_basemgr_test

When the test passes, this command prints output similar to the following:

Saw a screen that is not black; waiting for 0:00:59.924543 now.

...

[FINE]: Running over ssh: killall sl4f.cmx
01:46 +1: All tests passed!

General instructions

Run any end-to-end test

Use the fx test --e2e command to run an end-to-end test from your host machine:

fx test --e2e <TEST_NAME>

Some product configurations may include a set of end-to-end tests by default (see Examine product configuration files). However, if you want to run an end-to-end test that is not part of your product configuration, configure your Fuchsia image to include the specific test:

fx set <PRODUCT>.<BOARD> --with <TEST_DIRECTORY>:<TARGET>

For example, the following commands configure and build your Fuchsia image with all the end-to-end tests in the //src/tests/end_to_end/perf test directory:

$ fx set workstation.qemu-x64 --with //src/tests/end_to_end/perf:test
$ fx build

For the list of all available end-to-end tests in the Fuchsia repository, see the //src/tests/end_to_end directory.

Examine product configuration files

To find out which end-to-end tests are included in a specific product configuration, examine product configuration files (.gni) in the Fuchsia repository's //products directory.

The following example shows the product configurations files in the //products directory:

~/fuchsia/products$ ls *.gni
bringup.gni  core.gni  terminal.gni  workstation.gni

To see the list of all available product configurations, you can run the following command:

fx list-products

Among these product configurations, terminal and workstation include end-to-end tests by default. The following example shows the end-to-end tests included in terminal.gni:

cache_package_labels += [
  ...
  "//src/tests/end_to_end/bundles:end_to_end_deps",
  "//src/tests/end_to_end/bundles:terminal_end_to_end_deps",
]

...

universe_package_labels += [
  "//src/tests/end_to_end/screen_is_not_black:no_basemgr_test",
  "//src/tests/end_to_end/sl4f:test",
  "//src/tests/end_to_end/perf:test",
  ...
]