Mod integration testing with Topaz

Introduction

This step-by-step guide is for running integration tests using Flutter Driver in Topaz. If you are not looking to run integration testing on your mods, or if your mod is not written using Flutter, then you don’t need this guide.

This is different from unit testing with widgets because the expectation is that you will be testing simulated user interaction with your mod (tapping buttons, scrolling, etc, for example), which requires Scenic and cannot be run in QEMU.

The examples in this doc will be focused around a testing mod under //topaz/examples/test/driver_example_mod. The name is derived from how it is an example mod relating to the use of Flutter Driver. In addition, you'll see how to set up a hermetic test with Fuchsia component testing.

The ultimate goal of this document is to make it possible for you to add integration tests into //topaz/tests so that your mod can be tested in CQ and CI.

Setup

To start, this doc assumes you’ve already got a mod that you can run on Topaz (see here if you haven't). For simplicity, we’ll assume it is a standalone mod that doesn’t depend on other mods (in the future this will have been tested and verified).

Per the introduction section, we’ll be focusing on driver_example_mod as the mod under test.

Enabling Flutter Driver extensions

If you want to simulate user interaction with your mod, or capture a screenshot of a certain state you are interested in, you will need to enable the flutter driver extensions before you can start using the flutter driver. This is done by adding flutter_driver_extendable = true to your flutter_app target in the BUILD.gn for your mod:

flutter_app("driver_example_mod") {
  // ...
  flutter_driver_extendable = true
  // ...
}

In a debug JIT setting, this will generate a wrapper for your main that calls enableFlutterDriverExtension() from package:flutter_driver/driver_extension.dart. (See also: gen_debug_wrapper_main.py)

Writing your tests

Next you’ll get to the exciting part: writing the tests for your mod! These will require use of the aforementioned Flutter Driver library.

Tests live in a test subfolder of your mod and end in _test.dart. These requirements are stipulated by dart_fuchsia_test, described later. The tests for driver_example_mod are in driver_example_mod_test.dart.

Boilerplate

You’ll need some boilerplate to set up and tear down your code. The following helper function starts Modular with test shells and launches the mod under test in a new story.

import 'package:fidl_fuchsia_modular/fidl_async.dart' as modular;
import 'package:fidl_fuchsia_modular_testing/fidl_async.dart';
import 'package:fuchsia_modular_testing/test.dart';

const Pattern _isolatePattern = 'driver_example_mod.cmx';
const _testAppUrl =
    'fuchsia-pkg://fuchsia.com/driver_example_mod#meta/driver_example_mod.cmx';

final _addModCommand = modular.AddMod(
    modName: [_isolatePattern],
    modNameTransitional: 'root',
    intent: modular.Intent(action: 'action', handler: _testAppUrl),
    surfaceRelation: modular.SurfaceRelation());

// Starts Modular with test shells. This should be called from within a
// try/finally or similar construct that closes the component controller.
Future<void> _launchModUnderTest(TestHarnessProxy testHarness) async {
  final puppetMaster = modular.PuppetMasterProxy();
  await testHarness.connectToModularService(
      ModularService.withPuppetMaster(puppetMaster.ctrl.request()));

  // Use PuppetMaster to start a fake story and launch the mod under test
  final storyPuppetMaster = modular.StoryPuppetMasterProxy();
  await puppetMaster.controlStory(
      'driver_example_mod_test', storyPuppetMaster.ctrl.request());
  await storyPuppetMaster
      .enqueue([modular.StoryCommand.withAddMod(_addModCommand)]);
  await storyPuppetMaster.execute();
}

The Intent.handler defines the mod that will be launched via _addModCommand. After starting the Modular TestHarness, you can use PuppetMaster to create a new story and execute this command.

In your test setup, you'll need to launch and run the Modular TestHarness. You can specify any extra Modular configurations in the TestHarnessSpec. Once completed, you can launch your mod and connect to Flutter Driver.

// ...
import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';

// ... _launchModUnderTest ...

void main() {
  group('driver example tests', () {
    TestHarnessProxy testHarness;
    FlutterDriver driver;

    setUpAll(() async {
      testHarness = await launchTestHarness();
      await testHarness.run(TestHarnessSpec(
          envServices:
              EnvironmentServicesSpec(serviceDir: Channel.fromFile('/svc'))));
      await _launchModUnderTest(testHarness);

      driver = await FlutterDriver.connect(
          fuchsiaModuleTarget: _isolatePattern,
          printCommunication: true,
          logCommunicationToFile: false);
    });

    tearDownAll(() async {
      await driver?.close();
      testHarness.ctrl.close();
    });

    // ...
  });
}

When a Dart app starts in debug mode, it exposes the Dart Observatory (VM Service) on an HTTP port. FlutterDriver.connect connects to the Dart Observatory and finds the isolate for the mod named in fuchsiaModuleTarget. Behind the scenes, Flutter Driver uses FuchsiaCompat and FuchsiaRemoteConnection to search over all the Dart VMs running on the device to find the isolate that matches your fuchsiaModuleTarget. Once this is found, Flutter Driver opens a websocket over which to send RPCs which were registered by your mod under test in enableFlutterDriverExtension().

If you’d like to see an example test that pushes a few buttons, you can check here.

Component manifest

A component manifest allows the test to run as a hermetic test component under its own dedicated environment that will sandbox its services and tear everything down on completion or failure. This is particularly important for Flutter Driver tests and other graphical tests as only one Scenic instance may own the display controller at a time, so any such test that does not properly clean up can cause subsequent tests to fail.

The component manifest for our tests is driver_example_mod_tests.cmx.

{
    "facets": {
        "fuchsia.test": {
            "injected-services": {
                "fuchsia.auth.account.AccountManager": "fuchsia-pkg://fuchsia.com/account_manager#meta/account_manager.cmx",
                "fuchsia.devicesettings.DeviceSettingsManager": "fuchsia-pkg://fuchsia.com/device_settings_manager#meta/device_settings_manager.cmx",
                "fuchsia.fonts.Provider": "fuchsia-pkg://fuchsia.com/fonts#meta/fonts.cmx",
                "fuchsia.sysmem.Allocator": "fuchsia-pkg://fuchsia.com/sysmem_connector#meta/sysmem_connector.cmx",
                "fuchsia.tracing.provider.Registry": "fuchsia-pkg://fuchsia.com/trace_manager#meta/trace_manager.cmx",
                "fuchsia.ui.input.ImeService": "fuchsia-pkg://fuchsia.com/ime_service#meta/ime_service.cmx",
                "fuchsia.ui.policy.Presenter": "fuchsia-pkg://fuchsia.com/root_presenter#meta/root_presenter.cmx",
                "fuchsia.ui.scenic.Scenic": "fuchsia-pkg://fuchsia.com/scenic#meta/scenic.cmx",
                "fuchsia.vulkan.loader.Loader": "fuchsia-pkg://fuchsia.com/vulkan_loader#meta/vulkan_loader.cmx"
            },
            "system-services": [
                "fuchsia.net.NameLookup",
                "fuchsia.net.SocketProvider",
                "fuchsia.posix.socket.Provider"
            ]
        }
    },
    "program": {
        "data": "data/driver_example_mod_tests"
    },
    "sandbox": {
        "features": [
            "deprecated-shell",
            "deprecated-ambient-replace-as-executable"
        ],
        "services": [
            "fuchsia.auth.account.AccountManager",
            "fuchsia.devicesettings.DeviceSettingsManager",
            "fuchsia.fonts.Provider",
            "fuchsia.net.NameLookup",
            "fuchsia.net.SocketProvider",
            "fuchsia.posix.socket.Provider",
            "fuchsia.sys.Environment",
            "fuchsia.sys.Launcher",
            "fuchsia.sysmem.Allocator",
            "fuchsia.ui.policy.Presenter",
            "fuchsia.ui.scenic.Scenic",
            "fuchsia.vulkan.loader.Loader"
        ]
    }
}

The injected-services entry starts the hermetic services our mod will need, mostly related to graphics. In addition, the fuchsia.net.SocketProvider system service and deprecated-shell feature are needed to allow Flutter Driver to interact with the Dart Observatory.

BUILD.gn target

The test itself also needs a target in the BUILD.gn.

dart_fuchsia_test("driver_example_mod_tests") {
  deps = [
    "//sdk/fidl/fuchsia.sys",
    "//third_party/dart-pkg/git/flutter/packages/flutter_driver",
    "//third_party/dart-pkg/pub/test",
    "//topaz/public/dart/fuchsia_modular_testing",
    "//topaz/public/dart/fuchsia_services",
  ]

  meta = [
    {
      path = rebase_path("meta/driver_example_mod_tests.cmx")
      dest = "driver_example_mod_tests.cmx"
    },
  ]

  environments = []

  # Flutter driver is only available in debug builds.
  if (is_debug) {
    environments += [
      nuc_env,
      vim2_env,
    ]
  }
}

dart_fuchsia_test defines a Dart test that runs on a Fuchsia device. It uses each file in the test subfolder that ends in _test.dart as an entrypoint for tests. In addition, it links the component manifest for the tests and specifies the environments in which to run the test in automated testing (CI/CQ). (See also the predefined environments in //build/testing/environments.gni.)

Topaz Package

Once you have this target available, you can add it to the build tree. In the case of driver_example_mod, this can be done in //topaz/packages/examples:tests to be available in //topaz/bundles:buildbot and other configurations, like so:

group("tests") {
  testonly = true
  public_deps = [
    # ...
    "//topaz/examples/test/driver_example_mod",
    "//topaz/examples/test/driver_example_mod:driver_example_mod_tests",
    # ...
  ]
}

With that you’re ready to run your test on your device.

Running Your Tests

To run your tests, you first need to make sure you're building a configuration that includes your test packages. One typical way is to use the core product and //topaz/bundles:buildbot bundle, as that is what the CI/CQ bots use.

$ fx set core.arm64 --with //topaz/bundles:buildbot

(If you are on an Acer or Nuc, use x64 rather than arm64 as the board.)

Then, build and pave or OTA as necessary.

$ fx build
$ fx serve / ota / reboot

The tests can then be run using

$ fx run-test driver_example_mod_tests

If the test hangs after the module shows up, make sure you're building in debug mode (omit --release from your fx set) as the Dart Observatory will not be available in release builds.