How not to get old while waiting for tests

Context

I think there is no need to explain how important tests are, what types there are and when they need to be written. Today I would like to draw your attention to the problems we encountered during the launch of tests.

Problem statement

First of all, let's take a look at the architecture of our modules. Globally, we have core, uikit and of course feature packages. But each product can also have its own core and feature packages. This way we can independently deliver new features, remove and also refactor them with ease. If we go deeper, there are 3-4 more packages inside of each feature package. impl, ui, ui_desktop packages contain unit tests, hence by considering the fact that we have more than 500 packages in the monorepo, you may understand the problem. We are losing a lot of time in our pipelines. Also because of tests taking significant time, we have very huge queues which are causing the second problem of available resources, hence developers can see messages in Pipeline like:

The agent request is not running because all potential agents are running other requests. Current position in queue: 59

Let’s see the exact numbers and statistics. We have three products: SME, Retail, Broker, and we are going to look into 4 pipelines with tests Retail, Broker, SME (Legacy Mobile only), SME Common (New architecture for Web and Mobile).

Reason

Briefly, because of running tests for many packages. Why can’t we run tests for packages in parallel (just like melos)? We can speed up a bit with this approach and we have been using this way. But when you have 500+ packages and every day new features are coming it can’t help you to run under ~10 minutes.

Let’s analyze the reason and try to solve it. First of all, we need a basic understanding of what’s going on under the hood of 1 single package during tests.

1. Under the hood, everything starts with pub get. We need to load all dependencies to compile the tests.
2. Building test assets is a skippable operation which can be used to load some assets needed for the tests. For instance, for Golden/UI tests you may need some images and some JSON/XML files for unit tests. If this option is turned on then loads assets and puts them into unit_test_assets folder inside the build folder.
3. Starting a testing device is the launch of some device (which can be virtual) to run tests. You may think about integration tests with simulators, but we are now focused on unit tests. So during ordinary tests, FlutterTesterTestDevice is created to run one single .dart file (but it’s compiled as a dill file). It delegates running unit tests into the flutter_test process inside the flutter engine.
4. Creating and compiling/recompiling flutter_test_listener.dart is actually a huge step in the pipeline. In the 3rd step, we created a test device which accepts some .dill files, right? Originally this .dill was a kind of .dart test file, but actually, it was a wrapper file on top of our test file. Flutter generates this file and imports our original test file into it. For more details refer to this code snippet:

flutter/packages/flutter_tools/lib/src/test/flutter_platform.dart at stable · flutter/flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond - flutter/flutter

GitHubflutter

Now we found out that Flutter needs to compile that generated file. For that, it creates ResidentCompiler and sends a path to that. Therefore, that compiler uses frontend_server to compile the dart code. For the next Dart file it's going to be reused only. That's the main reason why tests for several packages are taking so much time.

sdk/pkg/frontend_server/lib/frontend_server.dart at main · dart-lang/sdk

The Dart SDK, including the VM, dart2js, core libraries, and more. - dart-lang/sdk

GitHubdart-lang

Solution

We thought that probably in CI some virtual package with all the unit tests could be set up to speed up everything. It’s possible because we have standard versions of dependencies across the monorepo which means that we are not going to face conflicts. Also, we are going to cache most of the libraries, and dependencies in the precompiled file once. No need to create isolates, servers, and sockets again and again for every package. So that was the idea behind our new script.

We should just merge all the dependencies + all the test files into one virtual temporary package.

Main executable dart file - is an interesting file which may remind flutter_test_listener.dart from flutter_tools. It also imports all the test files and wraps them with its own logic:

import 'features/feature_a/x_test.dart' as _i0;
import 'features/feature_b/y_test.dart' as _i1;

void main () {
  group('x_test.dart', () {
     _i0.main();
  });

  group('y_test.dart', () {
     _i1.main();
  });
}

That’s it. We just go through packages, take pubspec.yaml from there and parse yaml with our common script for that. Then we resolve conflicts by deleting duplicates and so on.

List<PackageDependencies> _getAllFeaturesDeps (List<Directory> packages) {
  return packages
    .map((e) => e.childFile('pubspec.yaml'))
    .map((e) => _parsePubspec(e))
    .toList()
}

Conclusion

The results are pretty impressive and the script is stable. Potential issues with global variables and golden tests exist, but we haven't faced them yet. We'll handle them collaboratively if they arise. Moving forward, let's stay focused and confident in our progress.

By Yerkebulan Yelzhan, Senior Flutter Engineer