By Wio Tech in Frontend — May 1, 2024

Building an automated testing platform for Flutter applications

In our journey with Flutter at Wio, not only have we embraced its versatility for developing both mobile and web applications but also encountered a significant challenge: establishing a unified automated testing framework suitable for all these applications. Considering the limited range of tools available for Flutter's automated testing a few years back, this was no small feat.

Our vision for this framework: Efficiency reimagined, flexibility empowered, and comprehensiveness unbound.

Cross-Platform Testing: Our framework is designed to handle tests across different environments, such as web and mobile applications. This is particularly important for our projects in Flutter, ensuring versatility and adaptability.
Gherkin Language for Clear Test Cases: We embrace Gherkin for our test cases, making them crystal clear for everyone on the team. No matter your technical background, you can understand and contribute. This shared language speeds up troubleshooting when tests falter, keeping you moving forward.
Advanced Reporting for CI/CD Pipelines: Integration with our Azure DevOps CI/CD pipelines was a key goal. Beyond basic integration, we wanted to achieve detailed and visually engaging reports, enhancing the insight and understanding of our testing processes.
Incorporating Screenshots and Logs in Test Steps: To deepen our analysis, our framework includes the capability to attach screenshots and logs directly to specific steps in our test cases. This feature is invaluable for diagnosing and resolving issues more effectively.
Cost Efficiency: Throughout the development of this framework, we were conscious of maintaining cost efficiency. We aimed to create an economical solution yet yielded a significant return on investment.
Tag-Based Test Execution: A critical feature of our framework is the ability to execute tests in batches based on tags. This functionality allows for more efficient and targeted testing, particularly in response to changes made in certain areas of the application. By running only the relevant tests instead of the entire suite, we save considerable time and resources, enhancing the agility of our testing process.

Choosing the Right Tool for Integration Testing.

While this young framework brings excitement, automated testing still holds challenges. Though tools exist for end-to-end testing, the ideal solution remains elusive. Each tool we consider often requires us to build upon it with our custom solutions, which can vary in stability and efficiency.

In our search for the best testing tool, we primarily focused on two options that seemed most suitable for our needs: Appium and integration_test. Appium stands out as a separate framework that's not specific to Flutter, offering a wider range of programming languages for writing tests. On the other hand, integration_test is a part of the Flutter SDK, providing a more integrated approach specific to the Flutter app.

Appium vs Integration_test: A Comparison for Flutter Testing

It has the ability to access native and WebView elements

There's no way to access native and WebView elements

When writing tests, you will have to define locators for two platforms, Android and IOS

It's required to define only one locator for two platforms

The speed of creating tests is low, you need to use additional tools to search for locators

The speed of creating tests is high because we use widgets and have the ability to quickly find locators in the widget inspector and add new keys if necessary

To run tests locally, you need to install and run the Appium server

To run tests locally, you do not need to install anything else, it is enough to have an Simulator or Emulator running

Slow execution of tests because all actions are performed through an external driver. Execution time of 10 identical tests locally 1 minute 12 seconds.

Fast interaction with the application, all actions are performed directly with Flutter widgets. Execution time of 10 identical tests locally 5.6 seconds.

Example of a locator for tests with Appium:

@AndroidFindBy(xpath = "//android.widget.Button[@content-desc=\"Order your starter kit\"]")
@iOSXCUITFindBy(xpath = "//XCUIElementTypeButton[@name=\"Order your starter kit\"]")
private MobileElement orderStarterKitButton;

Example of a locator for tests with integration_test:

final body = find.byKey(const ValueKey('LoginBodyKey'));
final close = find.byType(CloseButton);

After careful evaluation, we decided to opt for the integration_test package over Appium for our Flutter application testing needs. A key factor in this decision was the native integration of integration_test within the Flutter SDK, offering a more streamlined and efficient workflow, especially in terms of locator discovery and overall test execution speed. In contrast, Appium, while powerful, demands a more complex infrastructure, often relying on paid tools like Perfecto for optimal functionality. Although integration_test has its limitations, notably in interacting with native dialogues and web pages, we found ways to work around these challenges. Importantly, integration_test presents fewer risks associated with framework updates, for instance, third-party testing frameworks like Appium faced significant disruptions with the release of Flutter 3.0. Drawn to the future. Built for evolution. Our choice resonated with Flutter's ever-changing world. Adaptability and compatibility became the guiding lights, leading us to the perfect solution.

Enhancing Flutter Testing with flutter_gherkin Package

We needed clear and detailed reports for our tests, including Gherkin steps and more. That's why we chose to use the flutter_gherkin package, version 3.0.0-rc.17. The regular integration_test package is great, but it didn’t give us the kind of reports we were looking for. The flutter_gherkin offers these reporting features, making it easier for us to see and understand our test results.

The flutter_gherkin package allows us to leverage all the benefits of the classic Cucumber framework, a favourite in the Java world. It enables us to use tags for organizing tests, create parameterized test sets (Scenario Outlines), work with data tables, and more. This makes it a powerful tool for writing and managing our Flutter tests, much like we would in a traditional Cucumber-based environment.

flutter_gherkin 3.0.0-rc.17 | Flutter package

A Gherkin / Cucumber parser and test runner for Dart and Flutter

Dart packages

Putting Theory into Practice: Real-World Examples of Our Flutter Testing Approach

Get ready to experience the power of real-world testing! We'll peel back the curtain on our unique approach, showcasing how we write clear, concise Gherkin steps and effortlessly transform them into robust Flutter test code. Buckle up for a complete test journey, from start to finish, revealing every detail with captivating test reports. These examples will help you understand how we use these testing methods every day and how they help us make better apps.

To start, I'd like to mention that we've made testing easier by creating a custom extension for Finder and List<Finder>. In this extension, we included access to the WidgetTester object, which updates for every test scenario. This provides us with powerful capabilities for interacting with screen elements, like searching for them, waiting for them to appear, tapping, scrolling, and more. We keep updating this extension by adding new features as our testing needs to evolve. It's a dynamic tool that makes our testing process both more efficient and effective.

extension FinderExtension on Finder {
  Future shouldBeVisible({
    Duration timeout = Timeouts.defaultPumpTimeout,
    bool doThrow = true,
    bool isClickable = false,
  }) async {...}

  Future shouldNotBeVisible({
    Duration timeout = Timeouts.defaultPumpTimeout,
    bool doThrow = true,
    bool isClickable = false,
  }) async {...}

  Future shouldBeVisibleAmount({
    int amount, {
    Duration timeout = Timeouts.defaultPumpTimeout,
    bool isClickable = false,
  }) async {...}
}

Implementing Steps

All our test scenarios are built on pre-implemented steps in which we use locators from previously described pages of our application. There, we place locators for interacting with and verifying elements on the app screens.

Example of test screen:

class PhoneInputScreen {
  final phoneField = find.byKey(const ValueKey('PhoneFieldKey'));
}

Example of test step:

when1(
  RegExp(r'I enter {string} into the phone number field$'),
  (phoneNumber, context) async {
    await phoneInputScreen.phoneField.enterText(phoneNumber);
  },
)

Example of using the step:

Feature: User onboarding

@phone-number-input-validation
Scenario: Validate phone number input field
  Given I am on the phone number input screen
  When I enter "123-456-7890" into the phone number field
  Then the alert "Phone number must start with 5" appears below the field

Generating tests for flutter_gherkin

When using the flutter_gherkin framework, we need to generate our steps before running tests. Under the hood, flutter_gherkin employs the well-known package to Flutter developers, build_runner, which generates the steps. For more details on how it operates, you can read about it on the package's page.

build_runner | Dart package

A build system for Dart code generation and modular compilation.

Dart packages

Running tests

When running tests, the setup depends on the platform. For mobile testing, we use either an Android emulator or an iOS simulator. For web tests, a configured browser driver is necessary. In our automated testing system, we focus on Android emulators and iOS simulators. This approach allows us to seamlessly integrate testing into our workflow without relying on external services.

Our testing infrastructure includes a set of Azure agents specifically for running our integration tests. These agents can initiate either an Android emulator or an iOS simulator, providing a controlled environment for our tests. This setup not only streamlines our testing process but also enables us to scale up as needed.

Our strategy primarily involves automated testing, but we also conduct manual checks for the most complex and critical cases. This combination ensures that our application functions flawlessly, with both broad coverage and attention to detail. Our blend of automated and manual testing achieves a balance of efficiency and thoroughness, for a testing process that is both comprehensive and reliable.

Generating Detailed Test Reports with cucumber-html-reporter Plugin

After the tests are executed, detailed information about the completed steps, their execution times, screenshots, and error messages will be saved in the designated test folder. This enables the use of the cucumber-html-reporter, a powerful JavaScript plugin, to generate comprehensive HTML reports. The plugin transforms the test results into a visually appealing and easily navigable format, ideal for reviewing detailed test outcomes. It supports various themes for report customization and integrates screenshots for a more illustrative display of test failures or errors. The generated reports, which can be opened in any web browser, provide a clear and structured overview of the test scenarios, making it easier to analyze and debug the test results effectively.

Problems and Solutions in Flutter Integration Testing

One of the notable challenges we faced in our Flutter integration testing was the inability to interact with native dialog windows. This limitation posed difficulties in scenarios such as permission dialogs for access resolution, notifications, or geolocation services. Handling these native elements is crucial for comprehensive testing but is not directly supported in Flutter's testing environment.

Solutions for Overcoming Native Interaction Limitations

To overcome this hurdle, we implemented a creative and effective strategy. Recognizing the importance of maintaining the integrity of our tests while addressing these native interactions, we developed the following solutions:

Mocking Location Services: For geolocation tests, we created specialized versions of our service interfaces interacting with the native part of the application. These test-specific implementations return pre-mocked locations, ensuring consistent and predictable testing outcomes without the need for actual geolocation permissions or services.

class IntegrationTestLocationProvider implements DeviceLocationProvider {
  @override
  Future getCurrentLocation() async {
    return const UserLocation(24.4539, 54.3773, 100.0);
  }
}

Permission Resolver Handling: Addressing the challenge of permission dialogs, such as those for camera or storage access, we implemented a custom solution within our testing framework. Our approach involves always returning a 'true' response for any permission request during tests. This method allows us to bypass the native permission dialogs, ensuring uninterrupted test flows while still validating the application's response to granted permissions.

class MockPermissionResolver implements CompanyPermissionResolver {
  @override
  Future getPermissionStatus(
    PermissionType permission,
  ) async {
    return CompanyPermissionStatus.granted;
  }

  @override
  Future openDeviceSettings() async => true;

  @override
  Future requestPermissions(
    PermissionType permission,
  ) async {
    return CompanyPermissionStatus.granted;
  }

  @override
  Future shouldRequestPermission(
    PermissionType permission,
  ) async {
    return false;
  }
}

Notification Testing Approach: Testing notification functionality poses a unique challenge in the Flutter environment, as direct verification within integration tests is currently not feasible. To tackle this, we adopted a two-pronged approach. Firstly, for automated tests, we disable notifications to prevent their interference with test executions. Secondly, and most importantly, we conduct manual testing specifically for notification features. This manual testing ensures thorough verification of notification functionalities, which is critical for user experience and application reliability.

Efficient Use of Mock Objects in Flutter Integration Testing

In our integration testing for Flutter apps, we use mock objects to imitate how services work with the app's native features. This helps us keep our tests consistent and reliable, as we avoid issues caused by external factors.

When we come across services that need to work with native parts of the app and are hard to test, we create mocks for them. These mocks act just like the real services, but we can control what they do during tests. This makes our tests more predictable and precise. If you're having trouble testing certain services in your Flutter apps, using mock objects can be a good solution. They can help you handle and solve testing challenges effectively.

Executing Tests in the Pipeline

Our full suite of automated tests is executed via scheduled nightly jobs. Our test engineers analyze the outcomes of these runs and create bug reports if any issues are detected. They also update the automated tests if any flaws are found in the test scripts themselves. Additionally, we've implemented the execution of our most stable tests during pull request validation. The tests to run are automatically determined based on the files modified in each pull request. This targeted approach allows us to specifically test the changes made, avoiding the need to run the entire suite of tests.

To generate reports in Azure, we utilize a specialized plugin:

This plugin is configured to receive the path to the raw report generated in our test folder after each test run. This process ensures that our test results are seamlessly integrated into Azure, providing a streamlined and efficient reporting workflow.

A typical report in our pipeline presents a comprehensive overview of the test results. It includes a summary of the test cases executed, highlighting the number of passed and failed tests. Detailed information about each test case, such as execution time, test steps, and any associated error messages or screenshots, is also provided. Additionally, there's a section for any anomalies or bugs identified during the test runs. This report is visually organized for easy interpretation, with graphs or charts to represent data where applicable, making it a valuable tool for quickly assessing the health and stability of our application.

Conclusion

In closing, our work with Flutter at W has been a big step forward. We've built a strong system for testing our apps that works well for both mobile and web. Choosing to use flutter_gherkin has made writing and understanding our tests much easier for everyone on the team.

The cucumber-html-reporter plugin has really changed things for us. It makes our test results easy to understand by giving us simple, detailed reports. Having our tests run on their own every night, and choosing specific tests based on what we change in the code, has made our testing quick and effective.

We also know that some tests are so important that they need a human touch, so we still do some checks by hand. This mix of automated and manual testing means we can be really sure that our apps work well.

We're proud of the testing system we've set up. It's helping us keep our apps in great shape today and it's ready to grow with us as we keep improving them in the future. We hope our experience can help others who are also looking to improve their testing on Flutter.

By Roman Morozov, Flutter Engineer