Build your Android app faster than ever by fanning out - and parallelizing - tests on Bitrise. An example workflow and the necessary documentation to make it work for you.
This blog post describes a legacy solution. To run builds in parallel, learn more about Build Pipelines.
Speed and quality can go hand in hand, but sometimes you need a bit of tinkering to get both. Try this Bitrise setup to split tests up, run them in parallel and speed up your Android builds.
Just want to try it out for yourself? Find the code and assets used to make this post here:
Android Test App
bitrise.yml
Fan Out Calculator (How many "fan outs" are likely optimal for your use case?)
Fan Out for iOS
Storing your gradle dependencies on Bitrise can make sure that you can access all the gradle deps as soon as possible. When using our build steps setting up caching for gradle dependencies only requires the Cache:Push and Cache:Pull steps to be added to your workflow and to set the cache level of the build step to deps_only.
When you are using static versions you can prevent gradle from checking for updates online during the builds. How this can be done in the build.gradle file:
Use classpath "com.google.gms:google-services:3.2.0" instead of classpath "com.google.gms:google-services:3.+
When you are building an Android application having all the resources available in the generated apk is simply just an overkill. As (in most cases) you would only be running your test using a single language and a single screen density you can leave out any other resources from the generated apk. To achieve this you can create a new flavor for your application that only includes the necessary resources, decreasing your compilation time and the apk size as well.
android {
productFlavors {
ui {
...
resConfigs ("en", "xxhdpi")
}
}
}
Legacy Multidex is used when the minSdkVersion is set to 20 or lower. Simply create another variant to only use for testing and only running for your tests.
android {
dev {
...
minSdkVersion 21
}
prod {...}
}
}
As development builds don't require multiple APKs this option can simply be disabled.
eg.:
if (project.hasProperty('devBuild')) {
// Prevent multi apk generation on development
splits.abi.enable = false
splits.density.enable = false
}
And in the Additional Gradle Arguments input of the step add -PdevBuild
In most cases you won’t need to store your crash logs in a different tool when running your UI tests as you would get the results in the test reports as well.
android {
buildTypes {
debug {
ext.enableCrashlytics = false
}
}
}
Whether we like it or not, tests fail! Sometimes they fail due to code changes, but sometimes they fail due to a previous step or case failing. This makes it much harder to identify which steps/cases are really failing and which are working as expected.
First step is to meaningfully increase your testing time (yes, you read that right, increase)
Going through features and user stories during your UI tests provides a perfect ground for parallelisation later on based on your feature sets and adding setup steps to each would make the tests less prone to errors in other test sets. It can make it much easier to understand your test sets if those are focused on a specific user journey and it makes it easy to run your test sets one by one.
Tests ran on a CI environment are using different network settings/connection and the devices have different performance as well. Running your tests on the devices with a high speed internet connection can produce smaller wait times and fast loading of different UI elements while testing on an older phone or on an emulator can slow down builds. Make sure to always test if the element that you are planning to interact with is visible and make sure you give enough time to the device to display the desired screen completely.
You can start parallel builds and wait for them to complete using the Bitrise Start Build & Bitrise Wait for Build steps. This allows you to run more simulators at once.
Every workflow and every script below can be found in the bitrise.yml we linked on the top of the article. Feel free to start experimenting with your own setup based on the provided sample! 😎
The ui_test_fan_out_ftl workflow triggers the fan out builds using the Bitrise Start Build & Bitrise Wait for Build steps.
In order to not have to rebuild the app multiple times, we can leverage the Cache Push & Cache Pull steps, or other steps like the S3 File Uploader step.
The test_x_ftl workflows triggered by the primary build will run in parallel on the apks created in the ui_test_fan_out_ftl build. Each of these workflows can target a different test runner class or test package. The script step in the workflow generates the zip archive that is later on exposed by the Deploy to Bitrise.io step and which is also available on the Apps & Artifacts tab of the build.
The script that we used:
set -euxo pipefail
echo "-- compress --"
cd "${BITRISE_DEPLOY_DIR}/"
ls
exec zip -qr "${BITRISE_DEPLOY_DIR}/TestResults1.zip" "$BITRISE_TEST_RESULT_DIR"
The ui_test workflow runs the same number of tests in serial for performance comparison. The script steps (Test 1, etc.) contain a call to adb to start up the instrumentation tests.
Using the Bitrise API we can get the build artifacts back into the primary build for processing. We can do this in a script step easily but for convenience we created a custom step to do it for you.
Custom Bitrise step to automatically retrieve all artifacts created by the fan out builds. (Work in progress)
Running your fan out builds require a specific amount of concurrencies. If other builds are taking up the concurrencies you can run into a deadlock where builds would fail waiting for all the started builds to finish while the fan out builds are stuck in the queue. A possible solution can be:
Although this is a trivial example with very few tests, it shows how you can apply fan out builds to your own workflows.
Here you can see that the tests happened one after another and the total build time was 17.1 mins in this simple example. Depending on how many tests you perform your results may vary.
Here you can see that the tests happened in parallel and the total build time was 10.5 mins in this simple example. Depending on how many tests you perform your results may vary.
Any sequential action that could be split up to run in parallel, can be split out over workflows to cut down on the time spent waiting for a build to finish tremendously. Have you tried the fan out setup and want to share your experiences?
Find us on Twitter, drop by our LinkedIN or Facebook pages, or simply start a topic on Reddit - We're always happy to jump in on the conversation.
Happy building! 🚀
We made some improvements to our Reach Native demo app and reduced build time by nearly 60%. Check out what optimizations we made and how we sped up the build time on our demo app.
We recently explored improvements to Bitrise’s demo apps. In particular, we modified the React Native demo app — not only to optimize our DevOps platform for the fastest possible feedback and performance — but also to show you just how fast Bitrise can be. The goal was to make the React Native demo app build as fast as possible on Bitrise so users can get fast feedback from their builds. Read on to find out how we reduced build time by nearly 60%!
Demo apps are sample projects — maintained by Bitrise Mobile Experts — which customers can use to try features of Bitrise on a sample codebase. A user may want to explore Bitrise before demonstrating its benefits to their team. So, we created demo apps so users can explore Bitrise immediately. We optimize demo app workflows (found in the bitrise.yml in each demo app repository) to build as fast as possible in order to showcase Bitrise's ability to provide developers with fast feedback.
If you adopted continuous integration practices into your development workflow, your CI/CD pipeline builds, tests, and deploys your app. Therefore, your mobile CI/CD pipeline directly affects the rate you’re able to deliver new updates to your customers.
Imagine pushing, switching to a new branch (both in git and mentally), and then getting a notification 40 minutes later about a broken build on the original branch. Now you’ve interrupted the original problem and your new workflow — what a mess!
These things really add up. More importantly, they cause developers to move with caution. Push and then… do nothing until we get a green build. That’s a lot of wasted time, and developers feel like they’re not productive. It hurts morale, which is even worse!
We should consider the benefits of Sustainable Flow. It used to be that it took 15 minutes to get back into a state of Flow. More recent studies show that the true time is closer to 25-30 minutes. Fast feedback means that we don’t break Flow. We address things immediately and can then move on to new workloads with confidence!
Bottom line: In order to maintain a high-quality product, fast feedback is important. Fast feedback helps you to quickly test changes made to your app while shipping new features and bug fixes faster.
Before making any changes, we took an initial measure of the time it took to build our React Native demo app on Bitrise. We can see that build time is especially important when developing software, but React Native builds are not known for their speed.
From the above breakdown for this build, we can see that the first build took nearly 14 minutes to run!
Some notable rows here are Run Cocoapods install and Xcode Build for Simulator, which are really inflating our build time.
The first thing we did when looking into what optimizations we could make was to consider what we gained by updating to the latest version of React Native. At the beginning of the project, we started with an older React Native version of 0.70.5. When considering where to get our information, we decided to review official sources first; so from the official React Native blog post on the release of version 0.71.0, we found that there are a few build speed optimizations made specifically surrounding the javascript engine which React Native makes use of: Hermes.
As Bitrise mobile experts, we scrutinize everything that goes on in a build backed with the knowledge of some of the most important parts of our build systems. This includes the dependency management phase of an app.
While investigating our project with fresh eyes we noticed a dependency in our Cocoapods called Flipper which the iOS build was spending a lot of time on. Flipper is a dependency included by default when creating a new React Native project using the react-native create ... command. While referring to the official React Native documentation, we came across this page about speeding up CI builds. From the documentation:
Flipper is a debugging tool shipped by default with React Native, to help developers debug and profile their React Native applications. However, Flipper is not required in CI: it is very unlikely that you or one of your colleague would have to debug the app built in the CI environment.
The first thing suggested on that page is to disable Flipper. We accomplished this by updating our Podfile to make use of the CI environment variable exposed by Bitrise. To achieve the same result, in your project you can change the following line in your Podfile from:
to
After exhausting our search of build optimizations from official sources, we took a look at how our CI pipeline was set up. When we started we had the following primary workflow in our Bitrise config:
...
trigger_map:
- pull_request_source_branch: "*"
workflow: primary
primary:
description: Runs JavaScript tests, and verifies iOS and Android builds
before_run:
- setup
after_run:
- test-js
- build-ios
- build-android
setup:
steps:
- git-clone@8: { }
- restore-npm-cache@1: { }
- npm@1:
inputs:
- command: install
- save-npm-cache@1: { }
test-js:
steps:
- npm@1:
inputs:
- command: test
build-ios:
description: Builds the iOS version of the app for use in Simulators
steps:
- react-native-bundle@1:
inputs:
- entry_file: index.js
- platform: ios
- restore-cocoapods-cache@1: { }
- cocoapods-install@2: { }
- save-cocoapods-cache@1: { }
- xcode-build-for-simulator@0:
inputs:
- project_path: ios/BitriseReactNativeSample.xcworkspace
- scheme: BitriseReactNativeSample
- simulator_device: iPhone 13
build-android:
description: Builds the Android version of the app for use in emulators
steps:
- react-native-bundle@1:
inputs:
- entry_file: index.js
- platform: android
- install-missing-android-tools@3:
inputs:
- gradlew_path: android/gradlew
- restore-gradle-cache@1: { }
- android-build@1:
inputs:
- module: app
- project_location: android
- variant: debug
- save-gradle-cache@1: { }
meta:
bitrise.io:
machine_type_id: g2-m1.8core
This is a pretty straightforward configuration. You can read this from top to bottom and things will be executed in that order. In order to kick things off via a pull request we specify the primary workflow as the one to be triggered. In this workflow:
Note: We also recommend you to make use of caching in your build to speed things up. We were already making use of caching in our builds from the beginning for this project so we decided not to cover it in this article, but you can learn more here.
One thing that jumped out at us right away is that there’s no pipelines section in our configuration. This means we're running each phase of our workflow sequentially. We realized that the iOS and Android apps could be compiled independently. We then realized that the tests could also be run independently from the two builds. To reduce build time even further, we leveraged our new feature — Build Pipelines — that allows us parallelize parts of our build.
We wanted to run our tests and create our iOS and Android apps in parallel. So, we added them to our test-and-build stage. Workflows in stages are executed in parallel. Next, we need to tell Bitrise when to run this stage. We'll create a primary pipeline to run this stage.
Here’s what our config file includes after rewriting it to include parallelization:
...
trigger_map:
- pull_request_source_branch: "*"
pipeline: primary
pipelines:
primary:
description: Runs JavaScript tests, and verifies iOS and Android builds
stages:
- test-and-build: {}
stages:
test-and-build:
abort_on_fail: true
workflows:
- test-js: {}
- build-ios: {}
- build-android: {}
workflows:
setup:
steps:
- git-clone@8: {}
- restore-npm-cache@1: {}
- npm@1:
inputs:
- command: install
- save-npm-cache@1: {}
test-js:
before_run:
- setup
steps:
- npm@1:
title: Run tests
inputs:
- command: test
build-ios:
before_run:
- setup
description: Builds the iOS version of the app for use in Simulators
steps:
- react-native-bundle@1:
inputs:
- entry_file: index.js
- platform: ios
- restore-cocoapods-cache@1: {}
- cocoapods-install@2: {}
- xcode-build-for-simulator@0:
inputs:
- project_path: ios/BitriseReactNativeSample.xcworkspace
- scheme: BitriseReactNativeSample
- simulator_device: iPhone 13
- configuration: Debug
- save-cocoapods-cache@1: {}
build-android:
before_run:
- setup
description: Builds the Android version of the app for use in emulators
steps:
- react-native-bundle@1:
inputs:
- entry_file: index.js
- platform: android
- restore-gradle-cache@1: {}
- android-build@1:
inputs:
- module: app
- project_location: android
- variant: debug
- save-gradle-cache@1: {}
meta:
bitrise.io:
machine_type_id: g2-m1.8core Here are the changes we made in a nutshell:
Another thing we noticed is this bit at the bottom of our config:
This means that we’re utilizing the same M1 8-core stack for all of our workflows.
Since we’re utilizing pipelines now it’s important to understand that each workflow in a stage executes in a separate environment. This enables us to specify a different stack per workflow meaning we can do things like building our Android app on a Linux stack, and our iOS app on an M1 stack. Unfortunately, this also means that we introduce some redundancy in running our setup steps for each workflow executed by our stage.
We decided to utilize a Linux stack for stability and performance while building our Android app. Our build-android workflow changed to:
build-android:
before_run:
- setup
description: Builds the Android version of the app for use in emulators
steps:
- react-native-bundle@1:
inputs:
- entry_file: index.js
- platform: android
- restore-gradle-cache@1: {}
- android-build@1:
inputs:
- module: app
- project_location: android
- variant: debug
- save-gradle-cache@1: {}
meta:
bitrise.io:
stack: linux-docker-android-20.04
machine_type_id: elite As we can see, we have a total build time of less than 6 minutes after our above changes!
That includes unit testing and building both iOS and Android apps.
We set out to reduce the build time of our React Native app as much as possible and succeeded through the use of parallelization and a bit of mobile expertise. In addition to being a handy tool to evaluate Bitrise, we also attempted to encode some of the best practices in our example. Users are encouraged to not only try them, but also steal the best bits and make their builds better, faster, and stronger!
Since the iOS build was the limiting factor in our build time let’s take a look at the summary for it after all our changes:
By disabling Flipper in our CI builds, we reduced the build time from 6.3 min in our original build to 2.1 min in the final version. Parallelization helped us quite a bit in speeding up our build, but our mobile experts knew about this one special trick to really get the most out of the CI!
https://arxiv.org/pdf/1805.05508.pdf
Developer Flow State and Its Impact on Productivity
React Native 0.71: TypeScript by Default, Flexbox Gap, and more... · React Native
Using Hermes · React Native
Speeding Up CI Builds · React Native
Dependencies and caching
Build Pipelines