The Librarian – A centralised server for easy build distribution

Written by Siddhartha Bokaria and Sunil Kumar, Software Engineers at MyGate

1. Introduction

The pain of creation, managing and distribution of application builds is a well known topic. As it occurs during any company’s journey, we have struggled finding the right rhythm of what we term as ‘build automation’. The core necessities remain relatively standard; the need for a developer to seamlessly distribute builds to validation engineers while operating within rigid virtual private cloud (VPC) zones.

In tightly knit independent application squads, we tend to take small steps toward harmonising an application (app) distribution strategy. Provided the product build and flavour pipelines are present, we start with the bare necessity of issuing a command that allows the package flow through proxy zones and get published for consumption.

We do not plan on reinventing the wheel but favour assembling the right solutions to manage our app distribution needs. As such, first in the list of agenda is to perform a basic survey of available options.

2. Literature survey

There are a host of projects dedicated to solve this problem. Some of the prominent ones are listed below with a brief overview of their capabilities.

Fastlane [1]
As their official phrase states, “app automation done right”. This includes a suite of pipeline centric features of extracting screenshots, beta distribution, app store deployment and code signing. Definitely what we should be setting up in the long term. However this will be an overwhelming solution for a basic app distribution need amongst squad members.

Firebase [2]
Previously part of Fabric’s Crashlytics, the upcoming Firebase app distribution which is now in beta shows the most promise and falls right within our ‘goldie lock zone’. It focuses on distributing development builds to a group of testers. It allows various approaches including console, CLI and gradle based scripts reducing friction to adopt. It promotes access to builds through a simplified portal and targeted emails. It provides a basic dashboard for central management and activity overview. It supports both iOS and Android builds. And, the entire approach falls under free tier at the moment. Adding GCP to our ecosystem is a task by itself and once allowed this is a frontrunner and a panacea to our problems.

AWS mobile hub [3]
A collection of products rather than a targeted solution. It is a feasible option and would require you to align application building activities from the beginning to account for the chosen framework. A relatively steep learning curve might be a determining factor. For an ongoing set of projects such as ours, this would require additional push for a consideration.

Apple testflight and Android test tracks [4]
This is a good platform for sharing early stage releases. It is predominantly used during beta testing. You would expect a certain level of stabilisation prior to using this facility. The portals often carry download volume quotas for your app listing. We need a simpler alternative to this.

3. Our approach

Everything said and done, we were yet to figure out the right approach that fit into our workflow. Our current goal is to reduce repetitive and time consuming activities of sharing builds with the validation team. Just when velocity of iterations were making the task of sharing builds harder, we came across a simple do it yourself (DIY) app distribution platform called Librarian [1]. This looked like a good place to start with a simple command based submission interface and light UX. Though getting this to a central location was tricky whilst navigating through layers of aforementioned VPC zones, it allowed us the flexibility of carving out our own build flow. That led to the following deliberate steps to set up our own distribution pipeline targeted at an AWS environment.

3.1 Key modules

  • A shell script to allow preparation of the environment. An Android engineer would ideally not want to be distracted with environmental setup required to support background operations. The script helps with installation of python package manager and AWS CLI. It also triggers configuration of AWS environments which means the developer should keep the access tokens ready.
  • Gradle file extension that exposes the task to the developer. A simple configuration at project level file helps identify remote resources. The app level changes allow you to link tasks to the gradle list. Now you can chain the ‘assemblePackage’ and ‘publishRelease’ tasks for seamless build distribution.
  • The final module is an example implementation of a bridge API. Librarian does not support remote submission of build currently and the accompanying NodeJS based API helps activate that feature.

3.2 Quick start (Android)

With all the prerequisites satisfied [5], set up your build environment by following these simple steps.

  1. Place the 2 files (s3Upload.gradle and in the app root where the build.gradle (for app) is present. Synchronise project and add files.
  2. Edit build.gradle (app level), put this line at EOF: apply from: ‘s3Upload.gradle’. Synchronise file.
  3. Edit build.gradle (project level), put following lines below allprojects: repositories section of the file:
    • project.ext.AWS_S3_BUCKET = ‘s3-bucket-name’
    • project.ext.BRIDGE_HOST_PATH = ‘http://localhost:3000/api/librarian’
    • Synchronise file.
  4. Refresh gradle command list.
    • The newly available commands are ready for use.
    • ./gradlew tasks –all
    • You should be able to locate the publish tasks under ‘other tasks’ group labelled as ‘app:publish-variant-ToS3’
    • ./gradlew <publish-task-name>
  5. For 1st time users the utility will setup the necessary packages (pip, aws cli installation, aws environment configuration).

3.3 Illustration

And, here is the workflow illustration to assist the speed readers.

4. Roadmap

This approach is working for us and has reduced the hassle of dealing with unorganised sharing channels. Through this post, we plan to share it with other teams and seek inputs. Some may be using alternatives easier to adopt approaches that might interest us. Others may embrace it and contribute to its development. In either case, the following items have been planned.

  • Extend approach to support iOS builds.
  • Extend support for other cloud storage options.
  • Investigate alignment to Firebase app distribution channel. The preparation script should now ideally target Firebase CLI and gradle script can be replaced by the public plugin.

Head over to our Github repository [5] and try it out yourself!


[1] App automation done right
[2] Firebase App Distribution
[3] AWS Mobile Hub – Build, Test, and Monitor Mobile Applications
[4] Apple testflight and Android test tracks
[5] MyGate app build distribution utility
[6] Librarian

What do you think?