How we created our tool for delivering Android application builds
Hi all! My name is Danil Kiselev, I am an Android developer in AGIMA. In this article I will tell you how we implemented our own tool for delivering assemblies of Android applications. The purpose of the article is to save time for teams that are developing Android projects and that do not yet have a similar solution. I have also attached a repository with the project code to the article. You can use it as a starter version and modify it to suit your needs.
With the official Firebase App Distribution service blocked in Russia, our mobile development department was faced with the task of creating our own tool for delivering builds of Android applications to testers. Yes, we could use Firebase App Distribution using a VPN, but this is not very convenient. VPN services are unreliable because they are subject to blocking. In addition, creating your own solution allows you to add new features and adapt the tool to the specific needs of the team.
On many projects, it took a huge amount of time to get a build from GitLab or build it manually and send it to a tester. It also greatly distracts from working on tasks. The solution was to develop a tool that would accept, store and distribute assembly files.
The main requirements for the new tool were:
Receive build files from GitLab, save them and automatically distribute them to project participants.
Provide access to a list of assemblies and information about those assemblies.
What is important is to provide the ability to quickly integrate with various projects and work independently with them. Our company is a custom development studio and simultaneously works on many projects.
Choosing Ktor
Ktor was not chosen by chance: it is a framework based on Kotlin, and since we decided to develop the tool using the Android department, the choice was obvious. Thanks to this, any developer familiar with Kotlin will be able to quickly understand the code base, maintain the tool, and integrate it with various projects.
Telegram selection
We decided to choose Telegram as the interface. Our work chats are most often located there, so you don't have to go far for assemblies. In addition, the Telegram Bot API provides many features, although it has some limitations. Telegram made it possible to implement intuitive interaction with the tool using inline buttons. Thanks to this, it was possible to achieve the effect of transitioning between pages.
Build distribution process
Sending a file
First, let's implement sending the assembly from GitLab. In the gitlab.yaml file, you need to create an upload qa build stage, which will be launched after the build stage, find the assembly and send it to the desired address.
In the before_script block we need to install curl using the apk package manager.
In the script block, using the fastlane release_notes command, we create a changelog.txt file that will contain a list of the latest commits. Next, we need to find the paths to the files that we will send – changelog.txt and the assembly file. And using the previously installed curl, we send the files and project name to our server:
upload qa build:
stage: upload
needs:
- job: produce qa build
artifacts: true
variables:
FLAVOR: debug
only:
- develop
- /^release.*$/
tags:
- android
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
script:
- fastlane release_notes
- changelog_path=$(find . -name "changelog.txt")
- apk_path=$(find app/build/outputs/apk/gorzdrav/$FLAVOR -name "*.apk")
- curl -F file=@$apk_path -F file=@$changelog_path -F "appName=$APP_NAME" http://{Сюда нужно вставить url вашего сервера}/uploadBuildReceiving a file on the server
Now you need to define a request that will expect the build file, the name of the project, and a changelog.txt file with a list of the latest commits. This request can be modified to accept additional information. The number of files that a request can accept is no longer limited – all files that come from GitLab will be saved along with the build:
fun Route.uploadBuildModule() {
val controller by inject<UploadController>()
post("/uploadBuild") {
val multipart = call.receiveMultipart()
val parts = multipart.readAllParts()
val parameters = parts.filterIsInstance<PartData.FormItem>()
val files = parts.filterIsInstance<PartData.FileItem>()
val appName = parameters.firstOrNull { it.name == APP_NAME_KEY }?.value
controller.saveFiles(files, appName)
}
}File storage
We have received the files, now we need to save them. The saveFiles method processes files coming from GitLab:
override suspend fun saveFiles(
files: List<PartData.FileItem>,
appName: String?
) {
val dateTime = getCurrentDateTime()
val dateTimeString = dateTime.toDataString(pattern = DATE_TIME_FORMAT)
val dateTimeStringDisplay =
dateTime.toDataString(pattern = DATE_TIME_FORMAT_FOR_DISPLAY)
val appDirPath = "$ROOT_PACKAGE/$appName/$dateTimeString"
files.forEach { filePart ->
val fileName = filePart.originalFileName
val file = File("$appDirPath/$fileName")
val freeSpace = getFreeSpace().bytesToMb()
try {
file.saveFileFromPart(filePart)
} catch (e: IOException) {
e.printStackTrace()
if (freeSpace < MIN_FREE_SPACE) {
telegramBot
.sendTextToTestUser(MEMORY_AUTO_CLEARING)
val report = buildStore.clearOldBuilds(appName)
telegramBot
.sendTextToTestUser("Отчет: $report")
telegramBot
.sendTextToTestUser(RESENDING)
}
} finally {
file.saveFileFromPart(filePart)
}
}
}For each project, a separate directory is created with a name that we received from GitLab, and in it there are subdirectories with our assemblies. The subdirectory name corresponds to the time the assembly was created. Inside each subdirectory is the build file itself and the changelog.txt file. All other files that came from GitLab will be saved here. Perhaps a better solution would be to store this structure in the database, but for now it is implemented like this:
Also, if there is not enough memory to save the file, the method will be called clearOldBuilds and will remove some of the old assemblies.
Distributing builds to users
After successfully saving the files, you need to send them to users. To do this, we sent the project name in the request. Using this name we can get a list of users who are subscribed to the project. All we need to know from the user is his chatId – we can use it to send a message. Next, we go through the list of these ids and send everyone a notification about the new build.
Here we are faced with a limitation of the Telegram API – the maximum file size that we can send is 50 MB. Therefore, we decided to send a link to download the file from our server and a link to view changelog.txt.
val apkFile = File(appDirPath).findFileByExtensions(EXTENSIONS_LIST)
if (apkFile != null) {
val textMessage = configureBuildMessage(
title = NEW_BUILD,
time = dateTimeStringDisplay,
appName = appName,
apkDir = dateTimeString,
apkName = apkFile.name,
)
if (appName != null) telegramBot.sendTextToUsersFromProject(textMessage, appName)
}The sendTextToUsersFromProject method gets a list of users from the specified project and sends them a message like this:
Using these links, the user can download the assembly and view the changelog.txt file with a list of commits.
Request to download the file:
fun Route.downloadFileModule() {
val controller by inject<DownloadFileController>()
get("/$APKS_PATH/{$APP_NAME_KEY}/{$DATE_TIME_KEY}/{$FILE_NAME_KEY}") {
val appName = call.parameters[APP_NAME_KEY]
val dateTime = call.parameters[DATE_TIME_KEY]
val fileName = call.parameters[FILE_NAME_KEY]
val file = controller.getFile(
appName = appName,
dateTime = dateTime,
fileName = fileName
)
call.response.header(
"Content-Disposition",
"attachment; filename=\"${file.name}\""
)
call.respondFile(file)
}
}Request to show a file with commits to the user:
fun Route.showTextFileModule() {
val controller by inject<ShowTextFileController>()
get("/$SHOW_TEXT_PATH/{$APP_NAME_KEY}/{$DATE_TIME_KEY}/{$FILE_NAME_KEY}") {
val appName = call.parameters[APP_NAME_KEY]
val dateTime = call.parameters[DATE_TIME_KEY]
val fileName = call.parameters[FILE_NAME_KEY]
val file = controller.getFile(
appName = appName,
dateTime = dateTime,
fileName = fileName
)
val text = file.readText()
if (text.isEmpty()) {
call.respondText("Нет информации", ContentType.Text.Plain)
} else {
call.respondText(text, ContentType.Text.Plain)
}
}
}Interface and interaction with the bot
Main section
At the very beginning, the user is greeted with a list of projects to which he has subscribed. Telegram has no restrictions on the number of buttons in a message, so there will be no problems with displaying a large number of projects. Here you can get instructions for integrating with the bot, register or subscribe to the project.
Project registration
To register a project, you need to enter its name. This is the name we will pass in the request from GitLab. You also need to enter your name for this project. Additionally, you can enter a list of optional fields that are needed to be able to launch pipelines directly from the bot: link to GitLab, trigger token, project ID. After registration, the bot will return us a token, with which other users can join the project.
Subscription to the project
After creating a project, you need to provide an access token for colleagues so that they can subscribe to the project. They will enter the appropriate command with the token and their name for this project, and after subscribing, it will be displayed on the main page.
Project management
When you navigate to a project, a set of actions opens. Here we can run the project build, get the latest build or the last 10 builds. Also, depending on the user's status, additional options will be displayed in the project. For example, buttons with the ability to delete old assemblies, delete a project, and a list of users will only appear for the administrator.
When going to the list of users, the administrator has the opportunity to view information about them, change their status in the project, or remove them from the project.
Interaction with Telegram API
To interact with the Telegram API we used this library: https://github.com/InsanusMokrassar/ktgbotapi. Its advantage is that it uses Kotlin Coroutines.
This is what the processing of a simple command that the user enters in the bot looks like:
command(HELP_COMMAND) {
bot.sendMessage(
chatId = it.chat.id,
text = CHOOSE_PROJECT_MESSAGE,
replyMarkup =
getProjectsListButtons(
ChatToProjectTable.getProjectsForChat(it.chat.id.chatId)
)
)
}And this is what the implementation of the buttons looks like. A button is part of a message, and it has callback data with which we can process clicks:
fun getProjectDeleteConfirmButtons(projectName: String): InlineKeyboardMarkup {
return InlineKeyboardMarkup(
keyboard = matrix {
row {
+CallbackDataInlineKeyboardButton(
text = DELETE_PROJECT_CONFIRM.getDisplayText(),
callbackData = "$DELETE_PROJECT_CONFIRM:$projectName"
)
+CallbackDataInlineKeyboardButton(
text = DELETE_PROJECT_CANCEL.getDisplayText(),
callbackData = "$DELETE_PROJECT_CANCEL:$projectName"
)
}
}
)
}Next we can process the callbackData:
onDataCallbackQuery { callback: DataCallbackQuery ->
val callbackCommand = callback.data
when (callbackCommand) {
DELETE_PROJECT_CONFIRM -> {
}
}
}This is what navigation in the bot using inline buttons looks like in the end:
You can read more about the functionality of the library in the documentation: https://docs.inmo.dev/tgbotapi/index.html.
Launching builds directly from the bot
We have also implemented functionality that allows testers to independently launch builds without access to GitLab. To create and launch a pipeline at the click of a button, you need to execute a similar Post request in Gitlab from our server:
“https://gitlab.example.com/api/v4/projects/<project_id>/trigger/pipeline”To do this, we must know the link to our Gitlab and project ID. We need to pass two parameters to the request: the name of the branch on which we want to launch the pipeline, and the Trigger Token that we generated for the project.
Instructions for obtaining Trigger Token in GitLab
Trigger Token allows you to launch a CI/CD pipeline via the API. Below are step-by-step instructions on how to get it for your project in GitLab.
Open the project for which you want to create a trigger. In the menu on the left, select “Settings” and then “CI/CD”.
Scroll down to the “Pipeline Triggers” section and click “Expand”. Click the “Add Trigger” button.
Fill in the description field. Click “Add Trigger” to create.
After creating a trigger, you will see it in the list. The trigger token will be displayed there. This token can be used to run a pipeline via the API.
We can provide all this information when registering the project.
And here is what an example request looks like in Kotlin code:
class GitLabApiImpl : GitLabApi {
override suspend fun triggerPipeline(
projectId: Int,
refName: String,
triggerToken: String,
gitLabBaseUrl: String
): TriggerResult {
val url = "$gitLabBaseUrl/api/v4/projects/$projectId/trigger/pipeline"
val client = HttpClient(CIO)
return try {
val response = client.post(url) {
parameter(TOKEN_KEY, triggerToken)
parameter(REF_KEY, refName)
}
val status = response.status
TriggerResult(
success = status.isSuccess(),
message = "${status.value} ${status.description}"
)
} catch (e: Exception) {
println("Error: ${e.message}")
TriggerResult(
success = false,
message = "Упс... Произошла неизвестная ошибка \uD83D\uDE31"
)
}
}
companion object {
private const val TOKEN_KEY = "token"
private const val REF_KEY = "ref"
}
}We also need to make some changes to the gitlab.yaml file. Here you need to add a rule to the build stage.
– if: '$CI_PIPELINE_SOURCE == “trigger”' – this means that the stage will start if the pipeline was created using a trigger.
rules:
- if: '$CI_PIPELINE_SOURCE == "trigger"'
when: always
allow_failure: true
- if: '$CI_COMMIT_REF_NAME == "develop"'
when: manual
allow_failure: trueYou can learn more about Pipeline Triggers in the official documentation: https://docs.gitlab.com/ee/ci/triggers.
Also, when implementing such functionality, it is important to take into account that if access to GitLab is provided only through a corporate VPN, then this VPN must be running to complete the request on Ubuntu.
How can you use it
You can download the source code of the project, make the necessary settings and deploy a server to start using the tool and modify it to suit your needs!
You will need:
Enter your Telegram bot's API key
Prepare a server based on Ubuntu and configure it.
Deploy the server by specifying its host, username and password.
The first point is the easiest – you need to register the bot and get an API key. Instructions: https://handbook.tmat.me/ru/dev/botfather.
Then just go to the file AppConstants and substitute your API key into the constant TG_API_KEY.
For the next two points I have prepared more detailed instructions. I'll break it down below.
Instructions for deploying a server
To deploy the tool, you will need an Ubuntu-based server. Setting up Ubuntu to work with Ktor can seem daunting, especially if you don't have much experience with Linux. But, following these instructions, I was able to successfully configure the server:
https://gist.github.com/philipplackner/bbb3581502b77edfd2b71b7e3f7b18bd.
The server host needs to be substituted for 11.111.11.1 throughout the project. Use the search and replace it with your value.
You also need to install and configure PostgreSQL on Ubuntu. Here are the instructions you can use:
After this, you need to update the DbConstants file with information about your database. To see information about the database, use the command \conninfo.
Example:
object DbConstants {
const val DB_USERNAME = "{Имя пользователя бд}" // Пример postgresql
const val DB_PASSWORD = "{Пароль от вашей юд}" // Пример password
const val DB_URI =
"{Url базы данных}" // Пример: jdbc:postgresql://localhost:5432/builddistributiondb
}After setting up Ubuntu, you need to add a deploy task to build.gradle. This task connects to the server via SSH, sends the executable file build-distribution.jar to it and starts the service that executes this file. The service is created at the Ubuntu setup stage according to the instructions above. The connection password is read from the keys/password file in the project root. Paste your server password into this file.
I took the instructions and script from this guide: https://youtu.be/sKCCwl5lNBk. Server setup starts at 1:33 minutes.
task("deploy") {
dependsOn("clean", "shadowJar")
ant.withGroovyBuilder {
doLast {
val knownHosts = File.createTempFile("knownhosts", "txt")
val user = "{Имя пользователя}"
val host = "{Хост вашего сервера}"
val password = File("keys/password").readText() // файл с паролем к серверу
val jarFileName = "com.kiselev.build-distribution-all.jar"
try {
"scp"(
"file" to file("build/libs/$jarFileName"),
"todir" to "$user@$host:/root/build-distribution",
"password" to password,
"trust" to true,
"knownhosts" to knownHosts
)
"ssh"(
"host" to host,
"username" to user,
"password" to password,
"trust" to true,
"knownhosts" to knownHosts,
"command" to "mv /root/build-distribution/$jarFileName /root/build-distribution/build-distribution.jar"
)
"ssh"(
"host" to host,
"username" to user,
"password" to password,
"trust" to true,
"knownhosts" to knownHosts,
"command" to "systemctl stop build-distribution"
)
"ssh"(
"host" to host,
"username" to user,
"password" to password,
"trust" to true,
"knownhosts" to knownHosts,
"command" to "systemctl start build-distribution"
)
} finally {
knownHosts.delete()
}
}
}
}Conclusion
Creating your own tool for delivering Android app builds turned out to be a great solution after Firebase App Distribution was blocked. Now we don't spend a lot of time manually sending builds to testers. The tool can be easily and quickly integrated with projects with minimal dependence on third-party services such as VPN services.
Using Telegram made everything as convenient and fast as possible. Telegram allowed us to create an intuitive way to interact with the server. Integration with GitLab and the ability to launch builds directly from the bot saved us a lot of time and effort.
As I said, our tool easily integrates with a variety of projects, which is especially important for custom development studios working with different clients. I'm sure we're not the first to come to this decision, but the goal of this article is to help other teams, especially small ones, make the process of delivering builds more efficient, convenient, and faster.
In conclusion, I would like to remind you that I have attached the source code of this tool to the article so that anyone can use it and adapt it to their needs:
https://github.com/kiselyv77/com.kiselev.build-distribution.
It will be great if some developer or company improves our tool, adds new cool features and also makes it publicly available.
Thanks for reading! I hope this article will help someone improve and speed up the development and testing processes.
PS Write questions in the comments – I will definitely answer. And leave a review about our tool if you try it on your team.
I also invite you to Sasha Vorozhishchev's channel, Head of Mobile Development Department at AGIMA. You will find many useful things there.
