Implementing Machine Learning on the Server with Swift
In this tutorial, I'll show you how to work with a machine learning model on the Vapor server using Swift.
It’s no secret that Apple is looking to move its ecosystem toward maximizing value by bringing powerful machine learning processes to users’ devices. Core ML offers lightning-fast performance and simplifies the integration of machine learning models into apps, from creation to training and deployment. To better understand the intricacies of the Core ML framework, I recommend you check out with this guide.
However, in this endless progress of AI, new technologies related to generative algorithms complicate the situation somewhat: often these models are quite heavy and require significant resources to run on the device.
Improvements in machine learning models have become part of the daily news we see on our social media feeds. Apple has been the subject of rumors about possible developments and investments in artificial intelligence. And we find ourselves in a situation where the ability to understand models is becoming an increasingly important skill.
With all that said, if you've always wondered how to optimize your computing resources and implement a decent level of decoupling to get a more reliable product, then this article is for you.
In this tutorial, you'll learn how to host your machine learning models and support in a server-side project using Vapor and get immediate results in your SwiftUI client app without overwhelming your handheld device's memory.
Before we begin
To follow this tutorial, you will need a good understanding of Swift, Core ML, Vision, Vapor, and some knowledge of terminal commands.
Briefly about Vapor: is a web framework that allows you to write web applications, Rest APIs and HTTP servers in Swift.
In particular, it is designed taking into account three principles:
It uses the Swift language and all its benefits.
It is built on the basis of SwiftNIO with an eye on non-blocking event-driven architecture.
It positions itself as an expressive, protocol-oriented and type-safe framework.
Additionally, another incredible benefit of using Vapor as a backend is the community that has gathered around this project. It also includes over a hundred official and community-supported Swift packages that you can use to create an experience that suits your needs.
If you want to learn more about Vapor or better understand its architecture, I recommend you read officialwhich is quite comprehensive and contains several practical examples documentation.
Now let's move on to the tutorial itself.
Step 1: Install Vapor and create a project
To install and use the latest version of Vapor on macOS, you will need:
Swift 5.6 or later
Installed Homebrewto install Vapor and set up a server project
If you already have Homebrew installed, open the application Terminal and enter the following command, which will install Vapor:
$ brew install vaporTo verify that Vapor was installed correctly, try running the help command.
$ vapor --helpOnce all the required dependencies are installed, we can create the project using our Mac as a local host. To do this, navigate to the folder where you want to create the project (in this tutorial, we are creating the project on the desktop) and initialize the project in the Vapor CLI with the following command:
$ vapor new <server-project-name> -nFor simplicity's sake, we've chosen to call our server project “server” and will refer to it that way throughout this guide, but you can call it whatever you like.
If you want a blank base template that is fine for our purposes, you can specify the -n flag, which will automatically select “no” for all questions asked during setup. If you omit this flag, the Vapor CLI will ask you if you want to install Fluent (ORM) and Leaf (templating). If you still need any of this in your project, then it is better not to use the flag -nand specifically indicate whether you need this plugin or not, respectively, using flags --fluent/--no-fluent or --leaf/--no-leaf.
Let's get started by opening the project in Xcode. Navigate to the folder you just created and open the file Package.swift:
$ cd <server-project-name>
$ open Package.swiftNow it's time to set up the project and add the ML model.
Step 2: Setting up the project in Xcode and preparing the ML model
To get started, you'll need a Core ML model. In this tutorial, we'll use an image classifier MobileNetV2which is a pre-trained model available at Core ML machine learning models pageHowever, feel free to experiment with any other designs that catch your eye.
Let's start by creating a new folder “MLModel” in the root of the server package and place the file in it MobileNetV2.mlmodel. Next, create a new folder “Resources” in /Sources/App/.
Since we need to compile the model and create its Swift class, we navigate to the directory using the terminal /server (where is the file located Package.swift) and enter the following command:
$ cd MLModel && \\
xcrun coremlcompiler compile MobileNetV2.mlmodel ../Sources/App/Resources && \\
xcrun coremlcompiler generate MobileNetV2.mlmodel ../Sources/App/Resources --language SwiftThe compilation is complete and you should see a new Swift class and compiled model files in the Resources folder.
We can't reference the compiled ML model resource yet, because we need to add it to the package as an executable file. So make sure the file Package.swift includes it as follows:
.executableTarget(
name: "App",
dependencies: [
.product(name: "Vapor", package: "vapor"),
],
resources: [
.copy("Resources/MobileNetV2.mlmodelc"),
]
)The final step in setting up this server project is to make it discoverable and accessible on the network (for now, that will be localhost) by editing the application schema. Access the schema editor by following the path Product→Scheme→Edit Scheme…→Run→Arguments→Arguments Passed On Launchand add it there serve --hostname 0.0.0.0 as follows:
To immediately get rid of a number of errors, switch the project's Run Destination to “My Mac”.
Now the project is set up and we can start implementing the Rest API on our server to handle machine learning tasks.
Step 3: Create classification tasks and query flow for the ML model
So how do we structure machine learning tasks that will be called via the Rest API from a client application?
In this step we will define the structure MobileNetV2Managerwhich will manage the execution of machine learning tasks, in a new file that will need to be added to the folder /Sources/App/.
We will also define the structure ClassificationResult to store the results of image classification, since we need to transform the results to pass them to the client application in JSON format: as a result, such classifiers usually generate a label and a level of accuracy (denoted as “confidence”).
In addition, we need to add a function that will classify the uploaded image and determine the types of errors that may occur when uploading it. As a result, we will get something that will look like this:
import Vapor
import CoreImage
import Vision
struct MobileNetV2Manager {
enum MLError: Error {
case modelNotFound
case noResults
}
func classify(image: CIImage) throws -> [ClassificationResult] {
// Создание инстанса модели MobileNetV2
let url = Bundle.module.url(forResource: "MobileNetV2", withExtension: "mlmodelc")!
guard let model = try? VNCoreMLModel(for: MobileNetV2(contentsOf: url, configuration: MLModelConfiguration()).model) else {
throw MLError.modelNotFound
}
// Создание запроса с изображением для анализа
let request = VNCoreMLRequest(model: model)
// Создание обработчика, обрабатывающего этот запрос
let handler = VNImageRequestHandler(ciImage: image)
try? handler.perform([request])
guard let results = request.results as? [VNClassificationObservation] else {
throw MLError.noResults
}
// Преобразование результатов для возврата [ClassificationResult]
let classificationResults = results.map { result in
ClassificationResult(label: result.identifier, confidence: result.confidence)
}
return classificationResults
}
}
struct ClassificationResult: Encodable, Content {
var label: String
var confidence: Float
}
In order to determine the method of requesting the server, we needed to go to the file routes.swift.
Here we define the structure RequestContent to store the downloaded content that we will receive via URL requests from the client.
struct RequestContent: Content {
var file: File
}We determine the route app.post("mobilenetv2")through which the client application will load images to be classified. In this route, we transform the content sent to us and create a CoreImage instance from it.
And finally we pass the image to mobileNetV2.classify(image:). This is what the file should look like routes.swift in the end:
import Vapor
import CoreImage
func routes(_ app: Application) throws {
app.post("mobilenetv2") { req -> [ClassificationResult] in
// Преобразование содержимого запроса, которое было загружено
let requestContent = try req.content.decode(RequestContent.self)
let fileData = requestContent.file.data
// Получение данных из файла
guard let imageData = fileData.getData(at: fileData.readerIndex, length: fileData.readableBytes),
let ciImage = CIImage(data: imageData) else {
throw DataFormatError.wrongDataFormat
}
// Создание инстанса MobileNetV2Manager
let mobileNetV2 = MobileNetV2Manager()
// Здесь происходит классификация
do {
return try mobileNetV2.classify(image: ciImage)
} catch {
print(error.localizedDescription)
return []
}
}
}
enum DataFormatError: Error {
case wrongDataFormat
}
struct RequestContent: Content {
var file: File
}It remains to increase the maximum size of uploaded content to support heavier images. This can be done in the file configure.swift using the following line of code:
public func configure(_ app: Application) async throws {
app.routes.defaultMaxBodySize = "20mb"
try routes(app)
}That's it! Our server project is ready! We have an amazing server running the MobileNetV2 model, classifying uploaded images.
All that's left is to create a client application that will run these machine learning tasks via URL requests. We're almost there!
Step 4: Create a client application that can query your backend
Requesting and retrieving this information in an iOS app is the final step of this tutorial, and it’s the most intuitive part if you’ve worked with API calls before. All we need to do is upload the image we want to classify to the server’s IP address using the route mobilenetv2which we defined above, and get ready to transform the results of the image classification task that will be sent back to the client.
Let's create a new Xcode project and select iOS as the target.
Since we need to load the image into the request body in the correct format, we need to create a method that converts it to multipart/form-data Content-Type.
Create a new Swift file Extensions.swift and add an extension for the Data type to it. In this extension, create a method that will be responsible for adding rows to the Data object.
// Функция для добавления данных в тело multipart/form-data URL-запросов
extension Data {
mutating func append(_ string: String) {
if let data = string.data(using: .utf8) {
append(data)
}
}
}Create a new Swift file called APIManager.swift.
Then create a class APIManagera single shared instance of which will be responsible for interacting with the server via URL requests. Define a method to convert image data into a multipart/form-data body as follows:
import Foundation
import UIKit
class APIManager {
static let shared = APIManager() // Общий инстанс
private init() {}
// Создание тело multipart/form-data с данными изображения
private func createMultipartFormDataBody(imageData: Data, boundary: String, fileName: String) -> Data {
var body = Data()
// Добавление данных изображения к первичным данным http-запроса
body.append("\\r\\n--\\(boundary)\\r\\n")
body.append("Content-Disposition: form-data; name=\\"file\\"; filename=\\"\\(fileName)\\"\\r\\n")
body.append("Content-Type: image/jpeg\\r\\n\\r\\n")
body.append(imageData)
body.append("\\r\\n")
// Добавление закрывающей границы
body.append("\\r\\n--\\(boundary)--\\r\\n")
return body
}
}Next, inside the APIManager class, define the structure ClassificationResultwhich will help transform the classification results coming from our server as follows:
import Foundation
import UIKit
class APIManager {
static let shared = APIManager() // Общий инстанс
private init() {}
// Создание тела multipart/form-data с данными изображения
private func createMultipartFormDataBody(imageData: Data, boundary: String, fileName: String) -> Data {
...
}
// Структура для преобразования результатов с сервера
struct ClassificationResult: Identifiable, Decodable, Equatable {
let id: UUID = UUID()
var label: String
var confidence: Float
private enum CodingKeys: String, CodingKey {
case label
case confidence
}
init(from decoder: Decoder) throws {
let container = try decoder.container(keyedBy: CodingKeys.self)
let label = try container.decodeIfPresent(String.self, forKey: .label)
self.label = label ?? "default"
let confidence = try container.decodeIfPresent(Float.self, forKey: .confidence)
self.confidence = confidence ?? 0
}
}
}Finally, we need to define a method that SwiftUI View will call to request the classification of the loaded images.
We used http://localhost:8080/mobilenetv2 as the URL for our server service since we were only deploying it to the MacBook we are currently working on.
In this tutorial we test this mechanism on our MacBook acting as a localhost, but remember to change the IP address when you run the server application on a real host.
class APIManager {
static let shared = APIManager() // Общий инстанс
private init() {}
func classifyImage(_ image: UIImage) async throws -> [ClassificationResult] {
// Это URL IP-адреса вашего хоста: в данный момент, чтобы не усложнять пример, мы используем localhost, но не забудьте изменить его на соответствующий IP-адрес хоста, когда будете запускать приложение на реальном хосте
guard let requestURL = URL(string: "<http://localhost:8080/mobilenetv2>") else {
throw URLError(.badURL)
}
// Преобразование изображения в JPEG со значением compressionQuality, равным 1 (0 - наилучшее качество)
guard let imageData = image.jpegData(compressionQuality: 1) else {
throw URLError(.unknown)
}
// Граничная строка с UUID для загрузки изображения в URLRequest
let boundary = "Boundary-\\(UUID().uuidString)"
// Инстанс POST URLRequest
var request = URLRequest(url: requestURL)
request.httpMethod = "POST"
request.setValue("multipart/form-data; boundary=\\(boundary)", forHTTPHeaderField: "Content-Type")
// Создание тела multipart/form-data
let body = createMultipartFormDataBody(imageData: imageData, boundary: boundary, fileName: "photo.jpg")
// Загрузка данных в URL на основе указанного URL-запроса и получение результатов классификации
let (data, response) = try await URLSession.shared.upload(for: request, from: body)
// Проверка URL-ответа, кода состояния и генерация исключения по результатам
guard let httpResponse = response as? HTTPURLResponse, httpResponse.statusCode == 200 else {
throw URLError(.badServerResponse)
}
// Преобразование данных в массив ClassificationResult
return try JSONDecoder().decode([ClassificationResult].self, from: data)
}
// Создание тела multipart/form-data с данными изображения
private func createMultipartFormDataBody(imageData: Data, boundary: String, fileName: String) -> Data {
...
}
// Структура для преобразования результатов с сервера
struct ClassificationResult: Identifiable, Decodable, Equatable {
...
}
}Finally, to call the APIManager method to classify the images we will get from the Image Picker, we created an object ViewModel (so as not to overload View with these tasks).
In this project we used the classic UIKit Image Picker, but if you want to use Apple's new APIs to select an image from Photos Library using SwiftUI, I recommend you check out with this guide.
@Observable
class ViewModel {
private var apiManager = APIManager.shared
var results: [APIManager.ClassificationResult] = []
// Функция, вызывающая менеджер, который отправляет URL-запрос
func classifyImage(_ image: UIImage) async {
do {
results = try await apiManager.classifyImage(image)
} catch {
print(error.localizedDescription)
}
}
}And here, finally, is the image classification from SwiftUI View by pressing one button.
Button("Classify Image") {
Task {
await viewModel.classifyImage(image)
}
}The final result
After completing all these steps, you should have two different projects running: the Vapor server project and the iOS client app. The app will generate and send a request with an image for analysis, and the server will feed it to the ML model and return a response to the client.
At the moment, our image classifier is a fairly basic application, but it can be expanded to include different analysis modes, or combined with different model types for a more complex and customized experience.
You can see the full project code here in this repository. Also, let me remind you that you can always put a star if you found it useful.
What's next?
So far we've used a local machine to host the server project, but how do we move it to a remote server for production work? After all, the more productive your server is, the better Core ML will handle large models and use large temporary files.
For example, we can choose many different hosting providers for our server. The most popular of them are: AWS, Heroku, DigitalOcean…
Another very important topic that is worth spending time on is instance setup. NGINX as a proxy. This is very useful for requiring the client to use TLS (HTTPS), limiting the request rate, or even serving public files without going through your Vapor app (and much more). You can read more about this here find out here.
One of the best practices is to use a Docker image. If you want to learn more about it, I recommend you next resource.
Other services that let you manage machine learning models that you might want to take a closer look at include: AzureML, Amazon SageMaker And IBM Watson Machine Learning.
The material was prepared within the framework of practical online course “iOS Developer. Professional”.
