Flutter for car. Why not?

One of the problems in the development of on-board systems for a car is the creation of high-quality portable software that would work on on-board systems of different car manufacturers and can be compiled for different hardware and software architectures. Despite the obvious relevance, the task was complicated primarily by the fact that most automakers preferred to create their own proprietary operating systems, which made it difficult to create portable software. The situation has changed with the launch of the Automotive Grade Linux (AGL) project, supported by large companies such as Toyota, Mazda, Suzuki, Ford and Mercedes Benz and media system manufacturers (for example, Panasonic). And in the development plans of the Flutter project for 2022, the development of AGL support as a target platform for applications is indicated. In this article, we’ll cover the basic ideas behind building portable AGL applications with Flutter.

First of all, let’s look at the AGL project itself and its architecture. AGL is a specialized distribution of Linux, but unlike the desktop distributions we are used to, it uses the best practices of the project Yocto for building embedded systems. Yocto allows you to build a specialized distribution based on layers (bblayers.conf) consisting of recipes (recipesdescribe the rules for assembling components) and configuration settings (local.conf) that are compiled by the utility bitbake into a binary image for the target platform (which can be either one of the microcomputer boards, such as Renesas M3 or Raspberry PI 4, or an image to run in an emulator, such as vmdk or ext4 for qemu). To familiarize yourself with the system, you can use the virtual machine image from https://download.automotivelinux.org/AGL/snapshots/master/latest/qemux86-64/deploy/images/qemux86-64/agl-demo-platform-crosssdk-qemux86-64.ext4.xz (the archive needs to be unpacked) and the bootloader https://download.automotivelinux.org/AGL/snapshots/master/latest/qemux86-64/deploy/images/qemux86-64/bzImage. After downloading, you can run the AGL image and get acquainted with the system interface with the following command:

qemu-system-x86_64 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:35:02 \
-netdev user,id=net0,hostfwd=tcp::2222-:22 -drive \
file=agl-demo-platform-crosssdk-qemux86-64.ext4,if=virtio,format=raw \
-usb -usbdevice tablet -device virtio-rng-pci  -snapshot -vga virtio \
-display gtk,show-cursor=on -device intel-hda -device hda-duplex \
-machine q35 -cpu kvm64 -cpu \ qemu64,+ssse3,+sse4.1,+sse4.2,+popcnt \
-enable-kvm -m 2048 -serial mon:vc -serial mon:stdio -serial null \
-kernel bzImage  -append 'root=/dev/vda rw console=tty0 mem=2048M ip=dhcp oprofile.timer=1 console=ttyS0,115200n8 verbose fstab=no'

The system interface is shown in the figure. Here you can see the general HomeScreen interface, which provides access to the Launcher (application icons), climate control information, media player and navigator.

The system is assembled from several components, access to each of which requires permissions, which can be implemented in one of three ways:

binding – binding to native library
api – access to the subsystem API
permission – permission for a system-wide action

The following subsystems are implemented in the system:

platform (resource management, performance monitoring, interprocess communication) – based on Linux;
networking and Bluetooth (Connectivity);
multimedia system (Multimedia) – radio (Tuner), camera (Camera), sound and video playback;
voice services (Speech Services);
geolocation and navigation (Navigation and Location-Based Services);
graphics subsystem (Graphics) – window manager, graphic primitives, 3D-graphics (OpenGL-compatible);
security (Security);
vehicle control system (Automotive) – device bus (Vehicle Bus), diagnostics (Diagnostics);
application development frameworks:
- AGL Application Framework
- Native Application Framework (used including Flutter)
- Web Application Framework (web-based development)

Application permissions and bindings are requested through a manifest file, the structure of which depends on the framework used to build the interface. AGL currently runs on top of X11/DRM or Wayland and supports both HTML5 apps and native apps (including those built using Flutter).

To implement our own applications, we will need to build our own image using the necessary layers and integrating additional developer tools. Let’s start with the option of creating a Flutter Web application using AGL subsystems to access data or control vehicle systems.

Web Application Framework for AGL

Web applications for AGL are implemented in accordance with the Packaged Web Apps (Widgets) standard, which is described in the W3C standards here. To develop applications, it will be necessary to build your own AGL image, for this we will install the repo tool (a wrapper around git):

mkdir ~/bin export 
PATH=~/bin:$PATH 
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo 
chmod a+x ~/bin/repo

and then get the source texts and build:

repo init -u https://gerrit.automotivelinux.org/gerrit/AGL/AGL-repo 
repo sync 
source meta-agl/scripts/aglsetup.sh -m qemux86-64 agl-app-framework agl-demo\
  agl-pipewire agl-package-management agl-flutter agl-basesystem agl-devel\
  agl-app-framework agl-archiver agl-localdev agl-buildstats\
  agl-weston-remoting agl-demo-preload agl-drm-lease 
bitbake agl-ivi-demo-platform-html5

The assembly takes more than a day, so to simplify testing, I prepared an image for web development (for Intel Core i7 processor architecture or newer) and posted it here (archive with file system image and bootloader). In the -m option, you can specify an assembly for a hardware device (for example, for Raspberry PI) and then download the resulting image to the microcomputer. In source, we listed the necessary components (weston window manager is used over wayland, we add development and application launch tools).

To manage applications, the Application Framework Manager is used, which uses the application definition file appinfo.json (the structure is described in detail here), containing metadata and a list of requested permissions, APIs, and bindings to native libraries (bindings). Sample appinfo.json for our application:

{
  "id": "counter",
  "title": "HTML5 Counter",
  "description": "Counter application for AGL based on html5 technologies",
  "version": "1.0.0",
  "vendor": "Dmitrii Zolotov",
  "type": "web",
  "main": "https://habr.com/ru/company/otus/blog/674622/index.html",
  "uiRevision": "1",
  "icon": "https://habr.com/ru/company/otus/blog/674622/icon.svg",
  "extensions": [ ]
}

The generated web application can be placed in the /usr/lib/wam_apps/ subdirectory. When the emulator is restarted, an icon will be available to launch a new application. The application on Flutter is built in the usual way for the target Web platform, but it should be noted that the AGL subsystems are published as JavaScript objects (the implementation of the LG WebOS Open Source Edition platform integrated into the web version of the AGL build is used). Examples of web applications (not on Flutter) can be viewed here.

To install an application through the Web Application Manager, you can also use the Web Widget (.wgt) archive format, which contains a file that describes the application’s metadata and requested permissions. For example, you can indicate the need to access the list of installed applications and collect a web archive on Flutter. In this case, the metadata is defined in the config.xml file, which might look like this:

<?xml version="1.0" encoding="UTF-8"?>
<widget xmlns="http://www.w3.org/ns/widgets" id="webapps-flutter-homescreen" version="1.0.0">
  <name>Homescreen</name>
  <icon src="https://habr.com/ru/company/otus/blog/674622/icon.svg"/>
  <content src="https://habr.com/ru/company/otus/blog/674622/index.html" type="text/html"/>
  <description>HTML5 Homescreen flutter demo</description>
  <author>Igalia, S.L.</author>
  <license>APL 2.0</license>
  <feature name="urn:AGL:widget:required-permission">
    <param name="urn:AGL:permission::public:no-htdocs" value="required" />
    <param name="urn:AGL:permission::public:display" value="required" />
    <param name="urn:AGL:permission:afm:system:widget" value="required" /> <!-- list available apps -->
    <param name="urn:AGL:permission:afm:system:runner" value="required" /> <!-- run other apps -->
    <param name="urn:AGL:permission::public:applications:read" value="required" /> <!-- get app icons -->
  </feature>
  <feature name="urn:AGL:widget:required-api">
    <param name="homescreen" value="ws" />
    <param name="afm-main" value="ws" />
  </feature>
</widget>

This requests the Home Screen API and AFM (Application Framework Manager) and permission to launch applications and read the list of installed applications. Also in the API, you can request access to gps, wifi and other AGL subsystems (the list can be viewed here). An example of a Flutter application with access to system resources can be found in Github:

For a counter application, the configuration file might look like this:

<?xml version="1.0" encoding="UTF-8"?>
<widget xmlns="http://www.w3.org/ns/widgets" id="webapps-sample" version="0.0.0">
  <name>Sample App</name>
  <icon src="https://habr.com/ru/company/otus/blog/674622/icon.svg"/>
  <content src="https://habr.com/ru/company/otus/blog/674622/index.html" type="text/html"/>
  <description>Sample counter</description>
  <author>Dmitrii Zolotov</author>
  <license>GPL 2.0</license>
  <feature name="urn:AGL:widget:required-permission">
    <param name="urn:AGL:permission::public:no-htdocs" value="required" />
    <param name="urn:AGL:permission::public:display" value="required" />
  </feature>
  <feature name="urn:AGL:widget:required-api">
    <param name="homescreen" value="ws" />
  </feature>
</widget>

The archive can be installed via afm-util install <name.wmgt>; aft-util start <appname>

Native Application Framework

The multi-level architecture of Flutter and the separation of the dart framework and the low-level implementation of Embedder represent the potential for creating applications for any hardware architecture and software environment. Currently, among the stable platforms for running Flutter applications are Android, iOS, Web, Windows, Linux and MacOS. Development for Fuchsia OS is also experimentally supported. But of particular interest to us now is the project of Sony flutter-embedded-linux (and a set of tools flutter-elinux) based on Flutter for Tizen SDK, an extension for Flutter tools that allows you to build applications for embedded systems and is essentially an alternative solution for running Flutter applications on Linux platforms without using GTK (which is used in Flutter Desktop for Linux). The project implements an alternative Embedder, which is responsible for implementing the message loop, platform channels for using C / C ++ code, working with graphics, input devices and translating requests to Wayland or X11 / DRM. The generated image can be executed both inside an embedded device emulator (for example, AGL on QEMU) and on a microcomputer board that can be used in a real car.

The tool is installed as a wrapper around the Flutter SDK and implements the full set of flutter command line utility commands:

git clone https://github.com/sony/flutter-elinux.git
sudo mv flutter-elinux /opt/
export PATH=$PATH:/opt/flutter-elinux/bin
flutter-elinux --version

You can see that X11 or Wayland binding is supported on elinux targets, which can be specified when running the application. Let’s run a test application for X11 elinux-x11 (for ubuntu you can use elinux-wayland):

flutter create --platforms=elinux hello
cd hello
flutter-elinux run elinux-x11

To create runnable applications on AGL, you need to build the flutter bundle and transfer it via scp to the running virtual machine (in our case, via port 2222, the root user without a password). You can also prepare a ~/.flutter_custom_devices.json file describing how to deliver artifacts to a device, for example:

{
  "custom-devices": [
    {
      "id": "AGL",
      "label": "AGL",
      "sdkNameAndVersion": "Automative Grade Linux",
      "enabled": true,
      "platform": "x86_64",
      "backend": "wayland",
      "install": [
        "scp", "-P", "2222", "-r", "${localPath}", "root@127.0.0.1:/tmp/${appName}"
      ],
      "uninstall": [
        "ssh", "-p", "2222", "root@127.0.0.1", "rm -rf \"/tmp/${appName}\""
      ],
      "runDebug": [
        "ssh", "-p", "2222", "root@127.0.0.1", "/tmp/${appName}/${appName}"
      ],
      "forwardPort": [
        "ssh", "-o", "ExitOnForwardFailure=yes",
        "-p", "2222",
        "-L", "127.0.0.1:${hostPort}:127.0.0.1:${devicePort}", "root@127.0.0.1"
      ],
      "forwardPortSuccessRegex": "Linux"
    }
  ]
}

Now the new AGL device will appear in the list of devices (flutter-elinux devices) and you can run the application directly on it: flutter-elinux run elinux-wayland -d AGL

It should be noted that the use of flutter-elinux is not limited to AGL, but can be used to develop applications for other Linux-based embedded devices (and if the distribution allows you to build a Linux image on Yocto, then you can use the recipes https://github.com/sony/meta-flutter to integrate runtime and build if explicitly specified when running bitbake, Flutter apps on a device on top of Wayland or X11/DRM).

It is more convenient to build and run applications in the AGL virtual machine by building the runtime and SDK directly inside the image. To do this, in bitbake, we specify the target assembly agl-demo-platform and the populate_sdk configuration:

bitbake agl-demo-platform -c populate_sdk

As a result, we will be able to compile applications directly inside AGL (using the same implementation of Flutter eLinux). To access the functions of the operating system from the code, the mechanisms of platform channels are used, which are received on the system side through plugins.

We looked at the basics of application development on Flutter for the Automotive Grade Linux operating system. In the next article, we will look at examples of using device sensors using a simple climate control application in a car as an example.

The new version of Flutter 3.0 has added official support for the Flare and SpriteWidget game engines and the CasualGamingKit toolkit. I would like to invite you to a free lesson where we will explore the possibilities of Flutter for creating cross-platform games for mobile, web and desktop and create a simple arcade game from inception to preparation for publishing in app stores.