Flutter for Apple TV

This article is a translation of the original article by Aleksandr Denisov “Flutter for Apple TV“

I also run a telegram channelFrontend in a navy way”, where I talk about interesting things from the world of interface development.

Introduction

In March 2021, Flutter received a major update that allows developers to create beautiful, fast, and portable apps for a wide variety of platforms. With Flutter 2.x, you can use the same codebase to push native apps to mobile operating systems like iOS and Android, to desktop operating systems like Windows, macOS and Linux, and to browsers like Chrome , Firefox, Safari or Edge. Also, the Flutter team gave some information about Flutter for embedded devices, but nowhere was it officially described how you can develop applications for Smart TV operating systems using Flutter.

If you try to search StackOverflow or GitHub for information on the topic of Flutter for TV, you can find scattered pieces of information from the community, mostly about experiments with Android TV, and almost nothing about other platforms. The reason for this is that Android TV is not a Flutter platform supported for production use, and Apple TV is not supported at all.

But nevertheless, if you look into the Flutter repository on GitHub, you can find a lot of questions related to the development of an application for Smart TV, and not only Android TV, but also Apple TV. It can be concluded that, despite the lack of official support, developers continue to try to create applications for smart TVs.

Actually, I am no exception. My team was faced with the task of creating an application for several platforms at once, including Smart TV Platforms. We needed to support seven platforms with the same codebase: iOS, Android, Web, Android TV, Apple TV, Fire TV, and Tizen. We turned out to be developers who were trying to create an application not only for mobile and web applications, but also for TV platforms.

That is why I decided to write this article to share some of the experiences we had during development, what difficulties we encountered, and what problems we had.

But the first question that should arise is “why”, why would you even need to use Flutter to create TV applications. I have several answers to this question. Firstly, it’s very cool if you can just take an already written Flutter application and just run it on a TV. Secondly, and more importantly, if you are planning to develop an application that will support TV platforms, Flutter will save you a lot of resources.

In fact, the more platforms you support using a single codebase, the more resources you save. There is no need to have separate development teams for TV applications, everything can be done by one team. And it’s great! 🥳

In this article, I will show you in detail how to run Flutter apps for Apple TV and what is the difference between Android TV and Apple TV when working with these platforms.

So it’s always better not just to say words, but to show an example on a real application. Unfortunately, I can’t share examples from the app we’re working on as it’s against the NDA, so I’ve prepared an example of a simple app that works on mobile devices, web browsers, and TV devices. It’s just a gallery of movies and series. Tapping or clicking on any of the movies opens a page with a detailed description.

There is a result of running this application on iPhone, Android tablet and Chrome browser.

Let’s try to run it on Android TV.

Actually, with the launch of Android TV, everything is extremely simple, because Android TV is the same Android OS. You can run an Android TV emulator or connect a real device via Android Debug Bridge (adb) and press the start button. The application just starts up, without any additional steps.

But you need to take into account one detail, TV layout and mobile layout sometimes have differences, and you need to be able to distinguish between them. Flutter has a class Platform, which allows you to determine where the application starts. So we have a problem: Platform.isAndroid will true for both mobile devices and TVs, because Android TV is also Android. You need to provide additional platform information at build time. This can be done in several ways: make different entry points, like “main.dart” and “main_tv.dart” for example. But it seems to me that the easiest way is to use Flutter Environment Variables. In this example, I have created an environment variable TV_MODEto give the compiler different values: “ON” for TV and everything else for mobile devices. Then I implemented the class MyPlatform and used it instead Platform.

class MyPlatform {
  static const tvMode = String.fromEnvironment('TV_MODE');
  static bool get isTv => tvMode == 'ON';

  static bool get isIOS => !isTv && Platform.isIOS;
  static bool get isAndroid => !isTv && Platform.isAndroid;
  static bool get isTVOS => isTv && Platform.isIOS;
  static bool get isAndroidTV => isTv && Platform.isAndroid;
}

So don’t forget to include TV_MODE=ON when running the app on the TV to get the correct behavior of the app.

flutter run --dart-define TV_MODE=ON

So let’s check.

What about Apple TV?

Even though the Apple TV emulator or real device can also be selected as the target device in Android Studio, the Flutter app cannot be run on it.

Let me explain why.

If you look at the structure of the Flutter project, you can see different folders for ios and android applications. So, in the “android” folder there is an Android project, and in the “ios” folder there is a ready-made XCode project for an iOS application, but Apple TV has a different operating system – tvOS. And as you can see, there is no tvOS folder in the project, and Flutter can only work on iOS.

Actually, you can try to open this project in XCode and edit it, I mean change the deployment target to tvOS, change the storyboard and so on to make it a tvOS project. But even if you do this, the flutter application will still not start.

The fact is that tvOS is not iOS, it is a different operating system that is quite close to iOS in its structure and has the same kernel – Darwin. But there are differences, at least in the API, part of the iOS API is simply missing in tvOS, so the Flutter Engine will crash when building a tvOS application, trying to find non-existent methods.

But there is a solution. The Flutter engine is completely open source, which means that it can potentially be adapted to create projects for tvOS. Exist detailed guide how to get the source code for the Flutter engine and set up a development environment, as well as guides on how to compile and debug updated engine.

You can try to make the necessary changes: cut out the use of methods that are not in tvOS, and edit the calls to methods that have different signatures. But you need to have some experience with C and Objective-C.

It would be an interesting experience, but it’s not really needed. There are enthusiasts who have already made these changes and open-sourced the updated Flutter Engine. Actually, I am one of them. But you still need to compile the engine locally by following the guide mentioned above, or you can take a look at detailed guidecreated specifically for the Flutter approach for Apple TV, which can also help.

Finally, you can simply download the already compiled by me custom Flutter Engine (supported version of Flutter is 2.10.3), but be careful, the archive file is heavy, about 6.5 GB.

You can check how it works on my exampleby following these steps:

• clone project

• Download engine

• Change the value FLUTTER_LOCAL_ENGINE to the path to the folder of the downloaded engine in scripts/run_apple_tv.sh

• Select the desired engine type (debug_sim, debug or release) in scripts/run_apple_tv.sh

• Complete sh scripts/run_apple_tv.sh in a terminal window (when the script completes, XCode will open with the AppleTV project)

• Select the device and click the “Run” button in the open Xcode project.

Voila! The app should run on an Apple TV device or emulator, depending on your choice.

So, we figured out how to run Flutter apps on Apple TV. But just launching the application is not enough. There are many different questions. For example, how to organize user interaction? You can’t tap your finger on the TV screen, and don’t even make a mouse click, like in a browser, you have to interact with the TV using the remote control.

In the case of Android TV, everything is quite simple: the Remote Control Unit is connected to the device as a raw keyboard, and the events of button presses on the remote control are processed as keystrokes on the keyboard. The up, down, left, and right buttons are the same as the arrow keys on a keyboard. Flutter already has a mechanism thathandle keyboard events. This is a widget Focus. Widget controls focusnodeto allow keyboard focus to be passed to the widget and its children. Typically used to navigate between widgets using the keyboard, such as in a browser. You just need to implement the highlighting of focused widgets.

In the case of Apple TV, the difference is big. The standard Apple TV remote has no arrow buttons at all, only a touchpad. And you need to support not only taps and clicks on the touchpad, but also swipes.

But to support Apple TV, you need to use Flutter’s own engine, which means you can make changes there. Thus, to receive touchpad events on the Flutter side, you just need to add code to the Engine that provides information about touches, clicks, and swipes to the Flutter application through Flutter Channels.

In fact, the custom open source Flutter engine you compiled (or downloaded) earlier has this functionality already implemented. Left, right, up, and down presses are automatically converted to raw keyboard events as arrow key presses, the only thing left for you to do is handle swipes and clicks in your Flutter app.

Custom engine provides PlatformChannel flutter/gamepadtouchevent, which makes it possible to handle touchpad events. You can create your own touchpad controller to control focus. The channel provides three arguments: “x”, “y”, and “type”. The first two are the coordinates of your finger on the touchpad, and the third is the interaction type.

It can take the values: “started”, “move”, “ended”, “loc”, “click_s” and “click_e”, which means:

• started: the finger is on the touchpad, and a swipe is launched;

• move: swipe in progress;

• ended: swipe ended, finger released the touchpad;

• click_s: finger is on the touchpad, the click is launched;

• click_e: finger released touchpad, click ended;

• loc: just finger coordinates, independent of interaction;

Here is an example of how a simple touchpad controller can be implemented. There, the movement of the finger is caught by 400 pixels in some direction and the method is called triggerKey.

static const channel = BasicMessageChannel<dynamic>('flutter/gamepadtouchevent', JSONMessageCodec());

void init() {
  channel.setMessageHandler(_onMessage);
}

Future<void> _onMessage(dynamic arguments) async {
  num x = arguments['x'];
  num y = arguments['y'];
  String type = arguments['type'];
  late LogicalKeyboardKey key;

  if (type == 'started') {
    swipeStartX = x;
    swipeStartY = y;
    isMoving = true;
  } else if (type == 'move') {
    if (isMoving) {
      var moveX = swipeStartX - x;
      var moveY = swipeStartY - y;

      // need to move min distance in any direction
      // the 400px needs tweaking and might needs to be variable based on location of the widget on screen and duration/time of the movement to make it smoother
      if ((moveX.abs() >= 400) || (moveY.abs() >= 400)) {
        // determine direction horizontal or vertical
        if (moveX.abs() >= moveY.abs()) {
          if (moveX >= 0) {
            key = LogicalKeyboardKey.arrowLeft;
          } else {
            key = LogicalKeyboardKey.arrowRight;
          }
        } else {
          if (moveY >= 0) {
            key = LogicalKeyboardKey.arrowUp;
          } else {
            key = LogicalKeyboardKey.arrowDown;
          }
        }
        triggerKey(key);
        // reset start point (direction could change based on next cooordinates received)
        swipeStartX = x;
        swipeStartY = y;
      }
    }
  } else if (type == 'ended') {
    isMoving = false;
  } else if (type == 'click_s') {
    unawaited(
      simulateKeyEvent(
        PhysicalKeyboardKey.enter,
        isDown: true,
      ),
    );
  } else if (type == 'click_e') {
    unawaited(
      simulateKeyEvent(
        PhysicalKeyboardKey.enter,
        isDown: false,
      ),
    );
  }
}

TriggerKey is the method that fires the FocusManager to move the main focus in the desired direction.

void triggerKey(LogicalKeyboardKey key) {
  if (LogicalKeyboardKey.arrowLeft == key) {
    FocusManager.instance.primaryFocus!.focusInDirection(TraversalDirection.left);
  } else if (LogicalKeyboardKey.arrowRight == key) {
    FocusManager.instance.primaryFocus!.focusInDirection(TraversalDirection.right);
  } else if (LogicalKeyboardKey.arrowUp == key) {
    FocusManager.instance.primaryFocus!.focusInDirection(TraversalDirection.up);
  } else if (LogicalKeyboardKey.arrowDown == key) {
    FocusManager.instance.primaryFocus!.focusInDirection(TraversalDirection.down);
  }
}

Another thing to keep in mind is that the custom engine does not support the RCU back button press out of the box, you have to support that yourself. You must handle this event and send a message event popRoute to the navigation channel.

void init() {
  HardwareKeyboard.instance.addHandler(handleKeyMessage);
}

bool handleKeyMessage(KeyEvent message) {
  if (LogicalKeyboardKey.goBack == message.logicalKey) {
    final message = const JSONMethodCodec().encodeMethodCall(const MethodCall('popRoute'));
    ServicesBinding.instance!.defaultBinaryMessenger
        .handlePlatformMessage(SystemChannels.navigation.name, message, (_) {});
    return true;
  }
  return false;
}

These were the most basic points regarding RCU support for Apple TV.

That’s all, congratulations! You’ve launched your first Flutter app for AppleTV and now you know it’s possible!

So developing TV apps with Flutter is not unrealistic. We have covered AndroidTV and AppleTV, but there are other TV platforms such as Tizen, FireTV, webOS, RokuTV and so on. I am sure that Flutter can be used to develop applications for them as well, and we will definitely discuss this in future articles.