Hello, Habr! Together with Harmony OS, we introduced the fifth version of HMS Core, a set of tools that you can use to develop applications for the Huawei ecosystem. We added new possibilities for working with content, focused on data security, interaction between devices and expanded the possibilities for AI tools – we talked about all this in detail at our conference HDC.Together, and in this article we will give an overview of the new HMS features.
Tools for working with ML and AI
The main services for working with AI are included in ML Kit and allow you to work with text, voice, pictures, AR / VR technologies. In HMS Core 5.0, we have increased the number of supported languages to 50 and can translate between 20 languages, while both text and voice recordings are accepted as input. Also ML Kit can be used to filter out spam and any obscene images.
With ML Kit, you can convert voice recordings to text format and vice versa, and we have expanded this functionality with Video Course Creator, which automatically creates educational courses: it takes as input the educational program, teacher’s comments and preset audio and video parameters – during the broadcast, the teacher can refer to the necessary materials and all the recorded materials are assembled into a single course.
We have made a special emphasis on face recognition, gestures and poses. Thanks to this, using ML Kit, you can perform face authentication – the system is connected via API and can be used for quick access to applications without any additional actions.
You can work with AI and ML tools at different levels:
- A platform for working with artificial intelligence technologies on mobile devices HiAI allows you to train neural networks, create models and convert them into a binary file that can already be loaded onto the NPU chip.
- Platform Ability Gallery provides developers with out-of-the-box AI use cases and allows them to work with big data in their applications.
AR / VR
Our AR engine analyzes information about lighting, plane, shape of objects, type of surface, is able to search for specified objects in space. Separately, the system can build a 3D scheme using anchor points and track human movements, gestures and facial expressions. For example, 21 points are allocated to determine the position of the hand, and 23 points are allocated to the position of the body. Now the system can recognize 6 poses and analyze 2 people at once.
From the fifth version of HMS Core CameraKit provides various shooting modes: wide aperture, portrait mode, HDR, background blur, supernight mode and others like them. It also became possible to use AI in photo and video shooting for pre-selection of filters and color correction.
Collaboration of devices
The new version of HMS Core has several new engines for collaboration between different devices. We’ve simplified interactions between devices and smart things with DeviceVirtualization-engine. It converts peripheral devices into virtual objects and allows the phone to work with them in the same way as with normal functions.
In the same direction, we have developed Cast Enginewhich allows the phone to operate in a multiscreen mode and transfer images to TV in high definition. We also plan to add the ability to work with Cast Engine through other applications.
Another new engine OneHop Kit works with NFC and allows you to securely transfer data between devices with one touch. With it, you can sync Huawei devices with each other, transfer files and open applications on other devices with the same settings as on the main one. For now, it only works between Huawei phones and tablets, so for communication with other devices we provide Share enginewhich provides speeds up to 80 Mb / s via Bluetooth.
Video and audio
In the new version, we have developed several new SDK tools for working with audio and video. Audio Kit and Video kit support major file formats and codecs and can be easily integrated into the application and read, save and delete lists, play sequences, adjust the playback speed. All audio and video are protected by encryption algorithms.
The Audio Kit now has audio engine for high quality sound recording with delay optimization and other instruments. The video engine, in turn, supports the main protocols HTTP, HTTPS, HLS, DASH. It also allows you to organize streaming from third-party services using the Video Kit WisePlayer SDK.
Based on our service Huawei meetime appeared Caas kit for video calls, which ensures playback and transmission of HD images even with an unstable connection. The engine also allows you to share the screen, draw doodles and, using virtualization technologies, transfer images to any other devices, which can be used for business conferences.
Pictures and 3D rendering
For image processing appeared Image Kit – it offers over 20 filters and visual effects, including rendered animation. It includes 2 SDKs: Image Vision SDK for working with color filters and Image Render SDK to use animation effects.
Scene Kit designed to work with 3D objects: it uses the Physically Correct Rendering (PBR) method and allows applications to connect via API and get 3D models of complex objects. The engine offers three scenarios: SceneView for shared scenes (non-AR), ARView for shared AR scenes, and FaceView for working with faces in AR scenes.
Analytics and security
In Core 5.0 Huawei, we launched the tagging system Dynamic Tag Manager (DTM) to track the marketing activity of users: it integrates both with Huawei services themselves and with third-party platforms for sending and processing data. Using the DTM, you can dynamically update tags in the web user interface, track specific events, and send data to third-party analytics platforms. Our systems focus on security, which is why the DTM engine is also used as an antifraud system to track suspicious activity.
One of the features of the new version of HMS is face authentication using LocalAuthentication Engine… It works with an infrared camera that builds the model from the reference points and authenticates with the ML Kit.
Finally, we paid attention to content protection: for audio files, cache encryption is used, for video – an anti-lytic system and authentication during playback, and for working with a camera – end-to-end encryption. All data HMS stores in a trusted environment TEE, within which encryption is performed.
Where can I find out more?
All HDC.Together conference recordings are available via link… Here you can listen to reports on platform updates, the main scenarios for working with HMS tools and ask technical questions at Huawei Developer Forum or Stackoverflow tagged huawei-mobile-services…