This project is based on the ARM ML embedded evaluation Kit. The original project Readme.md file content is in the Section Arm ML embedded evaluation kit below. For details on how to use this project with the Ensemble Evaluation Kit, please refer to the "KWS and Image Classification on Dual M55 Cores Draft Note - Shared" document in the root of this source tree. Please note that this is meant for early beta engagement only.
This repository is for building and deploying Machine Learning (ML) applications targeted for Arm® Cortex®-M and Arm® Ethos™-U NPU. To run evaluations using this software, we suggest using:
-
an MPS3 board with Arm® Corstone-300 or Arm® Corstone-310 implementations.
- Arm® Corstone™-300 runs a combination of the Arm® Cortex™-M55 processor and the Arm® Ethos™-U55 NPU.
- Arm® Corstone™-310 runs a combination of the Arm® Cortex™-M85 processor and the Arm® Ethos™-U55 NPU.
-
a Arm® Corstone™-300 MPS3 based Fixed Virtual Platform (FVP) that offers a choice of the Arm® Ethos™-U55 NPU or Arm® Ethos™-U65 NPU software fast model in combination of the new Arm® Cortex™-M55 processor. You can also take advantage of Arm Virtual Hardware (AVH) and run the Fixed Virtual Platform in the cloud.
NOTE: While Arm® Corstone™-300 is available as an Ecosystem FVP and AVH, Arm® Corstone™-310 is available (for both Arm® Ethos™-U55 and Ethos™-U65 NPUs) only as AVH implementations.
This product conforms to Arm's inclusive language policy and, to the best of our knowledge, does not contain any non-inclusive language. If you find something that concerns you, email terms@arm.com.
The purpose of this evaluation kit is to allow users to develop software and test the performance of the Arm® Ethos-U NPU and Arm® Cortex-M CPUs for ML workloads. The Ethos-U NPU is a new class of machine learning (ML) processor, specifically designed to accelerate ML computation in constrained embedded and IoT devices. The product is optimized to execute mathematical operations efficiently that are commonly used in ML algorithms, such as convolutions or activation functions.
The evaluation kit adds value by providing ready to use ML applications for the embedded stack. As a result, you can experiment with the already developed software use cases and create your own applications for Cortex-M CPU and Ethos-U NPU. The example application at your disposal and the utilized models are listed in the table below.
| ML application | Description | Neural Network Model |
|---|---|---|
| Image classification | Recognize the presence of objects in a given image | Mobilenet V2 |
| Keyword spotting(KWS) | Recognize the presence of a key word in a recording | MicroNet |
| Automated Speech Recognition(ASR) | Transcribe words in a recording | Wav2Letter |
| KWS and ASR | Utilise Cortex-M and Ethos-U to transcribe words in a recording after a keyword was spotted | MicroNet Wav2Letter |
| Anomaly Detection | Detecting abnormal behavior based on a sound recording of a machine | MicroNet |
| Visual Wake Word | Recognize if person is present in a given image | MicroNet |
| Noise Reduction | Remove noise from audio while keeping speech intact | RNNoise |
| Generic inference runner | Code block allowing you to develop your own use case for Ethos-U NPU | Your custom model |
| Object detection | Detects and draws face bounding box in a given image | Yolo Fastest |
The above use cases implement end-to-end ML flow including data pre-processing and post-processing. They will allow you to investigate embedded software stack, evaluate performance of the networks running on Cortex-M55 CPU and Ethos-U NPU by displaying different performance metrics such as inference cycle count estimation and results of the network execution.
The evaluation kit primarily supports Arm® Corstone™-300 and Arm® Corstone™-310 reference packages as its primary targets. Arm® Corstone™-300 and Corstone™-310 design implementations are publicly available on Download FPGA Images page, or as a Fixed Virtual Platform of the MPS3 development board.
The Ethos-U NPU software stack is described here.
All ML use cases, albeit illustrating a different application, have common code such as initializing the Hardware Abstraction Layer (HAL). The application common code can be run on native host machine (x86_64 or aarch64) or Arm Cortex-M architecture thanks to the HAL. For the ML application-specific part, Google® TensorFlow™ Lite for Microcontrollers inference engine is used to schedule the neural networks models executions. TensorFlow Lite for Microcontrollers is integrated with the Ethos-U NPU driver and delegates execution of certain operators to the NPU or, if the neural network model operators are not supported on NPU, to the CPU. If the operator is supported, CMSIS-NN is used to optimise CPU workload execution with int8 data type. Else, TensorFlow™ Lite for Microcontrollers' reference kernels are used as a final fall-back. Common ML application functions will help you to focus on implementing logic of your custom ML use case: you can modify only the use case code and leave all other components unchanged. Supplied build system will discover new ML application code and automatically include it into compilation flow.
A high level overview of the different components in the software, and the platforms supported out-of-the-box, is shown in the diagram below.
For a more detailed description of the build graph with all major components, see Building.
There are source files in the repository that form the core of the Machine Leaning flow for all the use cases. These are exposed as APIs that the examples can use and even be combined to form chained use cases. The API sources are designed to be portable across platforms and provide functionality for preprocessing of data, running an inference, and postprocessing of results. These allow a common flow for all use cases with minor differences in how each of these blocks are instantiated.
As an independent CMake project, these APIs can be used by or integrated into other projects easily. We also produce CMSIS Packs with these sources, so they could be used in all tools/IDEs (for example, Arm® Development Studio and Keil® µVision®) that support the use of CMSIS Packs.
To run an ML application on the Cortex-M and Ethos-U NPU, please, follow these steps:
- Set up your environment by installing the required prerequisites.
NOTE: A Docker image built from the
Dockerfileprovided will have all the required packages installed.
- Generate an optimized neural network model for Ethos-U with a Vela compiler by following instructions here.
- Configure the build system.
- Compile the project with a
makecommand. - If using a FVP, launch the desired application on the FVP. If using the FPGA option, load the image on the FPGA and launch the application.
To get familiar with these steps, you can follow the quick start guide.
Note: The default flow assumes Arm® Ethos™-U55 NPU usage, configured to use 128 Multiply-Accumulate units and is sharing SRAM with the Arm® Cortex®-M55.
Ml embedded evaluation kit supports:
Ethos™-U NPU Default MACs/cc Other MACs/cc supported Default Memory Mode Other Memory Modes supported Ethos™-U55 128 32, 64, 256 Shared_Sram Sram_Only Ethos™-U65 256 512 Dedicated_Sram Shared_Sram For more information see Building.
For more details see full documentation:
Contributions are only accepted under the following conditions:
-
The contribution have certified origin and give us your permission. To manage this process we use Developer Certificate of Origin (DCO) V1.1. To indicate that contributors agree to the terms of the DCO, it's necessary "sign off" the contribution by adding a line with name and e-mail address to every git commit message:
Signed-off-by: John Doe <john.doe@example.org>This can be done automatically by adding the
-soption to yourgit commitcommand. You must use your real name, no pseudonyms or anonymous contributions are accepted. -
You give permission according to the Apache License 2.0.
In each source file, include the following copyright notice:
/* * SPDX-FileCopyrightText: Copyright <years additions were made to project> <your name>, Arm Limited and/or its affiliates <open-source-office@arm.com> * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */
This repository follows a set of guidelines, best practices, programming styles and conventions, see:
Contributions must go through code review. Code reviews are performed through the mlplatform.org Gerrit server. Contributors need to sign up to this Gerrit server with their GitHub account credentials. In order to be merged a patch needs to:
- get a "+1 Verified" from the pre-commit job.
- get a "+2 Code-review" from a reviewer, it means the patch has the final approval.
Prior to submitting a patch for review please make sure that all build variants works and unit tests pass. Contributions go through testing at the continuous integration system. All builds, tests and checks must pass before a contribution gets merged to the main branch.
Please, if you want to start public discussion, raise any issues or questions related to this repository, use https://discuss.mlplatform.org/c/ml-embedded-evaluation-kit forum.
The ML Embedded applications samples are provided under the Apache 2.0 license, see License Apache 2.0.
Application input data sample files are provided under their original license: