Freescale ARM Inter Core Communication for Linux - S32V234 (Treerunner)
=======================================================================
The InterCore Communication (ICC) is a subsystem which enables applications to run on
different (heterogenous) cores in multicore systems, located either on the same target
or on different targets, running different Operating Systems.
This package contains the S32V234 (Treerunner) port for the Linux part of the
ICC product.
Currently, ICC enables the communication between applications running on:
- ARM® Cortex®-A53 and ARM® Cortex®-M4 cores on the S32V234 platform
- ARM® Cortex®-A53 core on S32V234 PCIe and ARM® Cortex®-A53 on S32V234 EVB over PCI Express
- ARM® Cortex®-A53 core on S32V234 PCIe and ARM® Cortex®-A57 on LS2080A/LS2085A RDB over PCI
Express, in the BlueBox dual platform system.
It is assumed that this git repo has been cloned to a local directory, e.g.
'/path/to/icc'.
Prior to being used it needs to be integrated with the Operating Systems running
on each of the targetted cores.
This Linux package provides the Linux source code representing the ICC API implementation,
initialization of the ICC resources (shared memory, interrupts, PCIe communication etc.),
a sample for demonstrating ICC functionality on the Linux side (A5x) and a binary AUTOSAR
application for demonstrating ICC functionality on the AUTOSAR side (M4), which is required
to be loaded on the M4 core prior to loading the Linux modules in the M4 - A53 ICC setup.
The API implementation and Linux resource initialization are linked in the ICC module
(ICC_dev*.ko), which also exports the ICC API.
The ICC Sample module (ICC_Sample*.ko) uses the API exported by the ICC module and executes
a loop which exchanges messages with the other end (M4 or another A5x).
ICC module (ICC_dev*.ko) is located in
'/path/to/icc/release/iccs32v234/sample/linux_app/ICC_module',
the Linux ICC Sample module (ICC_Sample*.ko) in
'/path/to/icc/release/iccs32v234/sample/linux_app/ICC_Sample_module',
and the a binary image for the AUTOSAR ICC pre-built sample application
(icc_sample_asr.bin) in '/path/to/icc/release/iccs32v234/sample/autosar_app'.
For all supported setups, it is assumed that the kernel sources are located in
'/path/to/kernel/sources' and the kernel build output (binaries and Module.symvers),
if different from kernel sources directory, is located in '/path/to/kernel/build/artifacts'.
Building the ICC modules
------------------------
ICC modules must be built after the kernel is built.
In order to build the ICC modules, the following variables must be set in the Linux
shell:
ARCH=arm64
CROSS_COMPILE=the cross toolchain specific prefix for the build tools (e.g.:
/path/to/bsp/build_*_release/tmp/sysroots/x86_64-linux/usr/bin/aarch64-fsl-linux/aarch64-fsl-linux-
for yocto builds or
/path/to/toolchain/bin/aarch64-linux-gnu- for stand-alone toolchains)
KERNELDIR=path to the kernel binaries (containing Module.symvers)
ICC is built from the shell by running 'make' from the sources root directory
('/path/to/icc').
The result of the build consists in two modules:
'/path/to/icc/release/iccs32v234/sample/linux_app/ICC_module/ICC_dev.ko' and
'/path/to/icc/release/iccs32v234/sample/linux_app/ICC_Sample_module/ICC_Sample.ko'.
1. ICC between M4 and A53 on S32V234
====================================
This setup enables communication between applications running AutoSAR OS on M4, and
applications running Linux on A53. Supported platforms are S32V234 PCIe and EVB.
Running the AUTOSAR image on M4
-------------------------------
This can be achieved using the latest u-boot build, which has support for this task.
The AUTOSAR sample binary image (icc_sample_asr.bin) must be copied to a FAT
partition on your SD card (e.g. the one containing the kernel image and the dtb,
assumed to be the first partition on the SD card).
From the u-boot prompt, run the following commands:
=> initsram 0x3E809000 0x200000; fatload mmc 0:1 0x3E809000 /icc_sample_asr.bin; startm4 0x3e809200
Then boot the kernel.
Running the ICC demo
--------------------
The output of the demo consists in periodic messages in the console with the number
of executions of the main thread loop and average execution time of the loop.
If there's something wrong while running the demo, there will be error messages
displayed by the Linux modules.
After starting the M4 image, booting the kernel and inserting the ICC modules,
one could monitor the following:
- output from the modules module:
root@s32v234evb:~# insmod /icc/ICC_dev.ko
[ 30.174276] ICC_dev: found MSCM block: addr=0x40081000 size=4096
[ 30.183489] ICC_dev: reserved ICC_HW_MSCM_VIRT_BASE=0xffffff8001292000 size=4096
[ 30.190726] ICC_dev: reserved irq=30 from MSCM node
[ 30.196003] ICC_dev: Found SRAM block: addr=0x3e800000 size=4194304
[ 30.201996] ICC_dev: Reserved ICC_Shared_Virt_Base_Addr=0xffffff8001300000 size=1048576
[ 30.210479] ICC_dev: ICC Shared Config found at address 0xffffff8001301ca0
[ 30.217127] ICC_dev: ICC Shared Config local virtual address: 0xffffff8001301ca0
[ 30.224513] ICC_dev: ICC Shared Config remote virtual address: 0x0
root@s32v234evb:~#
root@s32v234evb:~# insmod /icc/ICC_Sample.ko
[ 38.060338] ICC_Sample: ICC_Initialize ... done
[ 38.064580] ICC_Sample: Opening all channels ... done
[ 38.070128] ICC_Sample: ICC data kthread ICC_Data_kthrea: started
[ 38.076045] ICC_Sample: exec 1 times; 20000 ns in average
root@s32v234evb:~# [ 40.673187] ICC_Sample: exec 12627 times; 205046 ns in average
[ 43.673362] ICC_Sample: exec 27251 times; 204784 ns in average
[ 43.673362] ICC_Sample: exec 27251 times; 204784 ns in average
[ 46.673177] ICC_Sample: exec 41861 times; 204765 ns in average
..........
- contents of the file '/proc/interrupts', which outputs how many times the
ICC interrupt occurred:
s32v234evb:~$ cat /proc/interrupts
CPU0
3: 85099 GIC 27 Edge arch_timer
15: 11155761 GIC 32 Level ICC <-- the ICC interrupt
18: 0 GIC 40 Level edma-tx_0-15
.....
If the line for ICC changes each time the cat command is run, then everything is running ok.
2. ICC between two A5 cores over PCI Express
============================================
2.1 ICC between two S32V234 platforms (PCIE and EVB) over PCI Express
---------------------------------------------------------------------
This setup enables communication between two Linux applications (kernel modules) running on two
A53 cores physically located on two different S32V234boards, an PCIE and an EVB. The PCIE board
must configured as PCI Express End Point, while the EVB must be configured as PCI Express Root Complex.
Configure S32v234 PCIe as PCI Express End Point
-----------------------------------------------
In the directory where the kernel sources are located, please run the kernel configurator
('make menuconfig').
In the kernel configurator window, enable Freescale S32V234 PCIe Host controller.
For S32V234 EVB (Root Compex), leave the End Point option unchecked:
Bus support -->
PCI host controller drivers -->
[*] Freescale S32V234 PCIeHost controller
[ ] Freescale S32V234 PCIeEP
For S32V234 PCIe (End Point), select everything under PCIe End Point support:
Bus support -->
PCI host controller drivers -->
[*] Freescale S32V234 PCIeHost controller
[*] Freescale S32V234 PCIeEP
[*] Ignore Freescale S32V234 erratum ERR009852
[*] Freescale S32V234 PCIeDW DMA
Exit and save as default configuration, then build the kernel (run 'make' from kernel
source directory).
Run Linux and the ICC Demo
--------------------------
For S32V234 PCIe:
- In uboot:
- Make sure you have the variable "ignoreERR009852" set:
=> setenv ignoreERR009852 1
- Update the kernel boot args, by adding "mem=255M" in order to reserve the upper most 1 Mb of
memory for ICC
- Boot the kernel
- From Linux, start ICC by inserting the two modules in this exact order:
ICC_dev_EP.ko and ICC_Sample_EP.ko:
root@s32v234pcie:~# insmod /icc/ICC_dev_EP.ko
[ 77.355486] ICC_dev_EP: reserved ICC_Shared_Virt_Base_Addr=0xffffff8001280018 size is 1048552
[ 77.363693] ICC_dev_EP: Inbound region setup successfully
[ 77.368948] ICC_dev_EP: EP 0x8ff00000 mapped to EP BAR 2
root@s32v234pcie:~# insmod /icc/ICC_Sample_EP.ko
[ 82.763732] ICC_Sample_EP: ICC Initialize ... done
[ 82.768140] ICC_Sample_EP: Waiting for peer ...
For S32V234 EVB:
- In uboot:
- Update the kernel boot args, by adding "mem=255M" in order to reserve the
upper most 1 Mb of memory for ICC
- Boot the kernel
- From Linux, start ICC by inserting the two modules in this exact order:
ICC_dev_RC.ko and ICC_Sample_RC.ko:
root@s32v234evb:~# insmod /icc/ICC_dev_RC.ko
[ 89.823659] ICC_dev_RC: reserved ICC_Shared_Virt_Base_Addr=0xffffff8001300018 size is 1048552
[ 89.832040] ICC_dev_RC: ICC Shared Config found at address 0xffffff8001300018
[ 89.838925] ICC_dev_RC: ICC Shared Config local virtual address: 0xffffff8001300018
[ 89.846584] ICC_dev_RC: ICC Shared Config remote virtual address: 0xffffff8001280018
root@s32v234evb:~# insmod /icc/ICC_Sample_RC.ko
[ 99.134143] ICC_Sample_RC: ICC Initialize ... done
[ 99.138580] ICC_Sample_RC: Notifying peer ...
[ 99.142983] ICC_Sample_RC: Peers connected
[ 99.164335] ICC_Sample_RC: Opening all channels ... done
[ 99.169392] ICC_Sample_RC: ICC data kthread ICC_Data_kthrea: started
After this, the two applications will begin to communicate.
Please look on the appropriate consoles.
For S32V234 PCIe:
[ 115.555309] ICC_dev_EP: Outbound region setup successfully
[ 115.560447] ICC_dev_EP: RC 0x72200000 mapped to EP BAR 2, size 1048576
[ 115.566947] ICC_dev_EP: EP 0x72000000 mapped to RC 0x8ff00000, size 1048576
[ 115.573943] ICC_Sample_EP: Peers connected
[ 115.579547] ICC_Sample_EP: Opening all channels ... done
[ 115.584640] ICC_Sample_EP: ICC data kthread ICC_Data_kthrea: started
root@s32v234pcie:~# [ 115.587125] ICC_Sample_EP: exec 1 times; 2471834 ns in average
[ 117.602453] ICC_Sample_EP: exec 637 times; 3167381 ns in average
[ 120.600430] ICC_Sample_EP: exec 1581 times; 3168702 ns in average
.....
For S32V234 EVB:
root@s32v234evb:~# [ 99.176055] ICC_Sample_RC: exec 1 times; 6645666 ns in average
[ 101.225409] ICC_Sample_RC: exec 642 times; 3193602 ns in average
[ 104.229683] ICC_Sample_RC: exec 1582 times; 3191250 ns in average
.....
2.2 ICC on BlueBox (S32V234 PCIe and LS2080A/LS2085A RDB over PCI Express)
-------------------------------------------------------------------------
This setup enables communication between the two boards part of the BlueBox, by running
two Linux applications (kernel modules) on the A5x cores physically located on both sides
of the PCI Express.
The S32V234 PCIe board by default is configured as PCI Express End Point, while the LS2
by default is configured as PCI Express Root Complex.
NOTE: In order to have PCI Express End Point support working for ICC, the End Point kernel
configuration must have "Ignore Freescale S32V234 erratum ERR009852" checked.
Run Linux and ICC demo
----------------------
For S32V234 PCIe:
- In uboot:
- Make sure you have the variable "ignoreERR009852" set:
=> setenv ignoreERR009852 1
- Update the kernel boot args, by adding "mem=255M" in order to reserve the upper most
1 Mb of memory for ICC
- Boot the kernel
- From Linux, start ICC by inserting the two modules in this exact order:
ICC_dev_EP.ko and ICC_Sample_EP.ko:
root@s32v234pcie:~# insmod /icc/ICC_dev_EP.ko
[ 77.355486] ICC_dev_EP: reserved ICC_Shared_Virt_Base_Addr=0xffffff8001280018 size is 1048552
[ 77.363693] ICC_dev_EP: Inbound region setup successfully
[ 77.368948] ICC_dev_EP: EP 0x8ff00000 mapped to EP BAR 2
root@s32v234pcie:~# insmod /icc/ICC_Sample_EP.ko
[ 82.763732] ICC_Sample_EP: ICC Initialize ... done
[ 82.768140] ICC_Sample_EP: Waiting for peer ...
For LS2 RDB:
- In uboot:
-Update the kernel boot args, by adding "mem=13568M" in order to reserve the upper most
block of memory for ICC
- Boot the kernel
- From Linux, start ICC by inserting the two modules in this exact order:
ICC_dev_BB_RC.ko and ICC_Sample_BB_RC.ko:
root@ls2bluebox:~# insmod /icc/ICC_dev_BB_RC.ko
[ 189.823659] ICC_dev_BB_RC: reserved ICC_Shared_Virt_Base_Addr=0xffff000009080018 size is 1048552
[ 189.832040] ICC_dev_BB_RC: ICC Shared Config found at address 0xffff000009080018
[ 189.838925] ICC_dev_BB_RC: ICC Shared Config local virtual address: 0xffff000009080018
[ 189.846584] ICC_dev_BB_RC: ICC Shared Config remote virtual address: 0xffffff8001280018
root@ls2bluebox:~# insmod /icc/ICC_Sample_BB_RC.ko
[ 199.134143] ICC_Sample_BB_RC: ICC Initialize ... done
[ 199.138580] ICC_Sample_BB_RC: Notifying peer ...
[ 199.142983] ICC_Sample_BB_RC: Peers connected
[ 199.164335] ICC_Sample_BB_RC: Opening all channels ... done
[ 199.169392] ICC_Sample_BB_RC: ICC data kthread ICC_Data_kthrea: started
After this, the two applications will begin to communicate. Please look on the appropriate
consoles.
For S32V234 PCIe:
[ 115.555309] ICC_dev_EP: Outbound region setup successfully
[ 115.560447] ICC_dev_EP: RC 0x1446100000 mapped to EP BAR 2, size 1048576
[ 115.566947] ICC_dev_EP: EP 0x72000000 mapped to RC 0x8350000000, size 1048576
[ 115.573943] ICC_Sample_EP: Peers connected
[ 115.579547] ICC_Sample_EP: Opening all channels ... done
[ 115.584640] ICC_Sample_EP: ICC data kthread ICC_Data_kthrea: started
root@s32v234pcie:~# [ 115.587125] ICC_Sample_EP: exec 1 times; 46261750 ns in average
[ 117.602453] ICC_Sample_EP: exec 64 times; 40318118 ns in average
[ 120.600430] ICC_Sample_EP: exec 139 times; 40103829 ns in average
.....
For LS2 RDB:
root@ls2bluebox:~# [ 199.176055] ICC_Sample_BB_RC: exec 1 times; 35930160 ns in average
[ 201.225409] ICC_Sample_BB_RC: exec 73 times; 39996015 ns in average
[ 204.229683] ICC_Sample_BB_RC: exec 148 times; 39955418 ns in average
.....
Appendix: Additional information
================================
For more information on the release, please see the release notes in
sub-directory release/iccs32v234/release_notes, the ICC User Manual in
release/iccs32v234/doc and the ICC section from the Linux BSP User Manual.