More than one action on the same FPGA card
Closed this issue · 10 comments
How it would be possible to have more than one action on the same FPGA card?
Hi, today it supports only one action. Because the snap_core bridge (PSL-to-AXI) only provides one config path (AXI Lite) and one data path (AXI MM) to the user logic (action_wrapper). For multiple actions, or multiple functional units, I suggest to instantiate Vivado AXI interconnect and Vivado AXI crossbar to connect multiple actions -- more precisely, here we call them kernels. They still stay in one action_wrapper.
Thanks Abbas for your interest and your question.
Let me add to the previous answer the following since I don't know what level of information you are looking for:
SNAP's architecture has been built to support multiple actions in the FPGA. Each action can be connected to a different process. Therefore there is a job manager on both sides managing the job queues.
snap_maint command will discover and list the different actions embedded in a FPGA board and this list will then be used by the snap job manager to assign a process to an action.
We have started implementing this multi-action but saw no real application that could use it so we lowered the priority of this development. Implementing larger actions which has more more throughput was the choice we made.
Hi, today it supports only one action. Because the snap_core bridge (PSL-to-AXI) only provides one config path (AXI Lite) and one data path (AXI MM) to the user logic (action_wrapper). For multiple actions, or multiple functional units, I suggest to instantiate Vivado AXI interconnect and Vivado AXI crossbar to connect multiple actions -- more precisely, here we call them kernels. They still stay in one action_wrapper.
This way still there is one action_start and idle in snap application and hence each functional unit in FPGA should be handled manually inside the FPGA maybe by looking at some MMIO flags which are set by application. in other words, snap_action_sync_execute_job function can not be used and low level programing like what is done in hdl_example using action_write function should be applied.
Thanks Abbas for your interest and your question.
Let me add to the previous answer the following since I don't know what level of information you are looking for:
SNAP's architecture has been built to support multiple actions in the FPGA. Each action can be connected to a different process. Therefore there is a job manager on both sides managing the job queues.
snap_maintcommand will discover and list the different actions embedded in a FPGA board and this list will then be used by the snap job manager to assign a process to an action.We have started implementing this multi-action but saw no real application that could use it so we lowered the priority of this development. Implementing larger actions which has more more throughput was the choice we made.
By reading this document "https://github.com/open-power/snap/blob/master/hardware/doc/SNAP-Registers.md" and looking at the snap_maint code, it is inferred that up to 16 actions are supported in each card and snap_maint automatically discover all of them by reading SSR register.
As far as I understood the axi_lite from FPGA side has only access to the last 4KB of the slave context space which is MMIO-Map Action Space, so there is no way to set SSR from FPGA side. It was unclear for me so I asked it here.
Thanks for replying.
Hi Abbas,
You are right the axi_lite of the action has only access to the last 4KB of the slave context space. In other words, every action will have a 16B header used mainly for the flags and the retc and a 108B area for action registers used for the configuration of the action. This size is fixed on purpose. If user needs more than 108B, he can use a pointer to any area in host memory.
snap_maint is a command that must be executed prior to start any action. It will send to a request to every action one after the other (if any) to get their ID and release number.
Action will answer (hls_helloworld example)
Action_Config->action_type = HELLOWORLD_ACTION_TYPE; //TO BE ADAPTED
Action_Config->release_level = RELEASE_LEVEL;
SNAP will assign to every Action_ID a Short_ID that will be used by the job manager.
Short | Action Type | Level | Action Name
------+--------------+-----------+------------
0 0x10141000 0x00000023 IBM HLS Memcopy
1 0x10141000 0x00000023 IBM HLS Memcopy
2 0x10141002 0x00000023 IBM HDL Hello World
The Action Name will be displayed using the https://github.com/open-power/snap/blob/master/ActionTypes.md file.
Let me know if this answers your question
Hi Abbas,
You are right the axi_lite of the action has only access to the last 4KB of the slave context space. In other words, every action will have a 16B header used mainly for the flags and the retc and a 108B area for action registers used for the configuration of the action. This size is fixed on purpose. If user needs more than 108B, he can use a pointer to any area in host memory.
snap_maintis a command that must be executed prior to start any action. It will send to a request to every action one after the other (if any) to get their ID and release number.
Action will answer (hls_helloworld example)Action_Config->action_type = HELLOWORLD_ACTION_TYPE; //TO BE ADAPTED Action_Config->release_level = RELEASE_LEVEL;SNAP will assign to every Action_ID a Short_ID that will be used by the job manager.
Short | Action Type | Level | Action Name
------+--------------+-----------+------------
0 0x10141000 0x00000023 IBM HLS Memcopy
1 0x10141000 0x00000023 IBM HLS Memcopy
2 0x10141002 0x00000023 IBM HDL Hello World
The Action Name will be displayed using the https://github.com/open-power/snap/blob/master/ActionTypes.md file.
Let me know if this answers your question
Thank you Bruno, but currently snap_maint -C 0 does not show more than one action because we could only have one action on C0. correct?
Short | Action Type | Level | Action Name
------+--------------+-----------+------------
0 0x10141000 0x00000023 IBM HLS Memcopy
Let me explain a little bit how the exploration phase works.
In that phase LIBSNAP tries to access the (potentially 16) implemented actions via the master address space where above address 0y10000 16 times 4KB address space is reserved for the actions.
The actions should be written such that register 0x10 (i.e., 0x10010, 0x11010, 0x12010, ... 0x1F010) should contain the action type (see Action Register Layout).
LIBSNAP is constructing a one to one relation between the found action types and short action types (LIBSNAP is simply enumerating the different action types it found). The result might be less short action types than actually implemented actions (in case the same action type is implemented multiple times).
So, you would not see a table like the one Bruno did show above.
LIBSNAP will then for each implemented action write the short action type into the 16 Action Type Registers (ATRi) 0x100 - 0x178 within the master address space.
At the end of the exploration phase LIBSNAP is completing that phase by issuing an exploration done cmd via the SNAP Command Register (SCR). With that command the Maximum Short Action ID is specified which from then on can be read via the SSR (you'll get the same value via each of the slave or master address spaces).
This is how the process should work for multiple actions. But, we never implemented multiple actions. Therefore, this process was never tested, and I am not sure, how far that process got implemented in LIBSNAP.
Thank you Bruno, but currently snap_maint -C 0 does not show more than one action because we could only have one action on C0. correct?
This is correct. Only one action is possible as of today so you will never see more than 1 action. I displayed the case you could have multiple actions but this is not implemented today in code.
Thanks all.