/rerocc

Primary LanguageScala

ReRoCC: Remote RoCC Extension for RoCC-enabled RISC-V Cores

The Remote RoCC (ReRoCC) extensions for RoCC-enabled RISC-V cores enables physical disaggregation of RoCC accelerators from control cores. To simplify adoption and exploration, ReRoCC is backwards compatible with existing RoCC accelerator implementations and software stacks. ReRoCC provides a path towards supporting "accelerator virtualization" on many-accelerator architectures, where threads can contend for and share limited physical accelerator resources without requiring user-level software modifications.

ReRoCC Accelerator Client

The accelerator client attaches to existing RISC-V cores to enable them to use remote ReRoCC accelerators. The client is itself a RoCC "accelerator", and can plug-in to existing cores like Rocket, (TODO) BOOM, and (TODO) Shuttle.

The client is responsible for tracking which physical accelerators are acquired by the attached hart, forwarding accelerator instructions, and maintaining shadowed architectural state on the remote accelerators.

ReRoCC Accelerator Manager

The accelerator manager wraps an existing RoCC accelerator implementation, providing the standard RoCC interface to the accelerator. The manager implements shadow copies of critical core architectural state, including the mstatus and ptbr registers, as well as a shadow page-table-walker, for the accelerator to access.

ReRoCC RISC-V Architectural Extensions

ReRoCC adds new user-level CSRs to the host CPU, which manage the availability of physical accelerators for the CPU thread.

The rrcfg registers track an "acquired" accelerator available for the hart to access. Implementations may support up to 16 rrcfg registers. Each rrcfg register contains a mgr field, which can be set to the physical ID of a remote RoCC accelerator manager, and a acq field, which, if set, indicates that the accelerator is available to access.

The rropc registers assign rrcfg registers to the four available RoCC custom opcodes. When a rropcX register is set to the index of a rrcfg register with the acq bit set, then instructions with opcode customX will be send to that accelerator.

The rrbar register is currently used to assist in fencing accelerator memory accesses. This will be removed in the future.

CSR Id Name
0x800-0x803 rropc0-rropc3
0x804 rrbar
0x810-0x81f rrcfg0-rrcfg15
ReRoCC CSR Field Description
rropcX[3:0] Index of rrcfg opcode customX is mapped to
rrbar[3:0] When written, indicates that the next memory fence should apply to accelerator managed by the indexed rrcfg
rrcfgX[7:0] Index of physical accelerator this register is mapped to
rrcfgX[8] If set, indicates this register should attempt to "acquire" the corresponding physical accelerator. Reading this bit returns whether acquisition was successful

ReRoCC Messaging Protocol

The ReRoCC messaging protocol is a simple, in-order, non-blocking, dual-channel packetized messaging protocol to facilitate communication and synchronization between accelerator clients and managers. A request channel is for client-to-manager traffic, while a response channel implements manager-to-client traffic. Both channels use the same message encoding.

The protocol can be implemented by a crossbar or NoC-based interconnect. The protocol implements a 64-bit data channel with multi-beat message types.

ReRoCC Message Fields

Field Width Description
opcode 3 Type of the message. See the opcode table.
client_id ? ID of the rrcfg register on one end of the message. Managers see unique IDs per rrcfg on a multi-hart system
manager_id ? ID of the physical manager on one end of the message.
data 64 See opcode table

Request Channel Opcodes/Data encodings

Opcode Value Total Beats Beat data[63:0]
mAcquire 0b000 1 0 Ignored
mInst 0b001 1-3 0 32-bit custom instruction
1 (If requested) rs1 operand data
2 (If requested) rs2 operand data
mUStatus 0b010 2 0 New csr_mstatus[63:0]
1 New csr_mstatus[127:64]
mUPtbr 0b011 1 0 New csr_ptbr
mRelease 0b100 1 0 Ignored
mUnbusy 0b101 1 0 Ignored

Response Channel Opcodes/Data encodings

Opcode Value Total Beats Beat data[63:0]
sAcqResp 0b000 1 0 1 if successful acquire, else 0
sInstAck 0b001 1 0 Ignored
sWrite 0b010 2 0 64-bit register write data
1 64-bit architectural register destination
sRelResp 0b011 1 0 Ignored
sUnbusyAck 0b100 1 0 Ignored

Message Flows

Request Message Response Message Description
mAcquire mAcqResp Attempt to acquire a ReRoCC-managed accelerator, manager responds whether acquire was successful
mInst mInstAck Sends an instruction stream to the manager, credit-based flow-control with Inst/InstAck
mUStatus Updates shadowed MStatus on the manager
mUPtbr Updates shadowed satp on the manager
mRelease mRelResp Relinquishes an attached accelerator
mUnbusy mUnbusyAck Wait for accelerator to unbusy (TODO: Remove)

Status

ReRoCC is currently under development. A brief list of TODOs:

  • Support BOOM/Shuttle cores, these cores need to support RoCC-provided CSRs
  • Upstream necessary rocket-chip APIs
  • Support supervisor-level accelerator management
  • Remove need for explicit rerocc memory fence via rrbar csr.