foool/opae-xilinx

OPAE porting to Xilinx FPGA devices.

Table of Contents

1. Overview of opae-xilinx
   • Original OPAE
   • Ported OPAE
     • FIM and AFU design
     • Using AXI instead of CCI-P
     • Minimum FIM design
   • Demo: Full-System Simulation with QEMU (On virtual machine)
   • Demo: Compatability with Linux dfl-pci driver (On physical machine)
   • Sample Applications: Xilinx CDMA
   • Directory Structure
   • Development Tools
2. Getting Started

Overview of opae-xilinx

The purpose of this project is to port OPAE to Xilinx FPGA devices.

The original OPAE and FPGA accelerator:

The ported OPAE and FPGA accelerator:

FIM and AFU design

To be discoverable and managable by the OPAE PCIe driver, the design of FIU(FPGA Interface Unit, the PCIe interface logic) should be compliant with the OPAE specification. So a major design effort is put into the FIM(FPGA Interface Manager) part, which is a static 'Shell' that resides persistantly in the FPGA.

Based on the 'Shell', a design flow(Partial Reconfiguration) of the AFU(Accelerator Function Unit) part, which is a dynamic 'Role' that can be swapped in and out of the FPGA, is setup to utilize open-source projects supporting HLS, such as FINN, VTA, BISMO, GEMM_HLS, hls4ml, HeteroCL, hlslib, pp4fpgas and Vitis Library L1 modules.

Using AXI instead of CCI-P

Although the OPAE specification mandates the use of CCI-P interface between FIM and AFU when targeting Intel MCP and DCP platforms, this project uses AXI interface instead. The inclusion of an industry standard interface makes the OPAE ecosystem truly Vendor Neutral and makes the many IPs targeting ASIC designs available to FPGA designers, such as MatchLib and HLSLibs.

Minimum FIM design

The FIM design is kept to a minimum. On KCU105 platform, over 91% logic area is reserved for the AFU. That is 458k LCs for the accelerator logic.

Demo: Full-System Simulation with QEMU (Tested with XSIM on virtual machine)

A full-system simulation, which involves application/driver software code and FIM/AFU hardware logic, not only speeds up the development and debugging process of the SW/HW interface, but also enables the evaluation of this full-stack solution without a physical FPGA acceleration card. Please take a look at the README for details.

OPAE-scan results in QEMU-HDL co-simulation:

OPAE-scan AXI-bus transaction waveform in QEMU-HDL co-simulation:

Demo: Compatibility with Linux dfl-pci driver (Tested with U50 on physical machine)

"FPGA Device Feature List(DFL)" framework is involved in Linux 4.19 kernel in 2018. On Ubuntu 18.04.4 LTS with kernel version 5.3, the FIM can be successfully identified and the AFU can be enumerated.

DFL enumeration result

OPAE Sample Applications: Hello_FPGA with Xilinx CDMA

Application of Xilinx CDMA IP in C.

Application of Xilinx CDMA IP in Python.

The above sample applications have been validated in the QEMU-HDL co-simulation environment.

Docker script to setup an OPAE run-time environment

Build OPAE 1.3.0 on Ubuntu 16.04 from source code.

Build OPAE 1.4.0 on Ubuntu 18.04 from source code.

The original script is here. Thank you so much akirajoeshoji, for the inspiration!

Directory Structure

.
├── doc
│   └── pics
├── hw
│   ├── prj
│   │   ├── afu
│   │   ├── blue_bs
│   │   ├── fim
│   │   ├── green_bs
│   │   └── qemu_hdl_cosim
│   └── src
│       ├── afu_customize
│       ├── boardrepo
│       ├── constraints
│       ├── hdl
│       ├── ip
│       ├── ipi
│       ├── qemu_hdl_cosim
│       └── sim
└── sw
    ├── Docker
    │   ├── ubuntu1604
    │   └── ubuntu1804
    ├── FME_xilinx_driver
    │   ├── app_pcie_hwicap
    │   ├── app_pcie_qspi
    │   └── drv_pcie_hwicap
    ├── OPAE
    │   ├── driver
    │   └── sdk
    ├── pcimem
    └── QEMU
        └── qemu_hdl_cosim

Development Tools

The FPGA projects are designed with 2018.3 release of Vivado and Vivado HLS.

The FPGA platforms currently supported are Xilinx Kintex UltraScale FPGA KCU105 Evaluation Kit and Alveo U50 Data Center Accelerator Card from Xilinx.

Getting Started

To get started with the design of FIM and AFU, or the generation of Blue and Green bitstreams, follow the README in ecah of the directories under ./hw/prj.

ToDo List

1. [Done] Add Container scripts to install OPAE driver/sdk/pyopae - Clean environment.
2. Use Verilator/GtkWave in the QEMU-HDL cosimulation - Truely open source tools based; Mixed C/Verilog simulation.
3. [Done] Port Xilinx PR driver to OPAE - FME functionality enhancement.
4. [Done] Add Ethernet interface to the FIM - FIM functionality enhancement/AFU BBB optional component logic.
5. Add DDR interface to the FIM - FIM functionality enhancement/AFU BBB optional component logic.
6. [Done] Add AFU BBB logic components and compilation flow scripts - A synthesis flow for automatic HLS IP integration.
7. [Done] Optimize FIM pyhsical constraints - Available area estimation for AFU.
8. Test compliance with IFPGA Rawdev Driver

Wish List

1. Support SR-IOV.
2. Support dual PF.
3. Add AFU BBB logic components for supporting OpenCL/SyCL/oneAPI - ?.
4. PYNQ/XRT compliant - ?.
5. VirtIO NIC compatible with ixy user space driver - Network attached accelerator (SmartNIC).
6. Add a RISC-V Core as an AXI-Lite master in the FIM - Autonomous task scheduling and hardware microservice.