/aws-fpga

Official repository of the AWS EC2 FPGA Hardware and Software Development Kit

Primary LanguageVHDLOtherNOASSERTION

Design Portability Between F1 and Alveo U200 Release Note

Xilinx and AWS developers can now seamlessly migrate between Alveo and F1 platforms to enable scaling in AWS cloud. The migration is now supported for Vitis flow and full custom RTL flow. A customer shared the following anecdote, highlighting the importance of enabling vivado design portability between Alveo and F1 development environments

“Having a basic XDMA interface that allows us to switch between U200, U250 and AWS F1 would make life very easy for us”.

Based on the customer feedback, AWS released F1.A.1.4 shell for Alveo U200 on github and Xilinx Alveo U200 page to enable customers migrate designs seamlessly between a U200 Alveo card and F1 platform in a full custom RTL development flow. This gives choice to the customers to use either a Vitis flow or a full custom RTL/Vivado design flow and seamlessly migrate designs between Alveo U200 and F1. The F1.A.1.4 shell is fully interface compatible and footprint compatible with AWS F1.X.1.4 shell and no changes to custom logic design are required. Similar to the existing F1 shells, the F1.A.1.4 shell provides all the required communication interfaces to the custom logic. This helps customers with the effort required to implement their own interfaces for Alveo U200 platform and also provides them with a seamless migration path to F1 in the custom RTL development flow.

Please refer to the getting started README guide for more details and contact on aws-fpga-f1-u200 github issues for further questions/support.

Table of Contents

  1. Overview of AWS EC2 FPGA Development Kit
  2. Amazon EC2 F1 platform features
  3. Getting Started
  4. Documentation Overview
  5. Developer Support

Overview of AWS EC2 FPGA Development Kit

AWS EC2 FPGA Development Kit is a set of development and runtime tools to develop, simulate, debug, compile and run hardware accelerated applications on Amazon EC2 F1 instances. It is distributed between this github repository and FPGA Developer AMI - Centos/AL2 provided by AWS with no cost of development tools.

⚠️ NOTE: The developer kit is supported for Linux operating systems only.

Development Flow

After creating an FPGA design (also called CL - Custom logic), developers can create an Amazon FPGA Image (AFI) and easily deploy it to an F1 instance. AFIs are reusable, shareable and can be deployed in a scalable and secure way.

Alt text

Development Environments

Development Environment Description Accelerator Language Hardware Interface Debug Options Typical Developer
Software Defined Accelerator Development using Vitis/SDAccel Development experience leverages an optimized compiler to allow easy new accelerator development or migration of existing C/C++/openCL, Verilog/VHDL to AWS FPGA instances C/C++/OpenCL, Verilog/VHDL (RTL) OpenCL APIs and XRT SW/HW Emulation, Simulation, GDB, Virtual JTAG (Chipscope) SW or HW Developer with zero FPGA experience
Hardware Accelerator Development using Vivado Fully custom hardware development experience provides hardware developers with the tools required for developing AFIs for AWS FPGA instances Verilog/VHDL XDMA Driver, peek/poke Simulation, Virtual JTAG HW Developer with advanced FPGA experience
IP Integrator/High Level Design(HLx) using Vivado Graphical interface development experience for integrating IP and high level synthesis development Verilog/VHDL/C XDMA Driver, peek/poke Simulation, Virtual JTAG HW Developer with intermediate FPGA experience

For on-premise development, SDAccel/Vitis/Vivado must have the correct license and use one of the supported tool versions.

FPGA Developer AMI

The FPGA Developer AMI is available on the AWS marketplace without a software charge and includes tools needed for developing FPGA Designs to run on AWS F1.

Given the large size of the FPGA used inside AWS F1 Instances, Xilinx tools work best with 32GiB Memory. z1d.xlarge/c5.4xlarge and z1d.2xlarge/c5.8xlarge instance types would provide the fastest execution time with 30GiB+ and 60GiB+ of memory respectively. Developers who want to save on cost, could start coding and run simulations on low-cost instances, like t2.2xlarge, and move to the aforementioned larger instances to run the synthesis of their acceleration code.

AWS marketplace offers multiple versions of the FPGA Developer AMI. The following section table describes the mapping of currently supported developer kit versions to AMI versions.

Xilinx tool support

Developer Kit Version Tool Version Supported Compatible FPGA Developer AMI Version
1.4.23+ 2021.2 v1.12.X (Xilinx Vivado/Vitis 2021.2)
1.4.21+ 2021.1 v1.11.X (Xilinx Vivado/Vitis 2021.1)
1.4.18+ 2020.2 v1.10.X (Xilinx Vivado/Vitis 2020.2)
1.4.16+ 2020.1 v1.9.0-v1.9.X (Xilinx Vivado/Vitis 2020.1)
1.4.13+ 2019.2 v1.8.0-v1.8.X (Xilinx Vivado/Vitis 2019.2)
1.4.11+ 2019.1 v1.7.0-v1.7.X (Xilinx Vivado/SDx 2019.1)
1.4.8 - 1.4.15b 2018.3 v1.6.0-v1.6.X (Xilinx Vivado/SDx 2018.3)
1.4.3 - 1.4.15b 2018.2 v1.5.0-v1.5.X (Xilinx Vivado/SDx 2018.2)
⚠️ 1.3.7 - 1.4.15b 2017.4 v1.4.0-v1.4.X (Xilinx Vivado/SDx 2017.4) ⚠️

⚠️ Developer kit release v1.4.16 will remove support for Xilinx 2017.4, 2018.2, 2018.3 toolsets. While developer kit release v1.4.16 onwards will not support older Xilinx tools, you can still use them using HDK releases v1.4.15b or earlier. Please check out the latest v1.4.15b release tag from Github to use Xilinx 2017.4, 2018.2, 2018.3 toolsets.

For deprecation notices, please check the End of life announces

For software-defined development please look at the runtime compatibility table based on the Xilinx toolset in use: SDAccel or Vitis

End of life Announcements

Xilinx Tool version State Statement
2017.1 🚫 Deprecated on 09/01/2018 Developer kit versions prior to v1.3.7 and Developer AMI prior to v1.4 (2017.1) reached end-of-life.
2017.4 🚫 Deprecated on 12/31/2021 Support for Xilinx 2017.4 toolsets was deprecated on 12/31/2021.

Hardware Development Kit (HDK)

The HDK directory contains documentation, examples, simulation, build and AFI creation scripts to start building Amazon FPGA Images (AFI).
The HDK can be installed on any on-premises server or an EC2 instance. The developer kit is not required if you plan to use a pre-built AFI shared from another developer.

AWS Shells

With Amazon EC2 FPGA instances, each FPGA is divided into two partitions:

  • Shell (SH) – AWS platform logic implementing the FPGA external peripherals, PCIe, DRAM, and Interrupts.

  • Custom Logic (CL) – Custom acceleration logic created by an FPGA Developer.

At the end of the development process, combining the Shell and CL creates an Amazon FPGA Image (AFI) that can be loaded onto the Amazon EC2 FPGA Instances.

The following table provides the shells currently available to develop your CL with. Each shell provides specific interfaces and features and currently needs to be used with the Dev Kit branch listed in the table.

Shell Name Shell Version Dev Kit Branch Description
F1 XDMA Shell F1.X.1.4 master Provides all the interfaces listed here, includes DMA
F1 Small Shell F1.S.1.0 small_shell Provides all the interfaces listed here. This shell does not include DMA engine and provides significant reduction in Shell resource usage.

For more details, check the FAQ

Software-defined Development Environment

The software-defined development environment allows customers to compile their C/C++/OpenCL code into the FPGA as kernels, and use OpenCL APIs to pass data to the FPGA. Software developers with no FPGA experience will find a familiar development experience that supercharges cloud applications.

In addition, this development environment allows for a mix of C/C++ and RTL accelerator designs into a C/C++ software based development environment. This method enables faster prototyping using C/C++ while supporting manual optimization of critical blocks within RTL. This approach is similar to optimizing time critical functions using software compiler optimization methods.

To get started with Xilinx SDAccel, review the Software-defined development environment readme. To get started with Xilinx Vitis, review the Vitis unified development environment readme.

Runtime Tools (SDK)

The SDK directory includes the runtime environment required to run on EC2 FPGA instances. It includes the drivers and tools to manage the AFIs that are loaded on the FPGA instance. The SDK isn't required during the AFI development process; it is only required once an AFI is loaded onto an EC2 FPGA instance. The following sdk resources are provided:

  • Linux Kernel Drivers - The developer kit includes three drivers:
    • XDMA Driver - DMA interface to/from HDK accelerators.
  • FPGA Libraries - APIs used by C/C++ host applications.
  • FPGA Management Tools - AFI management APIs for runtime loading/clearing FPGA image, gathering metrics and debug interface on the F1 instance.

Amazon EC2 F1 Platform Features

  • 1-8 Xilinx UltraScale+ VU9P based FPGA slots
  • Per FPGA Slot, Interfaces available for Custom Logic(CL):
    • One x16 PCIe Gen 3 Interface
    • Four DDR4 RDIMM interfaces (72-bit with ECC, 16 GiB each; 64 GiB total)
    • AXI4 protocol support on all interfaces
  • User-defined clock frequency driving all CL to Shell interfaces
  • Multiple free running auxiliary clocks
  • PCI-E endpoint presentation to Custom Logic(CL)
    • Management PF (physical function)
    • Application PF
  • Virtual JTAG, Virtual LED, Virtual DIP Switches
  • PCI-E interface between Shell(SH) and Custom Logic(CL).
    • SH to CL inbound 512-bit AXI4 interface
    • CL to SH outbound 512-bit AXI4 interface
    • Multiple 32-bit AXI-Lite buses for register access, mapped to different PCIe BARs
    • Maximum payload size set by the Shell
    • Maximum read request size set by the Shell
    • AXI4 error handling
  • DDR interface between SH and CL
    • CL to SH 512-bit AXI4 interface
    • 1 DDR controller implemented in the SH (always available)
    • 3 DDR controllers implemented in the CL (configurable number of implemented controllers allowed)

Getting Started

Getting familiar with AWS

If you have never used AWS before, we recommend you start with AWS getting started training, and focus on the basics of the AWS EC2 and AWS S3 services. Understanding the fundamentals of these services will make it easier to work with AWS F1 and the FPGA Developer Kit.

FPGA Image generation and EC2 F1 instances are supported in the us-east-1 (N. Virginia), us-west-2 (Oregon), eu-west-1 (Ireland) and us-gov-west-1 (GovCloud US) regions.

⚠️ NOTE: By default, your AWS Account will have an EC2 F1 Instance launch limit of 0. Before using F1 instances, you will have to open a Support Case to increase the EC2 Instance limits to allow launching F1 instances.

Setting up development environment for the first time

You have the choice to develop on AWS EC2 using the FPGA Developer AMI or on-premise.

ℹ️ INFO: We suggest starting with the FPGA Developer AMI with build instances on EC2 as it has Xilinx tools and licenses setup for you to be able to quickly get into development.

ℹ️ INFO: For on-premise development, you will need to have Xilinx tools and licenses available for you to use

  1. Start a Build Instance first to start your development.

    💡 TIP: This instance does not have to be an F1 instance. You only require an F1 instance to run your AFI's(Amazon FPGA Image) once you have gone through your design build and AFI creation steps.

    ℹ️ INFO: If you need to follow GUI Development flows, please checkout our Developer Resources where we provide Step-By-Step guides to setting up a GUI Desktop.

  2. Clone the FPGA Developer Kit on your instance. git clone https://github.com/aws/aws-fpga.git

  3. Follow the quickstarts from the next section.

Quickstarts

Before you create your own AWS FPGA design, we recommend that you go through one of the step-by-step Quickstart guides:

Description Quickstart Next Steps
Software Defined Accelerator Development using Xilinx Vitis Vitis hello_world Quickstart 60+ Vitis examples, Vitis Library Examples
Software Defined Accelerator Development using Xilinx SDAccel SDAccel hello_world Quickstart 60+ SDAccel examples
Custom Hardware Development(HDK) HDK hello_world Quickstart CL to Shell and DRAM connectivity example, Virtual Ethernet Application using the Streaming Data Engine
IP Integrator/High Level Design(HLx) IPI hello_world Quickstart IPI GUI Examples

ℹ️ INFO: For more in-depth applications and examples of using High level synthesis, Vitis Libraries, App Notes and Workshops, please refer to our Example List

How Tos

How To Description
Migrate Alveo U200 designs to F1 This application note shows the ease of migrating an Alveo U200 design to F1.

Documentation Overview

Documentation is located throughout this developer kit and the table below consolidates a list of key documents to help developers find information:

Topic Document Name Description
AWS setup Setup AWS CLI and S3 Bucket Setup instructions for preparing for AFI creation
Developer Kit RELEASE NOTES, Errata Release notes and Errata for all developer kit features, excluding the shell
Developer Kit Errata Errata for all developer kit features, excluding the shell
F1 Shell AWS Shell RELEASE NOTES Release notes for F1 shell
F1 Shell AWS Shell ERRATA Errata for F1 shell
F1 Shell AWS Shell Interface Specification Shell-CL interface specification for HDK developers building AFI
F1 Shell - Timeout and AXI Protocol Protection How to detect a shell timeout The shell will terminate transactions after a time period or on an illegal transaction. This describes how to detect and gather data to help debug CL issues caused by timeouts.
Vitis Debug Vitis Kernel Instructions on debugging Vitis Kernel
Vitis Create Runtime AMI Instructions on creating a runtime AMI when using Xilinx Vitis
Vitis XRT Instructions Instructions on building, installing XRT with MPD daemon considerations for F1
SDAccel Debug RTL Kernel Instructions on debugging RTL Kernel with SDAccel
SDAccel Create Runtime AMI Instructions on creating a runtime AMI when using Xilinx SDAccel
HDK - Host Application Programmer View Host application to CL interface specification
HDK - CL Debug Debug using Virtual JTAG Debugging CL using Virtual JTAG (Chipscope)
HDK - Simulation Simulating CL Designs Shell-CL simulation specification
HDK - Driver README Describes the DMA driver (XDMA) used by HDK examples and includes a link to an installation guide
AFI AFI Management SDK CLI documentation for managing AFI on the F1 instance
AFI - EC2 CLI copy_fpga_image, delete_fpga_image, describe_fpga_images, fpga_image_attributes CLI documentation for administering AFIs
AFI - Creation Error Codes create_fpga_image_error_codes CLI documentation for managing AFIs
AFI - Power FPGA Power, recovering from clock gating Helps developers with understanding FPGA power usage, preventing power violations on the F1 instance and recovering from a clock gated slot.
On-premise Development Tools, Licenses required for on-premise development Guidance for developer wanting to develop AFIs from on-premises instead of using the FPGA Developer AMI
Frequently asked questions FAQ Q/A are added based on developer feedback and common AWS forum questions

Developer Support

  • AWS Re:Post is the first place to go to post questions, learn from other users, to engage with the FPGA developer community, AWS and Xilinx engineers to get help.

  • You could also file a Github Issue for support. We prefer AWS Re:Post as this helps the entire community learn from issues, feedback and answers.

    • Click the "Watch" button in GitHub upper right corner to get regular updates.