Some errors when simulating DDR burst reads

[kkenshin1]

Hi,

When I design my own afu module, I modify the platform config to set "uClk_usr" as the clock in "local memory" and set "pClk" as the clock in "cci-p", here is my platform config:

"class": "ccip_std_afu_avalon_mm",
"module-ports" :
    [
        {
            "class": "cci-p",
            "params":
                {
                    "clock": "pClk"
                }
        },
        {
             "class": "local-memory",
             "params":
                  {
                      "clock": "uClk_usr"
                  }
         }
     ]

The afu top module is ccip_std_afu() and there is two ddr bank interface avalon_mem_if.to_fiu local_mem[NUM_LOCAL_MEM_BANKS]

When my afu module sends DDR read requests continuously in simulation，such as sending read requests of different addresses for 20 consecutive cycles, that means the ddr single local_mem[0].read asserts for 20 cycles, the ddr single local_mem[0].waitrequest in ccip_std_afu() does not assert, but there is some warning shown in ASE. The warning is as follows.

#15290625: WARNING: ase_top.ase_top_generic.local_mem_model.b_emul[0].emif_ddr4.avs_bfm_inst_ddra: Pipelined read commands 10 > AV_MAX_PENDING_READS 9
#15294375: WARNING: ase_top.ase_top_generic.local_mem_model.b_emul[0].emif_ddr4.avs_bfm_inst_ddra: Pipelined read commands 10 > AV_MAX_PENDING_READS 9
#15298125: WARNING: ase_top.ase_top_generic.local_mem_model.b_emul[0].emif_ddr4.avs_bfm_inst_ddra: Pipelined read commands 10 > AV_MAX_PENDING_READS 9
#15301875: WARNING: ase_top.ase_top_generic.local_mem_model.b_emul[0].emif_ddr4.avs_bfm_inst_ddra: Pipelined read commands 10 > AV_MAX_PENDING_READS 9
#15305625: WARNING: ase_top.ase_top_generic.local_mem_model.b_emul[0].emif_ddr4.avs_bfm_inst_ddra: Pipelined read commands 11 > AV_MAX_PENDING_READS 9
#15309375: WARNING: ase_top.ase_top_generic.local_mem_model.b_emul[0].emif_ddr4.avs_bfm_inst_ddra: Pipelined read commands 12 > AV_MAX_PENDING_READS 9

I looked at the platform_shim_avalon_mem_if.sv code that the afu module is connected, I found that mem_async_shim module is generated due to the modification of the local memory clock (change to uClk_usr).

In mem_async_shim module (in avalon_mem_if_async_shim.sv), platform_utils_avalon_mm_clock_crossing_bridge module is instantiated and COMMAND_FIFO_DEPTH is set to 128. I guess that there is a FIFO in platform_utils_avalon_mm_clock_crossing_bridge and it can store 128 command, so the ddr single local_mem[0].waitrequest does not assert.

But from the warning of ASE, the ddr_bfm can only deal with 9 commands. I found the emif_ddr4.sv file and the AV_MAX_PENDING_READS parameter of altera_avalon_mm_slave_bfm module is set to 9.

Now, I use OPAE-1-2-1 and Intel PAC with Arria10 FPGA.

I think these are all errors caused by modifying the clock of "local memory". However my design must modify this clock, how should these problems be solved?

And, Do these warnings affect the correctness of data when continuously sending DDR read and write requests?
What should I do to get rid of the warning in ASE about continuously sending DDR read requests?

[michael-adler]

You have declared CCI-P and local memory to be in different clock domains. Do you have logic in the pClk domain that is sending command to local memory? Are there paths between the PCIe I/O and local memory I/O? Have you instantiated your own clock crossing?

If you are simply sending reads and writes to local memory driven by pClk then the commands will not be received properly and the flow control mechanism will not work.

[kkenshin1]

Thanks for your reply. There is no path between the PCIe I/O and local memory I/O and all commands are sent to local memory is in the local memory clock domain.

I use EMIF_MODEL_BASIC in simulation Makefile to select the simulation model for local memory. And I change the paremeter AV_MAX_PENDING_READS (in build_sim/rtl/device_models/dcp_emif_model_basic/emif_ddr4.sv) from 9 to 128 to solve this problem. Now there is no warning when simulation.

However, when I use EMIF_MODEL_ADVANCED in sumulation Makefile, there is an error as below.

Error: (vsim-3033) Instantiation of 'twentynm_io_aux_encrypted' failed. The design unit was not found
Time: 0 ps  Iteration: 0  Instance: /ase_top/ase_top_generic/local_mem_model/b_emul[0]/emif_ddr4/ddr4b/ddr4b/arch/arch_inst/io_aux_inst/io_aux File: /opt/intelFPGA_pro/quartu
s_19.2.0b57/quartus/eda/sim_lib/twentynm_atoms.v Line: 6229#         Searched libraries:
#            ***/build_sim/work/verilog_libs/twentynm_hssi_ver
#            ***/build_sim/work/verilog_libs/altera_mf_ver
#            ***/build_sim/work/verilog_libs/twentynm_hip_ver
#            ***/build_sim/work/verilog_libs/altera_ver
#            ***/build_sim/work/verilog_libs/twentynm_ver
#            ***/build_sim/work/verilog_libs/altera_lnsim_ver
#            ***/build_sim/work/verilog_libs/lpm_ver
#            ***/build_sim/work/verilog_libs/sgate_ver
#            ***/build_sim/work/work

My software environment: OPAE-1-2-1, Quartus-19.2, ModelSim-Intel FPGA Edition

It seems that I am missing some files in /quartus/eda/sim_lib/twentynm_atoms.v.

Is there any solutions to solve this problem? Maybe I use the wrong file?