/dlm

Distributed locker manager

Primary LanguageVerilog

Hardware Transaction Processing for Multi-channel Memory Node

This version Upgrade SpinalHDL 1.7.3 - works well in WrapNodeNetSim & SysCoyote1T2N1C8PGen

The memory node in cloud computing usually has parallel memory channels and memory interfaces with unified data access. Building a management layer on top of the parallel interfaces is not trivial to ensure data integrity (e.g., atomicity). The concept of transaction origins from the database domain has been applied as a general memory semantic. However, software-based transaction processing suffers from a high cost of concurrency control (e.g., lock management). Building a hardware transaction processing layer for the multi-channel memory node is valuable. Our target is to run the hardware as a background daemon and use the expensive memory bandwidth for data access in transactions. The project provides transaction processing hardware with three concurrency control schemes (No wait 2PL, Bounded wait 2PL, and Timestamp OCC). It has been prototyped with Coyote (an FPGA OS) on Alveo FPGA cluster with HBM. The transaction tasks are injected via either TCP or PCIe from the clients.

.
├── src/main/             # source files
│   ├── lib/              # external lib
│   │   ├── HashTable.    # 
│   │   └── LinkedList.   # 
│   └── scala/            # design with SpinalHDL
│       └── hwsys/        # hwsys lib
│           ├── coyote    # interface and datatype to coyote
│           ├── dlm/      # distributed lock manager
│           │   └── test  # testbench of dlm
│           ├── sim       # helper function for testbench
│           └── util      # hardware utilities
├── build.sbt             # sbt project
└── build.sc              # mill project

Install Software

On Red Hat 9:

  • tcl sudo yum install ./tcl-8.5.13-8.el7.x86_64.rpm
  • tk sudo yum install ./tk-8.5.13-6.el7.x86_64.rpm
  • gtkwave: sudo yum install gtkwave

On Windows

Config enviroment variables:

  • VERILATOR_ROOT: C:\msys64\ucrt64\share\verilator + C:\msys64\ucrt64
  • PATH: + C:\msys64\ucrt64\bin + C:\msys64\usr\bin
  • JAVA_HOME: auto configed by java installation in winsods. export JAVA_HOME="/usr/lib/jvm/java"

Scripts

Hardware generation: SysCoyote1T2N1C8PGen is an example system config name defined in Generator.scala

$ mill dlm.runMain hwsys.dlm.SysCoyote1T2N1C8PGen
./mill-0.10.4 dlm.runMain hwsys.dlm.SysCoyote1T2N1C8PGen
.\mill-0.10.3-assembly.bat dlm.runMain hwsys.dlm.SysCoyote1T2N1C8PGen

Simulation: WrapNodeNetSim is a testbench defined in WrapNodeNetSim.scala

$ mill dlm.runMain hwsys.dlm.test.WrapNodeNetSim
./mill-0.10.4 dlm.runMain hwsys.dlm.test.WrapNodeNetSim
.\mill-0.10.3-assembly.bat dlm.runMain hwsys.dlm.test.WrapNodeNetSim
On Linux Mint:
./mill-0.10.12 dlm.runMain hwsys.dlm.test.TableSim
./mill-0.10.12 dlm.runMain hwsys.dlm.test.CoreSim
./mill-0.10.12 dlm.runMain hwsys.dlm.test.CoreNetSim
./mill-0.10.4 dlm.runMain hwsys.dlm.test.TableSim
./mill-0.10.4 dlm.runMain hwsys.dlm.test.CoreSim

Linux Mint / Ubuntu / Debian Setup

  • install JDK, Scala, and SpinalHDL using cs
  • change the scala version into 2.12.14 and uninstall other scala 3.x libs
  • install verilator<5.0 and gtkwave using apt, don't use oss-cad-suite
sudo apt-get update
sudo apt-get install openjdk-17-jdk-headless curl git
curl -fL "https://github.com/coursier/launchers/raw/master/cs-x86_64-pc-linux.gz" | gzip -d > cs
chmod +x cs
# should find the just installed jdk, agree to cs' questions for adding to your PATH
./cs setup
source ~/.profile
# check cs libraries 
cs list
java --version
scala -version
cs Install application commands:
  install    Install an application from its descriptor.
  list       List all currently installed applications.
  setup      Setup a machine for Scala development.
  uninstall  Uninstall one or more applications.
  update     Update one or more applications.

Error:
%Error: Cannot find verilated_std.sv containing built-in std:: definitions: /ucrt64/share/verilator\include\verilated_std.sv
%Error: This may be because there's no search path specified with -I

.

Solution:
C:\msys64\ucrt64\share\verilator\include\verilated_std.sv
verilator -Wall --cc <your_verilog_file> +incdir+<path_to_verilated_std.sv>
+incdir+"C:\msys64\ucrt64\share\verilator\include\verilated_std.sv"
+incdir+"C:/msys64/ucrt64/share/verilator/include/verilated_std.sv"

.\simWorkspace\TwoNodeNetTop\verilatorScript.sh
[15412:0111/102004.500:ERROR:cache_util_win.cc(20)] Unable to move the cache: ????? (0x5)
[15412:0111/102004.500:ERROR:disk_cache.cc(205)] Unable to create cache

Error:
Can't locate FindBin.pm in @INC (you may need to install the FindBin module) (@INC contains: /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /usr/bin/verilator line 28.
BEGIN failed--compilation aborted at /usr/bin/verilator line 28.

Solution:
sudo yum install perl

Error:
fatal error: jni.h: No such file or directory 5 | #include <jni.h>

Solution:
export JAVA_HOME="/usr/lib/jvm/java"

Error: error opening scalactic x.x.x...

Solution: https://get-coursier.io/docs/cli-fetch the scalactic package and paste the module dependency info into build.sc

LOG:

TableSim works fine with 128 txns with 30 read requests.
shaun@shaun-virtual-machine:~/Documents/dlm$ ./mill-0.10.4 dlm.runMain hwsys.dlm.test.TableSim
[31/42] dlm.compile 
[info] compiling 1 Scala source to /home/shaun/Documents/dlm/out/dlm/compile.dest/classes ...
[info] done compiling
[42/42] dlm.runMain 
[Runtime] SpinalHDL v1.7.3    git head : aeaeece704fe43c766e0d36a93f2ecbb8a9f2003
[Runtime] JVM max memory : 2478.0MiB
[Runtime] Current date : 2024.07.19 16:31:58
[Progress] at 0.000 : Elaborate components
[Progress] at 0.843 : Checks and transforms
[Progress] at 1.453 : Generate Verilog
[Warning] toplevel/table_1/ht : Mem[65536*16 bits].readAsync can only be write first into Verilog
[Warning] toplevel/table_1/ll : Mem[8192*24 bits].readAsync can only be write first into Verilog
[Warning] 297 signals were pruned. You can call printPruned on the backend report to get more informations.
[Done] at 1.883
[Progress] Simulation workspace in /home/shaun/Documents/dlm/./simWorkspace/OneTxnManOneLockTable
[Progress] Verilator compilation started
[Progress] Verilator compilation done in 4892.287 ms
[Progress] Start OneTxnManOneLockTable OneTxnManOneLockTable simulation with seed 99
(Txn Context Length: ,16384)
Adding page 0 at 0x0
Adding page 0 at 0x0
Adding page 1 at 0x100000
Adding page 2 at 0x200000
Adding page 3 at 0x300000
...
Adding page 63 at 0x3f00000
Handling AXI4 Master read cmds...
Handling AXI4 Master read resp...
Handling AXI4 Master write cmds...
Handling AXI4 Master write...
Handling AXI4 Master read cmds...
Handling AXI4 Master read resp...
Handling AXI4 Master write cmds...
Handling AXI4 Master write...
[txnMan] cntTxnCmt: 128
[txnMan] cntTxnAbt: 0
[txnMan] cntTxnLd: 128
[txnMan] cntClk: 38468
[Done] Simulation done in 12497.288 ms

shaun@shaun-virtual-machine:~/Documents/dlm$ ./mill-0.10.4 dlm.runMain hwsys.dlm.test.TableSim
[42/42] dlm.runMain 
[Runtime] SpinalHDL v1.7.3    git head : aeaeece704fe43c766e0d36a93f2ecbb8a9f2003
[Runtime] JVM max memory : 2478.0MiB
[Runtime] Current date : 2024.07.31 15:16:23
[Progress] at 0.000 : Elaborate components
[Progress] at 0.982 : Checks and transforms
[Progress] at 1.835 : Generate Verilog
[Warning] toplevel/table_1/ht : Mem[65536*16 bits].readAsync can only be write first into Verilog
[Warning] toplevel/table_1/ll : Mem[8192*24 bits].readAsync can only be write first into Verilog
[Warning] 297 signals were pruned. You can call printPruned on the backend report to get more informations.
[Done] at 2.455
[Progress] Simulation workspace in /home/shaun/Documents/dlm/./simWorkspace/OneTxnManOneLockTable
[Progress] Verilator compilation started
[info] Found cached verilator binaries
[Progress] Verilator compilation done in 4538.930 ms
[Progress] Start OneTxnManOneLockTable OneTxnManOneLockTable simulation with seed 99
(Txn Context Length: ,16384)
[txnMan] cntTxnCmt: 128
[txnMan] cntTxnAbt: 0
[txnMan] cntTxnLd: 128
[txnMan] cntClk: 133480
[Done] Simulation done in 22555.227 ms

With optimized waitEntryAddrFast:
[Runtime] SpinalHDL v1.7.3    git head : aeaeece704fe43c766e0d36a93f2ecbb8a9f2003
[Runtime] JVM max memory : 2478.0MiB
[Runtime] Current date : 2024.08.02 18:36:34
[txnMan] cntTxnCmt: 128
[txnMan] cntTxnAbt: 0
[txnMan] cntTxnLd: 128
[txnMan] cntClk: 89026
[Done] Simulation done in 34009.162 ms

Dense write, 128 txns with 30 sequential lockIDs. 
[txnMan] cntTxnCmt: 12
[txnMan] cntTxnAbt: 116
[txnMan] cntTxnLd: 128
[txnMan] cntClk: 7039
[Done] Simulation done in 24560.012 ms

CoreSim

Two TxnMan Two Tables simulate passed all read. Node 0 and 1 both 128 txns,
[txnMan] cntTxnCmt: 128
[txnMan] cntTxnAbt: 0
[txnMan] cntTxnLd: 128
[txnMan] cntClk: 41080
[Done] Simulation done in 28181.376 ms

Two TxnMan Two Tables passed Read-Write mixed simulation. Node 0 128 txns, node 1 16 txns.
shaun@shaun-virtual-machine:~/Documents/dlm$ ./mill-0.10.4 dlm.runMain hwsys.dlm.test.CoreSim
[31/42] dlm.compile 
[info] compiling 1 Scala source to /home/shaun/Documents/dlm/out/dlm/compile.dest/classes ...
[info] done compiling
[42/42] dlm.runMain 
[Runtime] SpinalHDL v1.7.3    git head : aeaeece704fe43c766e0d36a93f2ecbb8a9f2003
[Runtime] JVM max memory : 2478.0MiB
[Runtime] Current date : 2024.08.07 11:54:29
[Progress] at 0.000 : Elaborate components
[Progress] at 1.088 : Checks and transforms
[Progress] at 2.191 : Generate Verilog
[txnMan] cntTxnCmt: 128
[txnMan] cntTxnAbt: 0
[txnMan] cntTxnLd: 128
[txnMan] cntClk: 32433
[Done] Simulation done in 20409.428 ms

Two TxnMan Two Tables simulate passed all Write. Node 0 and 1 both 128 txns,
shaun@shaun-virtual-machine:~/Documents/dlm$ ./mill-0.10.12 dlm.runMain hwsys.dlm.test.CoreSim
[32/43] dlm.compile 
[info] compiling 1 Scala source to /home/shaun/Documents/dlm/out/dlm/compile.dest/classes ...
[info] done compiling
[43/43] dlm.runMain 
[Runtime] SpinalHDL v1.10.1    git head : 2527c7c6b0fb0f95e5e1a5722a0be732b364ce43
[Runtime] JVM max memory : 2476.0MiB
[Runtime] Current date : 2024.08.14 10:19:48
[Progress] at 0.000 : Elaborate components
[Progress] at 1.288 : Checks and transforms
[Progress] at 2.671 : Generate Verilog
[txnManA] cntTxnCmt: 128
[txnManA] cntTxnAbt: 0
[txnManA] cntTxnLd: 128
[txnManA] cntClk: 88917
[txnManB] cntTxnCmt: 128
[txnManB] cntTxnAbt: 0
[txnManB] cntTxnLd: 128
[txnManB] cntClk: 89197
[Done] Simulation done in 42199.175 ms

CoreNetSim Results:
Passed Node 0 128 Txns with all read locks.
shaun@shaun-virtual-machine:~/Documents/dlm$ ./mill-0.10.12 dlm.runMain hwsys.dlm.test.CoreNetSim
[32/43] dlm.compile 
[info] compiling 1 Scala source to /home/shaun/Documents/dlm/out/dlm/compile.dest/classes ...
[info] done compiling
[43/43] dlm.runMain 
[Runtime] SpinalHDL v1.10.1    git head : 2527c7c6b0fb0f95e5e1a5722a0be732b364ce43
[Runtime] JVM max memory : 2476.0MiB
[Runtime] Current date : 2024.08.12 15:08:00
[Progress] at 0.000 : Elaborate components
[Progress] at 2.220 : Checks and transforms
[Progress] at 6.384 : Generate Verilog
[Warning] toplevel/tableA/ht : Mem[65536*16 bits].readAsync can only be write first into Verilog
[Warning] toplevel/tableA/ll : Mem[8192*24 bits].readAsync can only be write first into Verilog
[Warning] toplevel/tableB/ht : Mem[65536*16 bits].readAsync can only be write first into Verilog
[Warning] toplevel/tableB/ll : Mem[8192*24 bits].readAsync can only be write first into Verilog
[Warning] 804 signals were pruned. You can call printPruned on the backend report to get more informations.
[Done] at 8.989
[Progress] Simulation workspace in /home/shaun/Documents/dlm/./simWorkspace/TwoTxnManTwoTableTwoNet
[Progress] Verilator compilation started
[Progress] Verilator compilation done in 32156.441 ms
[Progress] Start TwoTxnManTwoTableTwoNet TwoTxnManTwoTableTwoNet simulation with seed 99
[txnMan] cntTxnCmt: 128
[txnMan] cntTxnAbt: 0
[txnMan] cntTxnLd: 128
[txnMan] cntClk: 24240
[Done] Simulation done in 78835.027 ms

Node 0 and 1 both 128 write on same locks passed!
shaun@shaun-virtual-machine:~/Documents/dlm$ ./mill-0.10.12 dlm.runMain hwsys.dlm.test.CoreNetSim
[43/43] dlm.runMain 
[Runtime] SpinalHDL v1.10.1    git head : 2527c7c6b0fb0f95e5e1a5722a0be732b364ce43
[Runtime] JVM max memory : 2476.0MiB
[Runtime] Current date : 2024.08.14 19:24:51
[Progress] at 0.000 : Elaborate components
[Progress] at 1.493 : Checks and transforms
[Progress] at 4.305 : Generate Verilog
[txnManA] cntTxnCmt: 125 // I read out data when TxnManB.done, so it is normal if TxnManA is still in progress.
[txnManA] cntTxnAbt: 0
[txnManA] cntTxnLd: 128
[txnManA] cntClk: 134578
[txnManB] cntTxnCmt: 128
[txnManB] cntTxnAbt: 0
[txnManB] cntTxnLd: 128
[txnManB] cntClk: 134578
[Done] Simulation done in 122594.006 ms