/Project-Zipline

Defines a lossless compressed data format that is independent of CPU type, operating system, file system, and character set, and is suitable for compression using the XP10 algorithm.

Primary LanguageVerilogOtherNOASSERTION

Project Zipline

Introduction

Project Zipline is a program to accelerate innovation in lossless compression. With this release Microsoft is making available:

  • A new compression format called XP10 which is tailored for modern cloud datasets.
  • All specifications for the entire pipeline.
  • All RTL to support implementation for the entire pipeline.
  • A testbench environment specifically developed for the VCS simulator.

Project Zipline is open-sourced under the MIT License, see the LICENSE file.

Specifications

The following Project Zipline specifications have been included under the “specs” directory in this repository:

  • Project_Zipline_Compression_Specification
  • Project_Zipline_Huffman_Encoder_Micro_Architecture_Specification
  • Project_Zipline_Comp_lz77_Micro_Architecture_Specification
  • Project_Zipline_Crypto_Engine_Micro_Architecture_Specification
  • Project_Zipline_Decompression_Top_Micro_Architecture_Specification
  • Project_Zipline_Keyblob_Micro_Architecture_Specification
  • Project_Zipline_KME_Micro_Architecture_Specification
  • Project_Zipline_Prefix_Attach_Micro_Architecture_Specification
  • Project_Zipline_Prefix_Micro_Architecture_Specification
  • Project_Zipline_SSB_Micro_Architecture_Specification
  • Project_Zipline_Top_Micro_Architecture_Specification

Simulation Notes

Edit the file ./zipline.setup for VCS, SYNTH, and VERDI environment settings.

Then source the file as follows:

source zipline.setup

Go to either the CCE_64, CDD_64, or KME run directory:

cd dv/CCE_64/run

cd dv/CDD_64/run

cd dv/KME/run

To build the simv executable:

make build_simv

Example for simulating a test:

make run_simv TESTNAME=xp10

Example for simulating a test with waves (.vpd):

make run_simv TESTNAME=xp10 WAVES=1

Example for simulating a test with Verdi waves (.fsdb):

make run_simv TESTNAME=xp10 VERDI_WAVES=1

All of the tests for the CCE_64, CDD_64, and KME can be run via a regress script in the associated "run" directory:

dv/CCE_64/run/regress

dv/CDD_64/run/regress

dv/KME/run/regress

All test files are located in the associated engine directory "tests". Each directory also contains a README file with a description of each test.

dv/CCE_64/tests

dv/CDD_64/tests

dv/KME/tests

A description of the programmable registers in the CCE/CDD/KME can be found in the register_doc directory. Please read register_doc/README for more details.

KME Notes

The KME RTL (rtl/cr_kme) has been modified to remove the following modules:

  • AES engine in the random GUID generator
  • SHA engines within the KDF function
  • AES engine within the Key Decryption logic

With these reductions, the RTL only supports Key Types 1-6 without KDF and Key Type 0 without encryption/authentication

Additionally, the KME output has been reduced to support a single engine.

All modifications are identified with "KME_MODIFICATION_NOTE" comments in the RTL code.

Synthesis Notes

Note: For synthesis, please edit syn/Makefile to choose an LSF or dedicated machine with at least 16G of memory. Edit the contents of this line, before "dc_shell":

bsub -Is -q irv-cpx-M16 -R "rusage [mem=16000]" dc_shell -f syn_example.tcl | tee ./LOGS/syn_eample.tcl.date '+%m%d.%H:%M'.log

To run synthesis for CDD

cd syn/cr_cddip

make syn

To run synthesis for CCE

cd syn/cr_cceip_64

make syn

To run synthesis for KME

cd syn/cr_kme

make syn