HashDL (Hash-based Deep Learning)
1 Overview
This repository is non-official third-paty re-implementation of SLIDE1.
We provide
- Python package
- Hash based Deep Learning
- Parallel computing based on C++17 parallel STL
We don’t provide
- Explicit CPU optimized code like AVX (We just rely on compiler optimization)
- Compiled binary (You need to compile by yourself)
2 Install
There are two options, “Install from PyPI” and “Install from Source”. For ordinary user, “Install from PyPI” is recommended.
For both case, sufficient C++ compiler is neccessary.
2.1 Requirement
- Recent C++ compiler with parallel STL algorithm support
- Python 3
Requirements can be installed on Docker image gcc:10.
# On local machine
docker run -it gcc:10 bash
# On gcc:10 image
apt update && apt install -y python3-pip libtbb-dev
2.2 Install from PyPI
pip install HashDL
2.3 Install from Source
git clone https://gitlab.com/ymd_h/hashdl.git HashDL
cd HashDL
pip install .
3 Features
- Neural Network
- hash-based sparse dense layer
- Activation
- ReLU
- linear (no activation)
- sigmoid
- Optimizer
- SGD
- Adam3
- Weight Initializer
- constant
- Gauss distribution
- Hash for similarity
- WTA
- DWTA2
- Scheduler for hash update
- constant
- exponential decay
In the current architecture, CNN is impossible.
4 Implementation
The official reference implementation focused on performance and accepted some “dirtyness” like hard-coded magic number for algotihm selection and unmanaged memory allocation.
We accept some (but hopefully small) overhead and improve maintenability in terms of software;
- Polymorphism with inheritance and virtual function
- RAII and smart pointer for memory management
These archtecture allows us to construct and manage C++ class from Python without recompile.
We also rely recent C++ standard and compiler optimization;
- Parallel STL from C++17
- Because of RVO (or at least move semantics), returning
std::vector
is not so much costful as it was.
5 Footnotes
3 D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization”, ICLR (2015) (arXiv)
2 B. Chen et al., “Densified Winner Take All (WTA) Hashing for Sparse Datasets”, Uncertainty in artificial intelligence (2018)
1 B. Chen et al., “SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems”, MLSys 2020 (arXiv, code)