/HashDL

Hash based Deep Learning (Mirror of https://gitlab.com/ymd_h/hashdl)

Primary LanguageC++MIT LicenseMIT

HashDL (Hash-based Deep Learning)

1 Overview

This repository is non-official third-paty re-implementation of SLIDE1.

We provide

  • Python package
  • Hash based Deep Learning
  • Parallel computing based on C++17 parallel STL

We don’t provide

  • Explicit CPU optimized code like AVX (We just rely on compiler optimization)
  • Compiled binary (You need to compile by yourself)

2 Install

There are two options, “Install from PyPI” and “Install from Source”. For ordinary user, “Install from PyPI” is recommended.

For both case, sufficient C++ compiler is neccessary.

2.1 Requirement

  • Recent C++ compiler with parallel STL algorithm support
  • Python 3

Requirements can be installed on Docker image gcc:10.

# On local machine
docker run -it gcc:10 bash

# On gcc:10 image
apt update && apt install -y python3-pip libtbb-dev

2.2 Install from PyPI

pip install HashDL

2.3 Install from Source

git clone https://gitlab.com/ymd_h/hashdl.git HashDL
cd HashDL
pip install .

3 Features

  • Neural Network
    • hash-based sparse dense layer
  • Activation
    • ReLU
    • linear (no activation)
    • sigmoid
  • Optimizer
    • SGD
    • Adam3
  • Weight Initializer
    • constant
    • Gauss distribution
  • Hash for similarity
    • WTA
    • DWTA2
  • Scheduler for hash update
    • constant
    • exponential decay

In the current architecture, CNN is impossible.

4 Implementation

The official reference implementation focused on performance and accepted some “dirtyness” like hard-coded magic number for algotihm selection and unmanaged memory allocation.

We accept some (but hopefully small) overhead and improve maintenability in terms of software;

  • Polymorphism with inheritance and virtual function
  • RAII and smart pointer for memory management

These archtecture allows us to construct and manage C++ class from Python without recompile.

We also rely recent C++ standard and compiler optimization;

  • Parallel STL from C++17
  • Because of RVO (or at least move semantics), returning std::vector is not so much costful as it was.

5 Footnotes

3 D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization”, ICLR (2015) (arXiv)

2 B. Chen et al., “Densified Winner Take All (WTA) Hashing for Sparse Datasets”, Uncertainty in artificial intelligence (2018)

1 B. Chen et al., “SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems”, MLSys 2020 (arXiv, code)