jbalma/ck

Collective Knowledge framework helps to convert ad-hoc code, data and scripts into portable, customizable and reusable components with a simple Python API and an integrated package manager for Linux, MacOS, Windows and Android; assemble automated workflows; crowdsource complex experiments; generate interactive papers, etc:

Linux/MacOS: Windows: Coverage:

Note that we just completed a proof-of-concept stage for CK with our partners and plan to gradually improve CK documentation, specification and APIs in Q1 2019. Please be patient, stay tuned or help the CK community via this open CK forum!

Collective Knowledge (CK) is a community effort to develop the universal, portable, customizable and automated workflow framework with "plug&play" components to enable collaborative and reproducible R&D based on agile, DevOps, FAIR and Wikipedia principles (see CK motivation, main features and RESCUE-HPC workshop).

The idea is to let the community collaboratively improve common experimental workflows and components via Git while making them adaptable to new environments and platforms, and gradually fixing reproducibility issues!

CK supports our long-term vision to connect academia and industry to solve the real-world challenges. For example, see several important use cases from our partners:

• IBM: "Reproducing Quantum results from Nature - how hard could it be?"
• General Motors: "Collaboratively Benchmarking and Optimizing Deep Learning Implementations"
• Amazon: "Scaling deep learning on AWS using C5 instances with MXNet, TensorFlow, and BigDL: From the edge to the cloud"
• Arm: ARM Demonstration of Collaboratively Optimizing Deep Learning Applications
• Raspberry Pi foundation: interactive and reproducible article about "a Collective Knowledge workflow for collaborative research into multi-objective autotuning and machine learning techniques"
• ACM, IEEE and NIPS conferences: automated artifact evaluation, CGO'17 paper, ReQuEST'18 papers
• ACM and dividiti:ReQuEST tournaments to co-design efficient SW/HW stacks for deep learning across diverse models, data sets and platforms from cloud to edge, organizers' report
• dividiti: CK-based public benchmarking of Quantum systems and algorithms

Just give it a try and do not hesitate to provide your feedback to the CK community via this public CK discussion group!

CK resources

• Project website: cKnowledge.org
• Real world use cases from the CK community
• CK documentation
  • Motivation and main features
  • CK basics
  • Getting started guide
  • Adding new CK components
• Shared CK components
  • Program workflows
  • Repositories with reusable workflows and artifacts
  • Reusable CK modules (plugins)
    • Exposed CK kernel productivity functions
  • Reusable software detection plugins
  • Packages to automate installation of workflows across diverse platforms
• Examples of collaborative R&D
  • Public scoreboard with results from crowd-sourced experiments such as SW/HW co-design of deep learning
  • Reproducible SW/HW co-design competitions for deep learning and other emerging workloads using CK
  • Open benchmarking of quantum algorithms
• CK publications
• Open discussion group and Slack

Installation

Dependencies

The minimal CK installation requires:

• Python 2.7 or 3.3+ with PIP (limitation is mainly due to unitests)
• Git command line client
• wget (Linux/MacOS)

Ubuntu

$ sudo apt-get install python3 python3-pip git wget

MacOS

$ brew install python3
$ brew install git
$ brew install wget

Windows

• Download and install Git from https://git-for-windows.github.io
• Download and install Python from https://www.python.org/downloads/windows

Installation from GitHub

Linux/MacOS

You can install CK in your local user space as follows:

$ git clone http://github.com/ctuning/ck
$ export PATH=$PWD/ck/bin:$PATH
$ export PYTHONPATH=$PWD/ck:$PYTHONPATH

Windows

 $ git clone https://github.com/ctuning/ck.git ck-master
 $ set PATH={CURRENT PATH}\ck-master\bin;%PATH%
 $ set PYTHONPATH={CURRENT PATH}\ck-master;%PYTHONPATH%

Installation via PIP

Ubuntu

You can also install CK via PIP with sudo to avoid setting up environment variables yourself:

$ sudo pip install ck

Starting from Ubuntu 18.10, you can install it via apt:

$ sudo apt install python-ck
 or
$ sudo apt install python3-ck

MacOS/Windows

$ pip install ck

Installation customization

Check this documentation about CK customization. For example, you can change directories with CK repositories and packages or change search paths during software detection (useful for HPC setups).

Basic usage example

Test ck:

$ ck version

Get shared ck-tensorflow repo with all dependencies:

$ ck pull repo:ck-tensorflow

List CK repos:

$ ck ls repo | sort

Find where CK repos are installed on your machine:

$ ck where repo:ck-tensorflow

Detect your platform properties via extensible CK plugins as follows (needed to unify benchmarking across diverse platforms with Linux, Windows, MacOS and Android):

$ ck detect platform

Now detect available compilers on your machine and register virtual environments in the CK:

$ ck detect soft --tags=compiler,gcc
$ ck detect soft --tags=compiler,llvm
$ ck detect soft --tags=compiler,icc

See virtual environments in the CK:

$ ck show env

We recommend to setup CK to install new packages inside CK virtual env entries:

$ ck set kernel var.install_to_env=yes

Now install CPU-version of TensorFlow via CK packages:

$ ck install package --tags=lib,tensorflow,vcpu,vprebuilt

Check that it's installed fine:

$ ck show env --tags=lib,tensorflow

You can find a path to a given entry (with TF installation) as follows:

$ ck find env:{env UID from above list}

Run CK virtual environment and test TF:

$ ck virtual env --tags=lib,tensorflow
$ ipython
> import tensorflow as tf

Run CK classification workflow example using installed TF:

$ ck run program:tensorflow --cmd_key=classify

Now you can try a more complex example to build Caffe with CUDA support and run classification. Note that CK should automatically detect your CUDA compilers, libraries and other deps or install missing packages:

$ ck pull repo --url=https://github.com/dividiti/ck-caffe
$ ck install package:lib-caffe-bvlc-master-cuda-universal
$ ck run program:caffe --cmd_key=classify

You can see how to install Caffe for Linux, MacOS, Windows and Android via CK here.

Finally, compile, run, benchmark and crowd-tune some C program.

$ ck pull repo:ck-crowdtuning

$ ck ls program
$ ck ls dataset

$ ck compile program:cbench-automotive-susan --speed
$ ck run program:cbench-automotive-susan

$ ck benchmark program:cbench-automotive-susan

$ ck crowdtune program:cbench-automotive-susan

You can then browse top shared optimization results on the live CK scoreboard: http://cKnowledge.org/repo .

You can also add and later customize your own program/workflow using provided templates as follows:

$ ck add program:my-new-program

When CK asks you to select a template, please choose "C program "Hello world". You can then immediately compile and run your C program as follows:

$ ck compile program:my-new-program --speed
$ ck run program:my-new-program
$ ck run program:my-new-program --env.CK_VAR1=222

Trying CK from Docker images

You can try CK using the following Docker image:

 $ (sudo) docker run -it ctuning/ck-ubuntu-18.04

Note that we added Docker automation to CK to help evaluate artifacts at the conferences, share interactive and reproducible articles, crowdsource experiments and so on.

For example, you can participate in GCC or LLVM crowd-tuning on your machine as follows:

 $ (sudo) docker run ck-crowdtune-gcc
 $ (sudo) docker run ck-crowdtune-llvm

Top optimization results are continuously aggregated in the live CK repository: http://cKnowledge.org/repo .

Citing CK

See CK publications.

CK authors

• Grigori Fursin, cTuning foundation and dividiti
• Anton Lokhmotov, dividiti

License

• Permissive 3-clause BSD license. (See LICENSE.txt for more details).

Acknowledgments

CK development is coordinated by the cTuning foundation and dividiti. We would like to thank the TETRACOM 609491 Coordination Action for initial funding and all our partners for continuing support. We are also extremely grateful to all volunteers for their valuable feedback and contributions.