How We Work

Knocean is a small consulting company. This is a brief description of how we work, with some attempt at justification.

As people, we strive to live good lives. Good work is one part of a good life, so we:

treat people well: each other, our clients, our collaborators, the people who use the tools we build, etc.
choose good projects: open science, open source, specializing in Open Biological and Biomedical Ontologies
work publicly, so our work is shared
work asynchronously, so we can work when it suits us
work remotely, so we can live where it suits us
limit our work time: take proper time off, encourage part-time work, etc.
do work that we're proud of

Guiding Principles

programming is mostly about people
use text
use small, sharp tools
use the tools we build
don't repeat yourself
automate
use version control
use open source

Programming is Mostly About People

prefer public GitHub and mailing lists
use private Slack for quick questions
use private GitHub Projects for detailed scheduling
weekly one-on-one meetings, with audio
code review is important, and hard!
documentation is important, and hard!
avoid pointless arguments about code style by specifying a style guide and running a formatting tool

Code Review

I like this advice.

Documentation

I like this four-way distinction: tutorial, how-to, reference, explanation

Use Text

text is ubiquitous: documents, email, messaging, issues, web forms, terminals, etc.
use UTF-8 for pretty much everything
use your preferred text editor for pretty much everything
for a give task, pick a point on the spectrum from unstructured to highly structured text format

Use Small, Sharp Tools

use the right tool for the job
break complex jobs down into a series of simple jobs
collect a toolbox of good tools and master them
small, sharp tools are easier to learn and remember, and they stand the test of time

Use the Tools we Build

We're among the main users of the tools we build, and we have to live with them for years to come. This encourages use to build tools that are:

simple and focused
effective
reliable
pleasant to use
efficient

Don't Repeat Yourself

Life is easier if everything has one canonical source, with one canonical name.

Automate

There's a rough spectrum of automation:

ad hoc:
- Unix shell: coreutils, pipes, "one liners", history
- Excel
fully manual:
- use a checklist
- try a Do-Nothing Script in Bash or Python
semi-automatic:
- start with your Do-Nothing Script
- automate what you can
- when can't automate a step, print instructions and wait for the user
- example: ROBOT release.sh
fully-automatic:
- small jobs, simple control flow: Bash script
- medium jobs: Python script
- large jobs: new command-line tool
- job management: GNU Make
- orchestration: NixOS, Docker

Version Control

text is easy to diff and patch
git works best with text
don't store generated files in version control
store all the information needed to rebuild:
- versioned install details
  - NixOS shell.nix
  - Python requirements.txt
  - Maven pom.xml
- versioned code
- versioned build script (Makefile)
- versioned inputs, maybe external

Use Open Source

Life is easier if we can:

see the source of the tools we use, especially when there's a bug
reuse the tools we write for the next project

Arguably, good science requires open source tools.

At Knocean we prefer the BSD 3-Clause licence for our code, and CC0 for other resources.

Common Tools

Based on those principles, these are the common tools and platforms that we build on:

Text Formats
World Wide Web
Unix (Linux, macOS, NixOS): coreutils, command-line tools
Git and GitHub
GNU Make
Bash: a weird language, very good for simple scripts, but switch to Python for anything fancy
Python: an acceptable language, lingua franca, good for scripting
Java: a poor language with an excellent runtime (JVM) and many good libraries
Clojure: an excellent language, but "obscure"
SQLite, Postgres

Text Formats

document formats
- Markdown
- HTML, separating content from style
- LaTeX, for advanced layout
tabular data
- TSV, CSV
- SQL
- Excel: for aggregation and visualization, but not version control
- Google Sheets: for collaboration, but not version control
structured data
- EDN: perfect balance, but mostly limited to Clojure
- YAML: human readable, overly complex syntax
- XML: well-structured, hard to read
- JSON: poor balance of features but ubiquitous
- Turtle: for RDF
- Manchester: for OWL logical expressions
- RDFa: for RDF in HTML
- JSON-LD: for sharing RDF with tools that only speak JSON

World Wide Web

stick to basics!
use HTTPS whenever possible, prefer Let's Encrypt for SSL certificates
HTML is good for display, but prefer source data in simpler formats, e.g. Markdown
use Bootstrap 4, avoid custom CSS
prefer Nginx, but Apache 2 is fine
use static HTML as long as possible
use Python Flask for dynamic sites
use Jinja2 for templates
use a small set of mature libraries
- jQuery
- InspireTree
- MomentJS
- CodeMirror
- Twitter Typeahead
write minimal vanilla JavaScript "glue"
use JavaScript for progressive enhancement
- site must work without JavaScript, for cURL etc.
- corollary: avoid Single Page Web Applications

Unix

use Linux for servers, prefer NixOS
macOS is acceptable for desktops/laptops
provisioning
- prefer NixOS
- otherwise use Ansible 2
- prefer official packages
use AWS EC2 but be careful about platform lock-in
everything is a file, most files are streams of delimited text
Unix coreutils: ls, grep, find, sed, awk
standard Unix tools: ssh, curl
good command-line tools: tree, mosh, xlsx2csv, Pandoc, RDF rapper

Git and GitHub

use git!
- commit early, commit often
include: README, LICENSE, CONTRIBUTING, CHANGELOG
use GitHub
- prefer to work on the main repo, not forks
use GitHub issues
- try to assign every issue to somebody
- use GitHub's default labels: bug, question, enhancement, good first issue, etc.
- issue templates are a good idea
use GitHub Pull Requests
- title should start with 'WIP' until ready for review, then assign reviewers
- description should link to an issue "Resolves #1234"
  - otherwise you have to clean up issues manually
- update description as needed
- PR templates are a good idea
use GitHub Milestones to plan releases
use GitHub Releases to tag versions and attach binaries
use GitHub Pages for simple static content and documentation
- prefer the docs/ directory option
use GitHub Projects ("project boards") for coordinating issues and PRs
- three columns:
  - To do: for manager to fill and prioritize
  - In progress: for developer to keep up-to-date
  - Ready for review: for manager to empty
- order of preference for cards: PR link, issue link, text
use Travis CI

GNU Make

tell Make all the prerequisites, so it can build the full dependency graph
tell Make all the outputs, preferably just one
keep recipes to a handful of lines
convert larger recipes to Python scripts
try to match script arguments with prerequisites exactly, then just python3 $^ $@
use Make functions and automatic variables, but strive for clarity
empty files are OK
combine with watchexec

GNU Make Defaults

###Configuration
#
# These are standard options to make Make sane:
# <http://clarkgrubb.com/makefile-style-guide#toc2>

MAKEFLAGS += --warn-undefined-variables
SHELL := bash
.SHELLFLAGS := -eu -o pipefail -c
.DEFAULT_GOAL := all
.DELETE_ON_ERROR:
.SUFFIXES:
.SECONDARY:

Bash

use ShellCheck
use Unofficial Bash Strict Mode

set -euo pipefail
IFS=$'\n\t'

Python

ubiquitous, cross-platform
good for scripting, interactive development
strong ecosystem of libraries for almost anything
use for anything more than about 10 lines of Bash
use Python 3.6+
write imperative code, avoid Object Oriented design

Java

ubiquitous, cross-platform
strong ecosystem of heavy libraries, especially for the Semantic Web
- we mainly use Java for ROBOT, which relies on OWLAPI and Apache Jena
use Java 8 features
use Maven
use Google Java Style and fmt-maven-plugin
write imperative code, rely on collections, avoid excessive Object Oriented design

Clojure

Clojure is an excellent language, but its "obscurity" makes it hard to choose for collaborative projects.

mcourtot/practises