CEM - Code Examples Manager

Code example manager (CEM) is a software managing your notes, scripts and code examples. It provides publish mechanisms to github.com (as gists) or gitlab.com (as snippets). It also automates execution for testable examples, this is a quite useful to manage a high number of examples.

All my notes, scripts and code examples (my programming knowledge base) are now managed using this tool, you can take a look to my public gists overview on github to illustrate the publishing work achieved by CEM.

Current Code example manager (CEM) implementation is just a command line tool which compare locally available examples with already published ones in order to find what it should do (add, update, do nothing).

Why ?

Code examples are very important, each example is most of the time designed to focus on a particular feature/characteristic of a programming language, a library or a framework. They help us to quickly test, experiment and remember how bigger project or at least some parts of them are working.

See the rules for good code examples for more information.

Managing hundreds of published code example files as gists (github) and/or snippets (gitlab) is really not easy and time-consuming, in particular if you want to keep them up to date. This is the main issue addressed by this software.

My open-source examples evolution trend :
and the execution status trend for executable/testable ones :

As you can see through the previous charts, once you have industrialized your notes and code examples, analytics on your examples become quite easy, and a lot of advanced features become possible... So stay connected to this project ;)

Quick start

No particular prerequisites, just a Java >=8 JVM available, and it will run on your Linux, Windows or MacOSX

Instructions example with github.com publishing configuration :

Install the coursier from @alxarchambault to automate the download/update/start of code-examples-manager directly from maven repositories

Customize your configuration (see below for token configuration)

export CEM_SEARCH_ROOTS="/home/myuser/myexamples"
export CEM_GITHUB_TOKEN="xxxxx"

Create an example file in /home/myuser/myexamples such as hello.md

<!--
// summary : my markdown cheat sheet
// keywords : markdown, hello, example 
// publish : gist
// authors : someone, someonelse
// id : d27245fc-22fb-4c9e-9809-feb0826400e7
-->
# Hello world !
this is just an example

Run the following command from your terminal (cs is the coursier CLI command):
```
cs launch fr.janalyse:code-examples-manager_3:2.4.0
```
- you can even use cs launch fr.janalyse:code-examples-manager_3:latest.release to always use the latest release
- current release is :
Check the command output to get the overview URL

Code examples

In order to be published your code examples must come with a description header inserted using single line comments. You must provide a unique identifier (UUID) to each of your example, as well as a summary and publish keywords which define remote destinations.

Example for languages using // for line comments :

// summary : Simplest scalatest test framework usage.
// keywords : scala, scalatest, pi, @testable
// publish : gist
// authors : David Crosson
// license : Apache
// id : d24d8cb3-45c0-4d88-b033-7fae2325607b
// created-on : 2020-05-31T19:54:52Z
// run-with : scala-cli $file

// ---------------------
//> using scala  "3.1.1"
//> using lib "org.scalatest::scalatest:3.2.10"
// ---------------------

import org.scalatest._, matchers.should.Matchers._

math.Pi shouldBe 3.14d +- 0.01d

Supported keys in description header are the following :

summary : example summary in one line.
keywords : keywords describing your code features (comma separated). Some reserved keywords :
- @testable : allow automatic execution
- @fail : the example is expected to fail when executed
- @exclusive : all testable examples with this flag will be run sequentially (for scripts which open server sockets for example)
publish : publish destination keywords (comma separated)
- the default configuration file provide those activation keywords :
  - gist : for github.com
  - snippet : for gitlab.com
authors : code example authors list (comma separated).
license : the example license.
id : UUID for this code example. Generated using such commands :
- with linux command : uuidgen
- with scala-cli : scala-cli https://gist.github.com/dacr/87c9636a6d25787d7c274b036d2a8aad
- with ammonite : amm -c 'println(java.util.UUID.randomUUID.toString)'
attachments : List of secondary files (comma separated) which must be published with the current one
- must be placed in the same directory in your local storage
- secondary files do not require any headers
created-on : The ISO8601 date when this example has been created. Generated using such commands :
- with linux command : date -u +"%Y-%m-%dT%H:%M:%S.%3NZ"
- with scala-cli : scala-cli https://gist.github.com/dacr/4298fce08e12ba76ab91e9766be52acb
- with ammonite : amm -c 'println(java.time.Instant.now.toString)'
run-with : command used to execute this example
- Only examples with @testable keywords are eligible for automated execution
- on execution the exit code is used to compute execution success or failure
- Use $file (or $scriptFile) for example filename substitution
test-with : Command to test the example
- When @testable is set as keyword
- When your example is a "blocking service", you can specify an external command to test it
  - for example : test-with : curl http://127.0.0.1:8080/docs

CEM operations

Code examples manager operations :

It reads its configuration
It searches for code examples from the given directories roots
- select files matching the search pattern and not involved in the ignore mask
- Selects code examples if and only if they contain a unique identifier (UUID)
It publishes or updates remote code examples to remote destinations
- the code example publish scope (publish keyword) select target destinations
  - comma separated publish activation keyword (activation-keyword parameter in configuration)
- It adds or updates a global overview of all published examples for a given destination
  - this summary has its own UUID defined in the configuration file

Configuration

The configuration relies on configuration files, a default one named reference.conf is provided. This default configuration file defines default values and default behaviors and allow a simple configuration way based on environment variables which override default values.

Simplified configuration

env or property name	description	default value
CEM_CONFIG_FILE	Your custom advanced configuration file (optional)	undefined
CEM_SUMMARY_TITLE	The generated summary title for all published examples	Examples knowledge base
CEM_SEARCH_ROOTS	Examples search roots (comma separated)	""
CEM_SEARCH_PATTERN	Examples files regular expression pattern	".*"
CEM_SEARCH_IGNORE_MASK	Ignore file regular expression	"(/[.]bsp)
CEM_CHAR_ENCODING	Chararacter encoding for your examples or notes	"UTF-8"
CEM_GITHUB_ENABLED	To enable or disable standard GITHUB support	true
CEM_GITHUB_ACTIVATION_KEY	Example publish keyword for github	"gist"
CEM_GITHUB_TOKEN	Github authentication token for gists API access	more information below
CEM_GITHUB_API	Github API http end point	"https://api.github.com"
CEM_GITLAB_ENABLED	To enable or disable standard GITLAB support	true
CEM_GITLAB_ACTIVATION_KEY	Example publish keyword for the gitlab	"snippet"
CEM_GITLAB_TOKEN	gitlab authentication token for snippets API access	more information below
CEM_GITLAB_API	Gitlab API http end point	"https://gitlab.com/api/v4"
CEM_GITLAB_VISIBILITY	Gitlab published examples chosen visibility	"public"

Configuration examples :

export CEM_SEARCH_ROOTS="/tmp/someplace,/tmp/someotherplace"
export CEM_SEARCH_PATTERN="[.](sc)|(sh)|(md)|(jsh)$"
export CEM_GITHUB_TOKEN="fada-fada-fada-fada"

Advanced configuration

Take a look to the configuration file for more information about advanced configuration.

Once CEM installed you can modify the provided conf/application.conf file (whose content is by default the same as the default reference.conf file), remember that any unset parameter in application.conf will default to the value defined in reference.conf.

Note : It is also possible to provide a custom configuration file through the config.file java property or the CEM_CONFIG_FILE environment variable.

Authentication tokens

Gitlab authentication token configuration

Get an access token from gitlab :

Go to your user settings
- Select Access tokens
  - Add a Personal access token
    - Enable scopes : api and read_user
setup your CEM_GITLAB_TOKEN environment variable or token parameter in your configuration file with the generated token
Keep it carefully as it is not possible to retrieve it later.
And of course KEEP IT SECURE

Github authentication token configuration

Get an access token from gitlab.com :

Got to your user settings
- Select Developer settings
  - Select Personal access tokens
    - Then generate new token
      - Enable scopes : gist and read:user
setup your CEM_GITHUB_TOKEN environment variable or token parameter in your configuration file with the generated token, the value shown within curl json response
Keep it carefully as it is not possible to retrieve it later.
And of course KEEP IT SECURE

Project history

2019-06 - PoC#1 example proof of concept
2019-07 - PoC#2 example proof of concept (in prod)
2019-08 - Switch to a real world project
2019-09 - In prod for my own usage
2020-07 - First public release
2021-05 - Full refactoring to use ZIO - pure functional
2021-06 - Migration to Scala3
2021-12 - PoC#3 Search & Execution engines
2022-01 - Migrate to ZIO2 and add support for attachments
2023-04 - Use ZIO standard configuration
2023-05 - Add ZIO LMDB for local storage cache AND data sharing with external applications

dacr/code-examples-manager