Firmware 'scripts'

UPDATE: There is now a website for this. Click me!. This repository acts mainly as documentation. Please read through it.

Python scripts to make writing firmware faster/easier. The idea of the script/s is to remove the need to manually write a lot of the boring/repetitve/time-consuming firmware stuff.

Basic demonstration on how quick it is (placing this at the beginning because it seems some people are missing the point of the script):

firmware-script-guide.mp4

Disclaimer: The script utilizes some code from other places like pykle_serial (for the deserializer code) and some small util scripts from vial-qmk (for the uid generation) and qmk_firmware (for the json encoders; to match the official qmk info.json formatting). I translated the serialization code from kle-serial myself. The rest of the scripts to actually convert from the deserialized layout to the other firmware files is done by me.

Realistcally you can use just the serialization and deserialization code (serial.py) to write your own scripts in python too. What I've coded are just examples/what I think is useful to me.

You start with your KLE (colours for multilayout not necessary). Keep in mind that apart from the image below (and the images in the guidelines), all files/images have been generated entirely by the script.

The idea is to then follow the following guidelines to be able to output various firmware-related files (e.g. a whole via.json).

NEW: Convert existing VIA/L JSON to Firmware Script KLE

This allows you to convert an existing VIA/VIAL json file into the layout used for this firmware script.

(If there are multilayout values): Multilayout labels will automatically go into the widest key of any given multilayout

Note: The it outputs in the format of a KLE json, and not raw data. If you want to use the output as a raw data (either to paste into the main page of the site, or to paste into KLE's raw data to modify) you will have to remove the first set of [] brackets.

Example:

Guidelines:

0: (info.json) If there is text here, it is included as the "label" in the info.json
1: (keymap.c) QMK Keycodes go here for layer 0 of the keymap.c, if nothing is included keys default to KC_TRNS
2: (VIAL only) If there is a u here, the key is included as a key for the unlock combo
3: Multilayout index
4: See Encoder - VIAL: add e to represent encoder CW/CCW key, VIA: add e# (# being the encoder index e.g. e0, e1, etc) for it to be recognised as an encoder)
5: Multilayout value
6: Secondary Multilayout name (if there is a list of multilayout options i.e. more than 2 e.g. multiple bottom row layouts. See example board below.)
7: Primary Multilayout name/label (needs to be in at least one of the keys for any given multilayout option. If there is a list of multilayout options, at least one key of each value should have a secondary label in position 6. See example board below.)
9: Row
11: Col

Note: The name of the board can also be edited in KLE

Note: Rotated layouts are currently not tested/fully supported. If you want to use this tool with layouts with rotated keys, I would recommend doing the whole thing in two separate 'steps'. Once with a simplified, non-rotated KLE (ie in a grid) to generate the keymap.c, info.json, etc. And the second time you can use your rotated KLE for the VIA/L json.

Example of the initial board being converted as per above guidelines:

The script takes in the json file of that KLE, downloaded as shown:

Note: You can also copy the raw data and manually paste.

Usage

First, read the documentation.

Use the website. OR if you still intend on using the code from this repo, modify the code as required in run.py and run it (this may sometimes be outdated compared to the site)

This is what the structure of your keyboard folder should look like if you plan to compile VIAL firmware.
The <kbd-name> folder should be somewhere inside the keyboards/ directory.
The website/script should generate all the necessary files except the rules.mk, which is generally empty or simple to define.

<kbd-name>
│  info.json       (Generated by site, you can configure it more if you want, see QMK documentation)
│  config.h        (Generated by site, HOWEVER is generally useless if using default layer count of 4)
│  rules.mk        (NOT generated by the site and is NOT optional, but can be left empty if everything is configured as required in info.json)
│  <kbd-name>.c    (Optional, can be used for things like LED definitions)
│  <kbd-name>.h    (Optional, for the LAYOUT macro, thus it's not required if the matrix is already defined properly in info.json)
│
└─ keymaps
   │
   └─ default      (Optional)
   │  │ keymap.c   (Generated by site, HOWEVER generally for the default keymap, all layers except the first are removed)
   │
   │
   └─ via          (Use either via or vial, NOT both)
   │  │ keymap.c   (Generated by site, you can use the VIA Layout File to set default keycodes - see documentation below)
   │  │ rules.mk   (NOT generated by site, just define VIA_ENABLE = yes, etc)
   │
   │  # NOTE: You must use VIA's 'Design' tab to load the via.json 
   │  #       (remove lighting property if using VIA v3 - see documentation below)
   │
   │
   └─ vial         (Use either via or vial, NOT both)
      │ keymap.c   (Generated by site, you can use the VIAL Layout File to set default keycodes - see documentation below)
      │ config.h   (Generated by site, for defining unlock combo, uid)
      │ rules.mk   (NOT generated by site, just define VIAL_ENABLE = yes, etc, see VIAL documentation)
      │ vial.json  (Generated by site)

VIA files

Note: Since VIA V3 has released, VIA V3 and VIAL definition files (i.e. via.json and vial.json) have split and are no longer cross compatible. However, VIAL jsons should still work with VIA V2. If you plan on making use of VIA V3 (versions of firmware compiled on the latest version of QMK should be VIA V3 compatible by default), you should just remove the lighting parameter from the output via(l).json. Also note that setting a unique Vendor ID and Product ID is required for VIA definitions to function correctly. Refer to this page for any additional required changes.

VIAL files

The script can generate a vial.json and accompanying config.h with required settings (unlock combo and uid). It automatically randomly generates the UID. No need to run the script manually and pain-stakingly paste it. It also automatically adds the unlock combo based on which keys were marked with a u in the input KLE. If no keys are assigned, it defines VIAL_INSECURE instead.

Example of the initial board being converted:

QMK info.json file

UPDATE: As on the website, you can now define certain things and it will add various configuration options to the info.json. e.g. MCU presets (currently there's just RP2040, 32U4 and STM32).

Note: You will have to manually set the specific STM32 model (e.g. STM32F072)

The script also automatically generates an info.json. Matrix labels are automatically added. Key labels/names are automatically added if applicable. It automatically detects (based on the multilayout keys), which combination of multi-layout options produces a layout with the most amount of keys. In other words, a LAYOUT for use with VIA/L. The keys are also offset appropriately.

~~WIP: I want to be able to automatically detect (based on the multilayout keys), which combination of multi-layout options produces a LAYOUT for use with via (maximum amount of keys).~~ DONE

WIP: Add a more generic converter. ~~WIP: Add the option to create more layouts based on multilayouts picked by the user~~ DONE

Example of the initial board being converted:

Below is what the LAYOUT looks like (represented in KLE). You can see how the script automatically picked the layout option with the most keys for each multilayout (e.g. the split backspace). If all multilayouts have the same amount of keys (e.g. non/stepped capslocks), the default one (value 0) is picked. You can also see how the everything is offset appropriately.

Note: I haven't tested this with more complex multilayouts or larger boards.

Alternate Layouts

You can specify different layouts to add extra alternative layouts (e.g. for QMK PR merging, where community layouts are required) to the info.json. Takes a a json/dict structured as shown in the example below.

More specifically: each dict key (e.g. "tkl_ansi") corresponds to an extra layout named LAYOUT_<key> (e.g. "LAYOUT_tkl_ansi"), and takes one list. Each list should be a list of integers referring to the chosen multilayout value, and should correspond to the multilayout indexes (in order).

Example (based on same KLE as used above):

Thanks to @awkannan for writing this part of the code for me.

Kicad Netlist

You can also use the output KiCAD Schematic Netlist file to fill in the info.json MCU pinout automatically. Otherwise, you will have to manually set the pins in the info.json

Currently supports RP2040 (tested with Sleepdealr's symbol), 32U4 and STM32 (KiCAD defaults should work) as described in the below image. Columns and rows should be numbered with positive integers starting from 0.

See below example:

Output:

Note: diode_direction is set to COL2ROW by default.

KEYMAP.C

You have two options for setting the default keycodes:

Use label 1 to set QMK keycodes for the keymap. Currently only applies to layer 0. Keys without this label will default to KC_TRNS. The script splits the lines by rows, but it's up to you to do the rest of the formatting.
(NEW): Once you complete the firmware, you can set your keycodes in the VIAL or VIA GUI and then re-run the site using the output .vil VIAL Layout file OR the .json VIA Layout file, and it will automatically input the keycodes. (See below examples):

Note: Some complex keycodes may have to be adjusted manually, particularly if using VIAL.

In VIAL:

OR in VIA:

Output:

LAYOUT macro (in kb.h file)

~~I plan on creating this soon. However, you can compile firmware without this through the use of the info.json and the matrix labels.~~ DONE.

This conversion directly reflects what QMK generates when you use info.json and the matrix labels.

Note: This can be useful for creating/visualizing your keymap, however is NOT required if you are using the generated info.json.

Encoder

The script also generates encoder stuff, just use the appropriate VIA/VIAL setup and it should make the appropriate changes to the info.json and keymap.c to enable encoders as well as encoder mapping. Make sure you add ENCODER_MAP_ENABLE to your keymap's rules.mk.

Output in info.json (you have to manually set the pins):

Output to keymap.c:

Note: Do not change the [NUM_ENCODERS][2], regardless of how many encoders you have. The 2 is based on the number of directions (there's always only CCW/CW).

VIAL Encoder

How to set up encoder for VIAL:

How it shows up in VIAL:

VIA Encoder

How to set up encoder for VIA:

How it shows up in VIA:

Extra

I plan on further improving the scripts, and also adding more features.

I also plan on soon implementing some code to convert from ~~VIA -> KLE, VIA -> info.json~~ (done), info.json -> KLE, etc.

WIP:

be able to re-configure the "guidelines"/labels as you wish
~~the info.json stuff mentioned above~~
~~generate layout macro (kb.h) and keymap.c~~ DONE on the website
more conversions
generate more firmware files (e.g. kb.c rgb stuff, ~~main config.h file~~, rules.mk file, maybe even kicad projects?)

Maybes:

automatically detect/generate matrix from just a KLE
be able to input a kicad project/pcb? automatically detect everything from the matrix, switch positions, etc? UPDATE: See here for a kicad switch placer plugin.
be able to generate a kicad project
be able to output things useful in blender (for the renderers)

zykrah/firmware-scripts