OpenFn Kit

Kit noun /kɪt/

A set of articles or equipment needed for a specific purpose.

a football kit
the next-generation openfn data integration kit

This repo contains runtime, tooling, libraries and components to support the next generation core openfn data integration pipeline.

It is a kitbag of Javascript-based components to support Lightning.

Prerequisities

asdf
pnpm

We use asdf to configure our local environments and ensure consistency of versions.

You should install asdf and the NodeJs plugin.

We use pnpm, a fast, disk space efficient package manager, to handle node dependencies within the repo.

Installing

$ pnpm run setup
$ pnpm build

Running Tests

pnpm run test

Development Guide

Thanks for being here! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock ❤️

Releases & Changesets

We use changesets to manage releases: github.com/changesets

A changeset is a description of batch of changes, coupled with semver information.

Adding a change

When submitting a PR against this repo, include a changeset to describe your work.

pnpm changeset

For example changeset notes, look in the .changesets folder.

Releasing

New releases will be published to npm automatically when merging into main.

Before merging to main, check out the release branch locally and run the following steps:

Run pnpm changeset version from root to bump versions
Run pnpm install
Commit the new version numbers
Run pnpm changeset tag to generate tags
Push tags git push --tags

Rememebr tags may need updating if commits come in after the tags are first generated.

TypeSync

This repo uses typesync to ensure that all packages have an appropriate @types/ package.

This must be run MANUALLY (See OpenFn#333)

On every add, update and remove. you should do pnpm run typesync from the repo root.

Note that @types packages only synchronise with the major and minor versions of a package. So for @types/x@major.minor.patch, major and minor refer to the versions of the corresponding package x, and patch is the version number of the actual types package.

Testing the CLI on a branch

From the repo root, run pnpm install:global.

This will build the CLI into ./dist, set the version to the current branch name, and install it globally as openfnx.

Run openfnx to use this dev-version of the CLI without overriding your production install.

This uses a similar technique to the release CLI below.

To remove the dev cli, run npm uninstall -g @openfn/clix

Testing the release CLI

You can test the built CLI package to ensure it works before publishing it.

The build/pack-local script is an overly complicated solution which:

Packs all the openfn packages
Updates the dependencies to use peer packages in dist, rather than module names
Creates openfn-cli-<version>-local.tgz which you can install globally.

To run the test:

$ pnpm build
$ pnpm clean:local
$ pnpm pack:local

Run the install command as printed in your shell - something like npm -g dist/openfn-cli-local.tgz

You can run openfn test to exercise the runtime and compiler.

Building into Lightning

Some components are designed to be run from inside Lightning.

To test a local package without publishing it to npm, run:

pnpm export adaptor-docs

This will build adaptor-docs into a tarball and install it directly into Lightning (assuming that Lightning is a sibling dir of kit).

Code Formatting with Prettier

Prettier ensures consistent code style throughout the project.

Editor Integration

Install the Prettier extension for your code editor to enable automatic formatting on save:

Visual Studio Code: Install the "Prettier - Code formatter" extension.

Manual Formatting

You can manually format files using Prettier by running:

pnpm run format

SatyamMattoo/kit