neo4j/import-spec

Neo4j Import Specification Format

Scope

This library provides a uniform configuration facade for tools running imports to Neo4j. In particular, it offers:

• a user-friendly configuration surface (in JSON or YAML), called import specification, backed by a JSON schema
• the Java equivalent of the import specification, a.k.a. org.neo4j.importer.v1.ImportSpecification
• source plugins (see SourceProvider Service Provider Interface)
• action plugins (see ActionProvider Service Provider Interface and built-in plugins)
• validation plugins (see SpecificationValidator Service Provider Interface and built-in plugins)

The library does NOT:

• define any sources, you need to define and register at least one (see examples)
• implement any actual import to Neo4j (although some end-to-end tests just do that)
• expose any configuration to locate a Neo4j instance to import data to

Getting Started

First, implement and register a source provider for BigQuery.

Then, save the following import specification into spec.json:

{
  "version": "1",
  "config": {
    "key": "value"
  },
  "sources": [
    {
      "name": "my-bigquery-source",
      "type": "bigquery",
      "query": "SELECT id, name FROM my.table"
    }
  ],
  "targets": {
    "queries": [
      {
        "name": "my-query",
        "source": "my-bigquery-source",
        "query": "UNWIND $rows AS row CREATE (n:ANode) SET n = row"
      }
    ]
  },
  "actions": [
    {
      "name": "my-cypher-action",
      "type": "cypher",
      "stage": "start",
      "query": "CREATE CONSTRAINT a_node_id FOR (n:ANode) REQUIRE n.id IS UNIQUE"
    }
  ]
}

You can then deserialize it and run your import logic accordingly:

import org.neo4j.importer.v1.ImportSpecificationDeserializer;
import org.neo4j.importer.v1.targets.Targets;

import java.io.Reader;

class GettingStarted {

    public static void main(String... args) {

        try (var reader = new InputStreamReader(createReaderFor("/import/spec.yaml"))) {
            var pipeline = ImportPipeline.of(ImportSpecificationDeserializer.deserialize(reader));
            pipeline.forEach(step -> { 
                switch (step) {
                    case SourceStep source -> handleSource(source);
                    case ActionStep action -> handleAction(action);
                    case TargetStep target -> handleTarget(target);
                }
            });
        }
    }
}

Prerequisites

• Maven
• JDK 21 (21 is used for tests, 11 and 17 for production sources)