/Sql2Neo

A command-line tool to convert data between SQL, NoSQL and Neo4j databases

Primary LanguageHTML

Sql2Neo

A command-line tool to convert data between SQL, NoSQL and Neo4j databases

Due to lack of semantics, SQL records converted to Neo4j use FK (Foreign Key) as a general relationship.

Installation

  • Python 3.8
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.8

Setup a virtual environment and install packages from requirements.txt using python -m pip install -r requirements.txt.

  • MySQL: The simplest way to install MySQL server is to follow the DigitalOcean tutorial.

  • MongoDB: Install MongoDB by following the official instructions here.

  • Neo4j: It requires Java 11 to be installed. Instructions on installing both can be found here.

Usage

A .env file must exist in the project root with the following keys

  • MYSQL_DB_USER
  • MYSQL_DB_PASS
  • MYSQL_DB_NAME
  • MONGO_HOST (default: localhost)
  • MONGO_PORT (default: 27017)
  • MONGO_USER (default: None)
  • MONGO_PASS (default: None)
  • MONGO_DB_NAME
  • NEO4J_HOST (default: localhost)
  • NEO4J_PORT (default: 7474)
  • NEO4J_USER (default: neo4j)
  • NEO4J_PASS (default: neo4j)
  • NEO4J_SCHEME (default: http)

Run the commands from the Sql2Neo root directory.

After databases have been set up and environment variables have been provided, the following commands can be used:

  • Migrate source database to Neo4j: python -m src.sql2neo.py convert --src (mysql|mongo). Optionally, --db <db_name> option can be used to override the source database provided in .env
  • Translate SQL queries to Cypher: python -m src.sql2neo.py translate -q <query> [-f] <file.(txt|sql)> . If both the flags are provided, it defaults to -q. Furthermore, the translated queries can be run on Neo4j as well. To do so, pass --run flag. The default action is to use --dry-run.
  • Delete all created data (including indices and constraints): python -m src.sql2neo.py --delete-all.

Verbosity can be reduced by using the --suppress-logs flag with any of the commands.

Testing

  • SQL: Test database 'hosp' is used. This can be set up by running; mysql -u <username> -p < hosp.sql. (hosp.sql)
  • NoSQL 
    curl https://raw.githubusercontent.com/ozlerhakan/mongodb-json-files/master/datasets/students.json
mongoimport -d=test students.json
    curl https://raw.githubusercontent.com/ozlerhakan/mongodb-json-files/master/datasets/products.json
mongoimport -d=test products.json

Features

  • SQL to Neo4j conversion
    • Index creation
    • Node creation
    • Relationship creation
    • Indexing on non-primary attribute
  • NoSQL to Neo4j conversion
    • Node creation
    • Relationship creation
  • Query Translation
    • Support basic SELECT, WHERE, DISTINCT, AS, ORDER BY, and LIMIT queries
    • Support JOIN queries

Sql2Neo creates indices for all SQL tables on its primary key. Further, it creates a uniqueness constraint on all non-PK attributes that are unique. Finally, it creates FK relations from referencing table to the referenced table.