XMLSyncGo 📄🔄🐹

A Go tool for synchronizing job data between a remote XML file (via SFTP) and a local database (JSON files). It efficiently handles additions, updates, and removals of jobs, tracking changes and providing statistics. Ideal for automating data synchronization tasks.

Features

SFTP Retrieval: Connects to a remote SFTP server and downloads the XML file. Only downloads if the remote file has been modified since the last sync.
XML Parsing: Parses the XML file, extracting job data and flattening the structure for easier processing. Tracks the frequency of different XML tags.
Data Integrity Check: Ensures that all jobs have a required identifier (external reference key) for proper matching.
Database Synchronization:
- New Jobs: Adds new jobs found in the XML file but not in the local database.
- Updated Jobs: Updates existing jobs in the local database with data from the XML file. Tracks specific changes (added/removed/edited keys).
- Removed Jobs: Marks jobs as deleted in the local database if they are present locally but not in the remote XML file.
Change Tracking: Logs all changes to jobs, including timestamps and the type of change (added, removed, edited).
Statistics: Tracks and displays statistics on the number of jobs added, removed, and updated.
Local Data Storage: Stores job data and file metadata (remote file modification time) in local db file.
Configurable: Uses a JSON configuration file for SFTP credentials, file paths, and other settings.
Scheduled Runs (Optional): Can be run periodically to automate synchronization.
Cross-Platform: Compiles to a single binary for easy cross-platform deployment.
Web App: Access a local web app dashboard. Built with Vue.js and Bootstarp 5.

Getting Started

Prerequisites

Go 1.x
Access to an SFTP server with the XML file to track.

Installation

go get https://github.com/joaooliveirapro/xmlsyncgo

Create the config.json file.

// config.json example
{
  "root": "data",
  "job_node": "job",
  "external_reference_key": "job_id",
  "remote_filepath": "/path/to/remote/jobs.xml",
  "filename": "jobs.xml",
  "sftp_credentials": {
    "hostname": "your_sftp_host",
    "port": "22",
    "username": "your_sftp_user",
    "password": "your_sftp_password"
  }
}

License

The MIT License (MIT)