/csv-import-db

Just another import db by directory wathing from ftp to database using rust

Primary LanguageRust

Multi-Database CSV Importer

🚀 Overview

A robust, flexible Rust-based CSV importer that supports multiple databases, real-time file watching, data validation, and notification systems.

✨ Features

  • Multi-Database Support

    • PostgreSQL
    • MySQL
    • MariaDB
    • SQLite (Planned)

  • Advanced CSV Processing

    • Real-time directory watching
    • Automatic CSV file detection
    • Data validation
    • Duplicate detection
    • Batch insertions

  • Flexible Notifications

    • Telegram
    • Email (Planned)
    • Slack (Planned)
    • Custom notification channels (Planned)

🛠 Installation

Prerequisites

  • Rust (latest stable version)
  • Cargo
  • Database servers (PostgreSQL, MySQL)

Setup

  1. Clone the repository
git clone https://github.com/irfnrdh/csv-import-db.git
cd csv-import-db
  1. Install dependencies
cargo build

📝 Configuration

Create a config.toml file with your database and notification settings:

[database]
types = ["postgresql", "mysql"]
host = "localhost"
port = 5432
username = "your_username"
password = "your_password"

[notification]
telegram_token = "your_telegram_bot_token"
telegram_chat_id = "your_chat_id"

🚀 Usage

cargo run --release

Generate Dataset Samples

 cargo run --bin generate_csv

📂 Project Structure

multi-db-csv-importer/
│
├── src/
│   ├── main.rs             # Main application entry point
│   ├── config/             # Configuration handling
│   │   ├── mod.rs
│   │   ├── database.rs
│   │   └── notification.rs
│   │
│   ├── importers/          # CSV import logic
│   │   ├── mod.rs
│   │   ├── csv_processor.rs
│   │   └── validator.rs
│   │
│   ├── database/           # Database connectors
│   │   ├── mod.rs
│   │   ├── mysql.rs
│   │   ├── postgres.rs
│   │   └── sqlite.rs
│   │
│   └── notifications/      # Notification systems
│       ├── mod.rs
│       ├── telegram.rs
│       └── email.rs
│
├── tests/                  # Unit and integration tests
│   ├── database_tests.rs
│   └── importer_tests.rs
│
├── Cargo.toml              # Project dependencies
├── Config.toml             # Project env
├── README.md               # Project documentation
├── LICENSE                 # Project license
└── .gitignore              # Git ignore file

🧪 Testing

Run tests with:

cargo test

🤝 Contributing

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

📄 License

Distributed under the MIT License. See LICENSE for more information.

📧 Contact

Irfannur Diah - irfnrdh@gmail.com

Project Link: https://github.com/irfnrdh/csv-import-db