LSTM Time Series Forecasting with Drift Detection

This application demonstrates time series forecasting with LSTM neural networks, model drift detection, and automated retraining pipeline using InfluxDB 3 for data storage.

Features

Synthetic time series data generation
LSTM-based time series forecasting
Data drift injection and detection
Automated model retraining when drift is detected
InfluxDB 3 integration for data persistence
Hugging Face integration for model storage and retrieval
Streamlit web interface for visualization and interaction

Prerequisites

Python 3.11+
Docker (for running InfluxDB 3)
Hugging Face account (for model persistence)

Installation

1. Clone the repository

git clone https://github.com/Anaisdg/influxdb3_huggingface_forecasting_monitoring.git
cd influxdb3_huggingface_forecasting_monitoring

2. Set up a virtual environment

python -m venv venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

InfluxDB 3 Setup

1. Start InfluxDB 3 Core with Docker

docker run -it --rm --name influxdb3core \
  -v ~/influxdb3/data:/var/lib/influxdb3 \
  -p 8181:8181 \
  quay.io/influxdb/influxdb3-core:latest serve \
  --node-id my_host \
  --object-store file \
  --data-dir /var/lib/influxdb3

2. Create a database

In a new terminal, run:

influxdb3 database create --name timeseries

3. Create an API token

influxdb3 create token

Save the generated token for later use.

Configuration

Environment Variables

Create a .env file in the project root with the following variables:

# InfluxDB Configuration
INFLUXDB_HOST=http://localhost:8181
INFLUXDB_TOKEN=your_influxdb_token_here
INFLUXDB_DATABASE=timeseries
INFLUXDB_ORG=""

# Hugging Face Configuration
HF_TOKEN=your_huggingface_token_here
HF_REPO_ID=your_username/your_repo_name

You can also set these variables directly in your environment:

export INFLUXDB_HOST="http://localhost:8181"
export INFLUXDB_TOKEN="your_influxdb_token_here"
export INFLUXDB_DATABASE="timeseries"
export HF_TOKEN="your_huggingface_token_here"
export HF_REPO_ID="your_username/your_repo_name"

Hugging Face Repository

Create a new Hugging Face repository to store your models
Update the HF_REPO_ID in app.py to your repository ID (format: "username/repo-name")

Running the Application

streamlit run app.py

The application will be available at http://localhost:5000

Workflow

InfluxDB Configuration: Connect to your InfluxDB instance
Data Generation: Generate synthetic time series data
Initial Model Training: Train the LSTM model on the generated data
Drift Injection: Inject drift into the data to simulate changing data patterns
Drift Detection: Detect the drift using MSRE or MSE metrics
Model Retraining: Retrain the model on the drifted data
Model Persistence: Save and load models to/from Hugging Face

Project Structure

├── app.py                   # Main Streamlit application
├── utils/
│   ├── __init__.py
│   ├── data_generator.py    # Data generation utilities
│   ├── drift_detection.py   # Drift detection algorithms
│   ├── huggingface_utils.py # Hugging Face integration
│   ├── influxdb_utils.py    # InfluxDB utilities
│   └── model.py             # LSTM model definition
├── plugins/                 # Optional InfluxDB plugins
├── .streamlit/
│   └── config.toml          # Streamlit configuration
└── requirements.txt         # Project dependencies

Customization

Adjust LSTM hyperparameters in the Streamlit UI
Modify drift detection thresholds and metrics
Change data generation parameters
Connect to different InfluxDB instances

License

MIT

Acknowledgements

InfluxData for InfluxDB 3
Hugging Face for model hosting
PyTorch deep learning frameworks
Streamlit for the web interface

InfluxCommunity/influxdb3_huggingface_forecasting_monitoring