Browser-Based Data Transfer System

This repository contains a browser-based data transfer system that enables efficient movement of data between disparate systems using the browser as an intermediary with persistence capabilities.

Project Overview

The browser-based data transfer system provides a solution for transferring large datasets between systems by leveraging browser capabilities for:

Data fetching and submission via the Fetch API
Persistent storage using IndexedDB
Resilient processing with automatic retry mechanisms
Pause/resume capability for long-running transfers
Progress tracking and user feedback

Architecture Documentation

The system architecture is documented through a series of diagrams that explain different aspects of the implementation. All diagrams use Mermaid syntax, which can be rendered in GitHub, VSCode (with plugins), or using the Mermaid Live Editor (https://mermaid.live/).

sequenceDiagram
    participant Browser as Web Browser
    participant FetchAPI as Browser Fetch API
    participant IndexedDB as Browser IndexedDB
    participant SourceServer as Source Server API
    participant TargetServer as Target Server API
    
    Note over Browser: User initiates data transfer
    Browser->>IndexedDB: Create TransferSession
    IndexedDB-->>Browser: Session Created
    
    loop Until all data is transferred
        Browser->>FetchAPI: Fetch data chunk with pagination
        FetchAPI->>SourceServer: HTTP GET request
        SourceServer-->>FetchAPI: JSON response with data
        FetchAPI-->>Browser: Process response
        
        Browser->>IndexedDB: Store chunk in dataChunks store
        IndexedDB-->>Browser: Chunk stored with "pending" status
        
        par Process chunks in parallel
            loop For each pending chunk
                Browser->>IndexedDB: Query for pending chunks
                IndexedDB-->>Browser: Return pending chunks
                Browser->>FetchAPI: Send chunk to target server
                FetchAPI->>TargetServer: HTTP POST with chunk data
                TargetServer-->>FetchAPI: Process confirmation
                FetchAPI-->>Browser: Success/failure response
                
                alt Successful transfer
                    Browser->>IndexedDB: Update chunk status to "completed"
                else Failed transfer
                    Browser->>IndexedDB: Update chunk status to "failed"
                end
            end
        end
    end
    
    Browser->>IndexedDB: Update Session status to "completed"
    Note over Browser: Transfer complete notification to user

Available Diagrams

System Architecture
- Detailed sequence diagram showing interactions between system components
IndexedDB Schema
- Class diagram showing the database schema for persistence
Chunk State Machine
- State transitions for data chunks throughout the processing lifecycle
Event-Driven Flow
- Flowchart illustrating the event-driven architecture
Browser APIs Integration
- Detailed sequence diagram showing interactions with browser APIs
IndexedDB Operations
- Sequence diagram for database operations
State Management
- State diagram for session status with pause/resume logic
Event-Driven Components
- Class diagram showing the event-driven components
Flow Summary
- High-level graph summary of the data transfer flow
Architecture Description
- Detailed explanation of the architecture, including context, problem statement, and solution approach

Repository Structure

data-transfer-mfe/ - Main implementation of the browser-based transfer system
express-server/ - Example source server implementation
express-receiver/ - Example target server implementation
shell/ - Application shell for hosting the micro-frontend
generic-service-mfe/ - Generic service micro-frontend
data/ - Sample data files

Getting Started

To run the system locally:

Start the source server:

cd express-server
npm install
npm start

Start the target server:

cd express-receiver
npm install
npm start

Start the application shell:
```
cd shell
npm install
npm start
```
Navigate to http://localhost:3000 to access the application

Key Features

Event-driven architecture for efficient resource utilization
Chunked data processing for handling large datasets
Parallel processing with controlled concurrency
Persistent state management for resilience
Progress tracking and status reporting