/LLM-Automapper

Primary LanguageJavaScript

LLM RAG C# Automapper Data Mapper

A powerful data mapping tool that uses Large Language Models (LLM) with Retrieval Augmented Generation (RAG) to intelligently generate C# AutoMapper code for transforming data between different structures.

What is AutoMapper?

AutoMapper is a popular object-to-object mapping library for .NET that simplifies the code needed to transform data from one type to another. Rather than writing tedious mapping code by hand:

targetObject.Property1 = sourceObject.Property1;
targetObject.Property2 = sourceObject.Property2;
// ... many more mappings

AutoMapper lets you define mapping configurations once and then transform objects with a single line of code:

var result = mapper.Map<TargetType>(sourceObject);

How This Tool Uses LLM AI

This tool combines the power of AutoMapper with artificial intelligence to:

  1. Understand Data Structures: The LLM (Llama3) analyzes your data structures, whether they're in JSON, CSV, plain text descriptions, or other formats
  2. Semantic Field Matching: Instead of rigid string matching, the AI understands the meaning of fields and matches them based on semantic similarity
  3. Code Generation: Automatically generates complete, compilable C# code with the appropriate AutoMapper configurations
  4. Knowledge Enhancement: Uses a vector database with RAG (Retrieval Augmented Generation) to enrich mappings with knowledge from similar structures
  5. Smart Transformations: Handles nested objects, arrays, and type conversions intelligently

Prerequisites

  • Node.js (v14.0.0 or higher)
  • Ollama installed and running locally
  • Llama3 model pulled in Ollama (ollama pull llama3)
  • An embedding model in Ollama (ollama pull nomic-embed-text:latest)

Installation

  1. Clone the repository:

    git clone https://github.com/yourusername/rag3-data-mapper.git
cd rag3-data-mapper

  2. Install dependencies:

    npm install

  3. Ensure Ollama is running with the required models:

    ollama serve
ollama pull llama3
ollama pull nomic-embed-text:latest

Usage

Loading Data Files

Load and embed data structure files from a directory:

npm run load -- ./data

This command:

  • Scans the specified directory for data structure files
  • Extracts structure information from each file
  • Creates vector embeddings for semantic search
  • Saves the processed structures to embedded-structures.json

Mapping Between Data Structures

Map fields between source and target data structures:

npm run map -- ./data/example-source.json ./data/example-target.json

This command:

  • Generates semantic mappings between fields using AI understanding
  • Uses any previously embedded structures to enhance mapping accuracy
  • Outputs field-to-field mappings with confidence scores
  • Generates C# code with AutoMapper for performing the data transformation
  • Saves the mapping code to a .cs file

Analyzing a Data Description

Analyze a plain text description of a data structure:

npm run analyze -- ./data/example-description.txt

This command:

  • Extracts structured information from a textual description
  • Identifies fields, types, and relationships
  • Provides an analysis of the implied data structure
  • Saves the extracted structure to a .json file

Querying the Vector Database

Search for information in the vector database:

npm run query -- "customer address fields"

This command:

  • Searches the vector database for relevant documents and fields
  • Provides analysis of how the query relates to existing data structures
  • Shows which fields and structures are most relevant to the query
  • Generates a markdown report with the search results

AI-Generated AutoMapper Code

The C# code generated by this tool includes:

  1. Complete Class Definitions: C# classes for both source and target data structures
  2. AutoMapper Profile: A configuration class that defines all mappings
  3. Complex Mappings: Handles nested properties, collections, and flattening/unflattening
  4. Type Conversions: Intelligently handles type differences between source and target
  5. Usage Example: Demonstrates how to use the mappings in a real application

For example, when mapping customer data between systems, the LLM AI might generate code like:

CreateMap<SourceCustomer, TargetClient>()
    .ForMember(dest => dest.ClientId, opt => opt.MapFrom(src => src.CustomerId))
    .ForMember(dest => dest.Name, opt => opt.MapFrom(src => new TargetName {
        First = src.FirstName,
        Last = src.LastName
    }))
    .ForMember(dest => dest.ContactInfo, opt => opt.MapFrom(src => new ContactInfo {
        EmailAddress = src.Email,
        Phone = src.PhoneNumber
    }));

How It Works

  1. Parsing: Files are parsed using a combination of direct parsing and LLM assistance
  2. Embedding: Field names and descriptions are embedded using vector representations
  3. Semantic Matching: The system uses semantic similarity to match corresponding fields
  4. RAG Enhancement: Knowledge from the vector database enriches the mapping process
  5. AI Code Generation: The LLM generates complete, customized AutoMapper code
  6. Field Confidence: Each mapping includes a confidence score to identify uncertain matches

Example Session

S C:\source\RAG3> npm run map -- ./data/example-source.json ./data/example-target.json

> rag3-data-mapper@1.0.0 map
> node index.js map ./data/example-source.json ./data/example-target.json

Mapping data from ./data/example-source.json to ./data/example-target.json...
(node:90456) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Loaded 4 embedded structures from cache
Enriching mapping with knowledge from vector database...
Found 3 relevant documents for source and 3 for target
Found 6 relevant fields for source and 6 for target

Field Mappings:
  customerInfo.customerId -> purchase.purchaseId (confidence: 0.81)
  customerInfo.firstName -> client.name.first (confidence: 0.84)
  customerInfo.lastName -> client.name.last (confidence: 0.83)
  customerInfo.email -> client.contactInfo.emailAddress (confidence: 0.84)
  customerInfo.phoneNumber -> client.contactInfo.phone (confidence: 0.84)
  customerInfo.address.street -> client.contactInfo.phone (confidence: 0.76)
  customerInfo.address.city -> client.contactInfo.mailingAddress.city (confidence: 0.89)
  customerInfo.address.state -> client.contactInfo.mailingAddress.stateProvince (confidence: 0.83)
  customerInfo.address.zip -> client.contactInfo.mailingAddress.postalCode (confidence: 0.75)
  customerInfo.address.country -> client.contactInfo.mailingAddress.countryRegion (confidence: 0.85)
  orderDetails.orderId -> purchase.purchaseId (confidence: 0.69)
  orderDetails.orderDate -> purchase.purchaseTimestamp (confidence: 0.75)
  orderDetails.items -> purchase.products (confidence: 0.79)
  orderDetails.items[0].productId -> purchase.products[0].id (confidence: 0.83)
  orderDetails.items[0].productName -> purchase.products[0].description (confidence: 0.84)
  orderDetails.items[0].quantity -> purchase.products[0].count (confidence: 0.77)
  orderDetails.items[0].unitPrice -> purchase.products[0].price (confidence: 0.80)
  orderDetails.totalAmount -> purchase.purchaseTotal (confidence: 0.72)
  orderDetails.paymentMethod -> purchase.paymentType (confidence: 0.81)

Enrichment from Vector Database:
  Source-related documents:
    - example-source.json (relevance: 0.99)
    - example-target.json (relevance: 0.90)
    - example-description.txt (relevance: 0.73)
  Target-related documents:
    - example-target.json (relevance: 0.98)
    - example-source.json (relevance: 0.88)
    - example-description.txt (relevance: 0.66)

Generated C# AutoMapper Code:
Here is the complete C# solution:

```csharp
using System;
using AutoMapper;

public class CustomerInfo
{
    public string customerId { get; set; }
    public string firstName { get; set; }
    public string lastName { get; set; }
    public string email { get; set; }
    public string phoneNumber { get; set; }
    public Address address { get; set; }
}

public class OrderDetails
{
    public string orderId { get; set; }
    public DateTime orderDate { get; set; }
    public List<OrderItem> items { get; set; }
    public decimal totalAmount { get; set; }
    public string paymentMethod { get; set; }
}

public class OrderItem
{
    public string productId { get; set; }
    public string productName { get; set; }
    public int quantity { get; set; }
    public decimal unitPrice { get; set; }
}

public class Address
{
    public string street { get; set; }
    public string city { get; set; }
    public string state { get; set; }
    public string zip { get; set; }
    public string country { get; set; }
}

public class ClientContactInfo
{
    public string emailAddress { get; set; }
    public string phone { get; set; }
    public MailingAddress mailingAddress { get; set; }
}

public class MailingAddress
{
    public string addressLine1 { get; set; }
    public string city { get; set; }
    public string stateProvince { get; set; }
    public string postalCode { get; set; }
    public string countryRegion { get; set; }
}

public class ClientPurchase
{
    public string purchaseId { get; set; }
    public DateTime purchaseTimestamp { get; set; }
    public List<Product> products { get; set; }
    public decimal purchaseTotal { get; set; }
    public string paymentType { get; set; }
}

public class Product
{
    public string id { get; set; }
    public string description { get; set; }
    public int count { get; set; }
    public decimal price { get; set; }
}

// AutoMapper Profile
public class CustomerInfoMapperProfile : Profile
{
    public CustomerInfoMapperProfile()
    {
        CreateMap<CustomerInfo, Client>();
        CreateMap<OrderDetails, Purchase>();
        CreateMap<OrderItem, Product>();
    }
}

// Sample Usage
public class Program
{
    public static void Main(string[] args)
    {
        // Create the AutoMapper instance
        var config = new MapperConfiguration(cfg => cfg.AddProfile<CustomerInfoMapperProfile>());
        var mapper = new Mapper(config);

        // Map CustomerInfo to Client
        CustomerInfo customerInfo = new CustomerInfo();
        client client = mapper.Map<Client>(customerInfo);

        // Map OrderDetails to Purchase
        OrderDetails orderDetails = new OrderDetails();
        Purchase purchase = mapper.Map<Purchase>(orderDetails);

        // Map OrderItem to Product
        OrderItem orderItem = new OrderItem();
        Product product = mapper.Map<Product>(orderItem);
    }
}

The above code generates the necessary classes for both the source and target structures, defines an AutoMapper profile class that configures the mappings, includes a sample implementation showing how to use the mapper, handles nested objects and arrays properly, includes any necessary type conversions, and returns the complete C# solution including necessary using statements.

C# AutoMapper code saved to data\Mappingexample-sourceToexample-target.cs



## License

MIT