/ecommerce-product-search

Primary LanguageJupyter Notebook

Ecommerce Product Search

Fully functional application that demonstrates semantic search and semantic search + GenAI with a 150K product reference dataset.

The dataset is a CSV file that has the following record format

id name title ean short_description img_high img_low img_500x500 img_thumb date_released supplier price
3920564 56ABCD Some Title ['0095205603217'] Some Short Description http://someurl.jpg http://someurl.jpg http://someurl.jpg http://someurl.jpg 2009-12-10T00:00:00Z Some Supplier 19.99

The following technologies were used to build this demo

  1. Pinecone - Python Library
  2. Pinecone - Node.js Library
  3. LlamaIndex
  4. Llama Hub - Paged CSV Loader
  5. OpenAI - Python Library
  6. OpenAI - Node.js Library

Overview

Architecture - Semantic

Demo Architecture

Architecture - GenAI

Demo Architecture

Data Pipeline

  1. Load CSV into LlamaIndex + Generate meta-data for each record
  2. Generate vector embeddings for each record
  3. Persist batches of vector embeddings into Pinecone

Application

  1. User provides search query: "Samgsung"
  2. Application generates vector embedding from query
  3. Application queries pinecone for most similar vectors - Semantic Semantic Screenshot
  4. Application asks LLM to generate a response - GenAI Semantic Screenshot

Setup

There are 2 main components 1) Data Pipeline and 2 ) Application. The data pipeline is packaged as a notebook and the Application is a next.js based React application.

Data Pipeline

Prerequisites

  1. python 3.11+ or later installed on your system.
  2. Ability to run Jupyter notebook files. VS code is highly recommended.
  3. A "standard" or "enterprise" plan Pinecone account because this dataset exceeds the free plan limit.

Quickstart

  1. [OPTIONAL] Create a .env file that has the following entries

    PINECONE_API_KEY="YOUR_API_KEY"
PINECONE_ENVIRONMENT="YOUR_ENVIRONMENT"
PINECONE_INDEX_NAME="YOUR_INDEX_NAME"

OPENAI_API_KEY="YOUR_OPEN_API_KEY"

  2. Run the steps in this data pipeline notebook.

    ./data-pipeline/product-search-genai.ipynb

Application

You must have node.js 16.14 or later installed on your system.

Quickstart

  1. Open a terminal window
  2. Run: cd ./application
  3. Run: npm install
  4. Run: npm run dev

Authors

James Williams

Kevin Butler