/knnelasticsearch

Building a k-NN similarity search engine using Amazon Elasticsearch and SageMaker

Primary LanguageJupyter Notebook

Overview

This is sample project shows how to build a scalable similarity questions search Engine using Amazon Sagemaker, Amazon Elasticsearch, Amazon Elastic File System (EFS) and Amazon ECS.

https://towardsdatascience.com/building-a-k-nn-similarity-search-engine-using-amazon-elasticsearch-and-sagemaker-98df18d883bd

Jupyter notebook

You can use knn-search.ipynb to transform and index dataset to Elasticsearch.

Prerequisites

The following must be done before following this guide:

  • Setup an AWS account.
  • Configure the AWS CLI with user credentials.
  • Install AWS CLI.
  • jq.

Build docker image

Create a ECR repo in ECS console, copy repo url and build docker.

$docker build -t YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com/knnsearch .

Push docker image to ECR

$eval $(aws ecr get-login --region ap-southeast-2 --no-include-email)

$docker push YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com/knnsearch

Deploy infrastructure stack

$cd cf-templates
$bash deploy-infra.sh

Create Container task definition

$bash create-task.sh

Deploy ECS service stack

$bash deploy-service.sh