/Automated-Image-Workflow-Challenge

Build an image analyzer that will rate an image using results from Amazon Rekognition features and post high ranking image to social media.

Primary LanguageC#

Fotostatur

Photostat - noun a still picture taken with a camera Source

In this challenge, we'll build an image analyzer that will rate the image using different criteria of your choosing by running the image through most of the Amazon Rekognition features.

An image passing the required criteria is deemed worthy of a profile picture and subsequently update social media profiles.

Helpful Links

.NET AWS SDK Docs

Pre-requisites

The following tools and accounts are required to complete these instructions.

Tools, Infrastructure, and Code Setup

Setup LambdaSharpTool and Deploy Skeleton Code

  1. From any directory, install/update the λ# tool.

    dotnet tool install -g MindTouch.LambdaSharp.Tool --version 0.4.0.4

    -- Or if already installed, to update:

    dotnet tool update -g MindTouch.LambdaSharp.Tool

  2. Once installed/updated, the λ# CLI needs to be configured/updated.

    dotnet lash config

  3. Initialize/update deployment tier with the λ# runtime.

    dotnet lash init --tier Sandbox

  4. Clone this Repo.

  5. From the root of the cloned repo, deploy the skeleton code. NOTE: This is the command to deploy any future changes.

    cd Automated-Image-Workflow-Challenge/
dotnet lash deploy Fotostatur --tier Sandbox

    See Setup LambdaSharp CLI & Runtime for detailed instructions and more info about the LambdaSharpTool. Note: LambdaSharp Contributors need to unset the LAMBDASHARP environment variable if you want to use dotnet lash

  6. Find the FotostaturPhotoBucket bucket name in the CloudFormation outputs:

    Hint: From the AWS console navigate to Cloudformation > Sandbox-Fotostatur > Resources

  7. Test deployment by taking a picture or choosing an image to upload.

    In the command below replace SOURCE_IMAGE.JPG and FOTO_STATUR_PHOTOBUCKET for their respective names.

    From the directory where the image resides, run the command:

    aws s3 cp SOURCE_IMAGE.JPG s3://FOTO_STATUR_PHOTOBUCKET/images/SOURCE_IMAGE.JPG

    Note: It is being uploaded to an images path.

  8. Check the lambda function's CloudWatch logs to see if it was invoked with the S3 Upload. In the logs it should show the final score of 0.

  9. From the directory where the image resides delete the image from S3.

    rm s3://FOTO_STATUR_PHOTOBUCKET/images/SOURCE_IMAGE.JPG

    Note: It is being deleted from an images path.

Note: While you could use the AWS console to upload an image, it's highly recommended for speedier development to use the AWS CLI to upload and remove images from S3.

Understanding the Challenge Setup

Confidence score info/background and helper functions

Confidence Scores

About Confidence scores

A confidence score is a number between 0 and 100 that indicates the probability that a given prediction is correct.

Confidence scores are provided in most responses from Rekognition.

Helper Function

A helper function to add totals is provided. Use the following code snippet when needed:

AddTotals("criteria label", (float) 99.1234);

Where criteria label is any string to identify the criteria, and the float which is the confidence score.

Level 1

Detect Objects and Scenes

In Fotostatur/ImageAnalyzer/Functions.cs edit the DetectLabels and ScoreLabels methods.

DetectLabelsAsync Method

  1. Detect the labels from the image uploaded.
  2. Score the results from detect labels using the AddTotals helper function to keep a running total.

Deploy with dotnet lash deploy Fotostatur --tier Sandbox

Test your code by using the CLI commands above to deploy, upload and remove images from S3.

Use CloudWatch to see the final score change based on criteria added.

Level 2

Detect Text

In Fotostatur/ImageAnalyzer/Functions.cs edit the DetectText and ScoreText methods.

DetectTextAsync Method

  1. Detect any text from the image uploaded.
  2. Score the results from detect text using the AddTotals helper function to keep a running total.

Deploy with dotnet lash deploy Fotostatur --tier Sandbox

Test your code by using the CLI commands above to deploy, upload and remove images from S3.

Use CloudWatch to see the final score change based on criteria added.

Level 3

Compare Faces

In Fotostatur/ImageAnalyzer/Functions.cs edit the CompareFaces and ScoreCompare methods.

CompareFacesAsync Method

  1. Add a "headshot" of someone to the Fotostatur/headshots directory.
  2. In Fotostatur/Module.yml, find HeadshotFileName and replace the TODO with the filename (include extension) of the headshot.

    Note: These variables can be accessed from: _headshotFileName, _comparingImageBucket, and _comparingImageKey. See InitializeAsync in Fotostatur/ImageAnalyzer/Functions.cs.

  3. Compare the headshot and the image uploaded.
  4. Score the results from comparison using the AddTotals helper function to keep a running total.

Deploy with dotnet lash deploy Fotostatur --tier Sandbox

Test your code by using the CLI commands above to deploy, upload and remove images from S3.

Use CloudWatch to see the final score change based on criteria added.

Level 4

Detect Faces

In Fotostatur/ImageAnalyzer/Functions.cs edit the DetectFaces and ScoreFaces methods.

DetectFacesAsync Method DetectFacesAPI

  1. Detect faces from the image uploaded.
  2. See ScoreFaces for a commented list of objects to generate criteria with. Use at least one.
  3. Score the results from generated criteria using the AddTotals helper function to keep a running total.

Deploy with dotnet lash deploy Fotostatur --tier Sandbox

Test your code by using the CLI commands above to deploy, upload and remove images from S3.

Use CloudWatch to see the final score change based on criteria added.

BOSS

Upload Image to Twitter Profile

To obtain Twitter keys, use the steps provided at the presentation. To create tokens you must have a Twitter account and create an app on the Twitter Developer Site.

  1. Encrypt the Twitter keys TwitterConsumerKey, TwitterConsumerSecret, TwitterAccessToken, TwitterAccessSecret.

    lash encrypt SECRET

  2. In Fotostatur/Module.yml, uncomment the TwitterConsumerKey, TwitterConsumerSecret, TwitterAccessToken, TwitterAccessSecret and add each encrypted value to their respective location.

In Fotostatur/ImageAnalyzer/Functions.cs do the following

  1. In method InitializeAsync, uncomment _consumerKey, _consumerSecret, _accessToken, _accessTokenSecret

  2. In method ProcessMessageAsync, uncomment await DownloadS3Image();

  3. In method DownloadS3Image, save the image locally from S3

  4. In method ProcessMessageAsync, uncomment ResizeImage();

  5. In method ResizeImage, process the image

    See ImageSharp Resize Example

  6. In method ProcessMessageAsync, uncomment TwitterUpload();

  7. In method TwitterUpload, change the LOCAL FILE PATH

  8. In method ProcessMessageAsync, edit the _criteriaThreshold from 50 to desired number.

  9. (Optional) In method ProcessMessageAsync, uncomment UploadImage();

  10. (Optional) In method UploadImage, make backup in s3 of the image posted

Deploy with dotnet lash deploy Fotostatur --tier Sandbox

Test your code by using the CLI commands above to deploy, upload and remove images from S3.

Use CloudWatch to see the final score change based on criteria added.

Check Twitter for the new profile picture.

Teardown

Remove Fotostatur and Lambdasharp installation

Removing Fotostatur

  1. Empty the S3 bucket for Fotostatur (the one uploading images to).
  2. On the CloudFormation page in the AWS console, select the stack named {tier}-Fotostatur, click Actions and then Delete Stack.

Removing LambdaSharp Tier Installation

  1. On the S3 bucket page in the AWS console, empty the bucket that begins with {tier}-lambdasharp-s3package.
  2. Delete the stack {tier}-LambdaSharp

Removing the LambdaSharpTool Installation

  1. On the S3 bucket page in the AWS console, empty the bucket that begins with lambdasharptool-default-deploymentbucket.
  2. Remove the cli profile stack named LambdaSharpTool-{cli-profile}.