In this sample, we're trying to do some basic analysis on the HappyDB dataset, by doing word count and drawing word cloud, to highlight the words appear more frequently in the people's happy moments.
Table of content
Create Azure Storage Account and File Share
Making ETLing simpler using containers as a plug and play model.
The application with separate common pieces of the ETLing process into separate docker containers. For example, the unzip container in the project will take a link as an input then download and unzip the file at that link. A separate container takes a csv location as an input and puts it into a Postgres database. This allows for plug and play ETLing pipelines for the data.
Using ACI, a user can define container groups with the exact elements they want. For example put the unzip and postgres modules together and to download a zip file from a datasource, unzip it then feed it into a databases all without writing a line of code. This allows you to only pay per second using the ACI instance.
This document will guide you through the steps to deploy the solution to your environment.
An Azure Active Directory (AAD) is required to register the app registrations. In this document, the AAD will be called "ETL AAD", and an account in the ETL AAD will be called ETL work account.
- All app registrations are created in the ETL AAD.
An Azure Subscription is required to deploy the Azure components. The ARM Template deploys these Azure components automatically.
Note: The following commands are running on your own computer.
-
Clone the repository with Visual Studio Code.
-
Open View > Integrated Terminal.
-
Execute the commands below to build and push the extracting image to the Docker Hub.
cd cmd/extracting docker build -t YOURDOCKERACCOUNTNAME/extracting . docker push YOURDOCKERACCOUNTNAME/extracting
-
Execute the commands below to build and push the transforming image to the Docker Hub.
cd ../extracting docker build -t YOURDOCKERACCOUNTNAME/transforming . docker push YOURDOCKERACCOUNTNAME/transforming
-
Execute the commands below to build and push the loading image to the Docker Hub.
cd ../extracting docker build -t YOURDOCKERACCOUNTNAME/loading . docker push YOURDOCKERACCOUNTNAME/loading
-
Execute the commands below to build and push the rendering image to the Docker Hub.
cd ../extracting docker build -t YOURDOCKERACCOUNTNAME/rendering . docker push YOURDOCKERACCOUNTNAME/rendering
Note: The following commands are running on Azure Portal.
-
Open the Shell in the Azure Portal.
-
Execute the command below to choose your subscription.
az account set --subscription SELECTED_SUBSCRIPTION_ID -
Execute the commands below to create a new resource group.
Note: Change the placeholder
[RESOURCE_GROUP_NAME]to the name for the new resource group you will create.ACI_PERS_RESOURCE_GROUP=[RESOURCE_GROUP_NAME] ACI_PERS_STORAGE_ACCOUNT_NAME=mystorageaccount$RANDOM ACI_PERS_LOCATION=eastus ACI_PERS_SHARE_NAME=acishare az group create --location eastus --name $ACI_PERS_RESOURCE_GROUP
-
Execute the commands below to create the storage account.
az storage account create \ --resource-group $ACI_PERS_RESOURCE_GROUP \ --name $ACI_PERS_STORAGE_ACCOUNT_NAME \ --location $ACI_PERS_LOCATION \ --sku Standard_LRS
-
Execute the commands below to create the file share.
export AZURE_STORAGE_CONNECTION_STRING=`az storage account show-connection-string --resource-group $ACI_PERS_RESOURCE_GROUP --name $ACI_PERS_STORAGE_ACCOUNT_NAME --output tsv` az storage share create -n $ACI_PERS_SHARE_NAME
-
Execute the commands below to shows the storage account you created.
echo $ACI_PERS_STORAGE_ACCOUNT_NAME
-
Click this button to navigate to the Azure Portal deployment page.
-
Choose the resource group created in the previous steps.
-
Fill in the values on the deployment page.
- Storage Account Name: the name of the storage account created in the previous steps
- Storage Share Name: the name of the file share created in the previous steps, which is
acishareby default - Administrator Login: the user name of the Postgres database
- Administrator Login Password: the password of the Postgres database, it must meet the complexity requirements, e.g.
postgres1! - Extracting Container Image: the Docker image you built for the extracting container
- Transforming Container Image: the docker image you built for the transforming container
- Loading Container Image: the docker image you built for the loading container
- Rendering Container Image: the docker image you built for the rendering container
- Check I agree to the terms and conditions stated above.
-
Click Purchase
-
Open the resource group you just created.
-
Click the container group MS-ACIAKS-ETLContainerGroups, notice the state of the first 3 containers is Terminated, while the last one is always Running to serve the Word Cloud.
-
Copy the IP address from the container group blade, and open it in the browser to check the Word Cloud.
Note: The following commands are running on Azure Portal.
-
Open the resource group you just created.
-
Click the Postgres database, remember the Server name and Server admin login name.
-
Open the Cloud Shell on the top bar.
-
Run this command to add the updated Azure Database for PostgreSQL management extension.
az extension add --name rdbms -
Run the psql command to connect to PostgreSQL database, and enter the password chosen in previous steps when prompted for password.
Note: Please replace the <Server name> and <Server admin login name> with the values shown in Postgres resource blade.
psql --host=<Server name> --username=<Server admin login name> --dbname=postgres
-
Run the following SQL command to create index on the column name in the table words.
CREATE INDEX ON words(name);
-
Run the following SQL command to query the first 10 records in the table words.
SELECT * FROM words LIMIT 10;
-
Now you can see the word count data like this.
- Akari Asai, Sara Evensen, Behzad Golshan, Alon Halevy, Vivian Li, Andrei Lopatenko, Daniela Stepanov, Yoshihiko Suhara, Wang-Chiew Tan, Yinzhan Xu, ``HappyDB: A Corpus of 100,000 Crowdsourced Happy Moments'', LREC '18, May 2018. (to appear)