Oracle Functions is a serverless, highly scalable, fully managed Functions-as-a-Service platform built on Oracle Cloud Infrastructure and powered by the open-source Fn Project engine. Developers can use Oracle Functions to write and deploy code that delivers business value without worrying about provisioning or managing the underlying infrastructure. Oracle Functions is container-native, with functions packaged as Docker container images.
This reference architecture shows how you can use a serverless function to automate the process of extracting data from files generated by various databases or applications and loading the data into a data warehouse for analysis.
The architecture diagram shows the data flow. When zipped CSV files are uploaded to a specific bucket in Oracle Cloud Infrastructure Object Storage, the Events service is triggered. The emitted event invokes a function, which extracts the data from the uploaded files and loads the data into an Oracle Autonomous Data Warehouse instance. After the data is loaded to the data warehouse, the processed CSV files are moved to a different bucket in Oracle Cloud Infrastructure Object Storage.
For details of the architecture, see Automate loading data to a data warehouse using a serverless application
-
Permission to
managethe following types of resources in your Oracle Cloud Infrastructure tenancy:vcns,internet-gateways,route-tables,security-lists,subnets,autonomous-database-family, andfunctionss. -
Quota to create the following resources: 1 VCN, 1 subnets, 1 Internet Gateway, 1 route rules, 1 function, and ADW instance.
If you don't have the required permissions and quota, contact your tenancy administrator. See Policy Reference, Service Limits, Compartment Quotas.
-
Click
If you aren't already signed in, when prompted, enter the tenancy and user credentials.
-
Review and accept the terms and conditions.
-
Select the region where you want to deploy the stack.
-
Follow the on-screen prompts and instructions to create the stack.
-
After creating the stack, click Terraform Actions, and select Plan.
-
Wait for the job to be completed, and review the plan.
To make any changes, return to the Stack Details page, click Edit Stack, and make the required changes. Then, run the Plan action again.
-
If no further changes are necessary, return to the Stack Details page, click Terraform Actions, and select Apply.
Now, you'll want a local copy of this repo. You can make that with the commands:
git clone https://github.com/oracle-quickstart/oci-arch-load-file-into-adw-python.git
cd oci-arch-load-file-into-adw-python
ls
First off, you'll need to do some pre-deploy setup for Docker and Fn Project inside your machine:
sudo su -
yum update
yum install yum-utils
yum-config-manager --enable *addons
yum install docker-engine
groupadd docker
service docker restart
usermod -a -G docker opc
chmod 666 /var/run/docker.sock
exit
curl -LSs https://raw.githubusercontent.com/fnproject/cli/master/install | sh
exit
-
Complete the prerequisites described here.
-
Create a
terraform.tfvarsfile, and specify the following variables:
# Authentication
tenancy_ocid = "<tenancy_ocid>"
user_ocid = "<user_ocid>"
fingerprint = "<finger_print>"
private_key_path = "<pem_private_key_path>"
# Region
region = "<oci_region>"
# Compartment
compartment_ocid = "<compartment_ocid>"
# dbpwd-cipher
dbpwd-cipher = "<ADW_password>"
# OCIR
ocir_repo_name = "<ocir_repo_name>"
ocir_user_name = "<ocir_user_name>"
ocir_namespace = "<ocir_namespace>"
ocir_docker_repository = "<ocir_docker_repository>"
ocir_user_password = "<ocir_user_password>"
Run the following commands:
terraform init
terraform plan
terraform apply
When you no longer need the deployment, you can run this command to destroy the resources:
terraform destroy
