Thynkon/BI_ELIXIR_AWS

BI_ELIXIR_AWS

BI_ELIXIR_AWS

Table of Contents

1. Built With
2. Getting Started
   • Prerequisites
   • Configuration
   • Installation
3. Tests
4. Documentation
   • REST API
5. Debugging
6. License

Built With

• Elixir
• Phoenix
• SQLite
• Docker
• Espec
• ExAWS

(back to top)

Getting Started

This is microservice written in Elixir whose purpose is to AWS S3 as a data source to perform some techniques related to Business Intelligence.

Since this project is written in Elixir, you will need to install the BEAM.

Prerequisites

Erlang + Elixir 1.14.2 / OTP 25

• macOS

  brew install erlang elixir

• ArchLinux

  pacman -S erlang elixir

• Windows

  Checkout Elixir's documentation

Note that by installing Elixir's runtime, you will be able to use both mix (package management/build tool) and iex (interactive Elixir shell).

They both have the same version as the Elixir's compiler since they are part of Elixir's core. Unlike Python, Javascript or other programming languages, the package management tool is not a external tool that can be updaded individually.

mix are to Elixir what cargo is to Rust and go is to Go.

Configuration

In able to communicate with AWS servers, you have to store your credentials in environment variables.

Make sure the following environment variables are set:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_BUCKET

If you intend to run this project in a docker container, you have to define those credentials in the docker-compose.yml file.

cp docker-compose.yml.example docker-compose.yml

Then, assign your credentials to these variables:

environment:
  - AWS_ACCESS_KEY_ID=
  - AWS_SECRET_ACCESS_KEY=
  - AWS_BUCKET=
  - AWS_REGION=

Installation

In can either install Elixir's binaries directly on your machine or you can use a docker container that provides an already configured development environment.

On your machine

1. Install dependencies

   mix deps.get

2. Compile the dependencies

   mix deps.compile

3. Create database and lanch migrations

   mix ecto.create
   mix ecto.migrate

4. Finally, launch the web server

   mix phx.server

   Or you if want to have a console and execute Elixir code in live (just like Ruby on Rails console/irb):

   iex -S mix phx.server

Docker

If you don't want to clone this project in order to use the Docker images, you can pull them from the Dockerhub repository.

Development

1. Build docker's image

   docker compose build

2. Launch the container using docker compose

   docker compose --project-name bi_elixir_aws up -d

   This will open two ports: 4000 and 4369. The later is used by the observer. This tool allows to connect to a remote Elixir Node and watch the running processes (this is one of the reasons why Elixir is a perfect solution for distributed systems).

Production

Production docker image can be found under container/.

Rename docker-compose.yml.example to docker-compose.yml and set the environment variables.

NOTE: SECRET_KEY_BASE should have the string returned from mix phx.gen.secret.

1. Build docker's image

   docker compose build

2. Launch the container using docker compose

   docker compose up -d

Just like the development container, the production one exposes both 4000 and 4369 ports.

(back to top)

Tests

On your machine

In order to run all tests type:

MIX_ENV=test mix espec

If you want to run all tests of a file

MIX_ENV=test mix espec ./spec/<FILE_TEST>.exs

Or, if you only want to run a specific test

MIX_ENV=test mix espec ./spec/<FILE_TEST>.exs:<LINE_NUMBER>

If you want to generate a test coverage report, type the following:

MIX_ENV=test mix coveralls.html

On the container

In order to run all tests type:

docker compose exec -e MIX_ENV=test app mix espec

If you want to run all tests of a file

docker compose exec -e MIX_ENV=test app mix espec ./spec/<FILE_TEST>.exs

Or, if you only want to run a specific test

docker compose exec -e MIX_ENV=test app mix espec ./spec/<FILE_TEST>.exs:<LINE_NUMBER>

If you want to generate a test coverage report, type the following:

docker compose exec -e MIX_ENV=test app mix coveralls.html

Documentation

The documentation of this project can be genrated by ExDoc.

mix docs

Or, if you want to generate the documentation and open it in your browser, type:

mix docs --open

Directory structure

Just like most web applications written in Elixir, this project follows the directory structure defined for Phoenix. The excellent official documentation explains in great detail the purpose of each directory.

The unit tests (BDD) can be found under the spec/ directory.

And, both class (even though Elixir is a functional language and does not have classes) and sequence diagrams can be found under the docs/.

API

As this project implements a RESTful API, we implement the OpenAPI specification.

If you want to see the available routes, you can use the built-in Swagger UI by connecting to the following route: http://localhost:4000/swaggerui

Also, if you prefer to generate the specification file (in JSON), run the following command:

On your dev machine

mix openapi.spec.json --spec BusinessIntelligenceWeb.ApiSpec

Inside the container

docker compose exec app mix openapi.spec.json --spec BusinessIntelligenceWeb.ApiSpec

This creates a file named openapi.json at the project's root.

Debugging

Since Elixir has excellent debugging tools (thanks to the Erlang ecosystem), I thought I would be interesting to demonstrate how to use the observer. As you might have noticed, in this video, I am running this project inside a docker container.

Due to one of Elixir's core-values (distributed systems), we can easily connect to remote notes and debug them while they are running.

The following demonstration shows how to launch an Elixir remote session. Make sure you replace thynkon with your username when typing the following command:

iex --name thynkon@172.40.0.1 --cookie business_intelligence

debugging.mp4

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

To give you a better idea of what this project looks like, here the output of tree -f.

.
├── ./config
│   ├── ./config/config.exs       // Define application variables before compilation
│   ├── ./config/dev.exs          // Development configuration
│   ├── ./config/prod.exs         // Prod configuration
│   ├── ./config/runtime.exs      // Define application variables after compilation (load environment variables)
│   └── ./config/test.exs         // Test configuration
├── ./containers                  // Production containers
├── ./docker-compose.yml.example
├── ./Dockerfile
├── ./docs                        // UML diagrams
│   ├── ./docs/class_diagrams
│   └── ./docs/sequence_diagrams
├── ./entrypoint.sh
├── ./lib
│   ├── ./lib/business_intelligence      // Files related to business logic (models)
│   ├── ./lib/business_intelligence_web  // Files related to the web application (controllers, views)
│   │   ├── ./lib/business_intelligence_web/controllers
│   │   └── ./lib/business_intelligence_web/views
├── ./LICENSE
├── ./mix.exs
├── ./mix.lock
├── ./priv
│   ├── ./priv/gettext               // Translation
│   └── ./priv/repo                  // Migrations, seeders
│       ├── ./priv/repo/migrations
│       └── ./priv/repo/seeds.exs
├── ./README.md
└── ./spec                           // BDD test

If you want to better understand the difference between config.exs and runtime.exs, take a look at Elixir's documentation.

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Contact

• Thynkon

(back to top)