Cloud Storage Analyzer project.
Description
A web application providing features such as data aggregation, anomalies detection and data inspection of files residing in AWS S3 buckets The tool leverages AWS Athena by using its JDBC driver
- Creation 01 March 2017
- Author Tewfik Ghariani
Configuration
Hi there!
Before starting to use the application, you have to set your environment. Don't worry, you need to follow these steps and you'll be ready!
Python 3
JAVA 8
Pip
$ sudo apt-get install python3-pip
$ sudo pip3 install virtualenv
Virtual Environment
$ virtualenv venv
Activate your virtual environment, (You can turn it off afterwards by typing simply deactivate)
$ source venv/bin/activate
Postgres:
PostgreSQL 9.4 version is mandatory since the application uses the JsonB object type.
$ sudo apt-get install postgresql-9.4
$ sudo su - postgres
$ psql
CREATE DATABASE athena_db;
CREATE USER si_aps WITH PASSWORD {% _<password>_ %} ;
ALTER ROLE si_aps SET client_encoding TO 'utf8';
ALTER ROLE si_aps SET default_transaction_isolation TO 'read committed';
ALTER ROLE si_aps SET timezone TO 'UTC';
GRANT ALL PRIVILEGES ON DATABASE athena_db TO si_aps;
\q
$ exit
Ref : Django + Postgres
Bower Configuration
$ sudo apt-get install npm
$ sudo npm install -g bower
In case of error:
$ sudo ln -s /usr/bin/bower /usr/local/bin/bower
or
$ sudo ln -s /usr/bin/nodejs /usr/bin/node
Python Dependencies
$ pip install -r requirements.txt
Front-end Dependencies
$ python manage.py bower update
DataBase Migration
$ python manage.py makemigrations
$ python manage.py migrate
$ python manage.py collectstatic (Only in prod)
Usage
Admin account
$ python manage.py createsuperuser
Run the server!
$ python manage.py runserver
Your environment is set up Follow these instructions to use our prototype and contribute in the development process of our analyzer