SOLIDATA
backend

WARNING : almost deprecated

From now please refer to the cloned and enhanced repo : solidata-backend

PRESENTATION

part 2a/3 of the TADATA! sofware suite (ApiViz / Solidata_frontend / OpenScraper )

Building a public service for managing, consolidating, and sharing "api-fied" data

GOALS

allow you to consume data from your own csv/xls files or external APIs ;
consolidate your data : apply your own datamodel, simplify columns, apply metadatas, ... ;
share the consolidated datas on API endpoints with the level of openness you decide (opendata, commons, collective, private) ;
manage all your data and your recipes by projects ;
be able to manage projects by teams and share data/recipes/datamodels...

DEVELOPPERS

Nice to see you around ! For a more complete tour check out :

TECHNICAL POINTS

Tech stack

Language : Python 3.6... yes ... I know ... Python again ... hmmm ... gnnn ... don't judge me ?!
Framework : Flask... minimalistic Python framework
API : Flask-RestPlus... Swagger documentation integrated
Security : Flask-JWT-extended... wrapper JWT for Flask
Emailing : Flask-email... templating, sending, etc...
Data management : Pandas... to do operations on datasets...
Geocoding : Geopy... to use several geocoders : Nominatim, BAN, etc...

Features :

JWT (JSON Web Tokens) :
- access and refresh token for security over all the app
user :
- anonymous login (automatic)
- login / register user
- confirm user by sending a confirmation link (protected) in an email
- password forgotten by sending a link (protected) in an email with redirection to new password form
- reset password from client interface (protected) ...
Documentation
- on all API endpoints with Swagger (and some patience from the developer)

Current features TO DO :

user :
- edit user
- edit email
add/edit a project :
- edit team and access levels
- add project to users involved

WHAT'S NEW ?

check out our our changelog for more details

INSTALLATION WALKTHROUGH

LOCALLY

clone / fork the depo ...

create a virtual environment for Python3

 python3 -m venv venv
 pip install --upgrade pip
 pip install -r requirements.txt

create a secret config_file config_prod.py in the folder ./solidata_api based

 cp ./solidata_api/config_pro_exampled.py ./solidata_api/config_prod.py
 nano ./solidata_api/config_prod.py

pay attention at the MONGO_URI variable depending on your local mongodb configuration...
- The following is the standard URI connection scheme (from mongo documentation):
```
 mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
```
run the app in dev mode (without possibility to send email) :
```
 python appserver.py --mode=dev
```
run the app in dev mode (with possibility to send email) :
```
 python appserver.py --mode=dev_email
```

PRODUCTION

Check our snippets folder for more instructions
for now we are using the following configuration
- droplet in digitalocean.com
- ubuntu 18.04
- 3Go RAM / 2CPU
- 60Go memory
configure your server (user, firewall...):
- cf : docs 1
- cf : docs 2
- cf : docs 3
install mongodb :
- cf : docs 1
- cf : docs 2
- cf : docs 3
- cf : docs 4
install nginx :
- cf : docs 1
- cf : docs 2
- cf : docs 3

install git on your server

cf : docs 1 - basically :

 sudo apt-get update
 sudo apt-get install git
 git --version

go to your directory and init git :

 git init . 
 git remote add origin https://github.com/entrepreneur-interet-general/solidata_backend.git
 git pull origin master

same steps than for local installation (virtual env, install dependencies, config_prod.py file, ) ...
run the app in production mode (with possibility to send email) :
```
 python appserver.py --mode=production
```

INSPIRATIONS / BENCHMARK

Dataiku, Parabola, but they are proprietary solutions (and too complex for our pruposes)... Not to mention they are very expensive...
Match ID, but the backend doesn't resolve the "sharing" part with levels as ["opendata", "commons", "collective", "private"], and no BDD to back up recipes

CREDITS

Solidata's backend team thanks :

the SocialConnect project, aka "Carrefour des Innovations Sociales"
the EIG program by Etalab
the CGET
Christophe Ninucci for his help, advices and taste for memes
the creator and community of Flask RestPlus

Contacts - maintainance :

Julien Paris, developer (aka JPy on Twitter, or Jpy on Github)

entrepreneur-interet-general/solidata_backend

SOLIDATA backend