- Python 3 environment
- Restful Flask framework
- PostgreSQL 13 database
Click on each section down below to unfold more details.
Setup python virtual environment
# Windows
python -m venv \path\to\project\meteorology-api\venv
# Unix
python -m venv /path/to/project/meteorology-api/venv
\path\to\project\meteorology-api\venv\Scripts\activate.bat
source /path/to/project/meteorology-api/venv/bin/activate
deactivate
Install python dependency packages via pip
Notes:
- make sure you activate your virtual environment before installing, otherwise packages will be installed to your global python site packages
- Use path delimiting character corresponding to your OS (Unix, Windows)
pip install -r /path/to/project/meteorology-api/requirements.txt
Setup PostgreSQL database
Download PostgreSQL server here for your OS.
I'm using PostgreSQL 13, but the current project is version agnostic for any currently supported versions.
Once software is installed and server is started, create database and execute the schema file.
/path/to/postgres/bin/psql -U postgres -c "create database meteodb;"
Navigate to the project root directory meteorology-api.
psql -U postgres -d meteodb -f src/main/resources/database_schema.ddl
Note: schema DDL will create a role meteodba for a meteodb database with a default password meteodba123.
Starting API server
Navigate to the project root directory meteorology-api.
To start API server from the root project directory run the following command from the consul
python src\main\python\main.py
API Usage
Available paths and methods
Request
GET /sensors/
curl -X GET http://localhost:5000/sensors/ -H 'Content-Type: application/json'
Response
HTTP/1.1 200 OK
content-length: 567
content-type: application/json
date: Sun, 12 Dec 2021 21:59:21 GMT
server: Werkzeug/2.0.2 Python/3.9.7
[
{
"metadata": {
"city_name": "Galway",
"country_name": "Ireland"
},
"sens_id": 1
},
{
"metadata": {
"city_name": "Berlin",
"country_name": "Germany"
},
"sens_id": 2
},
{
"metadata": {
"city_name": "Berlin",
"country_name": "Germany"
},
"sens_id": 999
}
]
Request
POST /sensors/
curl -X POST http://localhost:5000/sensors/
-H 'Content-Type: application/json'
-d '{ "sens_id": 777, "metadata": { "country_name": "Germany", "city_name": "Berlin" }}'
Response
HTTP/1.1 201 CREATED
content-length: 129
content-type: application/json
date: Sun, 12 Dec 2021 21:56:15 GMT
server: Werkzeug/2.0.2 Python/3.9.7
{
"metadata": {
"country_name": "Germany",
"city_name": "Berlin"
},
"sens_id": 777
}
Request
GET /sensors/{id}/
curl -X GET http://localhost:5000/sensors/1/
-H 'Content-Type: application/json'
Response
OK
content-length: 111
content-type: application/json
date: Sun, 12 Dec 2021 22:02:38 GMT
server: Werkzeug/2.0.2 Python/3.9.7
{
"metadata": {
"city_name": "Galway",
"country_name": "Ireland"
},
"sens_id": 1
}
Request
GET /sensors/{id}/data/
curl -X GET http://localhost:5000/sensors/1/data/
-H 'Content-Type: application/json'
Response
HTTP/1.1 200 OK
content-length: 169
content-type: application/json
date: Sun, 12 Dec 2021 22:04:41 GMT
server: Werkzeug/2.0.2 Python/3.9.7
{
"sens_id": 1,
"data": [
{
"temperature": 13.5,
"humidity": 70,
"recorded": "2021-12-09 19:04:56"
}
]
}
Request
PUT /sensors/{id}/
curl -X PUT http://localhost:5000/sensors/1/
-H 'Content-Type: application/json'
-d '{"data": [{ "temperature": 14.1, "humidity": 17, "recorded": "2021-12-11 22:52:25.536249"},
{"temperature": 13.7,"humidity": 16,"recorded": "2021-12-11 21:52:25.536249"} ]}'
Response
HTTP/1.1 201 CREATED
content-length: 60
content-type: application/json
date: Sun, 12 Dec 2021 22:11:47 GMT
server: Werkzeug/2.0.2 Python/3.9.7
{
"message": "Recorded data for the sensor with id 1"
}
Request
DELETE /sensors/{id}/
curl -X DELETE http://localhost:5000/sensors/777/ -H 'Content-Type: application/json'
Response
HTTP/1.1 204 NO CONTENT
content-type: application/json
date: Sun, 12 Dec 2021 22:17:05 GMT
server: Werkzeug/2.0.2 Python/3.9.7
Challanges faced during development
My first and probably the most time-consuming challenge was to get up-to-speed with web server development. I had to carefully choose which REST API framework to use because it would be hard to pivot away due to time constraints.
To me, two apparent choices were Django and Flask Restful. I went ahead with the Flask Restful framework because:
- The Flask is WSGI, Django is a full-stack web framework. So since I don't need to design UI, Flask won here.
- Flask seems to be more flexible in design approach. Django is a feature-packed, heavier framework.
- Flask is more minimalistic, perfect for the timeframe I had.
- Django is monolithic; Flask is diversified. For RestFul micro-services, we don't do monoliths.
- As for ORM, both Django and Flask provide excellent support for it. Django has built-in ORM, providing native support; Flask uses SQLAlchemy. I decided not to use SQLAlchemy and designed ORM with PostgreSQL and psycopg2 driver.
The second challenge I faced was to pivot away from Flask's native marshaling feature.
I found it quite ugly and hard to understand. On top of that, Flask developers stopped developing that feature and recommended using something better.
That is where the Marshmallow came in handy. It is not only easy to use and grasp, but it also does a great job in encapsulation my model object.
Another time-consuming challenge was Marshmallow's struggle to serialize Decimal and datetime objects. Thankfully, it allowed me to implement a pre-dump method in which I could use simplejson library to serialize Decimal and datetime. https://github.com/eduards-v/meteorology-api/blob/main/src/main/python/models/sensor_model.py#L18
I had to take it further with the datetime object and implement a custom encoder extension for simplejson to cast it to a string because the psycopg2 driver returns the datetime object from the PostgreSQL database. However, it seemed to be the cleanest option. https://github.com/eduards-v/meteorology-api/blob/main/src/main/python/utils/json_encoders.py#L5
Desired improvements and features
- Add unit tests
- Exceptions handling, primerly from the database driver.
- Improve REST endpoints robustness. Drills down to exceptions handling.
- Partially move implementation from SensorsRepo class to SesnsorsService, specifically:
- SensorModel object instanciation
- Driver's results mutation, i.g., https://github.com/eduards-v/meteorology-api/blob/main/src/main/python/repo/sensors_repo.py#L21
- Driver can return simple dict structure that can be handled in service
- Methods input parameters can also accept a dict structure
- Repo can be decoupled from Model's implementation details
- These changes will make SensorsRepo class more tolerant to changes in the SensorModel structure
- Move API resources from the main.py to a separate package called resources
- DateEncoder should be renamed to DatetimeEncode. Can also handle date object in the same implementation.
- Add doc strings to the methods, explaining what each method does
- Fix a bug in DELETE sensor endpoint; does not return response body
- Probably much more I can't think of now
- Add sensor URL to the response body when creating a new sensor
{
"metadata": {
"country_name": "Germany",
"city_name": "Berlin"
},
"sens_id": 777,
"url": http://localhost:5000/sensors/777/
}
- Add sensors URLs to the response body when quering for all sensors
- Add sensor's data URL to the response body when quering specific sensor
{
"metadata": {
"city_name": "Galway",
"country_name": "Ireland"
},
"sens_id": 1,
"data_url": http://localhost:5000/sensors/1/data
}
- Allow to query sensor's data by date and calculate avarage metrics
Pseudo URL:http://localhost:5000/sensors/777/data?date=<date> - Allow to query sensor's data by the date range and calculate avarage metrics
Pseudo URL:http://localhost:5000/sensors/777/data?from=<date>&to=<date> - Query all sensors for the specified city
Pseudo URL:http://localhost:5000/city/<city_name>/sensors/ - Query all sensors for the specified country
Pseudo URL:http://localhost:5000/country/<country_name>/sensors/