Python web server for the Altair hands on test
This is the recommended way for testing the application as it doesn't need any additional configuration. Note that this is not a production ready configuration and should only be used in one node.
The app comes with a docker-compose.yml
file which can be used to deploy a stack of containers using Docker Swarm
spawning a server and three different clients within the same network.
First we need to start a Docker Swarm node using:
$ docker swarm init
Then we can deploy the stack using:
$ docker stack deploy -c docker-compose.yml altair-stack
Creating network altair-stack_altair-network
Creating service altair-stack_altair-client-2
Creating service altair-stack_altair-client-3
Creating service altair-stack_altair-server
Creating service altair-stack_altair-client-1
This will spawn a server, three clients, and a network which handles communication amongst them.
As we can see the containers don't start in a specific order, and as we didn't provide internal failover for the clients, they will fail if they start before the server. However, Swarm handles this by restarting dead containers until they are successfully running.
We can then check the state of the services using:
$ docker stack ps altair-stack
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
z3l3ma728ys8 altair-stack_altair-client-1.1 prodalia/altair-client:1.0.0 docker-desktop Running Running 17 seconds ago
ts8gl7y0qfnw altair-stack_altair-client-3.1 prodalia/altair-client:1.0.0 docker-desktop Running Running 15 seconds ago
mzan8v67vm1d altair-stack_altair-client-2.1 prodalia/altair-client:1.0.0 docker-desktop Running Running 15 seconds ago
ytg4b1phbxtk altair-stack_altair-server.1 prodalia/altair-server:1.0.0 docker-desktop Running Running 17 seconds ago
ngzqd8o4jlgw altair-stack_altair-client-3.1 prodalia/altair-client:1.0.0 docker-desktop Shutdown Failed 21 seconds ago "task: non-zero exit (1)"
7ug06lv585nm altair-stack_altair-client-2.1 prodalia/altair-client:1.0.0 docker-desktop Shutdown Failed 21 seconds ago "task: non-zero exit (1)"
Here we can see how two of the clients failed at first (as they tried to start before the server was up) but are now successfully running.
We can check the logs of any of our services using $ docker service logs <service>
:
Note: We can use the ID (
9phs3kei28cv
) or the NAME (altair-stack_altair-server.1
)
- Server logs:
$ docker service logs -f 9phs3kei28cv
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,740 [ MainThread ] [ INFO ] BeerDB Instance running
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,740 [ MainThread ] [ INFO ] Instance with id 140564541893072 is observing DB updates
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,740 [ MainThread ] [ INFO ] Instance with id 140564541893136 is observing DB updates
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,742 [ MainThread ] [ INFO ] Started Web Service listening for connections on port 8338
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,815 [ MainThread ] [ INFO ] 101 GET /sockets/beers (10.0.0.4) 0.47ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,815 [ MainThread ] [ INFO ] A new tap has been connected to the system
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,818 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 1 posted a new beer (id: 0)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:57,818 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 0.70ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:59,584 [ MainThread ] [ INFO ] 101 GET /sockets/beers (10.0.0.4) 0.35ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:59,584 [ MainThread ] [ INFO ] A new tap has been connected to the system
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:59,586 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 2 posted a new beer (id: 1)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:53:59,587 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 0.70ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:00,094 [ MainThread ] [ INFO ] 101 GET /sockets/beers (10.0.0.4) 0.33ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:00,094 [ MainThread ] [ INFO ] A new tap has been connected to the system
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:00,098 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 3 posted a new beer (id: 2)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:00,099 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 1.07ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:00,824 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 1 posted a new beer (id: 3)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:00,825 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 1.27ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:02,107 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 3 posted a new beer (id: 4)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:02,108 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 1.28ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:03,833 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 1 posted a new beer (id: 5)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:03,834 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 1.12ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:04,115 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 3 posted a new beer (id: 6)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:04,116 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 1.10ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:04,840 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 1 posted a new beer (id: 7)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:04,841 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 1.24ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:08,123 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 3 posted a new beer (id: 8)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:08,124 [ MainThread ] [ INFO ] 200 POST /api/beers (10.0.0.4) 1.04ms
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:09,595 [ MainThread ] [ INFO ] New DB Hook Notification: Tap with id 2 posted a new beer (id: 9)!
altair-stack_altair-server.1.ytg4b1phbxtk@docker-desktop | 2019-09-17 18:54:09,595 [ MainThread ] [ INFO ] {'tap 1': '40.0%', 'tap 2': '20.0%', 'tap 3': '40.0%', 'total beers': 10}
- Client logs:
$ docker service logs -f qij24he6jzv0
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:53:57,811 [ MainThread ] [ INFO ]
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop |
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | - Inet Addr: altair-server:8338
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | - Tap ID: 1
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop |
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:53:57,812 [ MainThread ] [ INFO ] Attempting to connect to: ws://altair-server:8338/sockets/beers
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:53:57,815 [ MainThread ] [ INFO ] Posting beer...
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:53:57,818 [ MainThread ] [ INFO ] Tap with id 1 posted a new beer (id: 0)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:00,820 [ MainThread ] [ INFO ] Posting beer...
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:00,821 [ MainThread ] [ INFO ] Tap with id 2 posted a new beer (id: 1)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:00,822 [ MainThread ] [ INFO ] Tap with id 3 posted a new beer (id: 2)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:00,825 [ MainThread ] [ INFO ] Tap with id 1 posted a new beer (id: 3)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:03,829 [ MainThread ] [ INFO ] Posting beer...
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:03,830 [ MainThread ] [ INFO ] Tap with id 3 posted a new beer (id: 4)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:03,833 [ MainThread ] [ INFO ] Tap with id 1 posted a new beer (id: 5)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:04,836 [ MainThread ] [ INFO ] Posting beer...
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:04,838 [ MainThread ] [ INFO ] Tap with id 3 posted a new beer (id: 6)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:04,841 [ MainThread ] [ INFO ] Tap with id 1 posted a new beer (id: 7)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:09,850 [ MainThread ] [ INFO ] Posting beer...
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:09,853 [ MainThread ] [ INFO ] Tap with id 3 posted a new beer (id: 8)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:09,853 [ MainThread ] [ INFO ] Tap with id 2 posted a new beer (id: 9)!
altair-stack_altair-client-1.1.z3l3ma728ys8@docker-desktop | 2019-09-17 18:54:09,854 [ MainThread ] [ INFO ] {"tap 1": "40.0%", "tap 2": "20.0%", "tap 3": "40.0%", "total beers": 10}
The app is meant to be run using a Dockerized environment. The latest image of the app is hosted in Docker Hub under
the tag prodalia/altair-server:latest
and it is public.
This scripts use the altair-network so it should be created first if we want to use it
docker network create altair-network
The following command is used to run the app using the latest image:
$ docker container run \
-d \
--name altair-server \
--network altair-network \
-p 8080:8338 \
prodalia/altair-server
The server can also be run as a python process using
$ PORT=8338 python server.py
This will download the image and run a container named altair-server
binding the app to the host port 8080
. This
binding can be done to any other host port, but the container port has to remain the same (8338
) as it is the port
the Web Service will be listening to.
We can then interact with the app by running one or more clients or using postman.
The App logs detailed information about the processes, and helps understand how it works. In order to see the logs in real time we can do so using the command:
$ docker container logs -f altair-server
The repository also contains a Dockerfile so it is possible to build new images after modifying the source code using the following command:
$ docker image build -t altair-server-mod .
The client is also meant to be run as a Docker container, and takes environment arguments so we can manually set the
tap_id
and the inet_addr
so it can communicate with the server.
$ docker run \
-d \
--env INET_ADDR=altair-server:8338 \
--env TAP_ID=1 \
--network altair-network \
--name altair-client \
prodalia/altair-client
The client can also be run as a python process using
$ INET_ADDR=localhost:8338 TAP_ID=1 python client.py
The scenario this Web App works on is something like the following:
Multiple beer taps are provided with an IoT device which can measure the amount of beer they pour on each glass and send this information to a centralized system. It will be the WebServer's task to identify which type of glass has been served to the customer based on the volume the devices report for each pouring. This communication gets done using the HTTP protocol and a REST interface.
On the other hand, the centralized system registers connected taps and sends a signal every time a new beer is poured somewhere. Ideally it should not notify the tap that has triggered the operation, but we will keep it simple so the message will be broadcasted to all connected devices. It also sends in a report with some metrics to all devices periodically. These feedback communication is performed using the WS protocol.
This hands on exercise has been developed using as fewer third party libraries as possible. The only dependency required is the Tornado Web Framework for python.
The main goal of this Web Service is to handle HTTP requests in a RESTful manner and notify the connected devices about
changes in the system using the WebSocket protocol. It opens a listener on port 8338
for incoming requests on the
endpoint http:/api/beers
and lets clients subscribe to notifications on the ws:/socket/beers
.
For simplicity the app only allows the POST
, GET
and DELETE
method using basic numerical ids and the database collection
only stores beers using the following model:
{
"tapId": "<integer>",
"volume": "<double[100, 1000]>"
}
The complete model for a beer
resource contains more data which is inferred from the volume (the type
property) or
created when saving the document (timestamp
and id
properties).
The three basic operations that can be performed against this API are:
- Create a beer: Using the
/api/beers
endpoint and thePOST
method with a valid beer model. - Get a list of served beers: Using the
/api/beers
endpoint and theGET
method. This endpoint is paginated and filterable and accepts query parameters such aspage
,page_size
andtype
. - Get a beer using its ID: Using the
/api/beers/:id
endpoint and theGET
method and a validID
(it will return a404 Not Found
if there are no matches). - **Delete a beer using its ID:**Using the
/api/beers/:id
endpoint and theDELETE
method and a validID
.
This API not have any kind of authentication handlers or validation methods for the incoming requests, so it is not robust and has not been hard tested.
Upon initialization, the app will spawn a new DB using the BeerDB class, which is a Singleton for this data structure and provides the database of its basic operations (Create, Read and Delete for simplicity). The use of the Singleton pattern here is used so that it guarantees that the DB instance is unique and the collection will be preserved after multiple instantiations of the database across the code. The BeerDB class is also built using the Observer pattern, so other classes can be attached to it and be notified when changes occur within the collection.
The Singleton implementation used here is not thread-safe, but as no multi threaded processes are being used we can live with it.
Database operations are treated here as asynchronous operations as they usually are when working with real databases, which justifies the use of coroutines for these methods although it can be technically incorrect.
As Request Handler objects are created on each request, we need another class that subscribes to the BeerDB in order to
notify connected devices. This way, the RealTimeHandler will just store the WebSocket connections in a set and we can use
the Notifier class to subscribe to the DB and broadcast messages to all the devices reading their connections from
the taps
set.
Another design pattern used is the Decorator pattern, which in this context helps reuse the pagination and serialization logic for different database read operations.
Most of the decisions taken while developing this App where based on simplicity and quickness, and as such there is a lot of room for improvement. Here I list some of these improvements that would logically follow the development in order to build a more robust and functional app.
- Some kind of authentication method should be applied to both REST operations and WebSocket connections. While the API
could be public for any
GET
operation, thePOST
andDELETE
methods should be protected. - Connected devices only exit when they are unable to connect to the server via WebSockets. This is intended because in a containerized environment we could retry this connection by restarting the service instead of introducing logic within the client app. However, there are many other failover techniques which should be implemented both on client and server side in order to make this a robust system.
- Error handling is not configured. Some exceptions are raised, but exceptions coming from the Tornado Framework are not mapped nor handled. However the right pattern seems to be extend the Tornado Web Handlers using a base class for common tasks such as error handling and user authentication and subclassing this custom handler for any specific request handling we want to perform.
- App structure is something I still struggle with, and possibly could be improved.