Here's the Devpost for our project! Maskus was built for Stanford's TreeHacks 2021 Hackathon, and built by a team of 4: Alex Cui, Charlie Luo, Akash Binoj, and Alex Shevchuk. The project won Best Hardware Hack and Best Use of ML. Check it out!
During modern times, the idea underlying a facemask is simple--if more people wear them, less people will get sick. And while it holds true, this is an oversimplification: the number of lives saved is dependent not only on the quantity, but also on the quality of the masks which people wear (as evidenced by recent research by the CDC). However, due to an insufficient supply of N95 masks, healthcare workers are forced to wear cloth or surgical masks which both leak from the sides, increasing the risk of infection, and are arduous to breathe through for extended physical exertion.
Maskus is the first mask bracket and fitter in one - custom-fitted and printed using accessible technology. It is designed to improve the baseline quality of facemasks around the world, with its first and most pressing use is for healthcare workers. The user starts taking a picture of their face through their computer/smartphone camera. We then generate an accurate 3D representation of the user's face and design a tight-fitting 3D printable mask bracket specifically tailored to the user's face contours. Within seconds, we can render the user's custom mask onto the user's face in augmented reality in realtime. The user can then either download their custom mask in a format ready for 3D printing, or set up software to print the mask automatically. We also have an Arduino Nano that alerts the user if the mask is secured properly, or letting them now it needs to be readjusted.
After the user visits the Maskus website, our React frontend sends a POST request to a Python Flask backend. The server receives the image, decodes it, and feeds it into a state of the art machine learning 3D face reconstruction model (3DDFA). The resultant 3D face model then goes through some preprocessing, which compresses the 3D data to improve performance. Another script then extracts the user's face contour/outline from the 3D model and builds a custom mask bracket with programmable CAD software. On the web app, the user gets to see both their own 3D face mesh as well as an AR rendering of the custom fitted mask onto their face (using React and three.js). Lastly, this data is saved to a standard 3D printing file format (.obj) and returned to the user so they can print it wherever they like. In terms of our hardware, the mask's alert system comprises of an Arduino Nano with a piezo buzzer and two push buttons (left and right side of face) wired in series. In order to get the push buttons to engage when the mask is worn, we created custom 3D parts that create a larger area for the buttons to be pushed.
This project was touched many disciplines, and posed many difficulties. We were determined to provide the user with the ability to see how their mask would fit them in real time using AR. In order to do this, we needed a way to visualize 3D models in the web. This proved difficult due to many misleading resources and weak documentation. Simple things (like figuring out how to get a 3D model to stop rotating) took much longer than they should have, simply because the frameworks were obfuscated. AR was also very difficult to implement, particularly due to the fact that it is a new technology and the existing frameworks for it are not yet mature. Our project is one of the first we've seen placing 3D models (not images) onto user faces.
From the machine learning side of the project, 3D face reconstruction is a very difficult problem. Luckily, our team was able to succesfully implement and use the 3DDFA state of the art machine learning model for face reconstruction. Installing and configuring the neccessary Python packages and virtual environments posed a challenge at the start, but we were able to quickly overcome this and get a working machine learning pipeline. Being able to solve this problem early on in the hackathon gave our team more time to focus on other problems, such as web 3D model visualization and constructing the facemask from our 3D face model.
Amusingly, during this project we found that things which were supposed to be difficult turned out to be easy to implement and, conversely, the easy parts turned out to be hard. Things like front end design and integrating web frameworks turned out to be some of the most challenging parts of the project, whereas things like machine learning were easier than expected. A takeaway is that the feasibility of quickly building a project should be based not only on the difficulty of the task, but also on the quality of existing resources which can be used to build it. Good frameworks make implementing difficult projects much easier.
Aside from refactoring the code and improving webpage design, we see several things for the project going forward. Perhaps the biggest points is developing a reliable algorithm to extract the facemask outline from a 3D face model. The one the group currently has works most of the time, but serves as the bottleneck of the system in terms of facial recognition accuracy. The UI design can be improved as well. Lastly, threeJS was found to be a pain, especially when trying to integrate it with React. It would be worth exploring simpler JavaScript frameworks. We would also love to add more functionality to the Arduino in the future, making it a 'smarter' mask. We hope to add sensors like AQS (Air Quality Sensor), creating alerts if the mask has been worn too long and needs to be replaced, and status LEDs in order to visually tell your mask is secure.
In terms of future growth, Markus can comfortably be deployed as a web app and used by healthcare workers around the world in order to decrease risk of COVID transmission. It is a low cost solution designed to work with existing masks and improve upon them. Opening up the software to open source contribution is a potential way to grow, and we hope it would lead to very fast progress.