Alset is a small-scale proof-of-concept autonomous car which can react to traffic signs and navigate via GPS, follow road lanes, and pathplan around obstacles. Moreover, Alset can be used as a fully universal kit to make any and all RC cars intelligent. It is highly modular and very safe.
- Traffic sign detection
- Road/lane following
- High-speed FPV live stream
- Extensive communications
- Redundant safety features
- PS4 controller support
- GPS navigation webapp
- Phone app
- Pathfinding
- Modularity
- LCD display
- React to other traffic signs
- Better power management
- Hardware revision
- Crossroad detection & steering
- Max speed slider (WIP)
- Speed control (WIP)
- GPS TTS directions
Alset can be operated using a PS4 controller connected via bluetooth. In this scope, the MAC address of the console bound to the controller must be obtained using this tool and assigned to the ESP32. Upon connecting the controller, the robot starts in safe mode, where all input is ignored. When entering safe mode the robot stops all movement immediately. To begin driving, press the square button. It toggles safe mode, functioning as a kill switch. In order to drive forward, hold the right trigger. To brake, hold both the right and left trigger. To reverse, first brake, then hold the left trigger. The acceleration is proportional to the strength applied to the trigger. There is a turbo mode available by holding the triangle button, which increases the minimum and maximum speed. Moreover, by pressing the cross button, the user can switch between fully manual and assisted mode. In fully manual mode, the user controls both the motor throttle and the servo direction. In assisted mode, the pathfinding algorithm controls the direction and the user remains in control of throttle. This is done as an additional safety feature. Regardless of mode, feedback on distance to obstacles is provided to the user in two ways: first, the RGB leds on the controller fade from green to red depending on distance to nearest object; second, the dual vibration motors of the controller vibrate proportional to the distance to the nearest object on each side (front left and front right).
Three individual switches for motors, arduino circuit and raspberry pi + router allow for easy testing without any risks, as well as disabling features not currently desired (e.g. disabling raspberry pi when not using opencv). If any device loses power while driving or if signal is lost or too outdated, the car stops immediately.
The Haar cascade model is used with opencv, thus traffic signs can easily be implemented with the scripts in the "tools" folder. The pipeline consists of creating a reasonable number of positive samples (500+ images that contain the sign) and at least half the number of negative images (images that do not contain the sign). The dataset we used can be found here and consists of ~40000 images in total, all labeled in JSON files.
Next, a pos.txt file must be created containing all the positive image filenames (this can be achieved via the tools/parse_json.py script). It will be used for creating the .vec file. To do that, you will also need the opencv toolkit. On UNIX systems, the package manager can install everything for you, but on Windows you have to download the 3.4.x version, not the latest one, since future versions no longer contain the cascade training toolkit. After downloading the opencv tools, you are ready to start. To generate the .vec file mentioned earlier, you need to use the openv_createsamples program.
For example:
opencv_createsamples -info pos.txt -w 24 -h 24 -num 1000 -vec pos.vec
With the .vec file you've just created and a neg.txt file containing all the negative images filenames, you can use the opencv_traincascade program:
opencv_traincascade -data YourCascadeFolder/ -vec pos.vec -bg neg.txt -w 24 -h 24 -numPos YourNumOfPosImg -numNeg YourNumOfNegImgComplete documention on these commands can be found on the opencv website.
The final cascade.xml file can be found in YourCascadeFolder, as well as the stages (stage0.xml, stage1.xml, stage2.xml etc), which you won't need at the moment. They are mainly used for downgrading your cascade or for saving progress when the training stops unexpectedly.
Alternatively, you can use the unofficial GUI version.
The HAAR cascades are loaded at runtime by the raspberry pi, which uses opencv to recognize the signs captured by the camera. The generated output(position, distance etc) is then processed.
This feature works only on marked roads. It detects centre lines using Canny edge detection. After processing (grayscaling the image, blurring, applying edge detection to get contours, cropping it accordingly), a list of geometric lines is generated. Those lines are then added into a single vector, which will decide the direction the car has to go. That way, medium turns should be manageable without the driver's assistance. However, using it on unmarked roads or poorly marked may result in unexpected behaviour. An angle relative to the vertical will be computed and used for determining the direction the car must steer.
For GPS navigation, a U-Blox Neo-6M module is connected to the Pololu 328PB, which extracts latitude, longitude, speed and direction information from NMEA sentences using the TinyGPS++ library. Destination coordinates will be sent by the ESP32 from the Raspberry Pi webapp. Once the waypoint is selected, navigation data/steering information will be obtained through the GPRMB NMEA sentence. For route planning, the starting location is provided by the phone.
The communication with the raspberry pi is done through a Flask server, which can be accessed while in range of the car by connecting to its wi-fi router. The webapp allows the user to enter the destination address and, should it exist, a route will be selected by the raspberry pi, and longitude & latitude are passed on to the arduino. The backend for this is implemented using HERE Maps REST API.
A phone app made in the Blynk platform is available, providing a joystick, kill switch, and speed slider, along with some features not currently implemented on the arduino side. While both wifi and bluetooth are available, we have found bluetooth to be more reliable, and it also has the added advantage of not requiring an internet connection on the ESP32. While this app remains available for use when lacking a PS4 controller, we have currently abandoned it due to it's unstable connectivity and closed-source nature.
The security of Alset is as efficient as it is simple. There are three ways of interacting with Alset: the PS4 controller, the phone app, and the wi-fi network. Both the phone app and the PS4 controller use bluetooth and require initial pairing, which implies close proximity to the device and knowledge of the pairing PIN. Aditionally, the device will only connect to the MAC address of the controller, or, when using the phone app, to the unique API key assigned to the phone. Moreover, the raspberry pi is connected to a portable wifi router placed on the car, which is protected with a strong password. These features make it hard for any attackers to cause damage to/with Alset.
Every board is connected to the ESP32 via UART (since ESP32 does not properly support I2C slave mode), with the exception of the 328PB, which connects to the 32u4 via I2C and is forwarded to the ESP32 via UART (due to the lack of serial ports). The ESP32 acts as the main hub and logic controller, receiving any data from modules not directly connected to it through the SerialTransfer library. This allows us to update the code in only one place, while still relying on the extra boards, so as not to overload the ESP32, both in terms of processing power and GPIO/UART interfaces.
A low-latency first-person-view video stream can be broadcast by the Raspberry Pi. The stream is generated by the MJPG Streamer library from individual jpeg files taken by the webcam, reducing the processing power required and enabling better performance. The stream can be enabled by running the command /usr/local/bin/mjpg_streamer -i "input_uvc.so -f 30 -r 1200x500" \ -o "output_http.so -w /usr/local/share/mjpg-streamer/www", or more conveniently, the startStream script. To view the stream, access the address ip:port/?action=stream.
As you might know, Alset v1 used differential steering and was of much smaller scale. However, since the purpose of this project is to serve as proof-of-concept, we decided to create v2 to be as close as possible to a real car. Using a single motor and Ackermann steering, while also being much larger (1/10 scale), Alset v2 easily reaches that goal.
Another enhancement from v1 is the high modularity, enabling simple addition, removal and replacement of parts and providing easy access to everything. The base is attached to the body using the standard rc car system of shell clips, allowing easy universal mounting and unmounting within a blink. Furthermore, all that is required for Alset to interface with any electric RC car is the connection of just two cables from the car radio receiver to their socket on the Alset board: the one for the ESC and the one for the servo. Both are then driven by Alset using PWM and PPM (pulse-position modulation). Electronics aside, everything is mounted with screws, the batteries are connected with screw terminals and the weight distribution is optimised for the perfect driving experience.
Speed control is a work in progress. Currently, wheel speed can be measured using an optical speed sensor connected to an interrupt-capable pin. The speed control algorithm itself remains to be implemented soon. A slider for setting min and max speed range will also be enabled.
There are three power sources used on Alset, as follows: RC car is powered by its built-in NiMH battery pack, arduino circuit is powered by two 18650 cells in parallel, and a powerbank provides for the raspberry pi and router. The reasons for this seemingly complicated scheme are:
- since RC Cars require special battery packs with very high discharge rates and almost no safety features, they are a bad choice for powering anything other than the motors.
- the arduino circuits cannot procure enough power from usb, plus so many cables would be quite messy
- for a raspberry pi and router, a commercial powerbank makes much more sense than a custom circuit
- furthermore, the aforementioned two devices might be considered irrelevant by other users, and may be skipped altogether
In order to charge the batteries of the arduino circuit, just connect a regular usb charger to the module on the board. Do the same for the powerbank. For charging the battery of the RC car, since there are countless charger, battery, and connector types, and since Alset tries to be as cross-platform as possible, you should use the OEM hardware.
Debugging Alset v2 is made easy by the board design that allows easy testing of electrical connections and quick removal of components, as well as the debug flag available to turn on usb communications for inspecting every value. The LCD screen also displays many of these values, offering helpful insight without the need for a PC. Status LEDS will soon be enabled to further assist in this scope. In addition, all usb ports are placed on the same outer side of the robot to facilitate debugging.
- Raspberry Pi 3 B (openCV, server)
- Sparkfun ESP32 Thing (main processor, comms, IMU)
- Pololu A-Star Micro 32u4 (motors, ultrasonic & speed sensors)
- Pololu A-Star Micro 328PB (GPS, LCD, ultrasonic & voltage/current sensors)
- Maverick Quantum MT (RC car)
- TP-Link Portable 4G Router (optional)
- PS4 DualShock Controller (primary input)
- 6x HC-SR04 Ultrasound Sensors
- U-blox NEO-6M GPS
- Adafruit NXP 9DOF IMU
- 2004 I2C LCD
- Pololu 3.3V/5V Step-Up Regulator
- TP4056 Battery Charger
- 2x Switches
- 2x I2C Level Shifters
- 18650 2P Battery Holder
- Some things that are not currently implemented, such as: LEDs, level shifters, 8p DIP switch, LED bar graph, buzzer
Copyright 2021 Robert Saramet, Bogdan Maciuca
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.
-
- PS4 Controller
- Pathfinding
- Communcations
- Bluetooth app
- Hardware & electronics
- Documentation
- All arduino code
-
- Road/lane following
- Image recognition
- GPS navigation webapp
- Flask & Node servers
- Documentation
- All raspberry pi code
