Conventional Lane Detection with OpenCV

Project requirements

Minimum is to detect the lanes on the Project Video
Segmentation of the image
Preprocessing (also Camera Calibration)
Use of OpenCV (Color Spaces, Histograms, ...)
In real time (at least 20 fps) without displaying the image
Increase speed of lane detection by some way
Show the curve radius

Our results

Lane Detection on not only the Project Video but also on the Challenge Video
- Project Video with Lane Detection
- Challenge Video with Lane Detection
- The Images are also in the Docs/Images folder
Over 70 FPS without visualization:
Frames: 1260
Time: 17.145667791366577
FPS: 73.48795131995107
Over 30 FPS with visualization
We also implemented Lane Detection on the Harder Challenge Video with Machine Learning, aswell as License Plate Detection, in our other repository: Additional Tasks

Installation

We recommend Python Venv to install the dependencies.

Linux:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Windows:

python -m venv venv
venv\Scripts\activate.bat
pip install -r requirements.txt

Usage

To run the lane detection on the Project Video please stay in the root directory of the project and run the following command:

Without visualization:

python ./src/main.py --video ./Images/Udacity/challenge_video.mp4

With visualization:

python ./src/main.py --visualize --video ./Images/Udacity/challenge_video.mp4

Step by Step Image Processing

To visually follow the steps of the image processing, check our Step by Step visualization Jupyter Notebook.
By doing so we also visually present most of the functions we calll and use.

More Details

For more details on the implementation and the functions we used, check our:

Discussion of the results

The results on the Project Video and the Challenge Video are quite good. Even curves, shadows, etc. are no problem for our algorithm.
This was achieeved by using a very exact ROI and good preprocessing. The parameters were tuned in a long iterative process.

The algorithm is also quite fast. We can process the Project Video with over 70 FPS without visualization.
This is achieved by leaving out the formerly included Hough Transform and using a very exact ROI.
The call graph enabled us to find the biggest slow down which was reading the image ino memory. Therefore we parallelize the Input reading of the next image with the processing of the current image.

However the algorithm is not perfect. It is not able to detect the lanes on the Harder Challenge Video. The reasons for this are the following:

The algorithm is not able to detect lanes on the very different lighting conditions
We split the image into two parts and detect the lanes on each part. This is not possible on the Harder Challenge Video because the curves are too sharp.
Therefore we would need to implement another way of grouping the lanes, e.g. by using the color information (--> Left Lane is always yellow ), or another more sophisticated way.

Some of our ideas did not find their way into the final algorithm:

Hough Transform: We tried to use the Hough Transform to detect the lanes. However this ended us up with only lines and not with a curve.
Also we reduced calculation cost by removing the Hough Transform.
We tried to not use Birds Eye View, which ended up working. But for the curve radius calculation we needed to use itanyway so we decided to use it for the whole algorithm.

Optimization

Our first optimization was to remove one of our key features: The Hough Transform. We used

cv2.HoughLinesP()

but ended up removing it not mainly because of performance concerns but because it did not give us the results we wanted. We ended up with lines instead of curves.
Nonetheless it was a good idea to remove it because it reduced the calculation cost.

Our second optimization was to use a very exact ROI and color everything outside black.

Afterwards we checked the call graph (check it here) to find the biggest slowdowns. We found out that the biggest slowdown was reading the image into memory. Therefore we parallelized the reading of the next image with the processing of the current image. This makes sense, in comparison to processing all images parallel, because also in the real world the camera is always recording and not finished with recording when the image is processed.

One alternative Optimization we did not try is to use Parallelization for our two lanes. Because we anyway split the image in half and detect one lane left and one right, we could use multiprocessing and do all the steps parallel. We did not try it because of time limitations.

Lessons Learned

Do not underestimate the time needed for the project. Especially for finetuning parameters.
This project is extremely time consuming, although it made fun. Sometimes it is exhausting when you spend hours on finetuning parameters and the result is not as expected.

Also we learned a lot about the OpenCV library and how to use it.

We also learned how to use the call graph to find the slowest functions and how to analyze the call graph. Check it here

What we would do differently next time and what we want to try in the future

Try the Sliding Window approach to detect the lanes. We did not use it because we thought it would be too boring. We got it recommended but thought it would be nice to try to use our knowledge from the course.
Try to search for the lane only close to the lane from the previous frame. This would be a good idea to speed up the algorithm and improve results.
Try to use the color information to group the lanes. This would be a good idea to improve the results on the Harder Challenge Video.
One big possible time improvement we did not try is to use Parallelization for our two lanes. Because we anyway split the image in half and detect one lane left and one right, we could use multiprocessing and do all the steps parallel.