result image(watch the full video below)
This is Advanced lane finding project of Udacity's Self-Driving Car Engineering Nanodegree. We already completed lane finding project in the first project. In that project, we could find lane lines and made robust algorighm for shadow and some of occlusion. It might be enough in the straight highway. But there are many curve lines in the road and that's why we need to detect curve lanes. In this project we'll find lane lines more specifically with computer vision.
Windows 10(x64), Python 3.5, OpenCV 3.1.0
main.py : main code
calibration.py : get calibration matrix
threshold.py : sobel edge & hls color
finding_lines.py : find & draw lane lines with sliding widow search
finding_lines_w.py : find & draw lane lines with sliding window search using weighted average method (for the challenge_video)
- Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
- Apply a distortion correction to raw images.
- Use color transforms, gradients, etc., to create a thresholded binary image.
- Apply a perspective transform to rectify binary image ("birds-eye view").
- Detect lane pixels and fit to find the lane boundary.
- Determine the curvature of the lane and vehicle position with respect to center.
- Warp the detected lane boundaries back onto the original image.
- Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.
When a camera looks at 3D objects in the real world and transforms them into a 2D image, it's not perfect because of a distortion. And the distortion brings an erroneous information.(e.g. changed object shape, bent lane lines) So, we have to undo the distortion for getting useful data.
The code for camera calibration step is contained in the calibration.py.
I compute the camera matrix(intrinsic parameters) and distortion coefficients using the cv2.calibrateCamera() function with 20 9*6 sized chessboard images. And applied this distortion correction to the test image using the cv2.undistort() function.
General Process
If an image loaded, we immediately undo distortion of the image using calculated calibration information.
In image, a bonnet and background are not necessary to find lane lines. Therefore, I cropped the inconsequential parts.
I used two approaches to find lane lines.
a Gradient approach and a Color approach.
The code for lane finding step is contained in the threshold.py.
In gradient approach, I applied Sobel operator in the x, y directions. And calculated magnitude of the gradient in both the x and y directions and direction of the gradient. I used red channel of RGB instead of grayscaled image.
And I combined them based on this code :
gradient_comb[((sobelx>1) & (mag_img>1) & (dir_img>1)) | ((sobelx>1) & (sobely>1))] = 255
In Color approach, I used red channel of RGB Color space and H,L,S channel of HSV Color space. Red color(255,0,0) is included in white(255,255,255) and yellow(255,255,0) color. That's way I used it. Also I used HLS Color space because we could be robust in brightness.
I combined them based on this code :
hls_comb[((s_img>1) & (l_img == 0)) | ((s_img==0) & (h_img>1) & (l_img>1)) | (R>1)] = 255
With this method, I could eliminate unnecessary shadow information.
This is combination of color and gradient thresholds.
We can assume the road is a flat plane. Pick 4 points of straight lane lines and apply perspective transform to the lines look straight. It is also called Bird's eye view.
The code for Sliding window search is contained in the finding_lines.py or finding_lines_w.py.
In the video, we could predict the position of lane lines by checking previous frame's information. But we need an other method for a first frame.
In my code, if the frame is first frame or lost lane position, found first window position using histogram. Just accumulated non-zero pixels along the columns in the lower 2/3 of the image.
In the course, we estimated curve line by using all non-zero pixels of windows. Non-zero piexels include color information and gradient information in bird's eyes view binary image. It works well in project_video.
But it has a problem.
This is one frame of challenge_video.
In this image, there are some cracks and dark trails near the lane lines. Let's check the result.
If we fit curve lines with non-zero pixels, the result is here.
As you can see, we couldn't detect exact lane positions. Because our gradient information have cracks information and it occurs error of position.
So, I used weighted average method.
I put 0.8 weight value to color information and 0.2 to gradient information. And calculated x-average by using weighted average in the window.
This is the result.
In my output video, I included some road informations.
- estimate lane status that is a straight line, or left/right curve. To decide this, I considered a radius of curvature and a curve direction.
- for calculating a radius of curvature in real world, I used U.S. regulations that require a minimum lane width of 3.7 meters. And assumed the lane's length is about 30m.
- Estimated current vehicle position by comparing image center with center of lane line.
- The small mini map visualizes above information.
Project Video (Click for full HD video)
Challenge Video (Click for full HD video)
I gave my best effort to succeed in challenge video. It wasn't easy. I have to change most of the parameters of project video. It means that the parameters strongly influenced by road status(bright or dark) or weather.
To keep the deadline, I didn't try harder challenge video yet. It looks really hard but It could be a great challenge to me.