result image(watch the full video below)
This is Vehicle Detection project of Udacity's Self-Driving Car Engineering Nanodegree. In this project, I approached with 2 methods for a vehicle detection. HOG + SVM approach and YOLO approach. In HOG + SVM approach, we classified vehicle using hog feature and color feature. And to search a position of vehicles, we implement a sliding window search. In YOLO approach, YOLO applies a single neural network to the full image.
Windows 10(x64), Python 3.5, OpenCV 3.1.0
CPU : i5-6500 3.20GHz, GPU : GTX 980Ti, Memory : 8GB
YOLO ver
main.py: main code
tiny.py: main code of tiny yolo
yolo_small.py: yolo model composition
yolo_tiny.py: tiny yolo model composition
calibration.py: get calibration matrix
threshold.py: sobel edge & hls color
finding_lines.py: find & draw lane lines with sliding widow search
HOG + SVM ver
main_hog.py: main code
svc_save.py: extract feature & save SVM classifier result
feature_extract.py: hog & color feature extraction
find_car.py: sliding window search method
svc_save.p: saved SVM result
- Perform a Histogram of Oriented Gradients (HOG) feature extraction on a labeled training set of images and train a classifier Linear SVM classifier
- Optionally, you can also apply a color transform and append binned color features, as well as histograms of color, to your HOG feature vector.
- Note: for those first two steps don't forget to normalize your features and randomize a selection for training and testing.
- Implement a sliding-window technique and use your trained classifier to search for vehicles in images.
- Run your pipeline on a video stream (start with the test_video.mp4 and later implement on full project_video.mp4) and create a heat map of recurring detections frame by frame to reject outliers and follow detected vehicles.
- Estimate a bounding box for vehicles detected.
In this project, I used the labeled data for vehicle and non-vehicle examples to train your classifier. These example images come from a combination of the GTI vehicle image database, the KITTI vision benchmark suite.
The code for this step is contained in the feature_extract.py and save_svc.py
The histogram of oriented gradients (HOG) is a feature descriptor used in computer vision and image processing for the purpose of object detection. The technique counts occurrences of gradient orientation in localized portions of an image.
This is my final choice of parameters of HOG
color_space = 'HLS'
spatial_size = (16, 16)
hist_bins = 32
orient = 9
pix_per_cell = 8
cell_per_block = 2
hog_channel = 'ALL'
spatial_feat = True
hist_feat = True
hog_feat = True
Extracted features using hog info & color info
car_features = extract_features(cars, color_space, spatial_size, hist_bins, orient, pix_per_cell,
cell_per_block, hog_channel, spatial_feat, hist_feat, hog_feat)
notcar_features = extract_features(notcars, color_space, spatial_size, hist_bins, orient, pix_per_cell,
cell_per_block, hog_channel, spatial_feat, hist_feat, hog_feat)To normalize a difference in magnitude between the color-based and gradient-based feature, I used StandardScaler() function of Python's sklearn package.
# Create an array stack of feature vectors
X = np.vstack((car_features, notcar_features)).astype(np.float64)
# Fit a per-column scaler
X_scaler = StandardScaler().fit(X)
# Apply the scaler to X
scaled_X = X_scaler.transform(X)Trained using LinearSVM classifier
# Split up data into randomized training and test sets
rand_state = np.random.randint(0, 100)
X_train, X_test, y_train, y_test = train_test_split(
scaled_X, y, test_size=0.2, random_state=rand_state)
svc = LinearSVC()
svc.fit(X_train, y_train)And according to svc.score(X_test, y_test), my Testest Accuracy was 0.9896
The code for this step is contained in the find_car.py
I used 3 square window sizes of 128x128, 96x96, 64x64 pixels size. And each adjacent sliding windows have an ovelap of 75%. Also upper half of image and lower bonnet's position are ignored to search vehicles. Each windows check the Hog features and compared with SVM result.
During sliding window search, there are many false positives and Multiple Detections. To resolve this problem I used heatmap. Simply added "heat" (+=1) for all pixels within windows where a positive detection is reported by your classifier.
This is my result of test images.
When I use a SVM + Hog method, the elapsed time for a vehicle detection was about 4.28 second per 1 frame. detection result was not bad but we never detect vehicles in real-time with 4.28s/frame. That's why I searched a YOLO.
YOLO(You Only Look it Once) is famous real-time object detection technique. Originally YOLO is codded with C language but I found a tensorflow version of YOLO and I revised it.
My code does not support training. Used darknet's weight file. On average, I could get 0.035 second per 1 frame with YOLO_small(GPU : GTX 980Ti) and about 0.025 second per frame in tiny YOLO model.
Yolo's approach is really impressive. Prior detection systems(e.g. faster-R-CNN) apply the model to an image at multiple locations and scales. High scoring regions of the image are considered detections. but YOLO applies a single neural network to the full image. This network divides the image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities.
To understand how yolo works, please watch this youtube video. Language is Russian but you can understand easily.
By using YOLO, i could real-time detect vehicles in video. To check a real-time lane-finding & vehicle detection, I merged this project with advanced lane finding project. This is the result.
Final real-time Result (click for full video)
I could learn and implement SVM classifier and yolo. Before starting this project, I had decided to use a YOLO for vehicle detection because I was really interested in YOLO. But I did't have experience about SVM-based detection and wondered the result. That's why I used both methods in this project. SVM classifier could detect vehicle well but the elapsed time was terrible. To search a position of object, I needed to search almost every frame using sliding window search. But I could resolve it by using YOLO.
In this project, I didn't retrain a model and used tiny YOLO. So, according to my result, it couldn't detect small vehicles and showed some of false positives. For a best result, I'll fine-tune yolo or ssd method to mine. That's my future work.