Self-Driving Cars: Implementing Real-Time Traffic Light Detection and Classification in 2017

Quick, where are the traffic light(s)?

What the AI thinks:

The top image was one of the results from the bosch test data set. This image was not available to the network at training time. Running time was 77ms. It also shows that while the system has has incredible performance, much much more efforts would be needed for a production level car. For example at this distance it did not detect the traffic light on the left at a confidence level greater than 50%. Code available here.

Google’s approach, circa 2011

Image credit Fairfield et. al.

Can it be done in one network, exclusively with an image and no prior information?

Left: Internal network architecture, credit tensorflow team, screenshot for a Faster R-CNN sub component. Right: Credit Huang arrows added.
A far away light, only a few pixels wide being detected.

Adapting Bosch data for the Udacity self driving car

Image credit Udacity
Results from Udacity self driving car test site. The model was trained on 14 classes. So it has additional classes like Off, Red Left, etc.

Why a deep learning based approach?

Credit wikipedia. The left most image is a traffic light that uses different shapes to aid colour blind recognition. The center example is from new york where vertical mounting is common and the right most image from Canada where it’s common to see horizontally mounted traffic.

Motivation for high accuracy localization:

For example, in this image, the nearer traffic lights are clearly detected with a larger bounding box than the far intersection. A less accurate localization would be more prone to errors here. In a more sophisticated system this could be used for all manner of things such as confirming the light is indeed for the lane the car is in, that the light matches expected locations based on prior mapping knowledge, etc.

Real time performance (10+ Hz)

Spectacular failure cases.

One last thing



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store