Passenger Safety: Augmented Reality for Blind Spots

In the process of looking to buy a car, I noticed that different cars have different blind spots. Some worse than others.
The two cars I tried, a Ford Focus and a Toyota Prius have a very different feel in terms of blind spots.

For example, the rear-view mirror on the Prius isn't all that useful. The back window is small, and more importantly narrow, so it is hard to get a sense of vehicles that are behind you to the left or right that might be attempting to pass you from either of those directions.

On the other hand, the back window on the Focus is large and wide, and the rear view mirror provides a much grander view of the status of the road behind you.

However, I noticed with the Focus that that wide view from the back provides a false sense of security, because vehicles may still be in your blind spots but you are led to believe that no one is trying to pass you, while on the Prius due to the narrow back window, you are constantly reminded to look behind your shoulder before and during a lane change.

That got me thinking... if my rearview mirror could "see through" the frame of the car, and show me what the road looks like from all directions, as a driver, my road sense would keep me alert to trafic that is approaching behind me on neighbouring lanes. Therefore, if I had a couple of cameras installed at either side of the roof of my car (on the outside), aimed at the "blindspot" area, and an ability to superimpose the image somehow on what is seen on the rearview mirror (with little distortion), then my rear-view mirror would essentially eliminate the blindspots.

While searching for more info on the topic, I bumped into a paper that covers current technology for eliminating blindspots as well as how to use Augmented Reality in a low cost and efficient way for that purpose.

Superman isn't the only one who can See Through Walls - Improving Vehicle Passenger Safety with Augmented Reality

For us ordinary humans who do not possess the power to see through walls, there's technology. For examplee MIT Lincoln Lab's project for seeing through walls using microwave radio frequencies.

Or a more passive "see through walls" radar system which takes advantage of wireless routers developed by University College London. When wireless radiation from WiFi routers passes through moving objects it gets slightly modified by the Doppler effect. A sensitive receiver can pick up on those frequency shifts and locate moving objects.

However, one might not need to see through walls to improve passenger safety with Augmented Reality. One might only need to see around them, or rather, around corners.

In many situations when line-of-sight is obstructed by buildings, approaching an intersection or turning into one may be dangerous. Crossing vehicles, pedestrians, or cyclists might surprise a driver who's braking distance might be too far into crossing traffic.


Some intersections are deployed with Convex Mirrors for this purpose.

However, with such mirrors the distance to an object is harder for a driver to estimate, as its speed. It also requires the driver to shift their focus to the mirror rather than the road, which can create other problems with hazards that are directly in the driver's line-of-sight.



A system developed by Carnegie Mellon University is using Augmented Reality techniques for overlaying live footage taken from cameras positioned on the walls of buildings in a problematic intersection onto live footage taking from the driver's point-of-view and combines them in real-time, such that they both appear to be taken from the driver's point-of-view.

With many cities installed nowadays with networks of CCTV cameras, it should be possible, according to the researchers, to integrate such sources of live footage into the system.

The idea has been explored before by researchers from Japan in a system that integrates footage from surveillance cameras allowing the user of the device to see obscure and hidden areas, essentially "through walls" and other obstacles.

The disadvantage of the approach is that it takes a lot of processing power to get right, and depending on where the CCTV cameras are positioned it may be hard to impossible to perfectly overlay the image without distortion. Therefore, some researchers (also from Japan) have explored the option of using the data from CCTV cameras but instead of overlaying the image as if the driver can "see through walls", they use abstract objects (such as a circle) to mark an object approaching from an obscure area. The approach may be more practical to implement, but its advantages over the "see-through" effect are yet to be fully studied. Although the processing power needed may be reduced, and the accuracy increased (due to the simplified problem), it remains to bee seen whether drivers can respond to the information in real-time with a similar effectiveness as the "see-through" approach.

I would imagine the best application for this being integrated onto a vehicle's heads-up-display, noting the driver of possible collision. For safety purposes this would have to be instantaneous (or "real-time"). Take a look at the linked paper on a practical application of this approach - the intent there is to make it simple enough for it to become real-time.

Current pre-collision detection systems  use varying methods for detecting a potential collision with objects that are in the driver's line-of-sight (e.g. radar or real-time video analysis).
As an extension to the model of pre-collision warning systems, it would be great if such information garnered from either cameras or sensors in problematic junctions would present itself in much the same way on the driver's windshield (perhaps noted by an audible signal as well) to warn the driver of a potential collision (or for the car to take action and brake in emergency situations).

Another interesting extension of this model is for fully-autonomeous cars, such as Google's Driverless Car. If you could detect approaching objects around corners it would provide the car with more time to react to hazardous situations.

Augmented Reality for Improving Passenger Safety, or Fully Autonomous Cars?

Are Fully Autonomous Cars the wave of the future? We've seen them in many Sci-Fi films and show, from KITT in the Knight Rider TV show

Total Recall (1990) with the robot Taxi driver

Minority Report

And let's not forget this Audi beauty from I, Robot

But before we give up the driver's seat, there are some interesting advances in Augmented Reality that are available today as built-in features or add-ons in some car designs, or as apps for your phone.

For example, Audi Pre Sense that identifies emergency braking situations, following a vehicle too closely, etc.

Or the Pre Collision System and Lane-Keep Assist features on the more affordable Toyota Prius

However, if you're not into buying a new car yet for any reason, you might want to check out iOnRoad. The App is free to download for iPhone and Android phones.

A quote from Wikipedia:
iOnRoad is a free augmented reality driving safety app.^[1] It received the International CES 2012 innovation award.^[2][3][4] The program uses the GPS feature, gyroscope and video camera stream of the native mobile device to monitor a vehicle’s position on the road, alerting drivers of lane departures and potential collisions with audio and visual cues.^[5][6] iOnRoad has been downloaded by over 500,000 people.

The application seems to use some nifty realtime image analysis on the video camera stream in combination with your GPS and accelerometers to determine the vehicle's speed.

The realtime stream processing is interesting and I'm quite impressed with its potential. I mean, cars on the road don't all look the same, and it takes quite a bit of an abstraction of the image in front of you to determine whether what you're looking at is a "car" versus other objects. And what about trucks of all sorts, or trailers that could be carried by cars or trucks, etc. Each of these may have very different "looks".

Some of the integrated safety systems (such as radar based ones) are also more impervious to varying weather conditions, and exposure sensitivity at night time.

However, as a start, it might be well worth it to at least try it out. If you buy the app with the windshield holder, the app will automatically detect that the phone is getting mounted on the windshield and would start monitoring. It can also work less intrusively in the background, so you only get to hear form it when you follow too closely. Watch though that you'd need to fiddle with it for a bit until you get your phone mounted at the right elevation and angle, because if the camera sees too much of the car hood, the app won't work.

Drive safe