Augmented Reality scopes - missing fewer shots

Target Acquisition Systems for fighter jest and helicopters have been around for years. In recent years, some developments in integrating Augmented Reality into infantry-scale arms have been added to the market. One example is the EOTech  Holographic Weapons System

The sight uses a combination of a laser and a holographic reflective film to form the visible dot or circle in infinite space over the target. In practice it has been said to provide faster target acquisition.

Sophisticated targeting systems have been around for a while. Systems such as VTAS, which allow aiming via a head mounted display, but even older systems that used head up displays.

With all that technology available for fighter jets, someone just had to come up with a way to squeeze that into a rifle size scope...

Meet DARPA's "One Shot" system. The system uses a laser to estimate distance and crosswind profile and shift the target cross-hairs to where it expects the bullet to actually hit.

And there's Tracking Point's cool technology for a 3-stage targeting system. First, tag a target in your scope, then the scope tracks the target. If you're unhappy with the tagged target, you clear the tag and tag the target again. When ready, just press the trigger and aim, and the targeting system takes care of releasing a shot when the rifle is aimed correctly.

Tracking Point's solution can deal with crosswinds via user input, compared to DARPA's OneShot solution that uses a laser to estimate crosswind effects.

Augmented Reality for Improving Passenger Safety, or Fully Autonomous Cars?

Are Fully Autonomous Cars the wave of the future? We've seen them in many Sci-Fi films and show, from KITT in the Knight Rider TV show

Total Recall (1990) with the robot Taxi driver

Minority Report

And let's not forget this Audi beauty from I, Robot

But before we give up the driver's seat, there are some interesting advances in Augmented Reality that are available today as built-in features or add-ons in some car designs, or as apps for your phone.

For example, Audi Pre Sense that identifies emergency braking situations, following a vehicle too closely, etc.

Or the Pre Collision System and Lane-Keep Assist features on the more affordable Toyota Prius

However, if you're not into buying a new car yet for any reason, you might want to check out iOnRoad. The App is free to download for iPhone and Android phones.

A quote from Wikipedia:
iOnRoad is a free augmented reality driving safety app.^[1] It received the International CES 2012 innovation award.^[2][3][4] The program uses the GPS feature, gyroscope and video camera stream of the native mobile device to monitor a vehicle’s position on the road, alerting drivers of lane departures and potential collisions with audio and visual cues.^[5][6] iOnRoad has been downloaded by over 500,000 people.

The application seems to use some nifty realtime image analysis on the video camera stream in combination with your GPS and accelerometers to determine the vehicle's speed.

The realtime stream processing is interesting and I'm quite impressed with its potential. I mean, cars on the road don't all look the same, and it takes quite a bit of an abstraction of the image in front of you to determine whether what you're looking at is a "car" versus other objects. And what about trucks of all sorts, or trailers that could be carried by cars or trucks, etc. Each of these may have very different "looks".

Some of the integrated safety systems (such as radar based ones) are also more impervious to varying weather conditions, and exposure sensitivity at night time.

However, as a start, it might be well worth it to at least try it out. If you buy the app with the windshield holder, the app will automatically detect that the phone is getting mounted on the windshield and would start monitoring. It can also work less intrusively in the background, so you only get to hear form it when you follow too closely. Watch though that you'd need to fiddle with it for a bit until you get your phone mounted at the right elevation and angle, because if the camera sees too much of the car hood, the app won't work.

Drive safe

3D Holographic Maps anyone? "Return of the Jedi"-style

In case you were wondering about 3D holographic maps...

Some of the technology is possible with today's technology and relies on persistance of vision (it was demonstrated successfully in 2006).

Since it provides only surface volumetric capabilities, it won't be able to show the Death Star under construction being shielded by a station on the surface of the planet as in the film.

A year later, there was an attempt to create a fully volumetric display using two rotation axises, called "ultra ORB", and an early demo presented, but I haven't found any indication that this project succeeded beyond that stage.

3D Medicine (almost) no longer the realm of Sci-Fi

Not strictly Augmented Reality specific but close enough...

Anyone remember the DNA regeneration scene from the film "The Fifth Element" ?

In the film, some remains of a sole "survivor" are found (a heavily gloved hand)ת from which an apparently very complex DNA is extracted and used to regenerate the entire body.

At first the skeleton is printed (in a form similar to 3D printing), then the rest of the organic tissue is built on top of it, etc.

Well we might not be able to regenerate every organ yet, but the technology is already here in the form of biological 3D printing, to "print" specific organs.

Check out this talk by surgeon Antony Atala on "Printing a Human Kidney" on TED2011.


While we're on the topic of Sci-Fi reaching medicine... anyone who watched the Star Trek Next Generation series, probably saw one or more instances of this scene

Check this article on The Air Force Treating Wounds With Lasers and Nanotech [wired.com].


How about a virtual dissection tables as a teaching tool for future surgeons?
Check out this talk by Jack Choi on TED2012.

HCI from Minority Report - not Sci-Fi

I mentioned in my presentation that the interaction technology used in Minority Report was not Sci-Fi.

John Underkoffler from Oblong Industries presented the interface at  TED2010

Holo-tables ("Avatar" style) are cool and seem very useful, but is the technology there to make them real today?

Like many others, I loved the Holo table concept presented in the film Avatar (2009).

I think the applications for showing terrain in 3D could be very useful for civilian and military uses.
For example, a quick look at a 3D enabled map such as that could help determine whether line-of-sight exists between two points, or whether some part of the terrain is going to be harder to traverse, a potential ambush spot, etc. I believe there are possibilities for strategic planning, beyond what 2D terrain maps enable.

One of the key aspects of such a "holo table" is the ability of having multiple viewers collaborate, while each viewer may be watching the map from a different angle.

A 3D display may be useful, although it would require lots of calculations to produce a large number of potential view points. However, most 3D displays today use vertically aligned lenticular lenses, which would enable multiple views from different horizontal points.

Using concentric elipses, one would be able to get the Parallax effect on the up-down motion, but not while walking around the table.

Therefor, one would need to organize the lenticular lenses along the perpendicular lines to these concentric ellipses. That would enable Parallax on the horizontal movement, which might be sufficient.
However, there is still a problem with the density of the lenticular lenses close to the centre of the display. Obviously, such lenses can not cross each other's paths.
The solution, both hardware, software and compute power, seems pretty expensive. An elliptic 3D display used horizontally as a "holo table", but price aside, it seems doable with today's technology unless I'm missing something.


Another interesting implementation idea, which shows more promise, although I believe it is at least a few years away from commercial use, is by using a Laser Plasma Volumetric display.

By setting up an overhead "projector" (infrared laser plasma emitter) that is capable of heating up the plasma in the air above the table, it should be able to display a 3D image over the table that resembles the picture above from Avatar. Currently the technology is limited to a monochromatic blueish like colour, but as it gets developed further we might see such displays in the future.


As a third, and perhaps simpler, alternative for implementing something similar to the holo table above, we might want to take a look at an old and familiar toy.

Using the same idea horizontally, and if we can effectively and electronically control the height of the pins, using a combination of elastic and electro magnetic forces as in this Pin-array tactile module, or even by using mechanical alternatives, one could overlay a reflective material over the pins (or just paint their heads white), use an overhead projector to project the surface map, and what we should get is a fairly realistic 3D representation of the terrain. Of course, the density of the pins would determine the resolution, and using the elastic/electro-magnetic combo solution would probably also allow some degree of animation (zoom, rotate, slide, etc.).
I can't help imagining how noisy this will be ;-) But at least it is a start in the right direction, until laser plasma volumetric displays are ready for prime time.

Update:

Zebra Imaging, a long-time producer of 3D holographic prints has been awarded a contract by DARPA back in 2005 to develop a real-time interactive holographic display map. The Urban Photoinc Sandtable Display (UPSD) is the result of that. It supports up to 20 participants, 360 degree view points, 12 inch depth and displays that scale up to 6 feet in length, enabling full Parallax without requiring special glasses or goggles.

Personally, I haven't heard of Zebra Imaging prior to searching for holo table implementations. Even with all my research into augmented reality and 3D display technology. However, I can only assume that the limit imposed on the number of participants suggests that the display does some face and eye tracking, in order to limit the amount of computation needed to create the holograms.

360 degree holograms take up a lot of compute power. Basically, as I understand it, one would need to calculate the interference patterns for multiple view points from all angles to create the full Parallax effect (otherwise the hologram may be invisible or look incoherent from some angles).
By being able to track the viewer's eyes, the display is able to calculate the relevant part of the holographic model only for that viewer. If you add more viewers, the patterns become more complex (and I assume super-linear complexity, although I'm not familiar with algorithms for creating holograms), thus if the system at its current compute power needs to deal with participant 21, the real-time effect would be lost, since refreshes would reduce the fram rate significantly.

I can only assume that with improved compute power, UPSC should be able to deal with a larger number of participants.

Conclusion:
I believe it is safe to tag "Holographic map tables" as Science-Fact.

Marker-less (General Image Recognition) based tracking

There's a nice presentation by Matt Mills on TED, where he demos a technology that uses image recognition as a reference point for Augmented Reality videos and models.

The demo looks nice, but can't help wondering what would happen when the number of image reference points on the server reaches critical mass...

The idea is that when your set of pictograms is small, or by some way easily decodes into a text string (or some other key), then it should be relatively easy to locate the appropriate data or model to show to the user.

However, when we allow multiple images, then some form of fast image search needs to happen. Since the reference point could potentially be any part of the scene (although higher priority would probably be given to the focus point of the scene), it requires some pretty powerful encoding or abstraction to be able to find those.

I guess that there should be optimizations for detecting "frames" (i.e. rectangles), and then cut that down by the length/width ratio to narrow down the number of possible images. However, if the image is a 4x6 (or any 2:3 ratio image), and since the phone has no data regarding the distance to the image due to lack of stereoscopy, I would assume that for such images the number of potential hits in the database would be quite large.

Other tricks, such as relying on geolocation, or abstracting parts of the image further (i.e. by recognizing elements in the encoded image such as rectangles, faces, other body parts, etc.) and doing the same on the input image, would probably allow faster lookup. But then again, if the database is full of portraits that would still be a hard problem (even with parallelisztion). Other elements like colour coding, de-rezing the image (into a much lower resolution), and some statistics on the image could probably be used in combination with Fuzzy Logic to allow yet faster image lookup.

If anyone has more details on how the performance is achieved in services such as Layar (as well as many others) that support general image recognition for Augmented Reality, I would love to hear about it.

On a side note:  +Augmented Reality has some good intro material and links for those new to AR, as well as links to lots of good demo videos.