Semiconductor IP News and Trends Blog

The IP of IP for AR

Intellectual property (IP) for image-processor (IP) cores and related technology is enjoying huge growth, thanks to the popular augmented-reality market.

During my research for a story on the design of augmented-reality (AR) systems, I came upon another term to add to my list of “IP” phrases – namely, image processors (IPs). And yes, there is intellectual property (IP) for image processors (IPs).

A quick search of the Chipestimate.com intellectual-property repository revealed a variety of image-processing-related IP, ranging from camera and CCD pattern de-mosaicing cores to image characterization, jpeg decoders, and the like.

This is good news for both IP communities, as augmented-reality technology is growing in popularity and applications. A recent Semico Research report estimated that the AR market will reach almost $620 billion by 2016.

Image-processing IP is needed for augmented- and virtual-reality systems to identify and recognize real-world images and objects to attach digital and virtual content to them in real time (see “Augmented Tools Design Reality“).

Designers of end-user applications need to consider both the software and hardware aspects of their augmented-reality implementations. Most vendors provide augmented-reality software application-development kits (ADKs) that work on the majority of iOS and Android platforms. “Beyond the basic needs of front-facing camera and reasonable performance, many of the newer platforms offer new compute resources, such as programmable image processors that promise improved computer-visioning capabilities,” says Trak Lord at Metaio. “The ongoing improvement in graphic-processing-unit (GPU) and general-purpose-GPU (GP-GPU) processing also provides more opportunity to improve augmented-reality user experiences.”

In addition to improved performance, hardware must provide more power-efficient depth-of-field imaging sensors and greater ease of programming for synchronized, multi-sensor data streams. High-performance, low-power GPUs and associated computing engines are a critical part of the design of AR systems. Companies like Metaio offer dedicated hardware image processors for accelerating augmented-reality experiences. Dubbed the “AREngine,” the acceleration chip works by taking on much of the processing required to run augmented-reality experiences from the general CPU. The company claims a drastic reduction in battery power consumption and an increase in initialization speeds.

Many designers use the compute power of the existing mobile-device GPU to enhance performance and minimize power for their AR applications. This requires careful integration of the GPU, video, and camera-vision processing to ensure optimal performance.

What is the difference between image and graphics processors? Image processing deals with the manipulation of images acquired through some device, like a camera. The emphasis is on analysis and enhancement of the image. Today’s popular computer-vision systems require the use of image analysis.

Conversely, graphic processing deals with synthesizing images based upon geometry, lighting, materials, and textures.

“Augmented-reality applications usually blend live video with computer images, where 3D graphics rendering is performed using OpenGL software APIs,” explains David Harold at Imagination Technologies. “Powerful cores can provide high-quality 3D graphics rendering, which can then be blended into the real-world camera capture. Also, by implementing features like camera-image texture streaming, GPUs are capable of processing camera images as textures to enable 3D and reality integration with minimal CPU loading.” Efficient integration of camera images into the 3D rendering flow is essential for good performance and efficiency in AR designs.

Harold shows a tablet connected to a screen to demonstrate the real-time computer power of GPUs.

Check out these cool apps based on augmented reality:

• Dassault Systemes provided the missing piece between comics and animation.
• This Metaio browser app uses ordinary objects as markers to get virtual information.
• Imec and UGent have unveiled a breakthrough augmented-reality contact lens.
• An IBM app marries augmented reality to comparison shopping.
• Users interact with augmented reality via hand gestures.
• Common Wi-Fi signals are used for augmented-reality gesturing.