Artificial Intelligence and XR | NINJAR Blog

Artificial Intelligence and XR

A series of essays from exploring how living on the Reality-Virtuality Continuum will change how we live, work, love and play.

Read Part 3 in this series for how Culture is changing with constant access to computers

Cameras are the eyes of AI, and they are everywhere.

The first steps into the brave new world of Mixed Reality will happen through the existing hardware of mobile phones. The gaze of algorithms will see everything.

This will allow everyday users to leverage the advantages of spatial computing and allow current developers to prepare today’s businesses for the upcoming paradigm shift into spatial computing and away from screen.

Along with automation of the skilled workplace with AI, it will fundamentally change how we work and play with computers.

This first stage - the conversion of a dumb selfie camera into the Eyes of AI that will understand the world and augment has has a massive surface area. Around 2 billion camera phones are already in the world, many of them equipped with the technology needed to open up Mixed Reality on a gigantic scale. And outside mobile phones we see cameras embedded in Cars, Laptops, TV’s and fridges.

The world will become seen, understood and augmented by Mixed Reality.

At we are interested in exploring these ideas and are working with interested parties to help them succeed through this forthcoming change. Initially we see the mobile phone being used to augment the world with extra data. Initially it will be frivelant and temporary and then it will become a backbone of daily life.

We are laying the foundations for building Mixed Reality. Imagine an app running on your mobile phone which uses computer vision to recognise every object and offer relevant information and offers to complete cognitive task for the User. This is a new kind of computer interaction subsuming the layers of Browser and Android and IOS apps- just as as they subsumed the linux, assembly and machine code beneath them.

We saw glimpses into the functionality of this world when we helped Ferrari and Byton build Augmented Reality Car configurators:

Examples of Augmented Reality

The first wave of the Mixed Reality functions, beyond naively Augmenting Reality with 3D objects will likely by nothing more than contextual icons floating over real objects. These objects being recognised by computer vision and helped, but not locked to, GPS positions. At first this will allow a new kind of Wikipedia - a kind of auto-foursquare - where a user can point a camera anywhere and instantly find out the relevant details about anything, with the conventional widgets for businesses like customer ratings, opening hours etc etc. The same system can be applied to products - such as informing the user about the amount of sugar in a can of coke of coke or or the going market rate for a second hand car. Whilst this is interesting, and certainly a scope to allow products in competitive markets to differentiate themselves, it is only the first stage to what will be done with this transformative technology.

The technological progress for an Operating System for Reality has already produced value, with a normal mobile phone equipped with AI showing its able to do the labelling of items in realtime:

The objects are identified, but not yet understood in the context of inter-object relationships, be them hierarchical or flat, or how one object can cause an event which changes the state of another. And then predict the effects of the changes of the system… This next stage of assigning behaviours and events to real objects and virtual will link a simple form of cognition and simple utility with the labelling of these real objects. An example would be to use your camera to keep a simple track of your automobiles maintenance, without having to do much or think much. I.E. Just point a phone at your car and the system will show all relevant details about tire wear, fuel amount, next maintenance date, current finance options etc etc. Your camera won’t need to interface with your car’s complex onboard computer to do this, and you won’t need a new app for each car you want to look at - the AI looking through the camera will recognise the car model, confirm via the number plate, then compare it online to crowd sourced details regarding similar cars. It will know realtime details such as the last time the car gas was filled up because when you went to the gas station you just pointed your camera and the car and pump. Of course, of you are wearing a clip-on camera, then all of this is done for you. The magic of Mixed Reality, and the reason it will push other forms of computing such as mobile phones to the side is because it is easier. The Mixed Reality Interface uses Natural Mapping of data onto the objects. This creates delightfully frictionless experiences for the user, with no abstraction between object, concept and user interface. Examples can be seen with apps like Golfshot, which allows the user to point their phone at a golf course and get detailed information about shot length and augmentations to the course:

Golfshot - Frictionless Augmentation

The natural mapping of information into the world has the educational advantage that it also allows specialist concepts such as car maintenance to be easily understood.
For example, modern cars have an after-service need for replacing their Air Filter after a set time. Most people do not know what an air filter is, where it is, or how much is costs, or even how to search on the internet to find details. But all these facts, and more, can easily be communicated using Mixed Reality. The owner can simply point their camera at their car and the MR system will inform them what an Air Filter does, where it is located, why you need to change it every few years and why yours needs to be changed soon. Wikipedia will become WikiMR.

The tools are here today to scan 3D objects with a mobile phone and the AI software is available to recognise real objects and connect the two. The next stage for the user will be to assign A.I. behaviour to these identified and virtual objects. The AI behaviours - such as “Give me the market price of a car when you see it” will probably come from crowdsourced platforms like Wikipedia and specialist services like Ninjar. The use of A.I. to identify and augment real world objects will provide a frictionless interface to the power of computing and it will be the killer feature of Mixed Reality.

This frictionless, active interface will be great way for companies to provide a more complete services - from product design, market testing, customisation, customer feedback and after sales service. For every product - the user would just point their camera at the subject they need augmenting and the platform would do the rest.

Once the world can be labelled by a computer it can be programmed. An advantage of Mixed Reality is that the world is the interface and linking behaviours to real world objects can happen naturally.

For instance if I have a plant in my office I could point my camera and the plant, the Mixed Reality Camera would recognise the plant and then offer suitable ‘augmentations’ for the species. Visually, this might look like a group of 3D objects floating around the plant.

These in world icons could be as simple as a pack of seeds (to buy online), an animating plant showing the growth and death cycle (a growers guide to the species) or a rain cloud with a clock on it. As a user I might pick the raincloud and see all the other items disappear and the raincloud lock itself in position above the plant. The Mixed Reality system has now locked the concept of reminding me to water the plant with a persistent MR icon. The cloud will be fully transparent when the plant is watered and only come into view when the plant needs watering. The AI system has recognised the plant species and has a good idea how often it needs water. And any other cameras in the area will also be checking the plant regularly because they were notified that it has been augmented. All computer vision systems will then reset the timer if anyone waters the plant in between scheduling times.

The programming of the real world via augmented behaviours will have an impact everywhere you can put a camera.

Already A.I cameras are appearing on the market for installation in homes and public areas like coffee shops which provide a constant set of eyes for AR Avatars to see through and understand the world. Amazon already sells one:

Amazon Deeplens

And Google can easily catch up with their portable AI camera:

Google Clips

For instance if I am in a city, trying to get a bus I won’t have to decode the bus timetable or search through google results to find the correct bus company app to download. Instead I will point a camera at the bus stop with GPS co-ords to the Mixed Reality platform which will augment the area with relevant content (crowdsourced or created by the bus company through a platform like Optionally Mixed Reality Avatars would appear and tell me which bus to take - customer service representatives that add a human touch.

Everywhere there is a camera there will be a Mixed Reality opportunity. And there will be cameras everywhere:

360 Camera’s for All Seeing AI

Even cameras flying in the sky are helping send augmented reality data down to firefighters to help combat Forest Fires:

But what if there is no subject for the camera? Nothing for the A.I. to compute? In this case the system can be leveraged for entertainment. Just as computers play games when they are not being used for work - the spaces of Mixed Reality will become playgrounds for digital avatars…

Update: Read Part 5 for how A.I. will merge with reality to create a world full of avatars

For companies looking to get into VR/AR/MR our Virtual Reality services offer guidance on how these technologies can enhance and support your brand strategy.

Connect with us on LinkedIn and follow @ninjar on Twitter or send us an email to

Recommended Articles

Ninjar Creators Marketplace

21 Mar by Chi

"The next step for us is to establish a Creators Marketplace where brands and businesses looking for help with their XR products can find the exact person or studio they need in a quick and easy way..."

XR Outsourcing Trends 2019

4 Jan by Marco

"While software is said to be eating the world, immersive software developers and other technical talent remains in short supply. This shortage has led to intense competition among major tech companies to hire the best engineers.."

When we look back at AR ten years from now we’ll laugh

21 Dec by Marco

"‘beep-beep-beep…. eeee …. oooo’ - that’s the sound that we used to hear every time we wanted to connect to the internet during the dial-up phase..."

Connect with Your Users or Perish

16 Nov by Marco

"Looking at a study done by Harvard Business Review and Microsoft showing the growing trend of MR that is too strong to deny.."

Immersive Communication

12 Nov by Matt

"Visualising imaginary places and objects is fine - But what about people? How can immersive technology help bring people together, and improve our social lives?.."

How 3D & VR makes automotive design 100x better

7 Nov by Sabina

"Years ago if you wanted to buy a car, you would walk onto the lot and be shown models by some salesman, who would disclose the information he wanted to, often times at the expense of the buyer.."

VR Sickness — what it is & what we can do about it

15 Oct by Sabina

"What is it that makes putting on a headset and stepping into a virtual world feel like you are on a rocking sailboat about to be hurled into the ocean?"

The Future of XR is Serious Business

12 Oct by Marco

"Instagram already has virtual humans which attract over a million followers. The use of X.R. together with A.I. mixes together to make a brilliant new show, broadcast 24/7 with you as the star.."

Virtual Avatars

23 Sep by Matt

"Instagram already has virtual humans which attract over a million followers. The use of X.R. together with A.I. mixes together to make a brilliant new show, broadcast 24/7 with you as the star.."

Artificial Intelligence and XR

21 Sep by Matt

"Cameras are the eyes of AI, and they are everywhere.The first steps into the brave new world of Extended Reality will happen through the existing hardware of mobile phones.."

Digital Reality and Culture

18 Sep by Matt

"There is an interesting bias in modern culture – people are mainly hired for scientific and technical knowhow — not for humanity, morality, religion or creative talent.."

Technology Extends Reality

15 Sep by Matt

"We are creating the hive mind. Wires linking everything. We are building another world inside our heads, and we are teaching A.I how to thrive there. But is it what we ultimately want?.."

How Can Reality Be Extended?

12 Sep by Matt

"When we built the computers to help us - we created an exo-cortex. A part of our mind outside our bodies. It extends our thoughts, our consciousness, via digital signals.."

VR | AR | MR — What do these buzz words really mean

12 Sep by Sabina

"In 2018 we are now able to mix the real world with virtual worlds in very realistic ways — and this technology is just getting started!"

Why ninjAR is a Platform and Why That Matters

11 Sep by Matt

"Everywhere we look we see the world becoming virtual. Everything we know, such as media, currencies, businesses, products, services and even people are being converted to digital assets..."

VR Creation Made Simple

10 Sep by Marco

"Our team spent seven years creating innovative digital products. During this time, we were lucky to establish and collaborate with a network of world-class developers, designers, and product managers to create beautiful, functional software..."

Back to Blog