David Amerland

Semantic Paint Makes Real-World 3D Labelling Child’s Play

Semantic Paint models 3D real-world on the go

When I was researching and writing Google Semantic Search I was impressed by the amount of original, ground-breaking research that had first come from Microsoft Research, on the subject. The Redmond corporation underwent its own internal, identity issues and allowed Google to steal a march on it, in search but it is now back with a vengeance.  

The new battleground is comprised of the twin fields of virtual reality (VR) and augmented reality (AR) both of which are united by the need for accurate real-world object recognition. In the age of interactive computing and smart environments, processing speed and computing efficiency are everything which is why semantic technologies are so critical. 

By learning associative values on the fly, semantic technologies enable interactive mapping systems to learn from every instance. So, having taught whatever system you’re working with that a chair is something that looks like a chair with four legs, a seat, a back and arms you do not then have to repeat the process for every single chair you’re going to come across ever again (or, at least, hardly ever). 

Microsoft’s Semantic Paint uses the simple notion of color recognition to create object segemtation in the real world in real-time. The achievement cannot be stressed enough. It is elegant, fast and relatively accurate with an easy way to correct errors as the video below shows: 

Microsoft’s Semantic Paint 


Why Is Semantic Paint Important?

Microsoft Research has done the clever thing here harnessing pre-existing technology from Kinect to scan the real world and run relational connection values. It’s important because it links it all to a wearable headset that allows the user to map the real world into a digital model. If that sounds a little like Google’s  Project Tango you’re not far from it. 

The moment the real world is mapped there are two concrete gains made instantly: First, the digital world has acquired practical, high-fidelity knowledge of the real world, gained in the mapping process. Second, the digital world can now allow better manipulation of the real world through either remodeling trials or the application of enhanced layers (which is where augmented reality comes in). 

Microsoft has successfully used similar technology to create a passable version of the Holodeck and a stylus that comes with a mind of its own, capable of reading ours.

The World is Changing – Marketing needs to, also

Companies that used to need to know dumb things like who we are and where we are (you know, all the data stuff that last century marketing is associated with) are now busy building libraries of data captures that are semantically associated which gives them additional layers of meaning. So, Microsoft and Google, Apple and even Amazon are busy understanding gestures, speech patterns, voice inflections and even body language. 

It is all driven by machine learning algorithms that allow the continued refinement of what is being learnt so that it becomes better and better as we progress. When devices will be able to tell what we think, or search engines tell how we think, presenting an array of options of a product or service in the hope that we will be enticed into making a purchase if the packaging and presentation are clever enough, sounds crazy. 

The disconnect is too much of a gap to successfully bridge, no matter how hard marketing will try. 

This means that the marketing of the future will not even feel like marketing, in the traditional sense of the word. Marketing will be something that happens in the space of capturing someone’s attention, helping them solve something or do something they need and, in the process, establishing a connection.  

Really clever marketing uses the device interface of what’s in your pocket, your wrist or your desktop to occupy the only part that really matters: mindspace. 

This means that marketing is changing everywhere. If you are selling a product, or a service the real battle lies in understanding and then delivering on customer intent through relevance and context. 

That requires real thinking about each customer like they are unique. That level of thinking is a relationship and whether it is then delivered through automation (like Google and Amazon) or design characteristics (like Apple) or ubiquitous familiarity (like Microsoft) the result is always the same: marketshare. 


The Future of Digital Marketing
Semantic Paint: Interactive 3D Labeling and Learning at your Fingertips (Microsoft Research pdf)


© 2019 David Amerland. All rights reserved