I was planning to write this post yesterday, however, a Canadian wasp decided that it was better to leave me almost immobilized by attacking my foot and forcing me to plan my agenda differently.
Well, Marc Pollefeys (director of Science in the Hololens team) shared some information about the new version of Hololens. Until the code name is made public, I will refer to the new device as Hololens 2. What he tells us is a simple and powerful message:
The new HPU chip included in Hololens 2 will have Deep Neural Networks capabilities. (It means Artificial Intelligence!)
Let’s not forget that this is not new for Microsoft, but let’s also keep in mind that Microsoft is not dedicated to the design and creation of chips such as those we know from Intel or AMD. For years until now Microsoft is investing in R&D for a new generation of chips. Those chips are currently used mostly in the Azure Data Centers. In fact, it all started back in 2012, when Doug Burger presented a risky bet to Steve Ballmer: Project Catapult.
Doug commented to Steve that, in the near future, Internet would be controlled by a handful of group companies that would provide essential services for users. This “new Internet” would require a different architecture as a base platform. If Microsoft wanted to be part of this “new Internet”, they should not only build the OS and the Software, but they also had to take care of the hardware of the servers, manage the networks and more. It seems that at this moment Steve Ballmer change his face into a Gear of Wars Bad Boss, his eyes were all red and he responded with a “I thought this would be a research meeting, non a strategy one.”
Note: I have been fortunate to meet Steve Ballmer face to face, and the energy that he has is impressive. While I have seen him in happy and animated mode, I imagine that a 1:1 in discussion mode should require special skills to pull the conversation forward.
And the Qi Lu appeared (he was in charge of Bing), he was added into the discussion. It seems that Qi Lu also had a similar idea in his head: the need to build re programmable chips, allowing upgrades much faster than those that were running at that time.
And there was more, the Bing team had already started to work on this, from here we began to read the term FPGA much more often in some areas. (FPGA: Field programmable gate arrays). And that’s me on this part of the story. This is quite interesting and I recommend reading the Wired article (see references).
Let’s go back to 2017 with Hololens 2 and the new HPU 2 (HPU: Holographic Processing Unit). The task performed by the HPU in Hololens version 1 is to coordinate, analyze and present a coherent result of the information obtained by the device of all sensors. In other words:
The HPU merge information from different sources: motion sensor, camera, depth sensors and infrared camera sensors, with all this information the HPU is capable to determine our position in the space around us. With this information, holographic projectors can determine how and where to position the holograms that are projected in our field of vision.
To this day, this type of processing is something that is unique to the Hololens. And if it is combined with a GPU, a CPU and a battery, it allows Microsoft to have a 100% autonomous untethered device: Microsoft Hololens 😀
Update: Thanks @AlexDrenea for some typos help here!
Now, what would happen if this processor, also has some kind of DNN capacity. In some blogs they called it “AI co-processor” and we can think that it could help in tasks such as voice recognition, face detection, shape detection, image analysis and more. The first thing they have presented during CVPR17 is how these new capabilities can be used to improve Hololens’s hand tracking and hand gestures capabilities. This is the demo recorded by a conference assistant
Now comes the time to think that we can do with a device that “does not have to constantly send all this information to the cloud”, many of these tasks will be done in local. This will allow for more fluid applications, much more natural interactions and another couple of interesting surprises.
What is true is that 2018 will be a year where we will see what’s new Hololens 2 and surely we will have many interesting surprises along the way.
Greetings @ Burlington
PS: This is my foot 12 hours after the “Bruno vs The Wasp” moment