#Python – Detecting #Hololens in realtime in webcam feed using #ImageAI and #OpenCV with performance improvements

Hi!

In my previous post I created a sample on how to use ImageAI and OpenCV to detect Hololens from a webcam frame (see references). I added some code to the last sample, and I found that the performance was not very good.

python using imageai to detect hololens less than 1 fps

With the previous sample code, I couldn’t process more than 1 frame per second. So, I started to make some improvements and I got this result

python using imageai to detect hololens little more than 1 fps

Not an amazing one, but still is nice to have more than 1 frame per second analyzed.

I even remove all the camera preview and still works in less than 1FPS.

python using imageai to detect hololens no opencv camera preview

So, now it’s time to read and learn of the deep code on ImageAI. Fun times!

Happy coding!

Greetings @ Burlington

El Bruno

References

#Python –Detecting #Hololens in realtime in webcam feed using #ImageAI and #OpenCV (thanks to @OlafenwaMoses)

Hi!

Let’s start with a very quick intro:

During the past months, I’ve been playing around with several Image Analysis tools. And ImageAI (see references) is one that deserves a full series of posts. Please take a look at the product and the source code in GitHub, and also please thank the one behind this: Moses Olafenwa (@OlafenwaMoses).

And now, my 2 cents. I’ve started to test ImageAI to create my own image detection models. Most of the times, this is a hard path to do, however ImageAI show me an interesting option.

… with the latest release of ImageAI v2.1.0, support for training your custom YOLOv3 models to detect literally any kind and number of objects is now fully supported, …

Wow! That’s mean that I can pick up my own set of images dataset and train on top of a YOLOv3 and use it as a trained model. Again, this is amazing.

So, I started to read the article [Train Object Detection AI with 6 lines of code, see references] where Olafenwa explains how to do this using a data set with almost 500 rows with images for Hololens and Oculus Rift.

The code is very simple and easy to read. There are also examples on how to analyze a single file, or a video, or even a camera feed. The output for the analysis can be also in a new file, in a processed video or even a full log file with the detected information.

I started to read the code samples and I realized that I’m missing a scenario:

Display the realtime feed from a webcam, analyze each webcam frame and if a device is found, add a frame to the realtime feed to display this.

I use OpenCV to access to my camera, and it took me some time to figure out how to convert my OpenCV2 camera frame to the format needed by ImageAI. At the end, thanks to the GitHub code I manage to create this (very slow but working) demo

As usual in this scenario, now it’s time to improve the performance and start testing with some tweaks to get a decent up and running App.

And of course, the code

Happy coding!

Greetings @ Toronto

El Bruno

Resources

#Hololens – Developer Resources for #Hololens2

Hi !

Hololens 2 are on the way to some selected group of developers, so it’s time to start to collect and group some dev resources for the not so near future.

Let’s start with a couple of tweets

There is also a new Mixed Reality Toolkit, available as usual in GitHub https://microsoft.github.io/MixedRealityToolkit-Unity/README.html

What is the Mixed Reality Toolkit

MRTK is a Microsoft driven open source project.

MRTK-Unity provides a set of foundational components and features to accelerate MR app development in Unity. The latest Release of MRTK (V2) supports HoloLens/HoloLens 2, Windows Mixed Reality, and OpenVR platforms.

  • Provides the basic building blocks for unity development on HoloLens, Windows Mixed Reality, and OpenVR.
  • Showcases UX best practices with UI controls that match Windows Mixed Reality and HoloLens Shell.
  • Enables rapid prototyping via in-editor simulation that allows you to see changes immediately.
  • Is extensible. Provides devs ability to swap out core components and extend the framework.
  • Supports a wide range of platforms, including
    • Microsoft HoloLens
    • Microsoft HoloLens 2
    • Microsoft Immersive headsets (IHMD)
    • Windows Mixed Reality headsets
    • OpenVR headsets (HTC Vive / Oculus Rift)

Input Simulation Service

The Input Simulation Service emulates the behaviour of devices and platforms that may not be available in the Unity editor. Examples include:

  • HoloLens or VR device head tracking
  • HoloLens hand gestures
  • HoloLens 2 articulated hand tracking

As usual, a video is the best way to describe this

Unreal Engine 4.22 released

New: HoloLens Remote Streaming Support

Unreal Engine 4 now supports Holographic Remoting through the Windows Mixed Reality plugin! This allows Unreal applications to run on a Windows desktop PC and stream the rendered result wirelessly to HoloLens over a Wi-Fi connection in real time.

Happy coding!
Greetings @ Toronto

Resources

#Hololens – After the huge announcement of #Hololens2, here are some interesting resources

 

 

 

Hi !

The official launch of the Hololens 2 was a couple of days ago during MWC 2019. Even if I miss my days supporting Avanade booth in the MWC, we are incredibly connected today, so I feel like I was there during the launch. So I may recap some resources to be used in the near future in an amazing surprise.

Main Microsoft Resources

Of course, let’s start with the official tweet announcement

The official video

The official post in Microsoft Blog: Microsoft at MWC Barcelona: Introducing Microsoft HoloLens 2

Some tech specs, https://www.microsoft.com/en-us/hololens/hardware

And from the original tweet some amazing Hololens 2 facts

  • #HoloLens2’s Field of View has increased by more than 2X.
  • With direct manipulation, you can now touch your holograms.
  • You can now log into your #HoloLens2 with Iris Recognition.
  • The new universal fit system works for everyone.
  • #HoloLens2’s front enclosure is made entirely out of carbon fiber.
  • #HoloLens2 will launch with a suite of solutions. Today we’re announcing Microsoft Dynamics 365 Guides.
  • Introducing #Azure Remote Rendering. A cross platform service that supports ARKit, ARCore, and #HoloLens. The birth of the internet of holograms. Leverage the power of Azure to directly stream high polygon content with no decimation to #HoloLens2.
  • We’re committed to openness.
    • We believe in an open app store model. Third party app stores are welcome within our ecosystem!
    • We believe in an open web browsing model. Firefox will be joining #HoloLens2.
    • We’ll continue to steer open standards such as Kronos & OpenXR.

And the full MWC session

Client Stories

 

Tech Coverage

  • The Verge

Happy coding!

Greetings @ Burlington

El Bruno

#WindowsML – Create Native AI apps for #Hololens #Windows10

Hi!

A couple of weeks ago I wrote about a new feature in Windows 10 to be able to use ML models natively in W10 Apps. For me, that would mean a breakthrough in the entire Windows 10 Device ecosystem. In example, as we already know, the new version of Hololens incorporates a chip specially dedicated to AI tasks (DNN specifically),
so I assumed that in Hololens V2 we could start using Windows ML using the device capabilities.

What I did not expect is that in the Preview version of Windows 10 RS4 for Hololens we already had access to that feature. I have not had time to update my Hololens to RS4, however, a couple of community experts have shared more than interesting experiences.

For example, let’s see this video by Rene Schulte, where he shows an App that performs a real time image classification using the SqueezeNet ONNX model

Mike Taulty also offers a series of posts where he does an excellent research job in Windows ML and Hololens. He started on how to create a model using Azure Custom Vision, export it to CoreML and then create an ONNX model.

Important: currently ONNX is the type of models supported by Windows ML. It is possible to export CoreML models, TensorFlow, CNTK and other frameworks to ONNX.

From here, Mike comments on how to work with UWP and C # with the exported model. In subsequent posts, Mike also explains how, thanks to Hololens RS4 Preview, these capabilities can also be used in applications for Hololens (see references)

As well, If you dare to upgrade your Hololens to RS4 Preview, it is a very interesting moment to start thinking about scenarios where you can combine WinML and Hololens!

Happy Coding!

Greetings @ Toronto

El Bruno

References

#WindowsML – Ya se puede crear apps con AI nativo en #Hololens #Windows10

 

Buenas!

Hace un par de semanas comente que la nueva capacidad en Windows 10 de poder utilizar modelos de ML de forma nativa en W10 Apps significaría un gran avance en todo el ecosistema de Devices de Windows 10. Como ya sabemos que la nueva version de Hololens incorporar un chip especialmente dedicado a tareas de AI (DNN específicamente), pues era de suponer que en Hololens V2 podríamos comenzar a utilizar Windows ML.

Lo que no esperaba es que en la version Preview de Windows 10 RS4 para Hololens ya tuviésemos acceso a esa capacidad. No he tenido tiempo de dedicarle a mis Hololens para actualizarlas a RS4, sin embargo, grandes referencias han compartido experiencias más que interesantes.

Por ejemplo, veamos este video de Rene Schulte, donde muestra una App que en tiempo real realiza una clasificación de imágenes utilizando el modelo ONNX SqueezeNet

Mike Taulty también ofrece una serie de posts donde realiza un trabajo excelente al respecto. En primer lugar, comenta como crear un modelo utilizando Azure Custom Vision, exportar el mismo a CoreML y luego crear un modelo ONNX.

Importante: actualmente ONNX es el tipo de modelos soportados por Windows ML. Es posible exportar modelos de CoreML, TensorFlow, CNTK y otros frameworks a ONNX.

A partir de aquí, Mike comenta como trabajar con UWP y C# con el modelo exportado. En siguientes posts, Mike también explica cómo, gracias a Hololens RS4 Preview, también se pueden utilizar estas capacidades en aplicaciones para Hololens (ver referencias)

Pues bien, si te animas a poner una version Preview en Hololens, es un momento muy interesante para comenzar a pensar en escenarios donde combinar WinML y Hololens!

Happy Coding!

Saludos @ Toronto

El Bruno

References

#Hololens – 29 new countries are added to the Hololens market, total of 39 and is still targeting developers

e4358372c737c7ac307eaebb86108d19-1024x576

Hello!

Today is a quick post, I’m still trying to understand how the Qubits work and this is consuming me a lot of time.

At the Future Decoded event in London, Microsoft announced that Microsoft Hololens will be available in 29 new markets.

The new countries that are added to the list are Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, Greece, Hungary, Iceland, Italy, Latvia, Liechtenstein, Lithuania, Luxembourg, Malta, the Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland and Turkey.

They are added to the markets where they could already be purchased; United States, Canada, Australia, France, Germany, Ireland, New Zealand, the United Kingdom and Japan. Officially the sale of Hololens was formalized also in China, however, it seems that they still can not be obtained in the Chinese market.

In general this is good news. If we consider some similar movements in the past,
we can think that when Hololens V2 is released to the market, it will be in these 39 countries where Hololens V1 is already commercialized.

 

Happy Coding!

Greetings @ Burlington

El Bruno

References

#Hololens – Se agregan 29 nuevos países al mercado de Hololens, en total 39 y se sigue apuntando a developers

e4358372c737c7ac307eaebb86108d19-1024x576

Hola!

Post corto de miércoles que intentar comprender como funcionan los Qubits me está consumiendo mucho tiempo.

En el evento de Future Decoded en Londres, Microsoft ha anunciado que Microsoft Hololens estará disponible en 29 nuevos mercados.

Los nuevos países que se agregan a la lista son Austria, Bélgica, Bulgaria, Croacia, Chipre, República Checa, Dinamarca, Estonia, Finlandia, Grecia, Hungría, Islandia, Italia, Letonia, Liechtenstein, Lituania, Luxemburgo, Malta, Países Bajos, Noruega, Polonia, Portugal, Rumania, Eslovaquia, Eslovenia, España, Suecia, Suiza y Turquía.

Se suman a los mercados donde ya se podían comprar las mismas; Estados Unidos, Canada, Australia, Francia, Alemania, Irlanda, Nueva Zelanda, el Reino Unido y Japón. Oficialmente se había formalizado la venta de Hololens también en China, sin embargo, parece que las mismas todavía no se pueden conseguir en el mercado chino.

En general esto es una buena noticia. Si tenemos en cuenta algunos movimientos similares en el pasado, podemos pensar que cuando Hololens V2 se lance al mercado, será en estos 39 países donde ya se comercializa Hololens V1.

Happy Coding!

Saludos @ Burlington

El Bruno

References

#MixedReality – #Hololens new, #Steam partnership, a new Halo, new Devices, maybe a #Minecraft VR and more!

I2

Hi !

Today I’ll share some updates in the Mixed Reality and Hololens world. Before the big announcement on IFA, there is a good idea related to Hololens:

LimpidArmor presented a hybrid between a helmet and Hololens. This device is integrated with cameras outside a tank to give the Hololens user a 360-degree view of both optical and thermal systems without exposing them to additional risk.

I really like the photo, and if you want to see more pictures and some field tests, their Facebook page have some other materials.

i3

And, after this quick intro, it’s time to share some of the big Microsoft announcements during the IFA keynote on Berlin, Germany.

Microsoft Mixed Reality to get support for SteamVR, Minecraft and Halo starting this holiday season

Wow! In the references section you can read some articles with more details. I’ll try to highlight the more important topics:

  • Microsoft announced a collaboration with 343 Industries to add Mixed Reality Experiences to Halo. I’m a long time Halo fan, so this is amazing for me!
  • As part of the presentation, Microsoft also shared a long term view on which types of devices will be required to use MR capabilities. This is important because usually we think on VR devices and Expensive Ones, and Microsoft took a different approach here. There will be 2 types of devices:
    • Windows Mixed Reality PCs: will consist of desktops and laptops with integrated graphics.  When plugged into these devices, our immersive headsets will run at 60 frames per second.
    • Windows Mixed Reality Ultra PCs: will consist of desktops and laptops with discrete graphics. When plugged into these devices, our immersive headsets will run at 90 frames per second.
  • And, IMHO the best one. Microsoft is partnership with Steam, I may mean SteamVR. You probably know Steam, and they been investing and working in their VR branch for a while. Virtual Reality is not new for them. Using SteamVR you can already play and use several titles for Oculus Right or HTC Vive. I’m sure the path for Mixed Reality will receive some great ideas and maybe more from the Steam group. Also the chance to have some titles in the Steam Store for the new Mixed Reality devices is a very interesting approach.

So, these partnerships, new projects and devices may open the chance to enjoy Minecraft VR in the new Windows 10 Mixed Reality devices. Also maybe in 4K. If you don’t know Minecraft VR, please take a look at this video

Greetings @ Burlington

El Bruno

References

 

#MixedReality – Algo de #Hololens, partnerships con #Steam, un nuevo Halo, nuevos Devices y tal vez un #Minecraft VR!

I2

Hola!

Hoy toca un repaso por un par de novedades relacionadas con el mundo de Mixed Reality y Hololens. Antes de la gran noticia que dio Microsoft ayer, comienzo con una que es bastante interesante:

LimpidArmor ha presentado un casco que integra unas Hololens y se integra con cámaras fuera de un tanque. El usuario del caso, gracias a Hololens tiene una visión de 360 grados tanto de sistemas ópticos como termales sin exponerlos a riesgos adicionales.

La foto es impresionante, y en su página de Facebook se pueden ver un par de fotos con pruebas de campo.

i3

Ahora vamos a la noticia principal después de las declaraciones y anuncios que ha hecho el equipo de Microsoft en la keynote del evento IFS en Berlín, Alemania.

Microsoft Mixed Reality to get support for SteamVR, Minecraft and Halo starting this holiday season

Wow! En las referencias dejo un par de artículos donde se pueden encontrar muchos detalles, pero mejore vamos por partes que hay un par de bullets más que interesantes:

  • Microsoft ha anunciado que están trabajando con 343 Industries para traer a Halo al mundo de Mixed Reality.
  • También se ha presentado un plan de ordenadores para poder utilizar los VR Devices:
    • Windows Mixed Reality PCs: will consist of desktops and laptops with integrated graphics.  When plugged into these devices, our immersive headsets will run at 60 frames per second.
    • Windows Mixed Reality Ultra PCs: will consist of desktops and laptops with discrete graphics. When plugged into these devices, our immersive headsets will run at 90 frames per second.
  • Finalmente el partnership que promete y mucho es el que han presentado con Steam. Si conoces Steam, seguramente sabes que desde hace un tiempo están trabajando en SteamVR. El mundo de realidad virtual no es nuevo para Steam, ya tienen muchos títulos que se pueden utilizar con HTC Vive o con Oculus Rift. Veremos que tal integra la realidad mixta.

Estos partnerships, proyectos y devices nos dan a entender que tal vez no tengamos que esperar mucho tiempo para poder disfrutar de un Minecraft VR en los nuevos devices de Windows 10 Mixed Reality. Y tal vez en 4K. ¿Todavia no sabes que es Minecraft VR? Pues toma un video

Saludos @ Burlington

El Bruno

References