#Hololens – Lunar Module, new sample App with some very cool PreFabs for motion controllers

p1.jpg

Hi !

Last week I spent too much time between Cognitive Services and UWP. Between all this AI stuff I also put some times to try, test and understand on of the latests samples available for Hololens Developers in the Holographic Academy: Lunar Module.

The sample is very complete, the main goal is to recreate a Lunar Module game here the objective is to help the Lunar Module land safely in the Moon. We can use hand gestures or and XBox One Controller to control the lunar module. The HoloApp will also scan the environment to find a “landing plane”.

So, the cool stuff, is to go to the Github repository and review some of the assets included in the sample. I will remark 2.

Simple Menu Collection. Finally we have an easy way to create 2D interactive menus to be used in our Hololens Apps..

p2

Hand Coach, other cool one!. Every time we create and App for Hololens, there is a chance to include a tutorial on “how to use this app”, which is usually knows as a tutorial. This Prefab have some animations on how a hand can interact using gestures with holograms in the Hololens world.

P3

Happy Unity3D coding!

Greeting @ Burlington

El Bruno

References

El Bruno, my posts

#Hololens – How To save a #3D model of the environment around the device

Hi !

During the Hololens tour, I got a couple of questions and I promised to answer them later. Here is one of them:

Can we save the surrounding environment of the Hololens in a 3D model and work with this model later?

The answer is YES, and now I’l share the necessary steps to perform this.

We need to access to the Hololens Developer Portal of our device and navigate to the option “3D View”

Clipboard01.png

When we click on the [Update] button we will see a 3D model of the current environment scanned by the device.

Clipboard02

We can use mouse and keyboard to “navigate” in this view and view details of this environment.

Clipboard03.png

Important: The Hololens device is always scanning the environment, and it’s also updating the mesh of this model. If we press again the [Update] button we will be able to see the updated changes to the model. Also, the model is not restrained to “only one room”, we can have and work with complex models including several rooms. The next image is a photo and 3D view of a corner in my home office.

Clipboard04.png

We can export this scene model in a file with OBJ format. We can import and work with this file using several tools like Unity 3D, Paint 3D or 3D Builder. The next screen shoot is my office while I polish the model in 3D Builder.

Clipboard01

Greetings @ Toronto

El Bruno

References

El Bruno, my posts

#Hololens – Como guardar un modelo #3D del entorno que nos rodea

Buenas!

Durante el Hololens tour, una de las preguntas que quedo pendiente de responder era

¿Es posible utilizar Hololens para guardar el mapeo 3D de la habitación en la que estamos trabajando?

La respuesta era: SI, aunque estaba pendiente comentar como realizar este proceso.
Lo primero que debemos hacer es navegar al Developer Portal de nuestro device y acceder a la opción “3D View”

Clipboard01.png

En este momento debemos presionar el botón [Update] y veremos cómo aparece el mapeo de la habitación donde estamos usando el device.

Clipboard02

En esta vista podemos utilizar el mouse y el teclado para “navegar” en la misma y ver los detalles de nuestro entorno.

Clipboard03.png

Es importante recordar que Hololens está todo el tiempo actualizando este mesh del entorno, si presionamos nuevamente el botón [Update] veremos cómo se actualiza el modelo de nuestra habitación. Otro detalle importante es que podemos tener un modelo completo con varias habitaciones conectadas.

En la siguiente imagen presento una esquina de mi oficina y el mapeo que realizo Hololens. Se puede ver que el proceso de escaneo simplemente mirando “un sitio” no es muy detallado, sin embargo, es lo suficientemente completo para poder realizar tareas de navegación en el mismo. En futuros posts comentare como realizar este proceso con más detalle.

Clipboard04.png

Finalmente, lo que podemos hacer es guardar este modelo en formato OBJ, que luego podremos, por ejemplo, importar desde Unity 3D o 3D Builder. Este es un vistazo de mi habitación trabajada en 3D Builder.

Clipboard01

Como siempre, no soy un experto, pero después de un par de minutos pude “pulir” y mejorar un poco el modelo para poder trabajar con el mismo. En el siguiente post, comentare como importar este modelo en Unity3D para que sirva como la base de una escena.

Saludos @ Toronto

El Bruno

References

El Bruno, my posts

#Hololens – How to use clases TapToPlace and InitialScanManager

Hi !

Yesterday I wrote about a custom solution I made to have Tap to Place feature in an Hololens App. The main set of features was

  • We can perform an initial scan of an environment, based on a time constraint. Also we can continue the scan process if we don’t find a minimum of number of planes for floor or wall
  • We can associate one class to an hologram to have the Tap to Place feature
  • This process also allow us to validate if we can place the hologram in a vertical (wall) or horizontal (floor) plane
  • The validation process also have some visual features like change the color of a box collider from red to green depending if we can or can’t place an object and also cast a shadow with the final location of the hologram

The “InitialScanManager.cs” is the one I use to perform the initial environment scan. I usually add this class to an Empty Game Object. The main properties to use are this ones:

  • Limit Scanning by Time and Scan Time, definen how the scan process will be performed. If we set a Scan Time value of N, and check the Limit Scan time, the app will scan the environment for N seconds and then will stop the scan process
  • Minimum Floor and Minimum Wall defines the minimum amount of frames to be searched in the floor and the wall
  • Player, this is optional for the main hologram. When the scan process start it will be hide and when the scan process ends the player will be show again
  • DebugTextPlayer, optional, a TextMesh used for debug actions

Clipboard02.png

The class “TapToPlacePlayer.cs” is the one we must associate to the element where we want this feature. The main properties are

  • Placeable Bounds and Non Placeable Bounds Materials are materials used when we start the “place mode”. I create a collider with this materials and the class change the materials based on if we can place or not the hologram at the selected position
  • Placeable Shadow and Non Placeable Shadow Materials are materials used to display the final position to be placed for the hologram
  • Display Shadow defines if we want to show (or not) the final position of the hologram
  • Placement Surface defines the positioning mode Horizontal or Vertical. I mean Floor or Wall
  • DebugTextPlayer, optional, a TextMesh used for debug actions

Clipboard04.png

The complete source code can be downloaded from GitHub (link).

Greetings @ Toronto

El Bruno

References

El Bruno, my posts

#Hololens – Cómo utilizar las clases TapToPlace y InitialScanManager

Hola !

En el post de ayer compartí una solución para tener las siguientes capacidades en un proyecto de Hololens

  • Realizar un scanner inicial del ambiente basado en tiempo y en cantidad de frames encontrados para piso y pared
  • Se puede iniciar un proceso de Tap to Place sobre un holograma
  • El proceso valida además si la posición a la que se quiere mover el holograma es válida.
  • La validación es en primer lugar sobre mover a planos horizontales o verticales y luego si hay espacio para el holograma

Para la acción de análisis del entorno, he creado la clase “InitialScanManager.cs”. La misma puede ser agregada en un Empty Game Object o en uno existente y las propiedades principales a tener en cuenta son:

  • Limit Scanning by Time y Scan Time, definen si el proceso de scan se realiza en base a N segundos de escaneo
  • Minimum Floor y Minimum Wall definen la cantidad mínima de frames que se deben encontrar en el suelo y en un muro
  • Player, define el holograma o grupo de hologramas que se ocultarán al comienzo del scan y se activarán al terminar el mismo
  • DebugTextPlayer, opcional, un TextMesh utilizado para acciones de Debug

Clipboard02.png

Por otra parte tenemos la clase “TapToPlacePlayer.cs”, que es debe asociarse al elemento que queremos que tenga la capacidad de “Tap to Place”.  Las propiedades principales a tener en cuenta son:

  • Placeable Bounds y Non Placeable Bounds Materials es el material con el que se mostrará un collider cuando estemos posicionando el holograma
  • Placeable Shadow y Non Placeable Shadow Materials es el material con el que se mostrará la posición final que tendrá el holograma sobre el que estamos trabajando
  • Display Shadow define si queremos utilizar las propiedades anteriores
  • Placement Surface define si queremos soportar el posicionamiento en modo horizontal o vertical, es decir en muros o en el piso
  • DebugTextPlayer, opcional, un TextMesh utilizado para acciones de Debug

Clipboard04.png

El código completo del ejemplo se puede descargar desde aquí (link).

Saludos @ Toronto

El Bruno

References

El Bruno, my posts

#Hololens – Alternative fix to the TapToPlace and WorldAnchor issue (until official #HoloToolkit release)

Hi !

If you work with Beta products you will find funny scenarios, like something working, then not working, then working again. The next Hololens forums thread is a must read if you want to understand the frustration you can get in this scenarios:

VS2017/U5.6 Crash – WinRTScriptingObjectWrapper::InternalAddRef Reference Count cannot be zero (link)

TLTR HoloToolkit current version is not officially supported for the combo Visual Studio 2017 and Unity3D 5.6. 

There are features like Tap To Place which are not working fine with VS2017 and Unity3D 5.6. If you implement this, as soon as the Hololens App starts, it will crash and close. Without any clear explanation on what´s happening. It seems to be something related to World Anchors and the way Unity are is working with this,  this issue is still alive on GitHub (link).

So, I need Tap To Place, and I decided to implement this myself. I pickup some code from HoloToolkit and some other pieces from Holographic Academy tutorials and I’ve created 2 main classes which fix the problem, and also add some additional value. Here are the main implemented features

  • We can perform an initial scan of an environment, based on a time constraint. Also we can continue the scan process if we don’t find a minimum of number of planes for floor or wall
  • We can associate one class to an hologram to have the Tap to Place feature
  • This process also allow us to validate if we can place the hologram in a vertical (wall) or horizontal (floor) plane
  • The validation process also have some visual features like change the color of a box collider from red to green depending if we can or can’t place an object and also cast a shadow with the final location of the hologram

In the next animation we can see a cylinder hologram with the following configuration:

  • Green material for the rigid box for valid place
  • Red material for the rigid box for invalid place
  • No shadows enabled

2017 04 26 Holo TapToPlace with Validation 01

The complete source code can be downloaded from GitHub (link).

Greetings @ Toronto

El Bruno

References

#Hololens – Solución al problema con TapToPlace agregando validación de planos (hasta que #HoloToolkit sea compatible)

Hola !

Cuando trabajas con productos en estado Beta puedes encontrarte con errores como el que se comenta en el siguiente post del foro:

VS2017/U5.6 Crash – WinRTScriptingObjectWrapper::InternalAddRef Reference Count cannot be zero (link)

TLTR la versión actual de HoloToolkit no está soportada por Visual Studio 2017 y Unity3D 5.6. 

En otras palabras hay funcionalidades como la de Tap To Place que al momento de lanzar una App hacen que la misma se cierre sin lanzar una excepción ni nada parecido. Parece que es por un problema de cálculo de Anchors, como comentan aquí (link).

Pues bien, tomando un poco de código deHoloToolkit y otro poco de Holographic Academy, he creado 2 clases que me permiten solucionar el problema. Por ahora soporta lo siguiente

  • Realizar un scanner inicial del ambiente basado en tiempo y en cantidad de frames encontrados para piso y pared
  • Se puede iniciar un proceso de Tap to Place sobre un holograma
  • El proceso valida además si la posición a la que se quiere mover el holograma es válida.
  • La validación es en primer lugar sobre mover a planos horizontales o verticales y luego si hay espacio para el holograma

Veamos el siguiente ejemplo. En el mismo trabajo sobre un holograma de tipo cylinder, cuando se activa el TapToPlace se crea un rigidBox para mostrar con colores si podemos posicionar o no el cilindro.

2017 04 26 Holo TapToPlace with Validation 01

El código completo del ejemplo se puede descargar desde aquí (link).

Saludos @ Toronto

El Bruno

References

#Hololens – #SpatialMapping, tutorial search for simple surfaces (1×1 floor tile)

Hi !

In my previous posts I wrote on how to use Spatial Understanding to have a better control of the environment scanning process with Hololens. In that post I created a demo with the following steps

  • As soon as the App starts, Hololens started the scanning process
  • We defined a minimmun number of planes to be found, when we reach this number the scan process stopped
  • We have a FPS displaying the scan process

Today’s sample is the next required steps after last post to

  • Search for a simple surface in the floor with a size of 1×1
  • We start this search process with an AirTap / Click on a hologram

The next ugly animation shows the scanning process (at X6 speed) and then the floor mapping process.

2017 04 25 Hololens Spatial Understanding 01.gif

Tutorial simple steps.

  1. Create a 3D project in Unity3D
  2. Import HoloToolkit package
  3. Configure project to
    1. Support HoloLens projects (UWP, VR, etc)
    2. enable Spatial Mapping feature
  4. Clean Scene elements
  5. Add
    1. Hololens Camera
    2. Cursor With Feedback
    3. Input Manager
    4. Spatial Mapping
    5. Spatial Understanding
    6. FPS Display
  6. Add Empty element
    1. Rename to CodeManagers
    2. Add new C# Script named “GameStartScanner.cs”
  7. Add Empty Element
    1. Rename to HoloCollection
    2. Add 3D Cube element to HoloCollection
    3. Rename to “InteractiveCube”
  8. Set this properties to Cube
    1. Position: x: 0, y:0, z:1.2
    2. Scale: x: 0.2, y:0.2, z:0.2
  9. Add a new C# Script to the cube named “Scanner Analyzer”

The final project must be similar to this one

Clipboard02

The source code for the ScannerAnalyzer.cs script is below. As always a couple of important notes on this script.

  • The sample is based on the Spatial Mapping sample included on HoloToolkit
  • I use several additional classes (mostly for drawing), you can find this classes as part of the project
    • AnimatedBox.cs
    • AnimationCurve3.cs
    • GameStartScanner.cs
    • Line.cs
    • LineData.cs
    • ScannerAnalyzer.cs
  • In simple terms, the new class starts when the scanning process stops. As a reminder the scanning process was defined in “GameStartScanner.cs”
  • In the Update() function we check if we are in “search tile process”. If we are looking, we draw blue tiles on the floor
  • The search process start on “OnInputClicked”, this one is triggered on an AirTap on he Cube
  • The variables minWidthOfWallSpace and minHeightAboveFloor defines the size of the tile to search
  • Line 54, this is the important section. Here we use the core of Spatial Understanding to start the search process. We use the functions
    • SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(_resultsTopology);
    • SpatialUnderstandingDllTopology.QueryTopology_FindPositionsOnFloor()
  • The 1st steps create a memory pointer with all the scanned elements. 2nd steps filter this for the elements which matches the query
  • The next lines are drawing lines. Most of this code is based on HoloToolkit Spatial Mapping samples. Tons on C# lines of code, which requires a nice refactoring

Sample code

You can download the complete code from GitHub (link).

Greetings @ Toronto

El Bruno

References

#Hololens – #SpatialMapping, tutorial sobre cómo buscar superficies simples (Un tile de 1×1 en el piso)

Hola !

En el post anterior expliqué como utilizar Spatial Understanding para tener más control sobre el proceso de escaneo que realiza Hololens. En ese ejemplo, se realizaban los siguientes pasos

  • Cuando se lanzaba la App, Hololens comenzaba el proceso de escaneo
  • Cuando se llegaba a un mínimo de planos encontrados, se finalizaba el mismo
  • En todo momento se mostraba el estado del escaneo en un FPS

El ejemplo de hoy es la continuación del anterior, en el que una vez finalizados los pasos anteriores

  • Buscaremos una superficie de tamaño 1×1 en el piso, cuando se haga AirTap / Click en un holograma

La siguiente animación muestra el proceso de scanning (X6 speed, no apto para personas que se mareen) y luego el mapeo en el piso donde se puede ver como se “pintan” los mosaicos del tamaño encontrado.

2017 04 25 Hololens Spatial Understanding 01.gif

Estos son los pasos a seguir.

  1. Create a 3D project in Unity3D
  2. Import HoloToolkit package
  3. Configure project to
    1. Support HoloLens projects (UWP, VR, etc)
    2. enable Spatial Mapping feature
  4. Clean Scene elements
  5. Add
    1. Hololens Camera
    2. Cursor With Feedback
    3. Input Manager
    4. Spatial Mapping
    5. Spatial Understanding
    6. FPS Display
  6. Add Empty element
    1. Rename to CodeManagers
    2. Add new C# Script named “GameStartScanner.cs”
  7. Add Empty Element
    1. Rename to HoloCollection
    2. Add 3D Cube element to HoloCollection
    3. Rename to “InteractiveCube”
  8. Set this properties to Cube
    1. Position: x: 0, y:0, z:1.2
    2. Scale: x: 0.2, y:0.2, z:0.2
  9. Add a new C# Script to the cube named “Scanner Analyzer”

El proyecto debe quedar similar al siguiente

Clipboard02

Y el script para la nueva clase está debajo, y lo mejor es revisar un par de apuntes interesantes sobre la misma

  • Esta clase está basada en el ejemplo de Spatial Mapping de HoloToolkit
  • La misma usa clases auxiliares que son parte del proyecto,
    • AnimatedBox.cs
    • AnimationCurve3.cs
    • GameStartScanner.cs
    • Line.cs
    • LineData.cs
    • ScannerAnalyzer.cs
  • La funcionalidad de esta clase se activa cuando la clase que creamos en el post anterior “GameStartScanner.cs” termina el proceso de scan del entorno
  • En el Update se verifica si se está “buscando” una superficie de 1×1 en el suelo, en caso afirmativo se dibuja la misma
  • El proceso de búsqueda se realiza en “OnInputClicked”, que se activa cuando hacemos AirTap o click sobre el Cube
  • En esta funcion también se define el tamaño de superficie a buscar con las variables, minWidthOfWallSpace y var minHeightAboveFloor
  • En la línea 54 comienza el proceso de búsqueda utilizando los tamaños a buscar y las funciones
    • SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(_resultsTopology);
    • SpatialUnderstandingDllTopology.QueryTopology_FindPositionsOnFloor()
  • En el 1er paso se crea un puntero en memoria con todos los elementos de trabajo y el 2do paso es el que “filtra” por los que estén en el suelo y con el tamaño deseado
  • Lo siguiente es dibujar los frames en el piso, este código está basado en lo ejemplos de Spatial Mapping. Son líneas y líneas que merecen una buena tarde refactoring.

Código de ejemplo

El código completo del ejemplo se puede descargar desde aquí (link).

Saludos @ Toronto

El Bruno

References

#Hololens – Spatial Understanding vs Spatial Mapping, and a step-by-step on how to use it

Hi !

HoloToolkit has 2 main assets to perform a Spatial Mapping scan in an environment with the Hololens.

  • Spatial Mapping
  • Spatial Understanding

Spatial Mapping is the asset that we use by default and is the one which allows us to have basic control over the scan process performed by the Hololens. For example, once we add this element to an Unity3D project and we enable the drawing of the Mesh, we can see in real time the scan of the environment where while we are using it.

Spatial Understanding allows us to go one step further and perform more complex tasks. For example:

  • Define the start and end time for the scan process. This is really useful for example, when we need to perform an initial scan and then we know that the environment will not change. If we disable the scan in real time, we give more resources to Hololens to be used on other tasks
  • Perform simple queries on different elements in our point of view. For example, to search for flat surfaces in a wall or in the floor to position holograms.
  • Perform more complex queries, to search for items such as a chair, a guitar, etc. I have not used this feature yet, but I have it in in my to do list for when I need it.

Important: Whenever we use Spatial Understanding, we must have added the Spatial Mapping prefab, if we don’t do this we find see some very funny errors.

Personally I think that the 2 best ways to learn about how Spatial Understanding are as follows

  • Read Jeff Evertt article Case study – Expanding the spatial mapping capabilities of HoloLens, he comments on how they created this asset for specific scenarios that where nor contemplated in the original HoloToolkit Spatial Mapping.
  • Review the sample Scene located in “HoloToolkit-Unity/Assets/HoloToolkit-Examples/SpatialUnderstanding/SpatialUnderstanding-FeatureOverview/Scenes/“.
    It uses Spatial Understanding to perform an initial scan of the atmosphere and then present some options of simple queries.

If you also want it to use in a local project, the next tutorial may help you. The end result is an App that shows in real time a HUD with the status of the scan and at the end disable the visualization of the Mesh.

Easy steps.

  1. Create a 3D project in Unity3D
  2. Import HoloToolkit package
  3. Configure project to
    1. Support HoloLens projects (UWP, VR, etc)
    2. enable Spatial Mapping feature
  4. Clean Scene elements
  5. Add
    1. Hololens Camera
    2. Cursor With Feedback
    3. Input Manager
    4. Spatial Mapping
    5. Spatial Understanding
    6. FPS Display
  6. Add Empty element
    1. Rename to CodeManagers
    2. Add new C# Script named “GameStartScanner.cs”

The final project may seems similar to this one

Clipboard02

You can download the complete source code from GitHub (link).

The script for the new class is below, and it is best to check a couple of interesting notes about the same

  • MinAreaForComplete, MinHorizAreaForComplete, MinWallAreaForComplete properties defined the minimum area to be scanned to give the process how valid
  • In the Start() function starts the Scan process with operation SpatialUnderstanding.Instance.RequestBeginScanning (); and we also subscribe to us to state changes
  • The DoesScanMeetMinBarForCompletion property is that is responsible for validating if the scan process is finished
  • In the Update 2 major operations are performed
    • The status of the scan process is displayed in a FPS Text
    • If the minimal elements, it has been found we finished the scan process, and disable visual scan process

Sample Code

Greetings @ Toronto

El Bruno

References