[#KINECT] HowTo: Use Face Recognition with #KinectSdk (II)

image

Buenas,

in the previous post I mentioned some of the main features to begin work with the kit for the recognition of gestures that included with the Kinect SDK .

Today we will see how to use this functionality in a WPF project. To do this follow these steps:

1 Create a new WPF project. In my case I called [ElBruno.FaceTracking02] and also I added the reference to [Microsoft.Kinect].

2. From the installation folder of the Kinect SDK Developer Toolkit, add references to the assemblies in Face Tracking. The location is similar to:

C:\Program Files\Microsoft SDKs\Kinect\Developer Toolkit v1.5.2\Samples\bin

and the references should be added to

  • Microsoft.Kinect.Tookit
  • Microsoft.Kinect.Tookit.FaceTracking

image

3 Change the project settings so that it is compiled into x 86 mode. In the project properties, select the “Build” section in Platform Target select “x 86”, and compile the project.

4. This step is a little “curious”, but in the next post I will put a more elegant way of doing this.From the folder

C:\Program Files\Microsoft SDKs\Kinect\Developer Toolkit v1.5.2\Redist\x86

copy the assemblies

  • FaceTrackData.dll
  • FaceTrackLib.dll

towards the compilation of our project directory.

image

5. Add a new UserControl in our project called [FaceTrackingViewer] and a class called [SkeletonFaceTracker]. As they have lot code inside, it is best download example zip and copy it complete.

6. The [SkeletonFaceTracker] class, the charge of using assemblies of the Toolkit, of recognizing and then paint the face points. For this, you subscribe to the [OnFrameReady] event of the Kinect sensor and it uses the Toolkit to get the points of the face. Better see some code that is fairly self-explanatory:

   1: internal void OnFrameReady(KinectSensor kinectSensor, ColorImageFormat colorImageFormat, 
   2:     byte[] colorImage, DepthImageFormat depthImageFormat, 
   3:     short[] depthImage, Skeleton skeletonOfInterest)
   4: {
   5:     _skeletonTrackingState = skeletonOfInterest.TrackingState;
   6:     if (_skeletonTrackingState != SkeletonTrackingState.Tracked)
   7:     {
   8:         return;
   9:     }
  10:  
  11:     if (_faceTracker == null)
  12:     {
  13:         try
  14:         {
  15:             _faceTracker = new FaceTracker(kinectSensor);
  16:         }
  17:         catch (InvalidOperationException)
  18:         {
  19:             Debug.WriteLine("AllFramesReady - creating a new FaceTracker threw an InvalidOperationException");
  20:             _faceTracker = null;
  21:         }
  22:     }
  23:  
  24:     if (_faceTracker == null) return;
  25:     var frame = _faceTracker.Track(colorImageFormat, colorImage, depthImageFormat, depthImage, skeletonOfInterest);
  26:     LastFaceTrackSucceeded = frame.TrackSuccessful;
  27:     if (!LastFaceTrackSucceeded) return;
  28:     if (_faceTriangles == null)
  29:     {
  30:         _faceTriangles = frame.GetTriangles();
  31:     }
  32:     _facePoints = frame.GetProjected3DShape();
  33: }

7. This class has another function which is which is responsible for drawing the mask on a canvas, where are the points of the face. Apart from the collection of points, creates an array of triangles and the paints on a work context.

   1: public void DrawFaceModel(DrawingContext drawingContext)
   2: {
   3:     if (!LastFaceTrackSucceeded || _skeletonTrackingState != SkeletonTrackingState.Tracked)
   4:     {
   5:         _startFaceRecognition = DateTime.MinValue;
   6:         return;
   7:     }
   8:  
   9:     var faceModelPts = _facePoints.Select(t => new Point(t.X + 0.5f, t.Y + 0.5f)).ToList();
  10:  
  11:     var faceModel = _faceTriangles.Select(t => new FaceModelTriangle
  12:     {
  13:         P1 = faceModelPts[t.First],
  14:         P2 = faceModelPts[t.Second],
  15:         P3 = faceModelPts[t.Third]
  16:     }).ToList();
  17:  
  18:     var faceModelGroup = new GeometryGroup();
  19:     for (var i = 0; i < faceModel.Count; i++)
  20:     {
  21:         var faceTriangle = new GeometryGroup();
  22:         faceTriangle.Children.Add(new LineGeometry(faceModel[i].P1, faceModel[i].P2));
  23:         faceTriangle.Children.Add(new LineGeometry(faceModel[i].P2, faceModel[i].P3));
  24:         faceTriangle.Children.Add(new LineGeometry(faceModel[i].P3, faceModel[i].P1));
  25:         faceModelGroup.Children.Add(faceTriangle);
  26:     }
  27:  
  28:     drawingContext.DrawGeometry(Brushes.BlueViolet, new Pen(Brushes.Red, 1.0), faceModelGroup);
  29:     if (_startFaceRecognition == DateTime.MinValue)
  30:     {
  31:         _startFaceRecognition = DateTime.Now;
  32:     }
  33:     else
  34:     {
  35:         if (DateTime.Now.Subtract(_startFaceRecognition).TotalSeconds > 5)
  36:         {
  37:             Application.Current.Dispatcher.BeginInvoke(new Action(() =>
  38:             {
  39:                 System.Threading.Thread.Sleep(300);
  40:                 _startFaceRecognition = DateTime.MinValue;
  41:             }));
  42:         }
  43:     }
  44: }

8. Now go to the viewfinder, which is that “paints” this surface [FaceTrackingViewer]. Inside the function [OnAllFramesReady] where processed data from the Kinect sensor is identified each of the skeletons in a State [SkeletonTrackingState.Tracked]. Once identified, creates a new instance of the SkeletonFaceTracker class and run the above steps to identify a face face. Each SkeletonFaceTracker instance is stored in a collection.

   1: foreach (Skeleton skeleton in this.skeletonData)
   2: {
   3:     if (skeleton.TrackingState == SkeletonTrackingState.Tracked
   4:         || skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
   5:     {
   6:         if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
   7:         {
   8:             this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker() { });
   9:         }
  10:  
  11:         SkeletonFaceTracker skeletonFaceTracker;
  12:         if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
  13:         {
  14:             if (this.Kinect != null)
  15:             {
  16:                 skeletonFaceTracker.OnFrameReady(this.Kinect, colorImageFormat, 
  17:                                     colorImage, depthImageFormat, depthImage, skeleton);
  18:                 skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
  19:             }
  20:         }
  21:     }
  22:  
  23: }

9. Finally, in the OnRender() of the control function is “painted” each one of the faces.

   1: protected override void OnRender(DrawingContext drawingContext)
   2: {
   3:     base.OnRender(drawingContext);
   4:     foreach (var faceInformation in this.trackedSkeletons.Values)
   5:     {
   6:         faceInformation.DrawFaceModel(drawingContext);
   7:     }
   8: }

10. The complete example includes a bit of code in the main to detect the kinect sensor and work with the same window. And another couple of details which I will comment on in a next post.

The example can be downloaded from http://sdrv.Ms/MZY0Eq

Saludos @ La Finca

El Bruno

image image image

3 thoughts on “[#KINECT] HowTo: Use Face Recognition with #KinectSdk (II)

  1. Buenos días Bruno,
    Me parece fantastico las posibilidades que ofrece kinect y el face tracking. No obstante no encuentro respuesta a mi pregunta un tanto simple viendo la potencia de la herramienta.
    Como puedo colocar una mascar 3D “estática” y simple cuando se detecte una cara???
    Me gustaría comenzar con un simple bigote, barba y sombrero. Esto lo puedo realizar con otros proveedores como T-inmersion donde sus modelos son 3d estáticos en un formato específico como*.obj.
    En este mundo kinect … que tipo de formato 3d puedo colocar como máscara?
    Muchas gracias por tu atencion.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.