Hi !
Custom Vision Allows us to create models of Object recognition. Once these models are trained, we can analyze an image and the model will offer us as an answer
- A list of [Tags] objects detected in each image
- For each TAG we will also have the probability [score] associated with it and a series of numerical values with the position of the object found within the analyzed image
In previous posts I wrote about how to perform object analysis from the feed From a Webcam In a Windows 10 application. The next step is to show the Frame of the recognized object.
The following code shows an example of how to show the frames In the Windows 10 App using a Canvas. The 2 main functions are
- DrawFrames(Where an iteration of the predictions made
- DrawFrame() This is the function that takes care of drawing the Frame in real time. There’s a little bit of math in it to adjust the ONNX values to the actual size of the Canvas and the Webcam.
For example, these are the values that I work with in a tag of Iron Fist In the image of this post.
- The Canvas size is Actual Width: 1356, Actual Height: 700
- The values returned by ONNX prediction process are Top: 20.80284, Left: 73.15757, Height: 54.41817, Width: 24.3813
- The Frame To show will be drawn with the following values Y: 140, x: 989, Height: 378, Width: 325
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| private async Task LoadAndEvaluateModelAsync(VideoFrame videoFrame) | |
| { | |
| _objectDetection.ProbabilityThreshold = 0.5F; | |
| _stopwatch = Stopwatch.StartNew(); | |
| _predictions = await _objectDetection.PredictImageAsync(videoFrame); | |
| _stopwatch.Stop(); | |
| await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => | |
| { | |
| DrawFrames(); | |
| }); | |
| } | |
| private void DrawFrames() | |
| { | |
| OverlayCanvas.Children.Clear(); | |
| var message = $"{DateTime.Now.ToLongTimeString()} – {1000f / _stopwatch.ElapsedMilliseconds,4:f1} fps{Environment.NewLine}============================={Environment.NewLine}"; | |
| if (_predictions.Count > 0) | |
| foreach (var prediction in _predictions) | |
| DrawFrame(prediction, OverlayCanvas); | |
| TextBlockResults.Text = message; | |
| } | |
| private void DrawFrame(PredictionModel prediction, Canvas overlayCanvas) | |
| { | |
| _overlayCanvasActualWidth = (uint)CameraPreview.ActualWidth; | |
| _overlayCanvasActualHeight = (uint)CameraPreview.ActualHeight; | |
| var x = (uint)Math.Max(prediction.Box.Left, 0); | |
| var y = (uint)Math.Max(prediction.Box.Top, 0); | |
| var w = (uint)Math.Min(_overlayCanvasActualWidth – x, prediction.Box.Width); | |
| var h = (uint)Math.Min(_overlayCanvasActualHeight – y, prediction.Box.Height); | |
| Debug.WriteLine($"\tOverLay Canvas \tActual Width: {_overlayCanvasActualWidth}, Actual Height: {_overlayCanvasActualHeight}"); | |
| Debug.WriteLine(prediction.GetExtendedDescription()); | |
| Debug.WriteLine($"\tOriginal\tY: {y}, x: {x}, Height: {h}, Width: {w}"); | |
| // fit to current size | |
| var factorSize = 100u; | |
| x = _overlayCanvasActualWidth * x / factorSize; | |
| y = _overlayCanvasActualHeight * y / factorSize; | |
| w = _overlayCanvasActualWidth * w / factorSize; | |
| h = _overlayCanvasActualHeight * h / factorSize; | |
| Debug.WriteLine($"\tScaled\tY: {y}, x: {x}, Height: {h}, Width: {w}"); | |
| var rectStroke = _lineBrushGreen; | |
| switch (prediction.TagName) | |
| { | |
| case "Venom": | |
| rectStroke = _lineBrushGray; | |
| break; | |
| case "Rocket_Racoon": | |
| rectStroke = _lineBrushYellow; | |
| break; | |
| case "Iron_Fist": | |
| rectStroke = _lineBrushGreen; | |
| break; | |
| } | |
| var r = new Windows.UI.Xaml.Shapes.Rectangle | |
| { | |
| Tag = prediction, | |
| Width = w, | |
| Height = h, | |
| Fill = _fillBrush, | |
| Stroke = rectStroke, | |
| StrokeThickness = _lineThickness, | |
| Margin = new Thickness(x, y, 0, 0) | |
| }; | |
| var tb = new TextBlock | |
| { | |
| Margin = new Thickness(x + 4, y + 4, 0, 0), | |
| Text = $"{prediction}", | |
| FontWeight = FontWeights.Bold, | |
| Width = 126, | |
| Height = 21, | |
| HorizontalTextAlignment = TextAlignment.Center | |
| }; | |
| var textBack = new Windows.UI.Xaml.Shapes.Rectangle | |
| { | |
| Width = 150, | |
| Height = 29, | |
| Fill = rectStroke, | |
| Margin = new Thickness(x, y, 0, 0) | |
| }; | |
| overlayCanvas.Children.Add(textBack); | |
| overlayCanvas.Children.Add(tb); | |
| overlayCanvas.Children.Add(r); | |
| } |
In following posts I’ll comment on final details on how to measure processing time and other tips.
The full app can be seen in https://github.com/elbruno/events/tree/master/2019%2001%2010%20CodeMash%20CustomVision/CSharp/CustomVisionMarvelConsole01
Happy Coding!
Greetings @ Burlington
El Bruno
References
- Custom Vision
- Quickstart: Create an image classification project with the Custom Vision .NET SDK
- Quickstart: Create an object detection project with the Custom Vision .NET SDK
- ONNX Documentation
- Sample application for ONNX1.2 models exported from Custom Vision Service
- Windows Community Toolkit
- Visual Studio Tools for AI
My Posts
- Object recognition with Custom Vision and ONNX in Windows applications using WinML (1)
- Object recognition with Custom Vision and ONNX in Windows applications using WinML (2)
Windows 10 and YOLOV2 for Object Detection Series
- Introduction to YoloV2 for object detection
- Create a basic Windows10 App and use YoloV2 in the camera for object detection
- Transform YoloV2 output analysis to C# classes and display them in frames
- Resize YoloV2 output to support multiple formats and process and display frames per second
- How to convert Tiny-YoloV3 model in CoreML format to ONNX and use it in a Windows 10 App
- Updated demo using Tiny YOLO V2 1.2, Windows 10 and YOLOV2 for Object Detection Series
- Alternatives to Yolo for object detection in ONNX format
El Bruno 
Leave a comment