# #Coding4Fun – How to control your #drone with 20 lines of code! (21/N)

Hi !

In my post series I already wrote about how to detect faces. We can do this with a camera and OpenCV. However, a drone can also be moved on command, so let’s write some lines to detect a face, and calculate the orientation and distance of the detected face from the center camera of the camera.

In order to do this, 1st let’s draw a grid in the camera frame, and once a face is detected, let’s show the distance and orientation from the center.

Let’s start with a Grid. The idea is to create a 3×3 grid in the camera frame, and use the center cell as reference for the detected objects. The code to create a 3×3 grid is this one:

```def displayGrid(frame):
# Add a 3x3 Grid
cv2.line(frame, (int(camera_Width/2)-centerZone, 0)     , (int(camera_Width/2)-centerZone, camera_Heigth)    , lineColor, lineThickness)
cv2.line(frame, (int(camera_Width/2)+centerZone, 0)     , (int(camera_Width/2)+centerZone, camera_Heigth)    , lineColor, lineThickness)
cv2.line(frame, (0, int(camera_Heigth / 2) - centerZone), (camera_Width, int(camera_Heigth / 2) - centerZone), lineColor, lineThickness)
cv2.line(frame, (0, int(camera_Heigth / 2) + centerZone), (camera_Width, int(camera_Heigth / 2) + centerZone), lineColor, lineThickness)

# Camera Settings
camera_Width  = 1024 # 1280 # 640
camera_Heigth = 780  # 960  # 480
centerZone    = 100

# GridLine color green and thickness
lineColor = (0, 255, 0)
lineThickness = 2

```

We use the line() function on OpenCV, and do some calculations to get the starting and endpoint for the 4 lines for the grid: 2 vertical lines and 2 horizontal lines. For this demo, I’ll implement this in my main webcam.

Based on my face detection samples and other samples in GitHub (see references), now I’ll calculate the position of the detected face (with x, y, h, w) from the center of the camera:

```def calculatePositionForDetectedFace(frame, x, y, h , w):
# calculate direction and relative position of the face
cx = int(x + (w / 2))  # Center X of the Face
cy = int(y + (h / 2))  # Center Y of the Face

if (cx <int(camera_Width/2) - centerZone):
cv2.putText  (frame, " LEFT " , (20, 50), cv2.FONT_HERSHEY_COMPLEX, 1 , colorGreen, 2)
dir = 1
elif (cx > int(camera_Width / 2) + centerZone):
cv2.putText(frame, " RIGHT ", (20, 50), cv2.FONT_HERSHEY_COMPLEX,1,colorGreen, 3)
dir = 2
elif (cy < int(camera_Heigth / 2) - centerZone):
cv2.putText(frame, " UP ", (20, 50), cv2.FONT_HERSHEY_COMPLEX,1,colorGreen, 3)
dir = 3
elif (cy > int(camera_Heigth / 2) + centerZone):
cv2.putText(frame, " DOWN ", (20, 50), cv2.FONT_HERSHEY_COMPLEX, 1,colorGreen, 3)
dir = 4
else: dir=0

# display detected face frame, line from center and direction to go
cv2.line     (frame, (int(camera_Width/2),int(camera_Heigth/2)), (cx,cy), colorRed, messageThickness)
cv2.rectangle(frame, (x, y), (x + w, y + h), colorBlue, messageThickness)
cv2.putText  (frame, str(int(x)) + " " + str(int(y)), (x - 20, y - 45), cv2.FONT_HERSHEY_COMPLEX,0.7, colorRed, messageThickness)

```

The output is similar to this one

And now with the base code completed, it’s time to add this logic to the drone samples !

Bonus: the complete code.

Happy coding!

Greetings

El Bruno

# #Coding4Fun – How to control your #drone with 20 lines of code! (20/N)

Hi !

We already have the drone camera feed ready to process, so let’s do some Image Segmentation today. As usual, let’s start with the formal definition of Image Segmentation

In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple segments (sets of pixels, also known as image objects). The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.[1][2] Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images. More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.

The result of image segmentation is a set of segments that collectively cover the entire image, or a set of contours extracted from the image (see edge detection). Each of the pixels in a region are similar with respect to some characteristic or computed property, such as color, intensity, or texture. Adjacent regions are significantly different with respect to the same characteristic(s).[1] When applied to a stack of images, typical in medical imaging, the resulting contours after image segmentation can be used to create 3D reconstructions with the help of interpolation algorithms like marching cubes.[3]

Wikipedia, Image Segmentation

The technique is amazing, and once is attached to the drone camera, we can get something like this:

I used a Python library to make most of the work: PixelLib. It was created by an amazing set of colleagues, so please check the references and take a look at the project description.

PixelLib: is a library built for an easy implementation of Image Segmentation in real life problems. PixelLib is a flexible library that can be integrated into software solutions that require the application of Image Segmentation.

PixelLib

Once I have all the pieces together, I pulled a Pull Request with a single change to allow the use of OpenCV and webcam camera frames and I got a basic demo up and running.

Let’s review the code

• Line 147. That’s it, a single line which performs the instance segmentation, and also display the bounding boxes.

#### Sample Code

I’ll show a couple of live demos of this in my next Global AI Community, Drone AI demos. Check my next event sections!

Happy coding!

Greetings

El Bruno

# #Coding4Fun – How to control your #drone with 20 lines of code! (19/N)

Hi !

Today I face another challenge: I needed to overlay an image on top of another. Something like this.

Lucky for me, and as usual, OpenCV allow us to do this with a few lines of code. Let’s take a look.

• Line 8. Define a custom size for all the images: background image and camera feed frame.
• Lines 10-12. Load and resize background image.
• Line 21. Overlay the camera frame and the background image.

Sample Code

And from here, I’ll update some posts with the drone camera.

Happy coding!

Greetings

El Bruno

# #Coding4Fun – How to control your #drone with 20 lines of code! (18/N)

Hi !

Today I’ll step back a couple of posts, and add 2 simple lines to allow me to save a video file from the Drone camera. This is a request, and it’s makes a lot of sense to have recorded a file with the drone camera.

The video will later contains detected objects and more, so let’s go with the code. All the magic happens here:

• Lines 97-103. Open the drone camera stream, and also opens a video output stream to save the video file.
• Lines 123-124. Display the camera feed and add the camera frame into the output video file.
• Lines 136-139. Dispose objects, and close the video output file.

Happy coding!

Greetings

El Bruno

# #Coding4Fun – How to control your #drone with 20 lines of code! (17/N)

Hi !

Once we have the a custom vision trained model instance, we can use it to recognize objects from the drone camera feed. Read my previous posts for descriptions on these.

Another interesting scenario, is to save local files for every detected object. In the following code, I’ll save 2 different files for every detected object

• A camera frame image, with a frame around the detected object
• A plain text file with the JSON information

In the sample code below, the save process is in the lines 122-129. And, not in a fancy way, the files have the same name to correlate them.

So let’s go to the full code:

And if you want to see this up and running, it’s much better to see this in a video (start at ):

The complete source code can be found here https://github.com/elbruno/events/tree/master/2020%2004%2018%20Global%20AI%20On%20Tour%20MTY%20Drone%20AI%20Mex

Happy coding!

Greetings

El Bruno

# #Coding4Fun – How to control your #drone with 20 lines of code! (16/N)

Hi !

In my previous post, I shared an example where I analyzed the camera feed using a Image Recognition model created using Custom Vision. Today I’ll expand the sample, and show in real time the detected MVPs logos with a frame in the drone camera feed.

Let’s take a look at the demo working in the following image.

In the top of the image, we can see the app console log, with the information received for each analyzed frame. When an image is detected, we can see the tag, the probability and the bounding box coordinates.

A sample JSON return string start like this one:

```{
"created": "2020-04-08T17:22:02.179359",
"id": "",
"iteration": "",
"predictions": [
{
"boundingBox": {
"height": 0.1979116,
"left": 0.3235259,
"top": 0.05847502,
"width": 0.20438321
},
"probability": 0.89171505,
"tagId": 0,
"tagName": "MVP"
},
{
"boundingBox": {
"height": 0.2091526,
"left": 0.65271178,
"top": 0.0433814,
"width": 0.17669522
},
"probability": 0.70330358,
"tagId": 0,
"tagName": "MVP"
},
```

In order to position the frames in the correct location, I need to make some math using the current camera and image size and the returned bounding box values for, height, left, top and width. Lines 87-110.

```resize_factor = 100

height = int(bb['height'] * resize_factor)
left = int(bb['left'] * resize_factor)
top = int(bb['top'] * resize_factor)
width = int(bb['width'] * resize_factor)

# adjust to size
camera_Width,
height = int(height * camera_Heigth / 100)
left = int(left * camera_Width / 100)
top = int(top * camera_Heigth / 100)
width = int(width * camera_Width / 100)

# draw bounding boxes
start_point = (top, left)
end_point = (top + height, left + width)
color = (255, 0, 0)
thickness = 2
cv2.rectangle(img, start_point, end_point, color, thickness)
```

So let’s go to the full code:

And if you want to see this up and running, it’s much better to see this in a video (start at ):

The complete source code can be found here https://github.com/elbruno/events/tree/master/2020%2004%2018%20Global%20AI%20On%20Tour%20MTY%20Drone%20AI%20Mex

Happy coding!

Greetings

El Bruno

# #Coding4Fun – How to control your #drone with 20 lines of code! (15/N)

Hi !

Let’s use Custom Vision to analyze the images from our drone camera. In this scenario, I created a custom model to recognize MVP awards from my MVP wall. I know, that’s bragging, but I like it.

Disclaimer: There are plenty of documentation and tutorials about Custom Vision. I won’t go deep on the steps about how to create a model. See references.

For my next scenario, I would assume that

• You have created a model in Custom Vision
• You have published the Custom Vision model, and have a HTTP endpoint
• Or the model is exported as a docker image, and it’s running in a docker container. And we have a HTTP endpoint.

The code is similar to the one we used before. OpenCV to hookup the camera, commands to take off and land. Let me remark a couple of important lines in this code:

• There are a couple of new references, mostly used for the process of the JSON response from the Custom Vision model.
• Lines 146-155. Get the frame from the drone camera and save a local file. Apply a specific format to the file name, for demo purposes.
• Lines 157-163. Make a HTTP POST call to analyze the saved file. Convert the result to a JSON object (room for improvement here), and analyze the JSON response.
• Lines 70-85. Analyzed the JSON response from the Custom Vision model. Sort the results by probability and filter the results using a threshold (75). Return a string with the detected object.
• Lines 165-178. Calculate and display FPS and detected objects.

A sample JSON return string start like this one:

```{
"created": "2020-04-08T17:22:02.179359",
"id": "",
"iteration": "",
"predictions": [
{
"boundingBox": {
"height": 0.1979116,
"left": 0.3235259,
"top": 0.05847502,
"width": 0.20438321
},
"probability": 0.89171505,
"tagId": 0,
"tagName": "MVP"
},
{
"boundingBox": {
"height": 0.2091526,
"left": 0.65271178,
"top": 0.0433814,
"width": 0.17669522
},
"probability": 0.70330358,
"tagId": 0,
"tagName": "MVP"
},
```

So let’s go to the full code:

And if you want to see this up and running, it’s much better to see this in a video (start at ):

The complete source code can be found here https://github.com/elbruno/events/tree/master/2020%2004%2018%20Global%20AI%20On%20Tour%20MTY%20Drone%20AI%20Mex

Happy coding!

Greetings

El Bruno

# #Personal – Kids and STEM: my 2 cents to #IWD2020 #PressforProgress #CDC2020

Hi!

Some time ago I posted about the amazing experience I had at the Caribbean Developer Conference (@caribbeandevcon) in Punta Cana.

And if you are wondering “How this is related to International Women Day?”, let me share this video:

That’s Martina (my 10yo daughter) with Scott Hanselman talking about Computer Vision. She was part of the interview! And as I wrote before:

Scott was an amazing host, and we talked about how we can use Image Recognition systems in day to day scenarios, like garbage bin detection, smart parking lots and even to track our cat at home.

This photo is much more important than you think ! InternationalDayoftheGirl DayoftheGirl

#### International Women Day

Even if we are still a couple of days away of the official International Women Day, let me share my contributions and plan for the future.

My main 2 cents to the International Women Days is kind of selfish, however I strongly believe that support and encourage new generations to get close to STEM topics is a great way to support them.

That’s why, if you can, I strongly encourage to bring your kids (and/or your kid’s friends) to tech events. Share moments with them, introduce them to the speakers, attendees, helpers, etc on the event. They will learn new stuff; they will share some amazing ideas, and this is an amazing path for them to learn and know STEM!

It’s also important for us to acknowledge that men and women have different workplace experiences, so that’s why I hope that next generations won’t see a difference there.

And finally, please, let’s keep talking about this. I’m not an expert in this area, and the best I can do is to connect the dots and support some very specific scenarios. I’ll keep doing this !

And yes, Martina is also helping me now with my Drone and AI pet projects. You can see how much fun our pet Goku is having with the drone

Thanks Channel 9 and On.Net !

Note: And it seems that the other half of the family is not happy to be excluded of this post.

Happy coding!

Greetings

El Bruno

Happy coding!

Greetings

El Bruno

# #Event – Download all #MSIgnite sessions from the [Developer’s guide to AI] Learning Path

Hi!

A few weeks ago I wrote a post where I explained how to download slides and videos for all the Microsoft Ignite 2019 sessions.

It was very simple, just search for your preferred session at

Just navigate to https://myignite.techcommunity.microsoft.com/sessions

And, in the session you got the link to download the slides and the video to watch it later. I also explained how to use the PowerShell file available in each session to automate the download process.

And now, someone asked about my choice on the best AI sessions from Microsoft Ignite. I have my personal choice, however I strongly advice everyone to visit the leaning path for AI:

## Developer’s guide to AI

##### Making sense of your unstructured data with AI

Tailwind Traders has a lot of legacy data that they’d like their developers to leverage in their apps – from various sources, both structured and unstructured, and including images, forms, and several others. In this session, learn how the team used Azure Cognitive Search to make sense of this data in a short amount of time and with amazing success. We discuss tons of AI concepts, like the ingest-enrich-explore pattern, search skillsets, cognitive skills, natural language processing, computer vision, and beyond.VIEW MORE

##### Using pre-built AI to solve business challenges

As a data-driven company, Tailwind Traders understands the importance of using artificial intelligence to improve business processes and delight customers. Before investing in an AI team, their existing developers were able to demonstrate some quick wins using pre-built AI technologies. In this session, we show how you can use Azure Cognitive Services to extract insights from retail data and go into the neural networks behind computer vision. Learn how it works and how to augment the pre-built AI with your own images for custom image recognition applications.VIEW MORE

##### Start building machine learning models faster than you think

Tailwind Traders uses custom machine learning models to fix their inventory issues – without changing their software development life cycle! How? Azure Machine Learning Visual Interface. In this session, learn the data science process that Tailwind Traders’ uses and get an introduction to Azure Machine Learning Visual Interface. See how to find, import, and prepare data, select a machine learning algorithm, train and test the model, and deploy a complete model to an API. Get the tips, best practices, and resources you and your development team need to continue your machine learning journey, build your first model, and more.VIEW MORE

##### Taking models to the next level with Azure Machine Learning best practices

Tailwind Traders’ data science team uses natural language processing (NLP), and recently discovered how to fine tune and build a baseline models with Automated ML. In this session, learn what Automated ML is and why it’s so powerful, then dive into how to improve upon baseline models using examples from the NLP best practices repository. We highlight Azure Machine Learning key features and how you can apply them to your organization, including: low priority compute instances, distributed training with auto scale, hyperparameter optimization, collaboration, logging, and deployment.VIEW MORE

##### Machine learning operations: Applying DevOps to data science

Many companies have adopted DevOps practices to improve their software delivery, but these same techniques are rarely applied to machine learning projects. Collaboration between developers and data scientists can be limited and deploying models to production in a consistent, trustworthy way is often a pipe dream. In this session, learn how Tailwind Traders applied DevOps practices to their machine learning projects using Azure DevOps and Azure Machine Learning Service. We show automated training, scoring, and storage of versioned models, wrap the models in Docker containers, and deploy them to Azure Container Instances or Azure Kubernetes Service. We even collect continuous feedback on model behavior so we know when to retrain.VIEW MORE

## Download all the slides and videos

And, finally if you want all these sessions material, just

• Access “Get the bulk session resource download script” at the bottom of the page in one of the sessions.
• Open a PowerShell window to the directory in which the script is located.
• Run the following script
`.\Download-Resources.ps1 -directory . -sessionCodes "AIML10, AIML20, AIML30, AIML40, AIML50"  `

A couple of seconds later you will see how each one of the sessions will be starting to be downloaded in a separated folder with the session keyname.

Happy coding!

Greetings @ Etobicoke

El Bruno

References