#Azure – How to push / export and run a docker python flask web app on Azure

Buy Me A Coffee

Hi !

Here is an important reminder before starting:

Please do not use as a guide to deploy to production environments.

I was reading a lot about Azure Container Registry and other container scenarios. And most of the demos / tutorials are based on .Net Container Web Apps. Which is great, however there are no much information about the same process using Python Container Web Apps. So, I take this as an excuse to learn something new and here I am 😁, Let’s start.

Python Web App using Flask

I started with a simple Python Flask app. For this demo, I will use a post I write about flask and multi threading (see references for the full post).

# Bruno Capuano
# start a webserver with flask in a thread
# start a different thread +1 a shared var

from flask import Flask                                                         
import threading
import time

iCounter = 0
data = 'foo'
app = Flask(__name__)

def mainSum():
    # increment counter every second
    global iCounter
    while True:
        iCounter = iCounter + 1
        t = time.localtime()
        current_time = time.strftime("%H:%M:%S", t)    
        print(str(f"{current_time} - data {iCounter}"))
        time.sleep(1)

def startWebServer():
     app.run(host='0.0.0.0', port=8100)

@app.route("/getdata")
def main():
    global iCounter
    t = time.localtime()
    current_time = time.strftime("%H:%M:%S", t)    
    return str(f"{current_time} - data {iCounter}")

if __name__ == "__main__":
    stateThread = threading.Thread(target=mainSum)
    stateThread.daemon = True
    stateThread.start()

    webThread = threading.Thread(target=startWebServer)
    webThread.start()

This a resource consuming app, so be careful with this code!

Create a Docker container for the app

So next step was a to create a container for the app. This is my docker file. I added the commands to build and run the image at the bottom.

FROM python:3.7-slim

RUN pip install -U pip
RUN pip install --no-cache-dir flask~=1.1.2

COPY app /app

# Expose the port, default 8100
EXPOSE 8100

# Set the working directory
WORKDIR /app

# Run the flask server for the endpoints
CMD python -u app.py

# Commands to build and run the image
# docker build --pull --rm -f "Dockerfile" -t dockertoazure-webapp-service "."
# docker run -d -p 8100:8100 --privileged --name dockertoazure-webapp-service dockertoazure-webapp-service:latest

So, build and run my docker image and everything works fine !

Double check in Portainer, because I’m not a command line person

docker to azure build and run local ok in portainer

So, here my 1st milestone:

A local python web app using flask, in a docker container ready to be used elsewhere.

Push local Image to Azure

Our next step will be to have a Azure container registry, that will be the place that we will use as repository in Azure to host our images. In the references section the official definition is:

An Azure container registry is a private Docker registry in Azure where you can store and manage private Docker container images and related artifacts. In this quickstart, you create a container registry with the Azure portal. Then, use Docker commands to push a container image into the registry, and finally pull and run the image from your registry.

Quickstart: Create an Azure container registry using the Azure portal

I will assume Azure CLI is installed, so next step will be to login to azure and also login to your container.

# LOGIN TO AZURE
az login
az acr login --name <container name>.azurecr.io

Next step will require to have a tagged image for the Azure Container Registry.

# TAG LOCAL IMAGE
docker tag dockertoazure-webapp-service <container name>.azurecr.io/dockertoazure-webapp-service:v1

And, time to push the local image to the cloud !

# PUSH LOCAL IMAGE
docker push <container name>.azurecr.io/dockertoazure-webapp-service:v1

And we have 2 ways to check if the image was successfully pushed. Using the command line

# LIST CURRENT REPOSITORIES
az acr repository list -n <container name>.azurecr.io
docker to azure list azure container registry repositories

We can also use the Azure Portal to check the available repositories

docker to azure azure container repositories list

Running the Docker image in a WebApp

And here comes the tricky part. Once you select a repository, you can see the tags of the image and you can deploy a WebApp directly from there.

docker to azure docker image available options

However, if you haven’t created a WebApp using other steps, you won’t have this option enabled.

Note: My guess is that until you have an Linux usage plan created, the [Run instance] and [Deploy to web app] options will be disabled. In this scenario, you may want to skip next steps and go directly to option 2.

Let’s see both options. From the context menu, from the previous image, create a web app is just a single step

docker to azure create using the context menu

A couple of seconds later we have our App up and running

The 2nd option to create your Web App is creating a new [Web App for Containers]

And here we found a couple of interesting options, like create a web site based on a docker container:

And in the next step, we can choose the image that we uploaded in previous steps

new web app docker image details

And, again, a couple of seconds later, we also have this new App up and running

Conclusion

So, nothing new here. I haven’t found an article or post, that describes this, so I put all the pieces together and dump this lines. Again an important warning

There are much better ways to do this ! Think on a DevOps approach.

I promise to write a follow up on this one, with the step-by-step details on this using GitHub actions. Which is the right way.

Happy coding!

Greetings

El Bruno

Resources

#VS2019 – ML.NET Model Builder training using GPU, CPU and … Azure !

Buy Me A Coffee

Hi !

I my previous posts on Model Builder, I show the differences between using the CPU and GPU to train a model. There is a 3rd option, which involves an Azure ML Experiment, and performs the training on the cloud.

It took me some time, to setup this environment, mostly because I tried to use an existing Azure Compute Instance that I already have, and Model Builder needs a Compute Cluster.

Here is also important to remark, that you need to create a Dedicated GPU based Compute Cluster. There are some expenses / costs associated to these resources, so make your numbers before you start.

And, here we go, now we can move forward with the Model Builder Assistant.

I made some tests using a small image data set, and it was awesome. Training an 24 images dataset took between 8 and 9 minutes. and the results were very good. A good insight here, is the chance to get more details directly in the Azure Machine Learning portal.

We can go deep in each experiment, and take a look at some metrics like F1 Score, Precision, Recall and more.

Each Model Builder Image Classification project, will trigger several Azure ML Experiments.

  • Automated ML
  • HyperDrive
  • Preparation
  • Script

The Script experiment is the one we can open to get access to some detailed logs, and also to the ONNX model.

So, I decided to go big and test this using the set of images from a Kaggle challenge [State Farm Distracted Driver Detection] (see references). This is a 1GB image set, 22424images, with ten categories.

The 1st step is to upload the 22424 images to an Azure resource, this took some time.

And then, start tracking the progress in the Azure Machine Learning portal.

And after some time, the process triggered a timeout.

The details on the 4 experiments suggests that some limit was exceeded. I’m not sure if from the IDE or on the Azure side.

However, the experiment in charge to train the model [Run 12], got some sucess models. Acuraccy, F1 and precision were getting better

Reading some log I can see how the error was triggered on Epoch 8. I need to spend more time here to figure out what’s happened !

Note: I already reported the issue to the GitHub Repo.

As final thought. Using Azure as the training environment in Model Builder is an amazing option. A big DataSet maybe a problem, or maybe my quota is the problem. Anyways, in smaller DataSets it worked great. I’ll keep an eye in this issue and update the blog with some news.

Happy coding!

Greetings

El Bruno

References

#Event – ML.Net in Spanish and back to XR with @ivanatilca to share some #Hololens2 and #AzureKinect dev experiences @globxrcommunity

Buy Me A Coffee

Hi !

Just a self reminder post of a couple of events in my calendar.

Introduction to Machine Learning.Net

31 July 2020

Machine Learning.Net 1.5.1 is already here, so that’s the perfect excuse to chat with my colleagues from the MUG Argentina about this. This session is designed for developers who wants to start in the Machine Learning world. Machine Learning.Net is an amazing framework for this!

Important: the session will be in Spanish.

More information https://mug-it.org.ar/event.aspx?event=530

XR Session – Lessons Learned creating a multiplatform AI project for Azure Kinect and Hololens 2

6 August 2020

It all started with a 10000 kms conversation between 2 friends about how easy is to port Mixed Reality projects between platforms. So, we choose Azure Kinect and Hololens 2 as the platforms to test this out. To make this more challenging, we also decided to place custom holograms in those different platforms based on some cool Image Recognition scenarios (custom Artificial Intelligence rocks!) During this session we will review how to use MRTK, Azure Kinect SDK, Computer Vision, and other cool technologies to make this happen. And, of course, be aware that this session is full of code, hardware and demos, do not expect a lot of slides. Let’s code / build this.
Speakers: Ivana Tilca & Bruno Capuano, Both Microsoft MVP

More information https://www.meetup.com/GlobalXRTalks/events/271140960/

Happy coding!

Greetings

El Bruno

#Event – Let’s hack a drone, hack the camera and use AI! virtual with Microsoft Reactor @MSFTReactor

Buy Me A Coffee

Hi !

Today I’ll share a new version of my drone and Artificial Intelligence session, hosted by Microsoft Reactor Toronto.

The registration link and details are available here

Happy coding!

Greetings

El Bruno

#Event – Resources used during the #GlobalAI Tour, Buenos Aires, May 30. Let’s code a drone to follow faces! Using AI, Python, containers and more.

Buy Me A Coffee
Globa AI Community on Virtual Tour Logo

Hi !

The event is complete, and now is time to share the resources I used during the session

Slides

Code

https://github.com/elbruno/events/tree/master/2020%2005%2030%20Global%20AI%20Tour%20BsAs%20Drone%20AI

Resources

Happy coding!

Greetings

El Bruno

#Event – #GlobalAI On Tour, Argentina. Vamos a programar a un dron en Spanish !

Buy Me A Coffee
Globa AI Community on Virtual Tour Logo

Buenas !

Cambios de ultimo momento. Mañana estaré como uno de los ponentes virtuales en el Global AI on Tour para Argentina. Estaré hablando de drones, y más que hablando más bien programando un poco un drone para pasarlo bien. Y utilizando un poco de AI para hacer esto más divertido.

Para subir el nivel, en la agenda pueden ver que los demás speakers van a tocar temas mucho más serios e interesantes.

Global AI On Tour Argentina

Happy coding!

Greetings

El Bruno

#Coding4Fun – How to control your #drone with 20 lines of code! (17/N)

Buy Me A Coffee

Hi !

Once we have the a custom vision trained model instance, we can use it to recognize objects from the drone camera feed. Read my previous posts for descriptions on these.

Another interesting scenario, is to save local files for every detected object. In the following code, I’ll save 2 different files for every detected object

  • A camera frame image, with a frame around the detected object
  • A plain text file with the JSON information

In the sample code below, the save process is in the lines 122-129. And, not in a fancy way, the files have the same name to correlate them.

drone recognized files

So let’s go to the full code:

# Bruno Capuano
# open camera with openCV
# analyze camera frame with local docker custom vision project
# draw bounding boxes for each reconized object
import socket
import time
import threading
import cv2
import urllib
import json
import requests
import os
from flask import Flask, request, jsonify
def receiveData():
global response
while True:
try:
response, _ = clientSocket.recvfrom(1024)
except:
break
def readStates():
global battery
while True:
try:
response_state, _ = stateSocket.recvfrom(256)
if response_state != 'ok':
response_state = response_state.decode('ASCII')
list = response_state.replace(';', ':').split(':')
battery = int(list[21])
pitch = int(list[1])
except:
break
def sendCommand(command):
global response
timestamp = int(time.time() * 1000)
clientSocket.sendto(command.encode('utf-8'), address)
while response is None:
if (time.time() * 1000) timestamp > 5 * 1000:
return False
return response
def sendReadCommand(command):
response = sendCommand(command)
try:
response = str(response)
except:
pass
return response
def sendControlCommand(command):
response = None
for i in range(0, 5):
response = sendCommand(command)
if response == 'OK' or response == 'ok':
return True
return False
# ———————————————–
# Local calls
# ———————————————–
probabilityThreshold = 75
def displayPredictions(jsonPrediction, frame, frameImageFileName):
global camera_Width, camera_Heigth
jsonObj = json.loads(jsonPrediction)
preds = jsonObj['predictions']
sorted_preds = sorted(preds, key=lambda x: x['probability'], reverse=True)
strSortedPreds = ""
resultFound = False
if (sorted_preds):
# open img to save results
img = cv2.imread(frameImageFileName)
detected = False
for pred in sorted_preds:
# tag name and prob * 100
tagName = str(pred['tagName'])
probability = pred['probability'] * 100
# apply threshold
if (probability >= probabilityThreshold):
detected = True
bb = pred['boundingBox']
resize_factor = 100
height = int(bb['height'] * resize_factor)
left = int(bb['left'] * resize_factor)
top = int(bb['top'] * resize_factor)
width = int(bb['width'] * resize_factor)
print(f'height = {height} – left {left} – top {top} – width {width}')
# adjust to size
camera_Width,
height = int(height * camera_Heigth / 100)
left = int(left * camera_Width / 100)
top = int(top * camera_Heigth / 100)
width = int(width * camera_Width / 100)
print(f'Adjusted height = {height} – left {left} – top {top} – width {width}')
# draw bounding boxes
start_point = (top, left)
end_point = (top + height, left + width)
print(f'MVP – {probability}')
print(f'start point: {start_point} – end point: {end_point}')
color = (255, 0, 0)
thickness = 2
cv2.rectangle(img, start_point, end_point, color, thickness)
print(jsonPrediction)
# save the detected image
cv2.rectangle(img, start_point, end_point, color, thickness)
if (detected == True):
detImageFileName = frameImageFileName.replace('tmp', 'det')
cv2.imwrite(detImageFileName, img)
detJsonFileName = detImageFileName.replace('png', 'json')
save_text = open(detJsonFileName, 'w')
save_text.write(jsonStr)
save_text.close()
return strSortedPreds
# instantiate flask app and push a context
app = Flask(__name__)
# ———————————————–
# Main program
# ———————————————–
# connection info
UDP_IP = '192.168.10.1'
UDP_PORT = 8889
last_received_command = time.time()
STATE_UDP_PORT = 8890
address = (UDP_IP, UDP_PORT)
response = None
response_state = None
clientSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
clientSocket.bind(('', UDP_PORT))
stateSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
stateSocket.bind(('', STATE_UDP_PORT))
# start threads
recThread = threading.Thread(target=receiveData)
recThread.daemon = True
recThread.start()
stateThread = threading.Thread(target=readStates)
stateThread.daemon = True
stateThread.start()
# connect to drone
response = sendControlCommand("command")
print(f'command response: {response}')
response = sendControlCommand("streamon")
print(f'streamon response: {response}')
# drone information
battery = 0
pitch = 0
# open UDP
print(f'opening UDP video feed, wait 2 seconds ')
videoUDP = 'udp://192.168.10.1:11111'
cap = cv2.VideoCapture(videoUDP)
time.sleep(2)
camera_Width = 640
camera_Heigth = 480
# open
i = 0
while True:
i = i + 1
imgNumber = str(i).zfill(5)
start_time = time.time()
sendReadCommand('battery?')
print(f'battery: {battery} % – pitch: {pitch} – i: {imgNumber}')
try:
ret, frame = cap.read()
img = cv2.resize(frame, (camera_Width, camera_Heigth))
# save image to disk and open it
frameImageFileName = str(f'tmp\image{imgNumber}.png')
cv2.imwrite(frameImageFileName, img)
with open(frameImageFileName, 'rb') as f:
img_data = f.read()
# analyze file in local container
api_url = "http://127.0.0.1:8070/image&quot;
r = requests.post(api_url, data=img_data)
with app.app_context():
jsonResults = jsonify(r.json())
jsonStr = jsonResults.get_data(as_text=True)
displayPredictions(jsonStr, frame, frameImageFileName)
fpsInfo = ""
if (time.time() start_time ) > 0:
fpsInfo = "FPS: " + str(1.0 / (time.time() start_time)) # FPS = 1 / time to process loop
font = cv2.FONT_HERSHEY_DUPLEX
cv2.putText(img, fpsInfo, (10, 20), font, 0.4, (255, 255, 255), 1)
cv2.imshow('@elbruno – DJI Tello Camera', img)
except Exception as e:
print(f'exc: {e}')
pass
if cv2.waitKey(1) & 0xFF == ord('q'):
break
response = sendControlCommand("streamoff")
print(f'streamon response: {response}')

And if you want to see this up and running, it’s much better to see this in a video (start at ):

The complete source code can be found here https://github.com/elbruno/events/tree/master/2020%2004%2018%20Global%20AI%20On%20Tour%20MTY%20Drone%20AI%20Mex

Happy coding!

Greetings

El Bruno

References

#Coding4Fun – How to control your #drone with 20 lines of code! (16/N)

Buy Me A Coffee

Hi !

In my previous post, I shared an example where I analyzed the camera feed using a Image Recognition model created using Custom Vision. Today I’ll expand the sample, and show in real time the detected MVPs logos with a frame in the drone camera feed.

Let’s take a look at the demo working in the following image.

drone camera image analysis using custom vision and drawing frames for detected objects

In the top of the image, we can see the app console log, with the information received for each analyzed frame. When an image is detected, we can see the tag, the probability and the bounding box coordinates.

A sample JSON return string start like this one:

{
  "created": "2020-04-08T17:22:02.179359",
  "id": "",
  "iteration": "",
  "predictions": [
    {
      "boundingBox": {
        "height": 0.1979116,
        "left": 0.3235259,
        "top": 0.05847502,
        "width": 0.20438321
      },
      "probability": 0.89171505,
      "tagId": 0,
      "tagName": "MVP"
    },
    {
      "boundingBox": {
        "height": 0.2091526,
        "left": 0.65271178,
        "top": 0.0433814,
        "width": 0.17669522
      },
      "probability": 0.70330358,
      "tagId": 0,
      "tagName": "MVP"
    },

In order to position the frames in the correct location, I need to make some math using the current camera and image size and the returned bounding box values for, height, left, top and width. Lines 87-110.

resize_factor = 100

height = int(bb['height'] * resize_factor)
left = int(bb['left'] * resize_factor)
top = int(bb['top'] * resize_factor)
width = int(bb['width'] * resize_factor)

# adjust to size
camera_Width, 
height = int(height * camera_Heigth / 100)
left = int(left * camera_Width / 100)
top = int(top * camera_Heigth / 100)
width = int(width * camera_Width / 100)

# draw bounding boxes
start_point = (top, left)                 
end_point = (top + height, left + width) 
color = (255, 0, 0) 
thickness = 2                
cv2.rectangle(img, start_point, end_point, color, thickness)            

So let’s go to the full code:

# Bruno Capuano
# open camera with openCV
# analyze camera frame with local docker custom vision project
# draw bounding boxes for each reconized object
import socket
import time
import threading
import cv2
import urllib
import json
import requests
import os
from flask import Flask, request, jsonify
def receiveData():
global response
while True:
try:
response, _ = clientSocket.recvfrom(1024)
except:
break
def readStates():
global battery
while True:
try:
response_state, _ = stateSocket.recvfrom(256)
if response_state != 'ok':
response_state = response_state.decode('ASCII')
list = response_state.replace(';', ':').split(':')
battery = int(list[21])
pitch = int(list[1])
except:
break
def sendCommand(command):
global response
timestamp = int(time.time() * 1000)
clientSocket.sendto(command.encode('utf-8'), address)
while response is None:
if (time.time() * 1000) timestamp > 5 * 1000:
return False
return response
def sendReadCommand(command):
response = sendCommand(command)
try:
response = str(response)
except:
pass
return response
def sendControlCommand(command):
response = None
for i in range(0, 5):
response = sendCommand(command)
if response == 'OK' or response == 'ok':
return True
return False
# ———————————————–
# Local calls
# ———————————————–
probabilityThreshold = 75
def displayPredictions(jsonPrediction, frame):
global camera_Width, camera_Heigth
jsonObj = json.loads(jsonPrediction)
preds = jsonObj['predictions']
sorted_preds = sorted(preds, key=lambda x: x['probability'], reverse=True)
strSortedPreds = ""
resultFound = False
if (sorted_preds):
for pred in sorted_preds:
# tag name and prob * 100
tagName = str(pred['tagName'])
probability = pred['probability'] * 100
# apply threshold
if (probability >= probabilityThreshold):
bb = pred['boundingBox']
resize_factor = 100
height = int(bb['height'] * resize_factor)
left = int(bb['left'] * resize_factor)
top = int(bb['top'] * resize_factor)
width = int(bb['width'] * resize_factor)
#print(f'height = {height} – left {left} – top {top} – width {width}')
# adjust to size
camera_Width,
height = int(height * camera_Heigth / 100)
left = int(left * camera_Width / 100)
top = int(top * camera_Heigth / 100)
width = int(width * camera_Width / 100)
#print(f'Adjusted height = {height} – left {left} – top {top} – width {width}')
# draw bounding boxes
start_point = (top, left)
end_point = (top + height, left + width)
color = (255, 0, 0)
thickness = 2
cv2.rectangle(img, start_point, end_point, color, thickness)
print(f'MVP – {probability}')
print(f'start point: {start_point} – end point: {end_point}')
print(jsonPrediction)
return strSortedPreds
# instantiate flask app and push a context
app = Flask(__name__)
# ———————————————–
# Main program
# ———————————————–
# connection info
UDP_IP = '192.168.10.1'
UDP_PORT = 8889
last_received_command = time.time()
STATE_UDP_PORT = 8890
address = (UDP_IP, UDP_PORT)
response = None
response_state = None
clientSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
clientSocket.bind(('', UDP_PORT))
stateSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
stateSocket.bind(('', STATE_UDP_PORT))
# start threads
recThread = threading.Thread(target=receiveData)
recThread.daemon = True
recThread.start()
stateThread = threading.Thread(target=readStates)
stateThread.daemon = True
stateThread.start()
# connect to drone
response = sendControlCommand("command")
print(f'command response: {response}')
response = sendControlCommand("streamon")
print(f'streamon response: {response}')
# drone information
battery = 0
pitch = 0
# open UDP
print(f'opening UDP video feed, wait 2 seconds ')
videoUDP = 'udp://192.168.10.1:11111'
cap = cv2.VideoCapture(videoUDP)
time.sleep(2)
camera_Width = 640
camera_Heigth = 480
# open
i = 0
while True:
i = i + 1
imgNumber = str(i).zfill(5)
start_time = time.time()
sendReadCommand('battery?')
print(f'battery: {battery} % – pitch: {pitch} – i: {imgNumber}')
try:
ret, frame = cap.read()
img = cv2.resize(frame, (camera_Width, camera_Heigth))
# save image to disk and open it
frameImageFileName = str(f'tmp\image{imgNumber}.png')
cv2.imwrite(frameImageFileName, img)
with open(frameImageFileName, 'rb') as f:
img_data = f.read()
# analyze file in local container
api_url = "http://127.0.0.1:8070/image&quot;
r = requests.post(api_url, data=img_data)
with app.app_context():
jsonResults = jsonify(r.json())
jsonStr = jsonResults.get_data(as_text=True)
displayPredictions(jsonStr, frame)
fpsInfo = ""
if (time.time() start_time ) > 0:
fpsInfo = "FPS: " + str(1.0 / (time.time() start_time)) # FPS = 1 / time to process loop
font = cv2.FONT_HERSHEY_DUPLEX
cv2.putText(img, fpsInfo, (10, 20), font, 0.4, (255, 255, 255), 1)
cv2.imshow('@elbruno – DJI Tello Camera', img)
except Exception as e:
print(f'exc: {e}')
pass
if cv2.waitKey(1) & 0xFF == ord('q'):
break
response = sendControlCommand("streamoff")
print(f'streamon response: {response}')

And if you want to see this up and running, it’s much better to see this in a video (start at ):

The complete source code can be found here https://github.com/elbruno/events/tree/master/2020%2004%2018%20Global%20AI%20On%20Tour%20MTY%20Drone%20AI%20Mex

Happy coding!

Greetings

El Bruno

References

#Coding4Fun – How to control your #drone with 20 lines of code! (15/N)

Buy Me A Coffee

Hi !

Let’s use Custom Vision to analyze the images from our drone camera. In this scenario, I created a custom model to recognize MVP awards from my MVP wall. I know, that’s bragging, but I like it.

Disclaimer: There are plenty of documentation and tutorials about Custom Vision. I won’t go deep on the steps about how to create a model. See references.

For my next scenario, I would assume that

  • You have created a model in Custom Vision
  • You have published the Custom Vision model, and have a HTTP endpoint
  • Or the model is exported as a docker image, and it’s running in a docker container. And we have a HTTP endpoint.

The code is similar to the one we used before. OpenCV to hookup the camera, commands to take off and land. Let me remark a couple of important lines in this code:

  • There are a couple of new references, mostly used for the process of the JSON response from the Custom Vision model.
  • Lines 146-155. Get the frame from the drone camera and save a local file. Apply a specific format to the file name, for demo purposes.
  • Lines 157-163. Make a HTTP POST call to analyze the saved file. Convert the result to a JSON object (room for improvement here), and analyze the JSON response.
  • Lines 70-85. Analyzed the JSON response from the Custom Vision model. Sort the results by probability and filter the results using a threshold (75). Return a string with the detected object.
  • Lines 165-178. Calculate and display FPS and detected objects.

A sample JSON return string start like this one:

{
  "created": "2020-04-08T17:22:02.179359",
  "id": "",
  "iteration": "",
  "predictions": [
    {
      "boundingBox": {
        "height": 0.1979116,
        "left": 0.3235259,
        "top": 0.05847502,
        "width": 0.20438321
      },
      "probability": 0.89171505,
      "tagId": 0,
      "tagName": "MVP"
    },
    {
      "boundingBox": {
        "height": 0.2091526,
        "left": 0.65271178,
        "top": 0.0433814,
        "width": 0.17669522
      },
      "probability": 0.70330358,
      "tagId": 0,
      "tagName": "MVP"
    },

So let’s go to the full code:

# Bruno Capuano
# open camera with openCV
# analyze camera frame with local docker custom vision project
# display recognized objects in output log
import socket
import time
import threading
import cv2
import urllib
import json
import requests
import os
from flask import Flask, request, jsonify
def receiveData():
global response
while True:
try:
response, _ = clientSocket.recvfrom(1024)
except:
break
def readStates():
global battery
while True:
try:
response_state, _ = stateSocket.recvfrom(256)
if response_state != 'ok':
response_state = response_state.decode('ASCII')
list = response_state.replace(';', ':').split(':')
battery = int(list[21])
except:
break
def sendCommand(command):
global response
timestamp = int(time.time() * 1000)
clientSocket.sendto(command.encode('utf-8'), address)
while response is None:
if (time.time() * 1000) timestamp > 5 * 1000:
return False
return response
def sendReadCommand(command):
response = sendCommand(command)
try:
response = str(response)
except:
pass
return response
def sendControlCommand(command):
response = None
for i in range(0, 5):
response = sendCommand(command)
if response == 'OK' or response == 'ok':
return True
return False
# ———————————————–
# Local calls
# ———————————————–
probabilityThreshold = 50
def getPredictionsSorted(jsonPrediction):
jsonObj = json.loads(jsonPrediction)
preds = jsonObj['predictions']
sorted_preds = sorted(preds, key=lambda x: x['probability'], reverse=True)
strSortedPreds = ""
if (sorted_preds):
for pred in sorted_preds:
# tag name and prob * 100
tagName = str(pred['tagName'])
probability = pred['probability'] * 100
# apply threshold
if (probability >= probabilityThreshold):
strSortedPreds = strSortedPreds + tagName + ": " + str(probability) + "\n"
return strSortedPreds
# instantiate flask app and push a context
app = Flask(__name__)
# ———————————————–
# Main program
# ———————————————–
# connection info
UDP_IP = '192.168.10.1'
UDP_PORT = 8889
last_received_command = time.time()
STATE_UDP_PORT = 8890
address = (UDP_IP, UDP_PORT)
response = None
response_state = None
clientSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
clientSocket.bind(('', UDP_PORT))
stateSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
stateSocket.bind(('', STATE_UDP_PORT))
# start threads
recThread = threading.Thread(target=receiveData)
recThread.daemon = True
recThread.start()
stateThread = threading.Thread(target=readStates)
stateThread.daemon = True
stateThread.start()
# connect to drone
response = sendControlCommand("command")
print(f'command response: {response}')
response = sendControlCommand("streamon")
print(f'streamon response: {response}')
# drone information
battery = 0
# open UDP
print(f'opening UDP video feed, wait 2 seconds ')
videoUDP = 'udp://192.168.10.1:11111'
cap = cv2.VideoCapture(videoUDP)
time.sleep(2)
# open
i = 0
while True:
i = i + 1
start_time = time.time()
sendReadCommand('battery?')
print(f'battery: {battery} % – i: {i}')
try:
ret, frame = cap.read()
img = cv2.resize(frame, (640, 480))
# save image to disk and open it
imgNumber = str(i).zfill(5)
frameImageFileName = str(f'image{imgNumber}.png')
if os.path.exists(frameImageFileName):
os.remove(frameImageFileName)
cv2.imwrite(frameImageFileName, img)
with open(frameImageFileName, 'rb') as f:
img_data = f.read()
# analyze file in local container
api_url = "http://127.0.0.1:8070/image&quot;
r = requests.post(api_url, data=img_data)
with app.app_context():
jsonResults = jsonify(r.json())
jsonStr = jsonResults.get_data(as_text=True)
predSorted = getPredictionsSorted(jsonStr)
fpsInfo = ""
if (time.time() start_time ) > 0:
fpsInfo = "FPS: " + str(1.0 / (time.time() start_time)) + "\n——————-\n" # FPS = 1 / time to process loop
# display FPS and Predictions, split text into lines, thanks OpenCV putText()
frameInfo = fpsInfo + predSorted
print(frameInfo)
j = 0
for j, line in enumerate(frameInfo.split('\n')):
print(f'{j}{line}')
cv2.putText(img, line, (10, 10 * j), cv2.FONT_HERSHEY_SIMPLEX, 0.4, (255, 255, 255), 1)
cv2.imshow('@elbruno – DJI Tello Camera', img)
except Exception as e:
print(f'exc: {e}')
pass
if cv2.waitKey(1) & 0xFF == ord('q'):
break
response = sendControlCommand("streamoff")
print(f'streamon response: {response}')

And if you want to see this up and running, it’s much better to see this in a video (start at ):

The complete source code can be found here https://github.com/elbruno/events/tree/master/2020%2004%2018%20Global%20AI%20On%20Tour%20MTY%20Drone%20AI%20Mex

Happy coding!

Greetings

El Bruno

References

#Event – Recorded session (spanish) and resources for the #GlobalAICommunity Virtual Tour, Monterrey. Vamos a programar a un dron para que siga rostros!

Buy Me A Coffee
Globa AI Community on Virtual Tour Logo

Hi !

The event is complete, and now is time to share the recorded session and the resources I used during the session

Recorded Session (Spanish)

Slides

Code

https://github.com/elbruno/events/tree/master/2020%2004%2018%20Global%20AI%20On%20Tour%20MTY%20Drone%20AI%20Mex

Resources

Happy coding!

Greetings

El Bruno