Python – El Bruno

#Coding4Fun – How to control your #drone with 20 lines of code! (12/N)

Coding4Fun Drone posts

Hi!

Today code objective is very simple, based on a request I received from internet:

The drone is flying very happy, but if the camera detects a face, the drone will flip out !

Let’s take a look at the program working:

This one is very similar to the previous one. I also realized that I may need a better camera to record the live action side by side with the drone footage, but I think you get the idea. The command to make the drone flip is “flip x”, where “x” is the direction. In example:

"flip l" # flip left
"flip r" # flip right
"flip f" # flip forward
"flip b" # flip back

Here is the code:

As I promised last time, in next posts, I’ll analyze more in details how this works, and a couple of improvements that I can implement.

Happy coding!

Greetings

El Bruno

References

My Posts

#Coding4Fun – How to control your #drone with 20 lines of code! (11/N)

Coding4Fun Drone posts

Hi!

Today code objective is very simple:

The drone is flying very happy, but if the camera detects a banana, the drone must land !

Let’s take a look at the program working:

drone flying and when detect a banana lands

And a couple of notes regarding the app

Still use Haar Cascades for object detection. I found an article with a Xml file to detect bananas, so I’m working with this one (see references).
Using Haar Cascades is not the best technique for object detection. During the testing process, I found a lot of false positives. Mostly with small portions of the frame who were detected as bananas. One solution, was to limit the size of the detected objects using OpenCV (I’ll write more about this in the future)
As you can see in the animation, when the drone is a few meters away, the video feed becomes messy. And because the object detection is performed locally, it takes some time to detect the banana.
I also implemented some code to take off the drone when the user press the key ‘T’, and land the drone when the user press the key ‘L’
The code is starting to become a mess, so a refactoring is needed

Here is the code

In next posts, I’ll analyze more in details how this works, and a couple of improvements that I can implement.

Happy coding!

Greetings

El Bruno

References

My Posts

#VSCode – Let’s do some #FaceRecognition with 20 lines in #Python (7/N)

Face Recognition and Detecting series in Python

Hi!

I’m writing a series of posts about how to control a drone with Python and 20 lines of code, and once I reach to the point to read the camera feed, I’ve added a face detection sample. However this time I didn’t use the face_recognition python package I’ve used in this series, I performed the face detection using OpenCV and Haar Cascades. So, let’s explain a little what’s this.

Let me start quoting an amazing article “Face Detection using Haar Cascades” (see references)

Object Detection using Haar feature-based cascade classifiers is an effective object detection method proposed by Paul Viola and Michael Jones in their paper, “Rapid Object Detection using a Boosted Cascade of Simple Features” in 2001. It is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images.
OpenCV comes with a trainer as well as detector. If you want to train your own classifier for any object like car, planes etc. you can use OpenCV to create one. Its full details are given here: Cascade Classifier Training.

And here we come to the cool part, OpenCV already contains many pre-trained classifiers for face, eyes, smile etc. Those XML files are stored in opencv/data/haarcascades/ folder (see references).

Let’s take a look at a really [20 lines] sample code for face detection:

Line 6, we use OpenCV to load the haar cascade classifier to detect faces
Lines 9-20, main app
Lines 10-12, open a frame from the camera, transform the frame to a gray color scaled image and use the face cascade detector to find faces
Lines 14-15, iterate thought detected faces and draw a frame
Lines 17-20, display the webcam image with the detected faces, and stop the app when ESC key is pressed

And a live sample using a drone camera instead of an USB Camera

Bonus. Viola Jones Face Detection and tracking explained video

This is a long video, however is an amazing entry point to understand how the Viola Jones algorithm works.

Happy coding!

Greetings

El Bruno

Resources

Visual Studio Code, https://code.visualstudio.com/
OpenCV, How to set camera resolution (webcam) with opencv?
OpenCV, Face Detection using Haar Cascades
OpenCV, GitHub Haar Cascades

#Coding4Fun – How to control your #drone with 20 lines of code! (10/N)

Coding4Fun Drone posts

Hi!

Back to some drone posts! I was kind of busy during the last weeks and now I can get back to write about the drone.

OK, in the last posts I described how to connect and work with the drone camera feed using OpenCV. Now with 2 extra lines of code we can also detect faces. Let’s take a look at the final sample.

drone camera and camera view performing face detection

In the previous image we can see 2 camera feeds. My computer webcam, where you can see how I hold the drone with the drone camera pointing to my face. And the drone camera feed, presented using OpenCV and drawing a frame over each detected face.

Let’s share some code insights:

As usual, I resize the camera feed to 320 x 240
The average processing time is between 40 and 70 FPS
I use a haar cascade classifier to detect the faces in each frame

Note: I need to write about Haar Cascades as part of my face detection post series.

In my next posts, I’ll add some drone specific behaviors for each face detected.

Happy coding!

Greetings

El Bruno

References

My Posts

Face Recognition and Detecting series in Python

#Coding4Fun – How to control your #drone with 20 lines of code! (9/N)

Coding4Fun Drone posts

Hi!

Let’s take some Frames Per Second measurements on the UDP and OpenCV connection. It seems that working with simple movements, the values moves between 30 and 60 FPS.

showing FPS information with the drone camera

Just added a couple of lines in the main While, to calculate the FPS.

# open
i = 0
while True:
    i = i + 1
    start_time = time.time()

    sendReadCommand('battery?')
    print(f'battery: {battery} % - i: {i}')

    try:
        ret, frame = cap.read()
        img = cv2.resize(frame, (640, 480))

        if (time.time() - start_time ) > 0:
            fpsInfo = "FPS: " + str(1.0 / (time.time() - start_time)) # FPS = 1 / time to process loop
            font = cv2.FONT_HERSHEY_DUPLEX
            cv2.putText(img, fpsInfo, (10, 20), font, 0.4, (255, 255, 255), 1)

        cv2.imshow('@elbruno - DJI Tello Camera', img)
    except Exception as e:
        print(f'exc: {e}')
        pass

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

As a final note, just need to mention that I make some tests using different camera resolutions and the FPS averages are similar. I tested with 640 * 480 pixels and 1024*768 pixels.

Next posts, let’s do some Face Detection and rock some AI with the drone!

Happy coding!

Greetings

El Bruno

References

My Posts

#Coding4Fun – How to control your #drone with 20 lines of code! (8/N)

Coding4Fun Drone posts

Hi!

Now that I started to understand how UDP works, I also did my research to find which are the best options to access an UDP video feed. Lucky for me, there are plenty of resources about doing this task using my old friend OpenCV.

Most of the OpenCV documentation is written in C++. However at the end, it all goes down to these basic lines of code

# open UDP
videoUDP = 'udp://192.168.10.1:11111'
cap = cv2.VideoCapture(videoUDP)

# read a frame from the feed
ret, frame = cap.read()
img = cv2.resize(frame, (320, 240))

# diplay the frame in openCV video window
cv2.imshow('Video', img)

Note: At the references section below I shared some of my posts with my experiences on how to install OpenCV in Windows 10.

Let’s go back to our sample Python App. Using the previous sample that display the battery level, I changed this code to be alive all the time and displaying the video feed in a 320*240 window.

The following video display how fast this works:

And of course the complete code with this notes:

Line 96-100. Open video feed and wait 2 seconds
Line 104-118. Main App.
- Get and display battery level
- Get UDP video frame
- Resize frame to 320×240
- Display frame
- When Q key is pressed, exit app
Line 121. Close video stream

Happy coding!

Greetings

El Bruno

References

My Posts

#Coding4Fun – How to control your #drone with 20 lines of code! (7/N)

Coding4Fun Drone posts

Hi!

No code today. Mostly because I spend a decent amount of time trying to understand how the DJI Trello camera feed works.

In order to access the camera feed remotely we need to perform 2 steps. First we need to send the command “command” to the drone, and then the command “streamon” to enable the video stream. Of course, there is also a command to stop the stream “streamoff“.

In the following sample, I enable the camera feed, then keep the camera feed live for 90 seconds and then disable the camera feed.

OK, once I got this, I needed to spend some time trying to figure out how to get this feed. Based on the SDK details, I realize that the url to access the video feed is:

udp://192.168.10.1:11111

First I make a try and try to access the UDP feed using VLC, however it didn’t work. So I did a little research, and found that I can use FFmpeg to do this. In case you don’t know about FFmpeg.

FFmpeg is the leading multimedia framework to decode, encode, transcode, mux, demux, stream, filter and play. All builds require at least Windows 7 or Mac OS X 10.10. Nightly git builds are licensed as GPL 3.0, and release build are licensed as GPL 3.0 and LGPL 3.0. LGPL 3.0 release builds can be found using the “All Builds” links.
FFmpeg Builds (see references)

I downloaded the latest FFMpeg build, and run the following command locally:

.\ffplay.exe -i udp://192.168.10.1:11111

And after a couple of seconds, I got my drone video feed displayed locally. It has a huge delay from the real action to the camera feed. So, there is some improvement opportunities here.

drone camera video feed using ffmpeg and udp

The video is at 3X speed, and I avoid the initial comments and setup. The main idea was to access the video feed and it’s done. Now, in next posts, I’ll try to use OpenCV to work with the feed and maybe process and display each frame independently.

Important: when you run the ffplay command, it will show an scary output, and we will get a great windows with the camera feed. This is the PowerShell output:

Happy coding!

Greetings

El Bruno

References

#Coding4Fun – How to control your #drone with 20 lines of code! (6/N)

Coding4Fun Drone posts

Hi!

Today is code time also! And a continuation from my previous sample.

Yesterday I show how to read a static value: the battery. And, when you work with a device like a drone there are other important values to analyze in order to send commands to the drone. Like altitude, position, time of flight etc.

So, based on yesterday sample, I’ll show how to create a simple Python app, that will display the accelerator X value. In the following video you can see the how the value start “static” during the 1st couple of seconds, until I pickup the device and I moved it around.

Important: don’t blame me about the low battery. Playing with the drone drains the battery very fast!

Once again, the code is very straight forward. It runs a loop furing 10 seconds, showing the battery and accelX information.

Just as a reminder, this is the information we get from the drone:

pitch:0;roll:1;yaw:0;vgx:0;vgy:0;vgz:0;templ:79;temph:82;tof:10;h:0;bat:39;baro:50.42;time:0;agx:-8.00;agy:-17.00;agz:-999.00

As you can read, all the information is condensed in a single line and we can split and get:

pitch
roll
yaw
vgx
vgv
vgz
templ (temperature low)
temph (temperature high)
tof (time of flight)
h (height)
b (battery)
baro (barometer)
time
agx
agy
agz

Happy coding!

Greetings

El Bruno

References

#Coding4Fun – How to control your #drone with 20 lines of code! (5/N)

Coding4Fun Drone posts

Hi!

Today is code time! In my previous post I share some code to send commands to the drone. Today I’ll show how to read the information from the drone. Before I start, someone asked if my kids are having fun with the drone. A image will be enough to answer this.

Reading drone information

So, as far as I understand, the drone is constantly sending information to the connected client. That’s why we have the following function running all the time in a separate thread:

def receiveData():
    global response
    while True:
        try:
            response, _ = clientSocket.recvfrom(1024)
        except:
            break
# ... more code
response = None
recThread = threading.Thread(target=receiveData)
recThread.daemon = True
recThread.start()

The response is stored in a global variable named response. And it’s very easy to understand the information that we the drone sends back. This is a sample received data.

pitch:0;roll:1;yaw:0;vgx:0;vgy:0;vgz:0;templ:79;temph:82;tof:10;h:0;bat:39;baro:50.42;time:0;agx:-8.00;agy:-17.00;agz:-999.00

As you can read, all the information is condensed in a single line and we can split and get:

pitch
roll
yaw
vgx
vgv
vgz
templ (temperature low)
temph (temperature high)
tof (time of flight)
h (height)
b (battery)
baro (barometer)
time
agx
agy
agz

In python this is a simple routine, and after split this into a list, the battery value is on index 21:

ls = 'pitch:0;roll:1;yaw:0;vgx:0;vgy:0;vgz:0;templ:79;temph:82;tof:10;h:0;bat:39;baro:50.42;time:0;agx:-8.00;agy:-17.00;agz:-999.00'
>>> ls1 = ls.replace(';', ':').split(':')
>>> ls1
['pitch', '0', 'roll', '1', 'yaw', '0', 'vgx', '0', 'vgy', '0', 'vgz', '0', 'templ', '79', 'temph', '82', 'tof', '10', 
'h', '0', 'bat', '39', 'baro', '50.42', 'time', '0', 'agx', '-8.00', 'agy', '-17.00', 'agz', '-999.00']
>>> ls1[21]
'39'
>>>

So with this, to get the battery level of the drone will require this code:

The output is not very amazing, but it works!

powershell console displaying the drone battery level.

Happy coding!

Greetings

El Bruno

References

#Coding4Fun – How to control your #drone with 20 lines of code! (3/N)

Coding4Fun Drone posts

Hi!

Today I’ll write the equivalent of a Hello World in the drone ecosystem. This is a very complex app which:

take off the drone
wait a couple of seconds
lands the drone

I’ve followed the Python code sample from the SDK and the final code is very complex (see below). And it deserves some remarks

Line 11. The function recv() run in a separated thread to receive messages from the drone. The thread is started on line 44
Line 19. The function sendMessage() sends messages to the drone. Important, the messages must be UTF-8 encoded. This took me some time, until I figure it out. This time implements a timeout with 5 second until the response is processed in a separated thread
Lines 31-41. Connection information and sockets to communicate with the drone
Line 48. Main code for the app. Start the SDK mode, Wait 5 seconds, send the take off message, and then send the land message. A very simple exception catch is implemented here.

Important: When using the SDK, the 1st command should be “command”, as I did in line 49.

Here is the code:

Here is the app running at 3X speed so you don’t spend all day watching boring drones videos:

Happy coding!

Greetings

El Bruno

Coding4Fun Drone posts

The drone is flying very happy, but if the camera detects a face, the drone will flip out !

References

Share this:

Like this:

Coding4Fun Drone posts

The drone is flying very happy, but if the camera detects a banana, the drone must land !

References

Share this:

Like this:

Face Recognition and Detecting series in Python

Bonus. Viola Jones Face Detection and tracking explained video

Share this:

Like this:

Coding4Fun Drone posts

References

Face Recognition and Detecting series in Python

Share this:

Like this:

Coding4Fun Drone posts

References

Share this:

Like this:

Coding4Fun Drone posts

References

Share this:

Like this:

Coding4Fun Drone posts

References

Share this:

Like this:

Coding4Fun Drone posts

References

Share this:

Like this:

Coding4Fun Drone posts

Reading drone information

References

Share this:

Like this:

Coding4Fun Drone posts

Share this:

Like this: