#ComputerVision – Object Detection with #YoloV4 (work in progress …) and let’s think about ethics in Computer Vision

Buy Me A Coffee

Hi !

So after yesterday post where I used YoloV3 and MobileNetSSD, I also remember that we have YoloV4 released on April. I managed to make my code work with YoloV4 with some poor FPS results.

If you are interested on the code, let me know and I’ll be happy to share the code. It’s still a mess, working but a mess.

Abstract: There are a huge number of features which are said to improve Convolutional Neural Network (CNN) accuracy. Practical testing of combinations of such features on large datasets, and theoretical justification of the result, is required. Some features operate on certain models exclusively and for certain problems exclusively, or only for small-scale datasets; while some features, such as batch-normalization and residual-connections, are applicable to the majority of models, tasks, and datasets. We assume that such universal features include Weighted-Residual-Connections (WRC), Cross-Stage-Partial-connections (CSP), Cross mini-Batch Normalization (CmBN), Self-adversarial-training (SAT) and Mish-activation. We use new features: WRC, CSP, CmBN, SAT, Mish activation, Mosaic data augmentation, CmBN, DropBlock regularization, and CIoU loss, and combine some of them to achieve state-of-the-art results: 43.5% AP (65.7% AP50) for the MS COCO dataset at a realtime speed of ~65 FPS on Tesla V100.

However, what I also learned is part of the story behind YoloV4. This is very relevant to our days. The next 10 min video, really nails an explanation about how YoloV4 works.

YOLO History

YOLO was developed by Joseph Redmon. It was 1st presented in 2016, and it was key for object recognition research. This led to better and faster Computer Vision algorithms.

The latest version, YOLO v4 is currently developed by three developers:

  • Alexey Bochkovskiy
  • Chien-Yao Wang
  • Hong-Yuan Mark Liao

No Joseph Redmon in YOLOv4?

Joseph Redmon quit developing YOLO v4 because of the potential misuse of his tech. He recently announced that he would stop doing computer vision research because of the military and ethical issues….

So, why this is important? It’s all about how we use this technology. There are amazing advances in the Computer Vision area, but we also are lacking some regulation about how to use this.

IBM announced that they will no longer offer facial recognition software

2 days ago, IBM announced that they will no longer offer facial recognition software. The Verge wrote an amazing article about this (see references). This sentences really hit a point regarding Ethics and more:

IBM will no longer offer general purpose facial recognition or analysis software, IBM CEO Arvind Krishna said in a letter to Congress today. The company will also no longer develop or research the technology, IBM tells The Verge. Krishna addressed the letter to Sens. Cory Booker (D-NJ) and Kamala Harris (D-CA) and Reps. Karen Bass (D-CA), Hakeem Jeffries (D-NY), and Jerrold Nadler (D-NY).

“IBM firmly opposes and will not condone uses of any [facial recognition] technology, including facial recognition technology offered by other vendors, for mass surveillance, racial profiling, violations of basic human rights and freedoms, or any purpose which is not consistent with our values and Principles of Trust and Transparency,” Krishna said in the letter. “We believe now is the time to begin a national dialogue on whether and how facial recognition technology should be employed by domestic law enforcement agencies.” Facial recognition software has come under scrutiny for issues with racial bias and privacy concerns

Facial recognition software has improved greatly over the last decade thanks to advances in artificial intelligence. At the same time, the technology — because it is often provided by private companies with little regulation or federal oversight — has been shown to suffer from bias along lines of age, race, and ethnicity, which can make the tools unreliable for law enforcement and security and ripe for potential civil rights abuses.

The Verge, IBM will no longer offer, develop, or research facial recognition technology

There it is, think about this.

Happy coding!

Greetings

El Bruno

Resources

#VSCode – Let’s do some #FaceRecognition with 20 lines in #Python (7/N)

Buy Me A Coffee

Hi!

I’m writing a series of posts about how to control a drone with Python and 20 lines of code, and once I reach to the point to read the camera feed, I’ve added a face detection sample. However this time I didn’t use the face_recognition python package I’ve used in this series, I performed the face detection using OpenCV and Haar Cascades. So, let’s explain a little what’s this.

Let me start quoting an amazing article “Face Detection using Haar Cascades” (see references)

Object Detection using Haar feature-based cascade classifiers is an effective object detection method proposed by Paul Viola and Michael Jones in their paper, “Rapid Object Detection using a Boosted Cascade of Simple Features” in 2001. It is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images.

OpenCV comes with a trainer as well as detector. If you want to train your own classifier for any object like car, planes etc. you can use OpenCV to create one. Its full details are given here: Cascade Classifier Training.

And here we come to the cool part, OpenCV already contains many pre-trained classifiers for face, eyes, smile etc. Those XML files are stored in opencv/data/haarcascades/ folder (see references).

opencv github haar cascades files

Let’s take a look at a really [20 lines] sample code for face detection:

  • Line 6, we use OpenCV to load the haar cascade classifier to detect faces
  • Lines 9-20, main app
  • Lines 10-12, open a frame from the camera, transform the frame to a gray color scaled image and use the face cascade detector to find faces
  • Lines 14-15, iterate thought detected faces and draw a frame
  • Lines 17-20, display the webcam image with the detected faces, and stop the app when ESC key is pressed

And a live sample using a drone camera instead of an USB Camera

Bonus. Viola Jones Face Detection and tracking explained video

# Bruno Capuano
# detect faces using haar cascades from https://github.com/opencv/opencv/tree/master/data/haarcascades
import cv2
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
cap = cv2.VideoCapture(0)
while True:
_, frame = cap.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (top, right, bottom, left) in faces:
cv2.rectangle(frame,(top,right),(top+bottom,right+left),(0,0,255),2)
cv2.imshow('Face Detection',frame)
k = cv2.waitKey(30) & 0xff
if k == 27:
break
cap.release()
cv2.destroyAllWindows()

This is a long video, however is an amazing entry point to understand how the Viola Jones algorithm works.

Happy coding!

Greetings

El Bruno

Resources

#Python – Let’s use a #FaceRecognition demo app for a performance comparison between #RaspberryPi3 and #RaspberryPi4

Buy Me A Coffee

Hi!

I started to do some tests with the new Raspberry Pi 4 and the results are amazing. I’m not a performance expert, so I decided to pick up some of the demos / apps I’ve creating for the Raspberry Pi and run them in both models: Raspberry Pi 3 B+ and Raspberry Pi 4.

I started with an amazing set of tutorials on how to perform Face Recognition from Adrian Rosebrock (see references). I’ve been using his Face Recognition python package for this scenarios and it’s an amazing one.

I added some code to a custom version of Adrian’s Face Recognition sample, and it looks great. The main idea was to track in real-time the current FPS (similar to the work I did with the Image AI and Hololens sample a couple of days ago, see references).

This sample load a file with 15 trained faces and analyze frame by frame to

  • Detect faces in the frame.
  • If a face is detected, draw a frame around it.
  • For each detected frame analyze if the face is a trained face.
  • If the face is part of the trained dataset, the app will add the name of the person on top of the frame.

I display in real-time the FPS processed with a USB camera in a Raspberry Pi 3 B+. Doing a lot of tweaks and getting the best performance in the device I could never process 1FPS. The average processing data were between 0.6 and 0.9 FPS in a Raspberry Pi 3B+.

python face recognition in raspberry py 3 with FPS live

IMHO, these results are great for a small device like a Raspberry Pi 3B+. But now it was time to test it in the new Raspberry Pi 4. And an important note here is to remark that even if I did this tests in a Raspberry Pi 4 with 4GB of Rams, the performance results are similar to a RPI4 with just 1 GB of ram. We have more memory, however the processor improvements are quite significant in the new version.

I installed all the necessary software in the Raspberry Pi 4 and I got 3X better results. I’ve even tun this in a 1080p resolution to get a sense of the real processing time. The average processing data were between 2.3 and 2.4 FPS in a Raspberry Pi 4.

python face recognition in raspberry py 4 with FPS live

Amazing! In this scenario the Raspberry Pi 4 is almost 3 times faster than the Raspberry Pi 3. And again, these are amazing times for a 50USD device.

The sample source code is https://github.com/elbruno/Blog/tree/master/20190819%20Rpi%203%20vs%20Rpi%204%20Face%20Recognition

I even have time for some BBQ time with family and friends!

Happy coding!

Greetings @ Toronto

El Bruno

References

#RaspberryPi – Running a #python script in a Python Virtual Environment on reboot / startup

Hi!

Adrian Rosebrock is a very smart person who has tons of great resources about Computer Vision in PyImageSearch.com. Most of them are with Python, and he also have some of them focused on how to perform CV using OpenCV in a Raspberry Pi.

In the post [Running a Python + OpenCV script on reboot, see resources] he explains how to automatically run a Python script when a Raspberry Pi starts. He uses python virtual environments, so the first 2 commands are focused on to load the virtual env. Then, move to the app folder and run the python script.

source ~/.profile
workon cv
cd /home/pi/pi-face-recognition
python startup.py

Something like this:

python source and cv on raspberry pi.

The suggested approach consists on create a Schell Script [.sh file] with these lines and add them to the auto start. However, once you create the file and test it, there seems to be an issue with the Source command.

python source not working on SH file

Ok, so no source command in an SH file. I started to think on install all my python dependencies directly in the main user, however the idea of working with virtual environments is very useful for me. It was to read online about Linux, python and more.

Note: Before moving forward, I may need to add some context. I need to run my python script in a Terminal. My device will always auto-start with a 3.5 inches touch screen and a camera, so I need some GUI loaded.

This is an excellent article on how to add actions to the Raspberry Pi start-up [How to Execute a Script at Startup on the Raspberry Pi, see resources]. So I added my SH file here and it didn’t work and I need to figure out how to load a virtual environment and run a python script.

After a couple of tests, I realized that all the files I need are part of the virtual env location in the device.

raspberry pi python folder for virtual envs

So, I only need to add the full path to my command to make it work without the and [workon] command. My complete command will became:

/home/pi/.virtualenvs/cv/bin/python /home/pi/pi-face-recognition/startup.py

So, I edited my autostart file adding this command

raspberry pi auto start file launching python script with a virtual env

And done! My python script running on a python virtual environment on the device startup is working!

Happy coding!

Greetings

El Bruno

Resources

My posts on Raspberry Pi

Dev posts for Raspberry Pi
Tools and Apps for Raspberry Pi
Setup the device
Hardware

#VSCode – Let’s do some #FaceRecognition with 20 lines in #Python (6/N)

Buy Me A Coffee

Hi!

In my last post I share some lines of code which allowed me to run some of the face recognition demos 6 times faster. I added a Frames per Second (FPS) feature in my samples. Later, thinking about performance, I realize that I don’t need to work with a full HD picture (1920 x 1080), so I added some code to resize the photo before the face detection process.

However, while I was coding around this solution I also realized that I may want to initialize my camera to start in a lower resolution. So, I searched online on how to do this with OpenCV and I found 3 beautiful lines of code.

open camera with opencv with lower resolution
# standard face detection sample with FPS in console
# open camera in low resolution to get better FPS
import face_recognition
import cv2
import time
video_capture = cv2.VideoCapture(0)
video_capture.set(cv2.CAP_PROP_FRAME_WIDTH,640)
video_capture.set(cv2.CAP_PROP_FRAME_HEIGHT,480)
while True:
start_time = time.time()
ret, frame = video_capture.read()
rgb_frame = frame[:, :, ::1]
face_locations = face_recognition.face_locations(rgb_frame)
for top, right, bottom, left in face_locations:
cv2.rectangle(frame, (left, top), (right, bottom), (0, 0, 255), 2)
cv2.imshow('Video', frame)
print("FPS: ", 1.0 / (time.time() start_time)) # FPS = 1 / time to process loop
if cv2.waitKey(1) & 0xFF == ord('q'):
break
video_capture.release()
cv2.destroyAllWindows()

So, I manage to improve my processing code from 20FPS to +30FPS … which is very good ! Later on this posts I’ll try to do some similar FPS tests on a smaller device and I’ll see and share how this works.

Happy Coding!

Greetings @ Burlington

El Bruno

Resources

#VSCode – Let’s do some #FaceRecognition with 20 lines in #Python (4/N)

Buy Me A Coffee

Hi!

Quick post today. I’ll pickup yesterday demo, showing the facial features and adding Face Recognition on top of that. In other words, we’ll move from this

To this

With a couple of extra lines for face recognition

import face_recognition
import cv2
import numpy as np
def LoadFaces():
bruno_image = face_recognition.load_image_file("d:\Faces\Bruno1.jpg")
bruno_face_encoding = face_recognition.face_encodings(bruno_image)[0]
valentino_image = face_recognition.load_image_file("d:\Faces\Valen1.jpg")
valentino_face_encoding = face_recognition.face_encodings(valentino_image)[0]
known_face_encodings = [
bruno_face_encoding,
valentino_face_encoding
]
known_face_names = [
"Bruno",
"Valentino"
]
return known_face_encodings, known_face_names;
video_capture = cv2.VideoCapture(0)
known_face_encodings, known_face_names = LoadFaces()
while True:
ret, frame = video_capture.read()
rgb_frame = frame[:, :, ::1]
face_landmarks_list = face_recognition.face_landmarks(rgb_frame)
for face_landmarks in face_landmarks_list:
for facial_feature in face_landmarks.keys():
pts = np.array([face_landmarks[facial_feature]], np.int32)
pts = pts.reshape((1,1,2))
cv2.polylines(frame, [pts], False, (0,255,0))
face_locations = face_recognition.face_locations(rgb_frame)
face_encodings = face_recognition.face_encodings(rgb_frame, face_locations)
for (top, right, bottom, left), face_encoding in zip(face_locations, face_encodings):
matches = face_recognition.compare_faces(known_face_encodings, face_encoding)
name = "Unknown"
face_distances = face_recognition.face_distance(known_face_encodings, face_encoding)
best_match_index = np.argmin(face_distances)
if matches[best_match_index]:
name = known_face_names[best_match_index]
#cv2.rectangle(frame, (left, top), (right, bottom), (0, 0, 255), 2)
cv2.rectangle(frame, (left, bottom), (right, bottom), (0, 255, 0), cv2.FILLED)
font = cv2.FONT_HERSHEY_COMPLEX_SMALL
cv2.putText(frame, name, (left + 6, bottom 6), font, 0.7, (255, 255, 255), 1)
cv2.imshow('Video', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
video_capture.release()
cv2.destroyAllWindows()

There is some room for performance improvement, so I’ll focus on this in next posts.

The complete project is available here https://github.com/elbruno/Blog/tree/master/20190528%20Python%20FaceRecognition

Happy Coding!

Greetings @ Burlington

El Bruno

Resources

My Posts

  1. Detecting Faces with 20 lines in Python
  2. Face Recognition with 20 lines in Python
  3. Detecting Facial Features with 20 lines in Python

#VSCode – Let’s do some #FaceRecognition with 20 lines in #Python (2/N)

Buy Me A Coffee

Hi!

Yesterday I explained how to write a couple of lines in Python to perform live face detection in a webcam feed [Post]. Check the resources section to find more about the tools I’m using.

Today, I’ll add some more code to perform face recognition. And as usual, I’ll work with my kids to test this out. I’ll start adding 2 face encodings for Valentino and myself. The code is simple enough, and I use a simple 300×300 head-shot photo to train and get the face encoding.

def LoadFaces():
bruno_image = face_recognition.load_image_file("d:\Faces\Bruno1.jpg")
bruno_face_encoding = face_recognition.face_encodings(bruno_image)[0]
valentino_image = face_recognition.load_image_file("d:\Faces\Valen1.jpg")
valentino_face_encoding = face_recognition.face_encodings(valentino_image)[0]
known_face_encodings = [
bruno_face_encoding,
valentino_face_encoding
]
known_face_names = [
"Bruno",
"Valentino"
]
return known_face_encodings, known_face_names;

The previous function returns an set of arrays with the face encodings and the face names. In the complete file, I’ll use this to analyze the camera frame (line 31) and later to check the matches for faces (lines 34 * 36)

import face_recognition
import cv2
import numpy as np
def LoadFaces():
bruno_image = face_recognition.load_image_file("d:\Faces\Bruno1.jpg")
bruno_face_encoding = face_recognition.face_encodings(bruno_image)[0]
valentino_image = face_recognition.load_image_file("d:\Faces\Valen1.jpg")
valentino_face_encoding = face_recognition.face_encodings(valentino_image)[0]
known_face_encodings = [
bruno_face_encoding,
valentino_face_encoding
]
known_face_names = [
"Bruno",
"Valentino"
]
return known_face_encodings, known_face_names;
video_capture = cv2.VideoCapture(0)
known_face_encodings, known_face_names = LoadFaces()
while True:
ret, frame = video_capture.read()
rgb_frame = frame[:, :, ::1]
face_locations = face_recognition.face_locations(rgb_frame)
face_encodings = face_recognition.face_encodings(rgb_frame, face_locations)
for (top, right, bottom, left), face_encoding in zip(face_locations, face_encodings):
matches = face_recognition.compare_faces(known_face_encodings, face_encoding)
name = "Unknown"
face_distances = face_recognition.face_distance(known_face_encodings, face_encoding)
best_match_index = np.argmin(face_distances)
if matches[best_match_index]:
name = known_face_names[best_match_index]
cv2.rectangle(frame, (left, top), (right, bottom), (0, 0, 255), 2)
cv2.rectangle(frame, (left, bottom 25), (right, bottom), (0, 0, 255), cv2.FILLED)
font = cv2.FONT_HERSHEY_COMPLEX_SMALL
cv2.putText(frame, name, (left + 6, bottom 6), font, 0.7, (255, 255, 255), 1)
cv2.imshow('Video', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
video_capture.release()
cv2.destroyAllWindows()

Last lines are cosmetic to mostly draw the frames for the detected faces, and show the names.

The complete project is available here https://github.com/elbruno/Blog/tree/master/20190521%20Python%20FaceRecognition

Happy Coding!

Greetings @ Burlington

El Bruno

Resources

My Posts