Face recognition using deep learning

Some time ago, I had this idea of making a hand gesture recognition glove using MPU6050 gyroscope and accelerator sensor. Though it was possible to detect some basic gestures I was unable to use it to recognize complex hand gestures. So then I turned my eyes towards image processing. Though I had not really taken a university course on this area, after talking with the supervisor I started following some online lessons on "Processing." Then I came across "Open cv" library. I started following the tutorials on "docs.opencv.org". Though "opencv" can be used in a wide range of applications I was particularly interested in "Object and feature detection" and "Machine learning." After extensively reading playing around with feature algorithms like SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features, Fast Algorithm and basic machine learning algorithms like k-Nearest Neighbor (kNN), Sample Vector Machines (SVM), k-Means Clustering, I wanted to combine those two branches and try a project that recognizes faces using machine learning.

Then there I came across this library named "face_recognition" written for python by Adam Geitgey. Click here to read about face_recognition. It is wrapped around "Dlib" another library written in C++ by Davis King which containing machine learning algorithms.

Test Image

Face_Recognition

Face_recognition divides the task of recognizing a face into several steps. In the first step it finds the locations of all the faces in a photo or a frame. To do this they use the algorithm called HOG (Histograms of Oriented Gradients). After locating the faces in the second step it solves the problem of faces being turned to different directions. There an algorithm called Face Landmark estimation is used. It works by training a machine learning algorithm to find 68 particular points on a face. The third step is encoding. There a deep neural network is trained to take 128 separate measurements(an encoding) of a face and write it into a file. Open face is the library they have used here. In the fourth step, once an example face is given finding the matching encoding and name is done by a linear SVM classifier.

128 measurements

Embedding 

PY Image search

Reading further on this topic brought me to this fantastic website by Adrian Rosebrock. He has written a collection of articles and tutorials on computer vision. His article titled Face recognition with OpenCV, Python, and deep learning provided me with a complete guide on how to effectively use face_recognition library along with machine learning.

Installing necessary libraries

The first step was installing all the necessary libraries. And I was so far using opencv in windows and I could run the programs with no hassle. However it proved to be a mammoth task getting dlib and face_recognition running on windows. Therefore I shifted to Ubuntu. As prescribed by many authors I created a virtual environment and isolated my project from the standard python. It was much easier from there onwards to install dlib, face_recognition and imutils. Once again the first two packages I built from the source rather than using PIP.

Creating the deep learning image data set 

To create the image data set I referred to a separate article, once again written by Adrian. There he has used Microsoft's Bing Image Search API to download images linked to a certain query. After registering myself at Microsoft cognitive services it was possible to get a free trial API for a week. Using the python "requests" library a program was run to get the images. Since I'm a big fan of the popular sitcom "The Big Bang Theory" I created a dataset of the main characters Sheldon, Leonard, Penny, Howard, Amy, Bernadette and Raj. Each character had about 40 pictures.

The program 

For the codes I mostly referred to the PYimagesearch.com and adapted with slight changes. So a python program called "encode_faces" was written to write the encodings into a file. To detect the faces I used the "HOG" algorithm, which is said to be faster but a little low on accuracy. Though I tried "CNN" algorithm which is supposedly more accurate, since I was running without a GPU my computer was stuck. Therefore decided to stick with HOG. So the program loops through all the images in the dataset, and first it will locate the faces in each image. Then it will write two lists, known encodings and known names. An encoding is a 128 measurements of a face. And then it writes the information into a .pickle file.


Then another program was run to to test and match the encodings that are already written. First, an image was tested. So the initial step was to find out the locations of the test image. Similar to the first code this was done by using the "Hog" algorithm which was relatively faster. Then the encodings of the faces were calculated. Once it was done we had to compare these encodings with our previously acquired data to see whether there is a matching name.

The .pickle file containing the data was loaded and then compare_faces function was used to do the comparison. This function returns a Boolean value(true/false) for each sample picture in the data set. After looping through the whole dataset, the number of votes (number of true values) in front of each name can be taken. Since there can be more than one face in a single picture, the names with true votes higher than a threshold are put in another list.

Then another loop was written to go through this list and draw rectangles around the faces and display names of the matches found.

A similar approach was followed in detecting faces in a video.

Test 2

Test 2

 

Read More

Labview based Pulse Analyzer

This is a data acquisition system which is capable of reading a pulse rate of a person. The pulse rate is read using a pulse sensor through an arduino uno and then the analysis is done by using LabView.

Pulse sensor

The sensor functions by examining hemoglobin, which is the oxygen-carrying pigment of red blood cells that gives them their red color and serves to convey oxygen to the tissues. Hemoglobin is found in two forms. The first is oxidized (oxy-) hemoglobin, which is denoted as HbO₂ (also called “oxygen-loaded”). The second is reduced-oxygen (deoxy-) hemoglobin, which is denoted as Hb (“oxygen-depleted”). Blood oxygen saturation (SpO₂) is the ratio of oxy-hemoglobin to deoxy-hemoglobin. This can also be expressed as: SpO₂=HbO₂/(Hb + HbO₂). An interesting thing about hemoglobin is how it reflects and absorbs light. For example, Hb absorbs more (and reflects less) visible red light. HbO₂ absorbs more (and reflects less) infrared light. Since blood oxygen saturation can be determined by comparing the values of Hb and HbO₂, one method for doing this is shining both a red LED and an infrared LED through a body part (such as a finger or wrist), and then comparing their relative intensities. There are two common methods of doing this.

1. measuring the light transmitted through tissue is called transmissive oximetry
2.  measuring the light reflected by tissue is called reflectance oximetry

Transmissive and Reflectance oximetry

Typical pulse oximeters monitor the SpO₂ of a person’s blood based on the red light (using a 600-750 nm wavelength) and infrared light (using a 850-1000 nm wavelength) absorption characteristics of HbO₂ and Hb. This type of pulse oximeter flashes the red and infrared lights alternately through a body part, such as a finger, to a photodiode sensor.The photodiode is normally utilized to receive the non-absorbed light from each LED. This signal is then inverted using an inverting operational amplifier, or op amp. The resulting signal represents the light that has been absorbed by the finger.

A look-up table is an important part of a pulse oximeter. Look-up tables are specific to a particular oximeter design and are usually based on calibration curves derived from, among other things, a high number of measurements from subjects with various SpO₂ levels.

In this project a pulse sensor manufactured by the pulsesensor.org was used.

Interfacing pulse sensor with Arduino Uno

The pulse sensor which consists of three pins, two for the ground and 5V and the third for the signal reading was easily interfaced with the arduino. Then using the library which is available on https://pulsesensor.com/pages/getting-advanced  
the data values that the sensor reads was transmitted using serial communication. In this step, using the serial monitor and serial plotter which is available on arduino IDE it was verified that the sensor is working fine.

VISA

In this step the task was to acquire the data which is being transmitted using serial communication by arduino. To do this, the VISA pluggin on LabView was used. VISA was downloaded from the National Instrument website and then it was installed. Then a using an example program which was available in LabView it was successfully tested. It was possible to get the data which was sent by the arduino on a LabView VI.

LabView Program

Once the data was acquired by the labview program the next step was the necessary analysis. Since the data was in String format it had to be converted to number. It was possible to plot the data on a graph. The signal was clearly visible with its peaks against a time axis. Then using a Peak Detector VI the peaks of the incoming signals were detected. This Peak detector VI has the ability to put the locations of the peaks on the time axis in an array. After this by using the distance on time axis between two successive peaks the BPM(Beats Per Minute) was calculated. And the program was set to give an alert using a Beep VI if the BPM value goes above 100 or below 60. (which is the standard BPM of a healthy person).
The user has the ability to control the settings such as sampling rate, baud rate, parity, data bits etc. And also the user can save the data in a file in .csv format. The user has also the ability to save these user settings into a text file and load whenever he or she wants to reuse the same settings.

The VI

You can access the full VI here. https://github.com/aswinda29/LabView-based-pulse-rate-analyzer

Read More

Raspberry pi weather station

This was an entry level study done on Raspberry pi in the Data Acquisition Lab to get a basic knowledge on Raspberry pi and also to get familiarized  with the linux operating systems.

First Raspbian was downloaded and a bootable SD card was created. Raspberry pi wa booted and  was connected to a wifi network. Then a little bit of study on static and dynamic IPs was done. For easy functionality a static IP was created using DHCPCD. Then the raspberry pi was connected to the computer using "Putty". 

Once the link was established between Raspberry pi and the computer files were transmitted to the pi and some basic codes were tested such as lighting an LED, lighting two LEDs, generating a PWM signal and blinking an LED etc.

As the next step a DS1820 temperature sensor was interfaced with the pi. Once it was up and running using the one wire protocol another DS1820 was interfaced.  Then apache server was installed and a temperature logger was created. There the python code will write the temperature data into a text file. And an html file was created to acquire data from the text file which was located at /var/www/html location. Therefore using the ip address 10.20.12.219 it was possible to view the web page.

Two sensor information

Click here to access the full report.

Read More

Robot Competition

The competition  was the final assessment of the course I took at the university on Robotics and automation. There were several tasks to be completed in the competition with three rounds.

The task

The track

The robot had to follow the track with which was designed with a black line. The track had a section with arrows pointing the directions. So robot should be able to identify the direction pointed out by the arrows. The track has red and green colored crossed lines. The robot should be able to identify these colored signals. There was a broken fence beside the track and the robot had to map the fence. And the robot should produce a graph which contains the accuracy of the line following. 

The robot

The participants had to make a robot car using two driving wheels using arduino nano. And the coding had to be done using the C language instead of using he arduino IDE. And any method could be used in the communication with the robot.

Driving the robot

L298N

The robot was driven using a an H bridge and PWM (Pulse width modulation). To control the rotating direction of a DC motor we should be able to control the direction of the current flow. The H bridge is a circuit which controls the direction of the current flow using 4 switching elements. L298N is a dual H bridge motor driver module. It has four ports to take inputs which corresponds to the two directional movements of the two motors. However there are only two PWM pins in arduino nano. Therefore the method used was to for each motor a PWM pin and a normal GPIO pin was connected. To move the motor forward the normal GPIO pin was set to logic one. So this gives the motor 100% power. To move it backward the first pin was set to logic 1 and the PWM pin was also set to logic one. Now according to the H bridge setting, the motor gets 100% power to go backward and 50% power to go forward(say the PWM is set to be 50%). Then the motor actually moves backward with 50% power. 

In the code separate functions were written to move forward, backward, turn left, turn right and get a 90 degree turn and was tested for each function.

Line following 

The IR sensor array

To do the line following an IR array sensor was created. The circuit was designed with 8 IR couples for a better line following. A circuit was designed, a pcb was printed and the circuit was soldered. IR couples work by emitting an IR ray and capturing its reflection. There only a white surface will reflect the array while a black surface will absorb the ray. So these outputs are amplified through an op amp (LM324n was used) and then fed to the arduino nano. The movement of the robot was altered based on this input.

In code implementation an algorithm written in a way that all the eight logic inputs were assigned a digit ranging from +4 to -4. And the algorithm always tries to make the sum of the inputs to 0. For example if the 2nd and 3rd pins are set to be logic one, then the sum becomes +5(sum of +3 and +2). Then the robot turns to the right side and makes the sum 0. (sum of +1 and -1 is zero). In this way it will be assured that the robot will always try to keep the black line in the middle of the IR sensor.

The color sensor

                   The color sensor

The color sensor was created on a basic theory in optics. How we see certain colors is determined by the wavelength of the light that particular color reflects. For example if we consider a red apple, it reflects the light with the corresponding wavelength of the red color. So in this sensor an RGB LED (an LED which can be lit in red, green and blue) is blinked respectively with the three colors. Based on the amount of the light reflected by each color is taken and compared. This was done by getting the voltage across an LDR. Based on these voltages the color of the target object is identified. The circuit was made on a PCB.

Ultrasonic sensor

Ultrasonic sensor

To map the broken fence beside the track an ultrasonic sensor was used. In the first stage the ultrasonic sensor was interfaced to measure the actual distance. The code was written in a way that at the moment when the sensor is triggered, a timer counter was started and it keeps checking the the signal has been reflected. At the moment the capture becomes positive the time is stopped and based on the time, the distance is calculated. Since the required information was binary, i.e. whether there is a fence or not, a threshold was given to the distance measurement. 

Utilization of timers

Though each of the sensors and functionalities of the robot were developed and tested separately at the end of the project the whole thing had to be run as a whole system. At the first phase most of the coding was done using the timer1 in Atmega328p. In Atmega328p microcontroller which is inside the arduino nano, there are three timers namely timer0, timer1 and timer2. Timer1 is a 16 bit timer and the other two are 8 bit timers. Therefore timer1 was used in PWM in driving the motors. Timer0 was used with the ultrasonic sensor. 

There was a requirement to sound a buzzer whenever the robot detects a red color signal. At first the buzzer was sounded using a basic delay statement. However when this was implemented in the driving robot, it was observed that because of this delay the accuracy of the line following is highly affected. Basically the robot was shaking then and there instead of a smooth driving. Therefore the delay statement was replaced by a toggle statement based on timer2.

Serial communication

At the competition it was expected to produce three graphs.

• The accuracy of line following
• Map of the fence
• The positions of the green colored cross signals on the track.

Therefore these information was sent from the robot using serial communication. The robot was connected to a computer with the use of a Bluetooth module which was interfaced to the robot.

The corresponding information was transmitted in the code of the robot and was received from the other end using a python program. By using python packages such as numpy, matplotlib the data was visualized. 

For an efficient serial communication data was sent using an encoding method. A serial data byte packet contains 8 bits. However there were 8 IR inputs, three color sensor inputs and the ultrasonic sensor input to be transmitted. So an algorithm was written in a way that the first two bits of the packet has an identification code and the rest contains information. 

00xxxxxx - 6 IR sensor inputs

01xxxxxx - 2 IR sensor inputs and ultrasonic sensor input

10xxxxxx - Color sensor information

11xxxxxx - Check whether all 4 packets are received

In the python program this was decoded to get the necessary information.

Plotting all three at the same time was a challenge. This task was achieved by writing the incoming data into a text file and plotting. However it is not considered a good practice to read ad write data at the same time rather than storing data in the memory and performing tasks. 

Click here for the codes

Read More

AVR based Guitar Distortion Pedal

An effect pedal is a device which can alter the sound of a music instrument that comes in a format of a pedal or a stomp box. Guitarists normally arranges several pedals in what is called a "pedal board." Each of these analog pedals activate different sound effects. At the same time multi effect digital guitar pedals are also present in the market. Being a guitarist, I have been using some guitar pedals such as DS -1 distortion by BOSS, SD-1 super drive by BOSS etc. With that experience I tried to build a pedal that is capable of producing the distortion effect my self. 

What is distortion?

The word distortion refers to any modification of wave form of a signal, but in music it is used to refer to nonlinear distortion (excluding filters) and particularly to the introduction of new frequencies by memoryless nonlinearities. Clipping is a non-linear process that produces frequencies not originally present in the audio signal. These frequencies can be harmonic overtones, meaning they are whole number multiples of one of the signal's original frequencies, or "inharmonic", resulting from general intermodulation distortion.   

"Soft clipping" gradually flattens the peaks of a signal which creates a number of higher harmonics which share a harmonic relationship with the original tone.

"Hard clipping" flattens peaks abruptly, resulting in higher power in higher harmonics. As clipping increases a tone input progressively begins to resemble a square wave, which has odd number harmonics. This is generally described as sounding "harsh".

Hard clipping and soft clipping

The design 

When I was doing the background research on the project I came across several other projects that other people have done on guitar pedals. Out of them, "Electrosmash pedalshield uno" which is an open source project based on arduino platform and "Homebrew digital effects pedal" which is a PIC micro controller based project particularly influenced me. So I designed a circuit using AVR ATmega32A microcontroller and the coding was done in C. 

The design

The first step was to understand the signal output of the guitar. So using a digital oscilloscope I tried to find what type of a signal does an electric guitar outputs. It was found out that the signal is an analog signal with a varying amplitude between 100mV and 1V with some noise. So it was important to use some high-pass filters to get rid of the noise. Then the analog signal had to be converted to a digital inside the microcontroller to do the processing of the signal. Since ATmega32A runs at 5V it was better to amplify the signal before feeding into the microcontroller. An operational amplifier was used to do that. When selecting an op amp there were two things to be considered. Because of the dynamic range of the signal it has to have a rail-to-rail operation. And the other thing was supply voltage below 5V. TL972 op amp from texas intruments was selected after consideration. So after the digital modification the signal leaves the micro controller and go through another circuit to set the signal back to its original voltage range. 

Since the user should be able to switch on and off the effect when desired a 3PDT switch was used. Its a switch with triple pole- double throw setting which is capable of handing two separate circuits. So this allows the input signal to bypass the main circuit when the effect is switched off. 

The powering of the system was done by a standard ac to dc adapter with a 15V dc output. So another circuit was designed using the 7805 power regulator to power the system with 5V. 

The maximum inbuilt oscillator of the Atmega32A is 8MHz. But it was important to run the microcontroller at least at 16MHz to ensure the better quality of the sound. Therefore a 16MHz crystal oscillator was used. 

The circuit

The code

The basic step of the code was to convert the Analog signal to a digital one. This was done by the ADC inside the microcontroller. Then the modulation was done in such a way that if the ADC value is higher than a threshold then you pass on the threshold value, if the ADC value is less than a particular value then the original value will be passed as the output. And this threshold value can be controlled using two push buttons generating interrupts. Since there is no DAC ( Digital To Analog converter) inside the microcontroller the output was taken out using PWM (Pulse Width Modulation).



The physical design

As a test run I re created the "Electrosmash pedalshield uno" in arduino. Since it was successful I went onto implement it on AVR and using the Atmega32A. So a PCB was designed using "Eagle" and it was created on a copper board using basic techniques. Once the soldering was done the circuits was fitted inside a plastic box which I bought from an electronic shop. The necessary holes were drilled using the university workshop. And it could provide a decent distortion effect!

Click here for the code, project report, user manual, presentation and the PCB layouts. 

Read More

De-do ( An android app to learn German language. )

This is an android app developed with the intention of assisting people who are learning the German language. De-do has it's special focus on German grammar since it's the key to get a good grasp on the language.

Start Page

The app has following functions. 

1. The Quizzes

 


Under this section the user is given grammatical questions in the form of multiple choice answers. The user can select the correct answer and then the app will evaluate the answer. At the end of each quiz the high-score will be calculated. And the quizzes are categorized based on the difficulty. The user can select from Beginner, Intermediate or Advanced levels.

 

 

2. Lessons

This section provides the user with a grammar lesson compatible with the quizzes. Pictures, tables and necessary explanations are used to enhance the knowledge of the user.

 

3. News from Germany

Reading plays a great deal in improving the vocabulary of an any language student. Therefore under this section real time news updates from all the main German news stations are given. The user has the ability to browse through, read and learn new words.

 

4. Picture dictionary 

A real time updated picture dictionary, once again to enhance the vocabulary.

 

5. Basic phrases

A collection of basic phrases in German language which are useful to the novice learners. The user can listen and then speak.

The technical aspects

The homepage of the app has used android grid view with four different icons for each sub section. The first icon, Lessons and Exercises opens up to an activity created using android fragments. The three fragmented activity has the options of selecting difficulty level. And also the ability to select the Lessons or Exercises. Android listview format is used to showcase the lessons each with an image. The data and images of the lessons are stored in an SQLlite database and loaded on the single activity. The questions are also stored in an SQLlite database with three different tables for each difficulty level. And the index number of the correct answer to each question is stored in a separate column and it is checked with the user's response in evaluation. The high-score is sent back to the start page after each exercise and updated.

The news is generated by a free API from newsapi.org and taken using a JSON format. Then the JSON is broken down and each detail is assigned to the particular field. Since the api can not be categorized based on the language the it was sorted out by the country(i.e Germany). 

The picture dictionary is created using an image slider. The pictures are stored in a blog and loaded in real time. And in the listening section short audio clips are stored in a builtin database and loaded. 

The following link has the project reports, presentations and the promotional video of the app. 


Click here for the reports
Click here for the video

Read More