Machine Vision at Petrol Stations
Ever had the urge to check for messages on your phone while refueling your car?
You know that’s probably not a very good idea as your mobile phone, in a cloud of gaseous vapor, could trigger an explosion. I took out my phone (could not resist) recently at a local unmanned fuel station while topping up and, to my surprise, the pump stopped. I put my phone away and restarted fueling. Then I took it out again to see if the pump would stop again and indeed, the pump stopped again.
The only way to make that work is if the security camera had spotted the phone and a signal was sent to the pump. This is a perfect example of machine vision. In this article, I will explain how that works.
Machine vision is another name for image recognition by computers without describing all the rules. A simple example is computers recognizing pictures of cats and dogs on the internet. Cats and dogs may not be that exciting but think of recognizing them when you are in a self-driving car.
It would be impossible to give a computer all the rules for recognizing cats or dogs as there are simply too many possibilities. This is where machine vision comes in. By labeling thousands of cat or dog images, it is possible to ‘train’ a computer to recognize a cat or a dog, simply by its characteristics.
Most training is done using Convolutional Neural networks (CNN). A CNN is a type of neural network very suitable for images, as it breaks down the images in components that can be reused; like the shape of eyes, corners, transitions, and so on. If a computer can learn the characteristics of cats and dogs, it can also learn to recognize mobile phones. By feeding a neural network with thousands of pictures of people with mobile phones (at unmanned fuel stations) it is possible for machines to recognize when a customer is checking his phone while filling up.
It is also possible to feed pictures of mobile phones on top of a variety of backgrounds (synthetic pictures) so that the neural network will pick up the typical shape of mobile phones, regardless of the background. If taking thousands of pictures of mobile phones puts you off, you’ll be pleased to learn that someone somewhere in the world already did this. And even better, in an open-source world, pre-trained neural networks for a wide variety of objects are freely available already. One popular open-source initiative on object detection is the TensorFlow Object Detection API.
I mention this one as it easy to set up in Google’s Colab environment, which is available (incl. free GPU’s) for everyone with a google account. First, you need to understand that video is just a series of pictures in real-time (called frames). Each frame has to be prepared to ‘flow’ through the trained neural network and the output of each run is a list of detected objects. In case a mobile phone is detected, a signal has to be sent to the pump’s control logic to switch off.
I thought it would be nice to show you how you can code the first part, i.e. take a picture yourself holding a mobile phone and then run it through a neural network to see if it detects it. As a true engineer, I’d rather copy something good than invent something poor. As such, I found this article on how to setup object detection in Colab and plug in my own images (link to article).
I simply downloaded the Colab Notebook to my Google Drive and opened it using Colab. You have to make sure you mount your Google Drive to Colab (https://colab.research.google.com/drive/1srw_HFWQ2SMgmWIawucXfusGzrj1_U0q).
To work with your own pictures, store them in a folder in Google Drive, and update the path in section 5 of the Colab Notebook. After running the model you should see your own picture with detected objects highlighted.
Conclusion
This is a very simple example using a single frame, but you can also set this up for video. I hope you can now understand how the pump switched off in the unmanned fuel station when the security camera spotted me picking up my mobile phone. Stay safe!
