The OpenMV H7 programmable AI camera can recognize shapes and lines and estimate color area ratios.

Evaluation of an AI Camera System for Environmental Monitoring Applications

Cecilia Sweeney, New Hampshire Academy of Science 2021

ABSTRACT

The OpenMV H7 camera is a small microcontroller-based camera equipped with AI and machine vision capabilities that is programmed using Micropython. The camera can be used with TensorFlow Lite, and a variety of applications such as: color tracking, line detection, circle detection, rectangle detection, image capture, and video recording. All applications are bundled with the OpenMV IDE, which provides the programming interface for Windows, MacOS, Linux, and Linux-Raspberry Pi systems. Given its capabilities, the camera has the potential to provide a new, low-power consumption, cost-effective tool for environmental monitoring studies. The purpose of the work reported here is to evaluate the capabilities and accuracy of the camera system quantitatively, specifically if applied to shape and color detection, to gauge its usability in a lab setting. Methods for evaluating shape and line detection involved making graphs with varying degrees of similarity to circles, squares, and straight lines. These graphs were designed and printed on paper, and then it was recorded which shapes the camera identified, as the camera was set to detect these shapes at different accuracy thresholds. The results of these studies show that with increasing accuracy threshold, the camera becomes more sensitive to what it considers a circle, rectangle, or straight line. Methods for color detection evaluated the camera’s ability to detect the ratio of the area of one color to another, by creating a program to give the coordinates of the box drawn around any color recognized. This was tested on a series of rectangles containing different ratios of red and green areas. The results compare the experimental ratios to the real, hand-measured ratios of green and red areas, and show the same general trend, but also an average 34% error rate. The shape and color data suggest that the OpenMV H7 camera is effective for identifying shapes and lines, and to find approximate shapes or perfect shapes. It is less effective at producing precise color area ratios, or exact measurements of an object. This camera is easy-to-use and doesn’t require much training or a great deal of safety precautions.

INTRODUCTION

The OpenMV H7 camera is a small, programmable, microcontroller camera equipped with AI and machine vision capabilities (OpenMV LLC, 2019). It is programmed using micropython, a version of Python 3 that uses only a small set of the standard programming library and is meant to run on microcontrollers (George, 2018). According to the OpenMV company, “[Their] goal with the OpenMV Cam is to make a flexible system for solving very straightforward computer vision problems” (George et al., 2021). The camera board contains an RGB LED and two 850nm IR LEDs, as well as the removable camera module. It originally comes with an OV775 image sensor, which “is capable of taking 640x480 8-bit Grayscale images or 640x480 16-bit RGB565 images at 75 FPS when the resolution is above 320x240 and 150 FPS when it is below” (OpenMV LLC, 2019), though it is also possible to buy more specialized lenses for the image sensor that the buyer can attach by themself. OpenMV also offers a global shutter camera module, for professional machine vision applications, and a FLIR Lepton adapter module, for thermal machine vision applications. This camera currently has applications for TensorFlow Lite, frame differencing, color tracking, marker tracking, face detection, eye tracking, person detection, optical flow, QR code detection/decoding, data matrix detection/decoding, linear barcode decoding, AprilTag tracking, line detection, circle detection, rectangle detection, template matching, image capture, and video recording (OpenMV LLC, 2019). These applications are possible through the OpenMV IDE, which can be downloaded on Windows, Mac, Linux-Desktop, and Linux-Raspberry Pi (George et al., 2021).

Despite its many capabilities, his camera is largely unexplored for research purposes. It specifically has not been used in the NHAS lab, though it may be a useful tool for several AI and environmental monitoring projects that are currently running. The structure of the OpenMV cam “allows you to write Python scripts to execute on the OpenMV which compile and load extremely fast while maintaining microcontroller benefits like low power consumption and instant on processing for embedded applications” (George et al., 2021). Given these benefits, as well as its low price at $65, the OpenMV H7 has the potential to provide a new, low-power consumption, cost-effective tool for certain environmental monitoring studies. The purpose of this study is to evaluate the capabilities and accuracy of this camera, specifically in shape and color detection, in order to see how it may be helpful in a lab setting.

METHODS AND MATERIALS

Feature Detection

Graphs of lines and shapes were created using the Desmos online graphing calculator. Lines were created using the equation y=asin(x), using a=0 through a=0.1 in intervals of 0.01 and y=sin(bx), using b=0 through b=0.1 in intervals of 0.01; circles were created using the equation x2a+y2=1, using a=1 through a=2 in intervals of 0.1; and squares were created using the equation xn+yn=5n, using n=2 through n=10 in intervals of 2, and n=20 through n=100 in intervals of 10. The grid and axis numbers were taken off, the shapes were translated to an area where they wouldn’t cross either axis, and the lines were translated to an area where they wouldn’t cross the x-axis and the window was changed so that the y-axis line couldn’t be seen. These graphs were saved by going to “export image” under “share graph” and selecting size: medium square and line thickness: medium. The images were transferred onto a google doc with about six shapes or eight lines per page, which was then printed out on standard 8.5x11 printer paper.

The find_lines, find_circles, and find_rects programs were run on the corresponding printed sheets, using threshold values of 1,000 through 11,000 for y=asin(x), 1,000 through 12,000 for y=sin(bx), 2,000 through 14,000 for the circles, and 10,000 through 150,000 for the squares, all in intervals of 1,000. The a or b value of the waviest line it detected as a straight line, the a value of the least circular “circle” it detected as a perfect circle, and the n value of the least square “square” it detected as a perfect square were recorded at each threshold value. (Unlike the lines and circles where a higher variable value corresponds to a less accurate straight line or circle, a lower n value for the squares corresponds to a less perfect square). This was repeated three times.

Color Tracking

The camera’s color tracking abilities were evaluated by using it to calculate the ratio of green area to red area in a rectangle (meant to represent the different colors in a changing leaf during Autumn). A series of rectangles with varying amounts of red and green were created by inserting 20 5x4 cell tables in a google doc. In the first table, all the cells were made green, and were one by one changed to red so that the last table was all red. The ratio of green versus red in each table was calculated using basic geometry, and the number of red cells in a table and the green versus red ratio for that table was recorded. These color tables were printed out on standard printer paper and cut out so that the camera would only see one table at a time.

Before the camera could be tested on its ability to detect the area of each color in these tables, a program had to be created to allow it to do that, as that wasn’t a feature of any of the programs that come with the OpenMV IDE. Starting with the single_color_rgb565_blob_tracking program, the threshold index line was taken out. The third color threshold was also taken out, as only two colors were being tested, but this could be left unchanged or more color thresholds could be added to test more colors. The first color threshold was set to the red color used in the tables and the second set to the green color using the threshold editor under tools → machine vision → threshold editor. The “for blob in img.find_blobs” loop was copy and pasted so that there were two, one for each color. “img.find_blobs([thresholds[threshold_index]]” was changed to “img.find_blobs([thresholds[0]]” in the first loop and “img.find_blobs([thresholds[1]]” in the second. In the first “for blob” loop, “img.draw_string(blob.x() + 2, blob.y() + 2, “r”)” was added under the “img.draw_cross” line, in order to display “r” for red in the corner of the rectangle that the program draws around any red area it finds. The same thing was done for the second loop, but using “g” for green instead. To display the coordinates of the color area to the serial terminal, “print(“r:” + str(blob.rect()))”  was added under “img.draw_rectangle” in the first loop and “print(“g:” + str(blob.rect()))” in the second. The coordinates are (x, y, w, h), with x and y being where the color is in the grame, and w and h being the width and height of the rectangle drawn around the color found relative to its size in the frame. This edited program was run on the created color tables, and the formula wg hgwr hrwas used to calculate the ratio of area of green versus area of red in each table. This was repeated three times and then compared to the real green versus red ratios calculated earlier.

RESULTS

Feature Detection

Fig. 1: Amplitude “a” of Waviest Sine Wave Detected by Camera as Straight Line for Various Recognition Thresholds

Fig. 2: “b” Value of Waviest Sine Wave Detected as Straight Line by Camera for Various Recognition Thresholds

Fig. 3: “a” Value of Least Circular Ellipse Detected as Circle by Camera for Various Recognition Thresholds

Fig. 4: “n” Value of Most Curved “Square” Detected as Rectangle by Camera for Various Recognition Thresholds

Figures 1–4 show how distorted a straight line, circle, or square can be before it stops being recognized as that shape or line at different threshold values. As the threshold value is increased, the program becomes more strict about what it considers to be a straight line, circle, or square.

Color Tracking

Fig. 5: Real Versus Experimental Green/Red Ratios

Figure 5 shows the real ratio of green to red in each color table compared to the experimental color ratio calculated using the results from the edited program. While each of the three tests follows the same trend as the real ratios, there is a fair amount of deviation, with an average percent error around 34%.

DISCUSSION

The purpose of this study was to evaluate the OpenMV H7 camera’s potential uses in a laboratory setting, specifically in regards to shape and color recognition for environmental monitoring. The data suggests that the camera is very effective for identifying shapes and lines. The threshold values in the programs used may also be changed based on how strict a researcher wants the camera to be, meaning that it is useful both for finding shapes that are approximately circular or rectangular or somewhat straight lines, as well as perfect circles and rectangles and completely straight lines, depending on the needs of the study

The camera also seems to be useful for identifying colors, given that the color can be seen first, in order to find the six-number threshold. Given the high percent error of the color ratio tests, it cannot be said that this camera can be used for advanced studies requiring precise measurements of color ratios, but it could be useful for those which only need more approximate measurements, as the results of the test did follow the same trend as the real ratios. This method is also limited in that it is currently unable to find the exact area of colors, only ratios of the areas on one color to another, as the coordinate system used for measurements is based on where the color is and how big the object is relative to the frame, meaning that an object farther away from the camera will be measured as smaller than one more close up. The program is also only able to draw rectangles around colors and give the coordinates of that box, rather than provide a more precise outline of the color. So, for areas of color that are not approximately rectangular, there is a large potential for error in area measurements.

This study was limited in that no factor was included for the time and proximity that the camera required to identify different versions of the shape or line at different thresholds. For example, when the camera was set to lower thresholds, it would often be able to identify all shapes or lines for several low thresholds, but varied in the amount of time it took to recognize the shapes and lines and how close it had to be to do so. Given time constraints, this was not accounted for in the results. Throughout the study, the camera also had a tendency to randomly disconnect from the OpenMV IDE, which could become problematic if the camera was used in more advanced studies. However, it is overall relatively easy to use in that it doesn’t require a lot of training or advanced safety techniques, which suggests that it may be a good tool for labs such as NHAS that work with young people and inexperienced researchers.

Paper References

George, D. (2018). MicroPython. Python for microcontrollers. Retrieved September 18, 2021, from https://micropython.org/.

George, D. P., Sokolovsky, P., & OpenMV LLC. (2021, June 27). 1. overview. 1. Overview - MicroPython 1.15 documentation. Retrieved September 18, 2021, from https://docs.openmv.io/openmvcam/tutorial/overview.html.

George, D. P., Sokolovsky, P., & OpenMV LLC. (2021, June 27). 2. software setup. 2. Software Setup - MicroPython 1.15 documentation. Retrieved September 18, 2021, from https://docs.openmv.io/openmvcam/tutorial/software_setup.html#linux-raspberrypi.

OpenMV LLC. (2019). OpenMV cam H7. OpenMV. Retrieved September 18, 2021, from https://openmv.io/products/openmv-cam-h7.

Slideshow References

https://openmv.io/collections/cams/products/openmv-cam-h7

https://docs.openmv.io/openmvcam/tutorial/index.html

https://docs.openmv.io/library/index.html#libraries-specific-to-the-openmv-cam

https://www.w3schools.com/python/default.asp

https://www.codecademy.com/learn/learn-python-3

I first became involved with research in 7th and 8th grade, when I joined the science club at my middle school, Crossroads Academy. This was a wonderful opportunity, as our science club also happened to be the site of the New Hampshire Academy of Science, run by Dr. Peter Faletra, who was also our science teacher at the time. After learning basic research skills, my research partner, Eleanor Press, and I created a project exploring the effects of BPA on Daphnia magna, and eventually presented our research at the AJAS annual conference in Austin, Texas.

After I graduated 8th grade and began at St. Johnsbury Academy, I stopped doing research for many reasons. St. Johnsbury was far away from the NHAS lab, I didn't have enough time, and I'd discovered that I didn't even really like biology, which was the main focus of NHAS at the time. I continued to love science, especially physics, and took many science classes at my school, but I didn't return to research until the summer of 2021, before the start of my senior year.

I became involved in research again for two main reasons: 1) I wanted to explore my growing interest in engineering, and 2) I wanted something that would look really good on my college applications. So, my older sister (who had been an active member of NHAS for a long time) convinced my to return to NHAS, and it ended up being an amazing experience. It was fascinating to get to work on a real, hands-on engineering project, rather than just learning from afar in school. I also made several new friends and reunited with an old one (shout out to Anna Testorf, Bo Blackburn, Alex Low, and Thomas Glass! Those first three are also presenting their research, go check it out!). And yes, it did help me get into college :).

Photo and profile picture by Mariah Crabb Photography, LLC.