Aerial Machine Vision has the potential to create useful maps and data for fragile ecosystems.

Eli Cowie is a senior at Hanover High School, graduating Spring 2022. He initially started working with the NHAS in the fall of 2020, initially pursuing engineering projects involving interferometry in the after school program. He started this project in the summer of 2021, aided by his interest in machine learning and computer vision. In college he hopes to continue applying these skills to other aspects of his studies in environmental science and theatre.

In the Strafford Fen, there are two plants that were the subject of research, Showy Lady’s Slipper Orchids (Cypripedium reginae) and Narrow Leaved Cattails (Typha angustifolia.) These orchids are endangered in New Hampshire, due to habitat loss and other factors. In New England as a whole, the cattails are a noninvasive plant, and are common. However, they have been growing at a rapidly increasing rate over the past couple years in the Strafford fen, leading to the question ‘How could one track the growth of these plants?’ Creating a labeled map of the cattails and the orchids would enable seeing where they grow, how well they grow in certain areas, and how they interact with the other species. This approach could be applicable to natural areas anywhere, though the specific training would only be applicable to this fen. The problem with creating a map by hand is that the scale would be such that it would take a large amount of time and resources. Additionally, most of the fen is inaccessible to foot traffic, confining visitors to boardwalks that cross the fen. Thus, this project required drone photography and image classification. Image classification is a type of machine learning. Data comes in the form of images, is processed for training through the use of filters, annotated, and analyzed for patterns and features that help to classify those images into their labels. The program changes its parameters over time, and thus hopefully gets better at classifying.

Methods

The Mavic 2 Pro drone (DJI) was used because of its automatic wind adjustment, gimbal camera, and high picture quality. The drone flew rows up and down the fen while taking 30 fps video, which required three batteries. To recharge, it flew to the landing area, had its batteries switched, then flew back to roughly the same spot. This step had to be done manually, but due to the downward facing camera it was not a significant source of error. The drone’s height varied slightly over the course of the flight due to changes in terrain elevation. The video was broken down into its component frames, which were saved individually.

Makesense.ai was used for human annotation of the images. Drone footage resulted in around 1000 images, of which ~150 were discarded because they were unfit to use. Some were blurry, some were taken too high to permit accurate annotation, and some were not orthogonally angled shots. The 861 frames that remained were scanned for cattails, orchids, people, and boardwalks. If an object of interest was found, a rectangle was drawn around it, fitting as closely as possible to the object. The annotations were all exported in a .csv file, and saved in a folder containing all the images.

Google Colab pro was used to write the program, as well as adapt code from other programs and tutorials to this application. First the code mounts the Google Drive folder containing the frames and .csv. Then it cuts the images into random 512 by 512 pixel squares that are then labeled by their majority constituent objects (tagged in the human annotation.)

Keras library and Tensorflow were used to develop the model. There were 2826 training squares, and 341 validation squares. It used 3 convolutions, thus 3 pooling layers. The first convolution extracted 16 3x3 layers, the second extracted 32, and the third extracted 64. After every convolution, it ran a 2x2 pooling layer. Finally, it ran a dropout layer to combat overfitting. It ran 30 epochs of training.

Results and Discussion

One possible reason for the low accuracy of both experiments could be the human annotation errors. It’s probable the annotations were not representative of the actual samples and thus difficult to classify in a way that would be applicable. If the 512 by 512 square did not reflect its label, that could throw off the training completely. A reason for the overfit (the difference between the training accuracy and the validation accuracy) could be the small sample size. When there isn’t enough image data, the chances of a random similarity in the training group causing the program to over specify that feature increase sharply. It then cannot generalize when fed new data, which is an ever present problem in machine learning. Future iterations of this project will have to take into account that accurate annotation is very important to getting good patterns and training. Also, more frames (and more human resources spent processing those frames) is a crucial part of getting consistently positive results. In the second experiment, even though the data was more representative of their labels, there simply was not enough of it to get consistent results. Every time the training was run with a random training/validation split, the validation accuracy changed by up to 0.0400, because the small training data set was insufficiently detailed/extensive to allow the model to find a consistent filter that would generalize to the smaller held-back validation set. This project produced a working model limited by overfitting, and with more aerial data and better labeling could in the future yield useful maps and data about this fragile ecosystem.