LeAF: Leveraging Convolutional Neural Networks for Plant Anomaly Detection and Classification for Farmers with Large Language Models for Natural Language Interaction

Overview

Farmers face numerous challenges in crop cultivation, particularly in monitoring and maintaining plant health. Plant anomalies such as pests, diseases, and weeds are critical indicators of poor plant health, leading to decreased crop yield. Over 40% of global crop production is lost to plant anomalies, costing $220 billion annually. As global food demand rises, manual surveillance for plant anomalies becomes increasingly difficult, resulting in excessive and indiscriminate use of fertilizers and pesticides. This not only escalates costs but also heightens environmental/consumer concerns due to chemical runoff, emissions, and residues. The lack of data on plant anomalies further compounds the issue, leaving farmers unaware of the efficacy of their practices. Leveraging recent advancements in multimodal Artificial Intelligence (AI) with Convolutional Neural Networks (CNNs) and Large Language Models (LLMs), I propose LeAF - a comprehensive system to survey crops in real-time. LeAF aims to achieve six objectives: (1) utilizing CNNs to analyze robot camera feeds for plant anomalies with bounding box detection and classification, (2) employing plant stem identification to attribut e anomaly data to specific plants and create field maps, (3) integrating a domain -specific LLM to provide farmers with optimal treatment suggestions, (4) supporting question-answering on farming techniques, (5) offering estimates on strategy -effectiveness, cost-savings, and environmental impact reduction, and (6) deploying a custom -made BRANCH robot (Budget -friendly Robot for Agricultural Nonintrusive Crop Photography) at local farms that costs under $500. This end -to- end solution addresses the challenges f aced by farmers, empowering them with actionable insights to enhance crop management efficiency while minimizing environmental impact. Serum Bilirubin Prediction for Neonates using Segmentation-Guided Neural Networks Om Shah Lakeside School, Seattle, WA Research Advisor: Michael Koenig, Seattle University Across the world, millions of newborns suffer from severe neonatal jaundice, a condition that can cause neurological damage and death. Approaches such as laboratory blood tests and transcutaneous bilirubinometers for assessing jaundice are financially inac cessible in developing countries, and current computer vision approaches suffer from ease -of-use issues. Furthermore, spectroscopy -based devices produce inaccurate total serum bilirubin (TSB) estimates for neonates with darker skin tones. This research develops novel multi -stage deep learning models to predict bilirubin levels from smartphone imagery of blood plasma test strips. The first task involved training a segmentation model to segment the region of extracted bilirubin from the test strips. In the se cond task, color and environmental features from the segments are calculated as inputs into a deep regression neural network to predict TSB. The machine learning models are integrated into an end-to-end mobile application for real-world clinical use. The results indicate the segmentation model can adapt to rotational, scale, and ambient lighting irregularities in blood- based bilirubin extraction test strips and successfully segment regions with high bilirubin concentration. The second neural network predicts TSB levels that strongly correlate with state-of-art laboratory measurements (cross-validated Pearson R = 0.83). The mobile application provides bilirubin predictions with an error of 2.38 mg/dL in under five seconds. BiliNet offers significant ease -of-use advancements due to the use of inherently skin-tone agnostic blood plasma test strips. The strong clinical relevance is demonstrated by using predicted TSB to classify whether a neonate requires phototherapy with 95% accuracy. The inexpensive (<$1) system can enable widespread proliferation of neonatal jaundice screening in low-middle income countries and reduce fatalities.