IoT Assisted Multimodal Deep Learning Framework for Real-Time Onion Spoilage Detection

Abstract

Onion storage suffers from major challenges due to post-harvest losses (PHLs) in agricultural supply chains, especially in developing countries where continuous monitoring of surrounding conditions is limited. Standard storage monitoring systems are typically based on traditional static threshold rules (e.g., an excess of 60% relative humidity), and as such, do not have intelligent capabilities to make decisions. To assist with this gap in the current available options for storage monitoring, this study presents a multimodal Internet of Things (IoT) system combined with deep learning for real-time detection of onion spoilage. This unique approach uses environmental data from Internet of Things (IoT) sensors connected via an ESP32 microcontroller to measure temperature, humidity, moisture content, and gas concentration. It also takes images of the onions and uses Convolutional Neural Networks (CNNs) to classify the photos of the onions as they spoil. The application of Grad-CAM visualization of the features contributing to image classification will also assist with the interpretability of the developed models. The evaluation of the proposed framework employed a dataset of 2400 onion images categorized into four classifications: healthy, sprouted, spoiled and with black mold. Precision, recall and F1-score were 91.8%, 93.2% and 92.4%, respectively, for each of the four classifications using the proposed multimodal IoT and CNN framework. Overall, the system achieved an end-to-end latency of approximately 1.3 seconds; therefore, providing almost real-time monitoring of stored onions.