基于AR眼镜采集的叶片图像上潜叶蝇为害等级自动诊断方法

doi:10.1016/j.jia.2025.02.008

摘要/Abstract

摘要：

植食性潜叶蝇对蔬菜和园艺植物造成严重的经济损失。为了准确科学防控潜叶蝇，调查人员一般通过目测植物叶片上潜叶蝇为害状的严重度来评估农药施用量。这种人工目测估计法客观性差，精准性低，数据难以追溯。为了客观精准诊断潜叶蝇为害等级，达到科学防控潜叶蝇目的，本文采用AR眼镜相机和AI算法结合的方法，建立了野外智能潜叶蝇为害调查系统。该系统利用穿戴式AR眼镜相机作为潜叶蝇为害叶片图像采集工具，便于调查人员利用双手展开受害叶片，来获得较平整的叶片图像。为了准确计算叶片上潜叶蝇为害区域，本文提出了DeepLab-Leafminer图像分割模型。针对潜叶蝇为害区域形状不规则，DeepLab-Leafminer在DeepLabv3+基础上添加了边缘感知模块和Canny损失函数，增强了对潜叶蝇为害状的边缘分割能力。与其他表现优越的图像分割模型相比，DeepLab-Leafminer模型达到了81.23%的IoU和87.92%的F₁ score，分割效果出色。根据模型分割出的潜叶蝇为害区域在整个叶片的面积占比计算得到的潜叶蝇为害等级，测试结果表明基于DeepLab-Leafminer的潜叶蝇为害等级诊断准确率为92.4%。我们设计并开发了潜叶蝇为害等级自动诊断APP和Web平台，实现了潜叶蝇为害等级诊断结果的可视化和数据分析。这种基于AR眼镜相机与AI模型DeepLab-Leafminer结合进行潜叶蝇为害等级自动诊断的方法为调查人员提供了高效、便捷、准确和数据可追溯的潜叶蝇为害等级自动诊断工具。此外，我们的方法也可以应用于其它作物叶片病虫为害等级自动诊断。

Abstract:

Agromyzid leafminers cause significant economic losses in both vegetable and horticultural crops, and precise assessments of pesticide needs must be based on the extent of leaf damage. Traditionally, surveyors estimate the damage by visually comparing the proportion of damaged to intact leaf area, a method that lacks objectivity, precision, and reliable data traceability. To address these issues, an advanced survey system that combines augmented reality (AR) glasses with a camera and an artificial intelligence (AI) algorithm was developed in this study to objectively and accurately assess leafminer damage in the field. By wearing AR glasses equipped with a voice-controlled camera, surveyors can easily flatten damaged leaves by hand and capture images for analysis. This method can provide a precise and reliable diagnosis of leafminer damage levels, which in turn supports the implementation of scientifically grounded and targeted pest management strategies. To calculate the leafminer damage level, the DeepLab-Leafminer model was proposed to precisely segment the leafminer-damaged regions and the intact leaf region. The integration of an edge-aware module and a Canny loss function into the DeepLabv3+ model enhanced the DeepLab-Leafminer model's capability to accurately segment the edges of leafminer-damaged regions, which often exhibit irregular shapes. Compared with state-of-the-art segmentation models, the DeepLab-Leafminer model achieved superior segmentation performance with an Intersection over Union (IoU) of 81.23% and an F₁ score of 87.92% on leafminer-damaged leaves. The test results revealed a 92.38% diagnosis accuracy of leafminer damage levels based on the DeepLab-Leafminer model. A mobile application and a web platform were developed to assist surveyors in displaying the diagnostic results of leafminer damage levels. This system provides surveyors with an advanced, user-friendly, and accurate tool for assessing agromyzid leafminer damage in agricultural fields using wearable AR glasses and an AI model. This method can also be utilized to automatically diagnose pest and disease damage levels in other crops based on leaf images.

Key words: agromyzid leafminer , plant leaf image , damage level , AR glasses , DeepLabv3+ model , image segmentation

. 基于AR眼镜采集的叶片图像上潜叶蝇为害等级自动诊断方法[J]. Journal of Integrative Agriculture, DOI: 10.1016/j.jia.2025.02.008.

Zhongru Ye, Yongjian Liu, Fuyu Ye, Hang Li, Ju Luo, Jianyang Guo, Zelin Feng, Chen Hong, Lingyi Li, Shuhua Liu, Baojun Yang, Wanxue Liu, Qing Yao. Automatic diagnosis of agromyzid leafminer damage levels using leaf images captured by AR glasses[J]. Journal of Integrative Agriculture, DOI: 10.1016/j.jia.2025.02.008.

参考文献

Badrinarayanan V, Kendall A, Cipolla R. 2017. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 2481–2495.

Bari B S, Islam M N, Rashid M, Hasan M J, Razman M A M, Musa R M, Ab Nasir A F, Majeed A P A. 2021. A real-time approach of diagnosing rice leaf disease using deep learning-based faster R-CNN framework. PeerJ Computer Science, 7, 432.

Borhani Y, Khoramdel J, Najafi E. 2022. A deep learning based approach for automated plant disease classification using vision transformer. Scientific Reports, 12, 11554.

Chen L C, Papandreou G, Kokkinos I, Murphy K, Yuille A L. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 834–848.

Chen S, Zhang K F, Zhao Y D, Sun Y Q, Ban W, Chen Y, Zhuang H F, Zhang X W, Liu J X, Yang T. 2021. An approach for rice bacterial leaf streak disease segmentation and disease severity estimation. Agriculture, 11, 420.

DB32/T 3714-2020. 2020. Technical Specification for investigation and forecast of Liriomyza trifolii (Burgess). Jiangsu Market Supervision and Administration Bureau, China. (in Chinese)

Fang W, Qian J, Wu Q, Chen Y,Yu G Z. 2017. ADAM-17 expression is enhanced by FoxM1 and is a poor prognostic sign in gastric carcinoma. Journal of Surgical Research, 220, 223–233.

Gao Y F, Yin F, Chen H, Chen X F, Deng H, Liu Y J, Li Z Y, Yao Q. 2025. Intelligent field monitoring system for cruciferous vegetable pests using yellow sticky trap images and an improved Cascade R-CNN. Journal of Integrative Agriculture, 24, 2-16.

Gao Y L, Reitz S, Xing Z L, Ferguson S, Lei Z. 2017. A decade of leafminer invasion in China: Lessons learned. Pest Management Science, 73, 1775–1779.

Goncalves J P, Pinto F A, Queiroz D M, Villar F M, Barbedo J G, Del Ponte E M. 2021. Deep learning architectures for semantic segmentation and automatic estimation of severity of foliar symptoms caused by diseases or pests. Biosystems Engineering, 210, 129–142.

Goyal A, Bochkovskiy A, Deng J, Koltun V. 2022. Non-deep networks. Advances in Neural Information Processing Systems, 35, 6789–6801.

Guo M H, Lu C Z, Hou Q, Liu Z N, Cheng M M,Hu S M. 2022. SegNeXt: Rethinking convolutional attention design for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 8575.

Han X T, Yang B J, Li S X, Liao F B, Liu S H, Tang J, Yao Q. 2022. Intelligent forecasting method of rice sheath blight based on images. Scientia Agricultura Sinica, 55, 1557–1567. (in Chinese)

Hatamizadeh A, Nath V, Tang Y, Yang D, Roth H R, Xu D. 2021. Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Springer, Cham, Switzerland. pp. 272–284.

He K, Gkioxari G, Dollár P, Girshick R. 2017. Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision. Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 2961–2969.

Hong Q H, Liu F M, Li D L, Liu J, Tian L, Shan Y. 2022. Dynamic sparse R-CNN. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 4713–4722.

Ji J, Lu X C, Luo M, Yin M H, Miao Q G, Liu X Z. 2021. Parallel fully convolutional network for semantic segmentation. IEEE Access, 9, 673–682.

Jin C L, Yuan J M, Shen J H, Wheeler C, Liu Z Y, Yan J H. 2024. A visual detection method for conveyor belt misalignment based on the improved YOLACT network. Particulate Science and Technology, 42, 541–552.

Li H, Liang Y X, Liu Y J, Xian X Q, Xue Y T, Huang H K, Yao Q, Liu W X. 2023. Development of an intelligent field investigation system for Liriomyza using SeResNet-Liriomyza for accurate identification. Computers and Electronics in Agriculture, 214, 108276.

Li K Y, Zhang L X, Li B, Li S F, Ma J C. 2022. Attention-optimized DeepLab V3+ for automatic estimation of cucumber disease severity. Plant Methods, 18, 109.

Li S X, Feng Z L, Yang B J, Li H, Gao Y F , Liu S H , Tang J, Yao Q. 2022. An intelligent monitoring system of diseases and pests on rice canopy. Frontiers in Plant Science, 13, 972286.

Liang Q K, Xiang S, Hu Y C, Coppola G, Zhang D, Sun W. 2019. PD2SE-Net: Computer-assisted plant disease diagnosis and severity estimation network. Computers and Electronics in Agriculture, 157, 518–529.

Liao F B, Feng X Q, Li Z Q, Wang D Y, Xu C M, Chu G, Ma H Y, Yao Q, Chen S. 2024. A hybrid CNN-LSTM model for diagnosing rice nutrient levels at the rice panicle initiation stage. Journal of Integrative Agriculture, 23, 711-723.

Liu B, Zhang Y, He D J, Li Y. 2018. Identification of apple leaf diseases based on deep convolutional neural networks. Symmetry, 10, 11.

Liu H J, Liu F Q, Fan X Y, Huang D. 2022. Polarized self-attention: Towards high-quality pixel-wise mapping. Neurocomputing, 506, 158–167.

Ronneberger O, Fischer P, Brox T. 2015. U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015. Springer International Publishing, Germany. pp. 234–241.

Saleh S, Anshary A, Yunus M, Hasriyanty. 2019. Compatibility of trap cropping system and insecticides in managing leafminers Liriomyza spp. IOP Conference Series (Earth and Environmental Science), 468, 12002.

Sheng H Y, Yao Q, Luo J, Liu Y J, Chen X F, Ye Z R, Zhao T Z, Ling H P, Tang J, Liu S H. 2024. Automatic detection and counting of planthoppers on white flat plate images captured by AR glasses for planthopper field survey. Computers and Electronics in Agriculture, 218, 108639.

Singh R P, Dixit M. 2015. Histogram equalization: A strong technique for image enhancement. International Journal of Signal Processing, Image Processing and Pattern Recognition, 8, 345–352.

Spencer K A. 2012. Divisions equisetophyta and polypodiophyta. In: Spencer K A, ed., Host Specialization in the World Agromyzidae (Diptera). Springer, Dordrecht, Netherlands. pp. 4–14.

Spencer K A. 2013. Leguminous crops 2: Leaf-miners. In: Spencer K A, ed., Agromyzidae (Diptera) of Economic Importance. Springer, Dordrecht, Netherlands. pp. 129–179.

Sun G J, Liu S H, Luo H L, Feng Z L, Yang B J, Luo J, Tang J, Yao Q, Xu J J. 2022. Intelligent monitoring system of migratory pests based on searchlight trap and machine vision. Frontiers in Plant Science, 13, 897739.

Sunil C, Jaidhar C, Patil N. 2021. Cardamom plant disease detection approach using EfficientNetV2. Ieee Access, 10, 789–804.

Wang C S, Du P F, Wu H R, Li J X, Zhao C J, Zhu H. 2021. A cucumber leaf disease severity classification method based on the fusion of DeepLabV3+ and U-Net. Computers and Electronics in Agriculture, 189, 106373.

Wang L, Chen X X, Hu L Y, Li H. 2020. Overview of image semantic segmentation technology. In: IEEE Joint International Information Technology and Artificial Intelligence Conference. Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 19–26.

Xu J J, Feng Z L, Tang J, Liu S H, Ding Z P, Lyu J, Yao Q, Yang B J. 2022. Improved Random Forest for the Automatic Identification of Spodoptera frugiperda Larval Instar Stages. Agriculture, 12, 1919.

Yan B, Li Y, Li L, Yang X C, Li T Q, Yang G, Jiang M F. 2022. Quantifying the impact of Pyramid Squeeze Attention mechanism and filtering approaches on Alzheimer's disease classification. Computers in Biology and Medicine, 148, 105944.

Yao N, Ni F C, Wu M C, Wang H Y, Li G L, Sung W K. 2022. Deep learning-based segmentation of peach diseases using convolutional neural network. Frontiers in Plant Science, 13, 876357.

Yao Q, Feng J, T J, Xu W G, Zhu H X, Yang B J. 2020. Development of an automatic monitoring system for rice light-trap pests based on machine vision. Journal of Integrative Agriculture, 19, 2500–2513.

Yuan H B, Zhu J J, Wang Q F, Cheng M, Cai Z J. 2022. An improved DeepLab v3+ deep learning network applied to the segmentation of grape leaf black rot spots. Frontiers in Plant Science, 13, 795410.

Yuan Y H, Chen X L, Wang J D. 2020. Object-contextual representations for semantic segmentation. In: Computer Vision-ECCV 2020. Springer International Publishing, Germany. pp. 173–190.

Zhang P, Yang L, Li D L. 2020. EfficientNet-B4-Ranger: A novel method for greenhouse cucumber disease recognition under natural complex environment. Computers and Electronics in Agriculture, 176, 105652.

Zhang T X, Xu Z Y, Su J Y, Yang Z F, Liu C J, Chen W H, Li J. 2021. Ir-unet: Irregular segmentation u-shape network for wheat yellow rust detection by UAV multispectral imagery. Remote Sensing, 13, 3892.

Zhang X R, Xing Z L, Lei Z R, Gao Y L. 2017. Recent status of the invasive leafminer liriomyza trifolii1 in China. Southwestern Entomologist, 42, 301–304.

Zhao H S, Shi J P, Qi X J, Wang X, Jia J Y. 2017. Pyramid scene parsing network. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 6230–6239.