Automatic diagnosis of agromyzid leafminer damage levels using leaf images captured by AR glasses

doi:10.1016/j.jia.2025.02.008

Advanced Online Publication | Current Issue | Archive | Adv Search

Zhongru Ye, Yongjian Liu², Fuyu Ye³, Hang Li², Ju Luo⁴, Jianyang Guo³, Zelin Feng⁵, Chen Hong², Lingyi Li², Shuhua Liu⁴, Baojun Yang⁴, Wanxue Liu³^#, Qing Yao²^#

¹School of Information Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, China

²School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, China

³State Key Laboratory for Biology of Plant Diseases and Insect Pests, Institute of Plant Protection, Chinese Academy of Agricultural Sciences, Beijing 100193, China

⁴ State Key Laboratory of Rice Biology and Breeding, China National Rice Research Institute, Hangzhou 311401, China

⁵ School of Information and Control, Keyi College of Zhejiang Sci-Tech University, Hangzhou 310018, China

Highlights

l An automatic diagnosis system based on wearable AR glasses and an AI model was developed to assess leafminer damage levels, and it achieved 92.38% accuracy.

l The DeepLab-Leafminer model incorporated an edge-aware module and the Canny loss function into the DeepLabv3+ model, which enhanced its ability to segment the leafminer damaged area in leaves.

l A mobile application and a web platform were developed to display the diagnostic results of leafminer damage levels for surveyors to guide their scientific decisions for leafminer prevention and control.

Abstract
References

Download: PDF in ScienceDirect
Export: BibTeX | EndNote (RIS)

摘要

植食性潜叶蝇对蔬菜和园艺植物造成严重的经济损失。为了准确科学防控潜叶蝇，调查人员一般通过目测植物叶片上潜叶蝇为害状的严重度来评估农药施用量。这种人工目测估计法客观性差，精准性低，数据难以追溯。为了客观精准诊断潜叶蝇为害等级，达到科学防控潜叶蝇目的，本文采用AR眼镜相机和AI算法结合的方法，建立了野外智能潜叶蝇为害调查系统。该系统利用穿戴式AR眼镜相机作为潜叶蝇为害叶片图像采集工具，便于调查人员利用双手展开受害叶片，来获得较平整的叶片图像。为了准确计算叶片上潜叶蝇为害区域，本文提出了DeepLab-Leafminer图像分割模型。针对潜叶蝇为害区域形状不规则，DeepLab-Leafminer在DeepLabv3+基础上添加了边缘感知模块和Canny损失函数，增强了对潜叶蝇为害状的边缘分割能力。与其他表现优越的图像分割模型相比，DeepLab-Leafminer模型达到了81.23%的IoU和87.92%的F₁ score，分割效果出色。根据模型分割出的潜叶蝇为害区域在整个叶片的面积占比计算得到的潜叶蝇为害等级，测试结果表明基于DeepLab-Leafminer的潜叶蝇为害等级诊断准确率为92.4%。我们设计并开发了潜叶蝇为害等级自动诊断APP和Web平台，实现了潜叶蝇为害等级诊断结果的可视化和数据分析。这种基于AR眼镜相机与AI模型DeepLab-Leafminer结合进行潜叶蝇为害等级自动诊断的方法为调查人员提供了高效、便捷、准确和数据可追溯的潜叶蝇为害等级自动诊断工具。此外，我们的方法也可以应用于其它作物叶片病虫为害等级自动诊断。

Abstract

Agromyzid leafminers cause significant economic losses in both vegetable and horticultural crops, and precise assessments of pesticide needs must be based on the extent of leaf damage. Traditionally, surveyors estimate the damage by visually comparing the proportion of damaged to intact leaf area, a method that lacks objectivity, precision, and reliable data traceability. To address these issues, an advanced survey system that combines augmented reality (AR) glasses with a camera and an artificial intelligence (AI) algorithm was developed in this study to objectively and accurately assess leafminer damage in the field. By wearing AR glasses equipped with a voice-controlled camera, surveyors can easily flatten damaged leaves by hand and capture images for analysis. This method can provide a precise and reliable diagnosis of leafminer damage levels, which in turn supports the implementation of scientifically grounded and targeted pest management strategies. To calculate the leafminer damage level, the DeepLab-Leafminer model was proposed to precisely segment the leafminer-damaged regions and the intact leaf region. The integration of an edge-aware module and a Canny loss function into the DeepLabv3+ model enhanced the DeepLab-Leafminer model's capability to accurately segment the edges of leafminer-damaged regions, which often exhibit irregular shapes. Compared with state-of-the-art segmentation models, the DeepLab-Leafminer model achieved superior segmentation performance with an Intersection over Union (IoU) of 81.23% and an F₁ score of 87.92% on leafminer-damaged leaves. The test results revealed a 92.38% diagnosis accuracy of leafminer damage levels based on the DeepLab-Leafminer model. A mobile application and a web platform were developed to assist surveyors in displaying the diagnostic results of leafminer damage levels. This system provides surveyors with an advanced, user-friendly, and accurate tool for assessing agromyzid leafminer damage in agricultural fields using wearable AR glasses and an AI model. This method can also be utilized to automatically diagnose pest and disease damage levels in other crops based on leaf images.

Keywords: agromyzid leafminer plant leaf image damage level AR glasses DeepLabv3+ model image segmentation

Online: 10 February 2025

Fund:

This work was supported by the National Key R&D Program of China (2021YFC2600400 and 2023YFC2605200), the National Key Research Program of China (2021YFD1401100), and the “San Nong Jiu Fang” Sciences and Technologies Cooperation Project of Zhejiang Province, China (2024SNJF010).

About author: Zhongru Ye, Mobile: +86-13735927281, E-mail: zhongruYE@163.com; #Correspondence Qing Yao, E-mail: q-yao@zstu.edu.cn; Wanxue Liu, E-mail: liuwanxue@caas.cn

	Service
	E-mail this article
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors

Cite this article:

Zhongru Ye, Yongjian Liu, Fuyu Ye, Hang Li, Ju Luo, Jianyang Guo, Zelin Feng, Chen Hong, Lingyi Li, Shuhua Liu, Baojun Yang, Wanxue Liu, Qing Yao. 2025. Automatic diagnosis of agromyzid leafminer damage levels using leaf images captured by AR glasses. Journal of Integrative Agriculture, Doi:10.1016/j.jia.2025.02.008

Badrinarayanan V, Kendall A, Cipolla R. 2017. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 2481–2495.

Bari B S, Islam M N, Rashid M, Hasan M J, Razman M A M, Musa R M, Ab Nasir A F, Majeed A P A. 2021. A real-time approach of diagnosing rice leaf disease using deep learning-based faster R-CNN framework. PeerJ Computer Science, 7, 432.

Borhani Y, Khoramdel J, Najafi E. 2022. A deep learning based approach for automated plant disease classification using vision transformer. Scientific Reports, 12, 11554.

Chen L C, Papandreou G, Kokkinos I, Murphy K, Yuille A L. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 834–848.

Chen S, Zhang K F, Zhao Y D, Sun Y Q, Ban W, Chen Y, Zhuang H F, Zhang X W, Liu J X, Yang T. 2021. An approach for rice bacterial leaf streak disease segmentation and disease severity estimation. Agriculture, 11, 420.

DB32/T 3714-2020. 2020. Technical Specification for investigation and forecast of Liriomyza trifolii (Burgess). Jiangsu Market Supervision and Administration Bureau, China. (in Chinese)

Fang W, Qian J, Wu Q, Chen Y,Yu G Z. 2017. ADAM-17 expression is enhanced by FoxM1 and is a poor prognostic sign in gastric carcinoma. Journal of Surgical Research, 220, 223–233.

Gao Y F, Yin F, Chen H, Chen X F, Deng H, Liu Y J, Li Z Y, Yao Q. 2025. Intelligent field monitoring system for cruciferous vegetable pests using yellow sticky trap images and an improved Cascade R-CNN. Journal of Integrative Agriculture, 24, 2-16.

Gao Y L, Reitz S, Xing Z L, Ferguson S, Lei Z. 2017. A decade of leafminer invasion in China: Lessons learned. Pest Management Science, 73, 1775–1779.

Goncalves J P, Pinto F A, Queiroz D M, Villar F M, Barbedo J G, Del Ponte E M. 2021. Deep learning architectures for semantic segmentation and automatic estimation of severity of foliar symptoms caused by diseases or pests. Biosystems Engineering, 210, 129–142.

Goyal A, Bochkovskiy A, Deng J, Koltun V. 2022. Non-deep networks. Advances in Neural Information Processing Systems, 35, 6789–6801.

Guo M H, Lu C Z, Hou Q, Liu Z N, Cheng M M,Hu S M. 2022. SegNeXt: Rethinking convolutional attention design for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 8575.

Han X T, Yang B J, Li S X, Liao F B, Liu S H, Tang J, Yao Q. 2022. Intelligent forecasting method of rice sheath blight based on images. Scientia Agricultura Sinica, 55, 1557–1567. (in Chinese)

Hatamizadeh A, Nath V, Tang Y, Yang D, Roth H R, Xu D. 2021. Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Springer, Cham, Switzerland. pp. 272–284.

He K, Gkioxari G, Dollár P, Girshick R. 2017. Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision. Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 2961–2969.

Hong Q H, Liu F M, Li D L, Liu J, Tian L, Shan Y. 2022. Dynamic sparse R-CNN. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 4713–4722.

Ji J, Lu X C, Luo M, Yin M H, Miao Q G, Liu X Z. 2021. Parallel fully convolutional network for semantic segmentation. IEEE Access, 9, 673–682.

Jin C L, Yuan J M, Shen J H, Wheeler C, Liu Z Y, Yan J H. 2024. A visual detection method for conveyor belt misalignment based on the improved YOLACT network. Particulate Science and Technology, 42, 541–552.

Li H, Liang Y X, Liu Y J, Xian X Q, Xue Y T, Huang H K, Yao Q, Liu W X. 2023. Development of an intelligent field investigation system for Liriomyza using SeResNet-Liriomyza for accurate identification. Computers and Electronics in Agriculture, 214, 108276.

Li K Y, Zhang L X, Li B, Li S F, Ma J C. 2022. Attention-optimized DeepLab V3+ for automatic estimation of cucumber disease severity. Plant Methods, 18, 109.

Li S X, Feng Z L, Yang B J, Li H, Gao Y F , Liu S H , Tang J, Yao Q. 2022. An intelligent monitoring system of diseases and pests on rice canopy. Frontiers in Plant Science, 13, 972286.

Liang Q K, Xiang S, Hu Y C, Coppola G, Zhang D, Sun W. 2019. PD2SE-Net: Computer-assisted plant disease diagnosis and severity estimation network. Computers and Electronics in Agriculture, 157, 518–529.

Liao F B, Feng X Q, Li Z Q, Wang D Y, Xu C M, Chu G, Ma H Y, Yao Q, Chen S. 2024. A hybrid CNN-LSTM model for diagnosing rice nutrient levels at the rice panicle initiation stage. Journal of Integrative Agriculture, 23, 711-723.

Liu B, Zhang Y, He D J, Li Y. 2018. Identification of apple leaf diseases based on deep convolutional neural networks. Symmetry, 10, 11.

Liu H J, Liu F Q, Fan X Y, Huang D. 2022. Polarized self-attention: Towards high-quality pixel-wise mapping. Neurocomputing, 506, 158–167.

Ronneberger O, Fischer P, Brox T. 2015. U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015. Springer International Publishing, Germany. pp. 234–241.

Saleh S, Anshary A, Yunus M, Hasriyanty. 2019. Compatibility of trap cropping system and insecticides in managing leafminers Liriomyza spp. IOP Conference Series (Earth and Environmental Science), 468, 12002.

Sheng H Y, Yao Q, Luo J, Liu Y J, Chen X F, Ye Z R, Zhao T Z, Ling H P, Tang J, Liu S H. 2024. Automatic detection and counting of planthoppers on white flat plate images captured by AR glasses for planthopper field survey. Computers and Electronics in Agriculture, 218, 108639.

Singh R P, Dixit M. 2015. Histogram equalization: A strong technique for image enhancement. International Journal of Signal Processing, Image Processing and Pattern Recognition, 8, 345–352.

Spencer K A. 2012. Divisions equisetophyta and polypodiophyta. In: Spencer K A, ed., Host Specialization in the World Agromyzidae (Diptera). Springer, Dordrecht, Netherlands. pp. 4–14.

Spencer K A. 2013. Leguminous crops 2: Leaf-miners. In: Spencer K A, ed., Agromyzidae (Diptera) of Economic Importance. Springer, Dordrecht, Netherlands. pp. 129–179.

Sun G J, Liu S H, Luo H L, Feng Z L, Yang B J, Luo J, Tang J, Yao Q, Xu J J. 2022. Intelligent monitoring system of migratory pests based on searchlight trap and machine vision. Frontiers in Plant Science, 13, 897739.

Sunil C, Jaidhar C, Patil N. 2021. Cardamom plant disease detection approach using EfficientNetV2. Ieee Access, 10, 789–804.

Wang C S, Du P F, Wu H R, Li J X, Zhao C J, Zhu H. 2021. A cucumber leaf disease severity classification method based on the fusion of DeepLabV3+ and U-Net. Computers and Electronics in Agriculture, 189, 106373.

Wang L, Chen X X, Hu L Y, Li H. 2020. Overview of image semantic segmentation technology. In: IEEE Joint International Information Technology and Artificial Intelligence Conference. Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 19–26.

Xu J J, Feng Z L, Tang J, Liu S H, Ding Z P, Lyu J, Yao Q, Yang B J. 2022. Improved Random Forest for the Automatic Identification of Spodoptera frugiperda Larval Instar Stages. Agriculture, 12, 1919.

Yan B, Li Y, Li L, Yang X C, Li T Q, Yang G, Jiang M F. 2022. Quantifying the impact of Pyramid Squeeze Attention mechanism and filtering approaches on Alzheimer's disease classification. Computers in Biology and Medicine, 148, 105944.

Yao N, Ni F C, Wu M C, Wang H Y, Li G L, Sung W K. 2022. Deep learning-based segmentation of peach diseases using convolutional neural network. Frontiers in Plant Science, 13, 876357.

Yao Q, Feng J, T J, Xu W G, Zhu H X, Yang B J. 2020. Development of an automatic monitoring system for rice light-trap pests based on machine vision. Journal of Integrative Agriculture, 19, 2500–2513.

Yuan H B, Zhu J J, Wang Q F, Cheng M, Cai Z J. 2022. An improved DeepLab v3+ deep learning network applied to the segmentation of grape leaf black rot spots. Frontiers in Plant Science, 13, 795410.

Yuan Y H, Chen X L, Wang J D. 2020. Object-contextual representations for semantic segmentation. In: Computer Vision-ECCV 2020. Springer International Publishing, Germany. pp. 173–190.

Zhang P, Yang L, Li D L. 2020. EfficientNet-B4-Ranger: A novel method for greenhouse cucumber disease recognition under natural complex environment. Computers and Electronics in Agriculture, 176, 105652.

Zhang T X, Xu Z Y, Su J Y, Yang Z F, Liu C J, Chen W H, Li J. 2021. Ir-unet: Irregular segmentation u-shape network for wheat yellow rust detection by UAV multispectral imagery. Remote Sensing, 13, 3892.

Zhang X R, Xing Z L, Lei Z R, Gao Y L. 2017. Recent status of the invasive leafminer liriomyza trifolii1 in China. Southwestern Entomologist, 42, 301–304.

Zhao H S, Shi J P, Qi X J, Wang X, Jia J Y. 2017. Pyramid scene parsing network. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronic Engineers Computer Society, USA. pp. 6230–6239.

No related articles found!

No Suggested Reading articles found!

Viewed

Full text

Abstract

Cited

Shared

Discussed

Cite this article:

About JIA

Editorial board

For authors