基于三重并行注意力的改进多尺度逆瓶颈残差网络苹果叶片病害识别

doi:10.1016/j.jia.2023.06.023

摘要/Abstract

摘要：

准确诊断苹果叶片病害对于提高苹果产量质量，促进苹果产业发展有着重要意义。但是不同苹果叶片病害在图像纹理和结构信息上差异不明显，且在复杂背景下病害特征提取存在困难，使得此类研究进展缓慢。针对上述问题，本文在ResNet-50的基础上，提出了一种基于三重并行注意机制的改进型多尺度逆瓶颈残差网络模型，同时改进并结合了Inception模块和ResNext逆瓶颈块，实现了七类苹果叶片识别（包括Alternaria_Boltch，Brown_Spot，Grey_spot, Mosaic，Rust, Scab六种病害和一种Healthy）。首先，用多尺度残差卷积代替部分残差模块中的3×3卷积，多尺度卷积的每个分支所包含的不同大小卷积核都应用于提取一部分特征图，最后这些卷积分支的输出通过相加进行多尺度融合，以丰富图像的输出特征。其次，采用全局逆瓶颈结构来减少网络特征损失，逆瓶颈结构让图像信息在不同维度特征空间之间转换时损失更小。多尺度和逆瓶颈的融合能够让模型在精细识别苹果叶片病害和面对复杂识别背景下以横向和纵向相结合的特点获得更高的鲁棒性。最后，在每一个改进的模块后，集成了三重并行注意力模块，通过旋转和残差变换实现通道间的跨维互动，在维度依赖性得到改善的同时，以相对较小的计算成本提高了重要特征的并行搜索效率和网络的识别率。为了验证本文模型的有效性，我们对从植物村、百度飞桨和互联网的公共数据集中筛选出来的苹果叶病图像进行了统一增强。最终处理后的图像数量为14000张。对处理后的数据集进行了消融研究、预处理比较和方法比较。实验结果表明，所提出的方法在所采用的数据集上达到98.73%的准确率，比经典的ResNet-50模型高1.82%，比预处理前的苹果叶病数据集好0.29%。与一些最先进的方法相比，它在苹果叶病识别方面也取得了有竞争力的结果。

Abstract: Accurate diagnosis of apple leaf diseases is crucial for improving the quality of apple production and promoting the development of the apple industry. However, apple leaf diseases do not differ significantly from image texture and structural information. The difficulties in disease feature extraction in complex backgrounds slow the related research progress. To address the problems, this paper proposes an improved multi-scale inverse bottleneck residual network model based on a triplet parallel attention mechanism, which is built upon ResNet-50, while improving and combining the inception module and ResNext inverse bottleneck blocks, to recognize seven types of apple leaf (including six diseases of alternaria leaf spot, brown spot, grey spot, mosaic, rust, scab, and one healthy). First, the 3×3 convolutions in some of the residual modules are replaced by multi-scale residual convolutions, the convolution kernels of different sizes contained in each branch of the multi-scale convolution are applied to extract feature maps of different sizes, and the outputs of these branches are multi-scale fused by summing to enrich the output features of the images. Second, the global layer-wise dynamic coordinated inverse bottleneck structure is used to reduce the network feature loss. The inverse bottleneck structure makes the image information less lossy when transforming from different dimensional feature spaces. The fusion of multi-scale and layer-wise dynamic coordinated inverse bottlenecks makes the model effectively balances computational efficiency and feature representation capability, and more robust with a combination of horizontal and vertical features in the fine identification of apple leaf diseases. Finally, after each improved module, a triplet parallel attention module is integrated with cross-dimensional interactions among channels through rotations and residual transformations, which improves the parallel search efficiency of important features and the recognition rate of the network with relatively small computational costs while the dimensional dependencies are improved. To verify the validity of the model in this paper, we uniformly enhance apple leaf disease images screened from the public data sets of Plant Village, Baidu Flying Paddle, and the Internet. The final processed image count is 14,000. The ablation study, pre-processing comparison, and method comparison are conducted on the processed datasets. The experimental results demonstrate that the proposed method reaches 98.73% accuracy on the adopted datasets, which is 1.82% higher than the classical ResNet-50 model, and 0.29% better than the apple leaf disease datasets before preprocessing. It also achieves competitive results in apple leaf disease identification compared to some state-of-the-art methods.

Key words: multi-scale module , inverse bottleneck structure , triplet parallel attention , apple leaf disease

. 基于三重并行注意力的改进多尺度逆瓶颈残差网络苹果叶片病害识别[J]. Journal of Integrative Agriculture, 2024, 23(3): 901-922.

Lei Tang, Jizheng Yi, Xiaoyao Li.

Improved multi-scale inverse bottleneck residual network based on triplet parallel attention for apple leaf disease identification [J]. Journal of Integrative Agriculture, 2024, 23(3): 901-922.

参考文献

Ahmed I, Din S, Jeon G, Piccialli F, Fortino G. 2021. Towards collaborative robotics in top view surveillance: A framework for multiple object tracking by detection using deep learning. IEEE/CAA Journal of Automatica Sinica, 8, 1253–1270.

Ai J Q, Mao Y X, Luo Q W, Jia L, Xing M D. 2022. SAR target classification using the multikernel-size feature fusion-based convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–13.

Cai G, Wang Y, He L, Zhou M. 2020. Unsupervised domain adaptation with adversarial residual transform networks. IEEE Transactions on Neural Networks and Learning Systems, 31, 3073–3086.

Chao X F, Hu X, Feng J Z, Zhang Z, Wang M L, He D J. 2021. Construction of apple leaf diseases identification networks based on xception fused by SE module. Applied Sciences, 11, 46–14.

Chao X F, Sun G Y, Zhao H K, Li M, He D J. 2020. Identification of apple tree leaf diseases based on deep learning models. Symmetry, 12, 17.

Chen H Y, Yi J Z, Chen A B, Peng D X, Yang J. 2023. Green technology innovation and CO2 emission in China: Evidence from a spatial-temporal analysis and a nonlinear spatial durbin model. Energy Policy, 172, 113338.

Chen Y P, Kalantidis Y, Li J S, Yan S C, Feng J S. 2018. A²-nets: Double attention networks. Advances in Neural Information Processing Systems, 31, 350–359.

El-Bendary N, Hariri E E, Hassanien A E, Badr A. 2015. Using machine learning techniques for evaluating tomato ripeness. Expert Systems with Applications, 42, 1892–1905.

Feng D, Ren F. 2021. Data-driven channel pruning towards local binary convolution inverse bottleneck network based on squeeze-and-excitation optimization Weights. Electronics, 10, 1329.

Gigot C, Turechek W, McRoberts N. 2017. Analysis of the spatial pattern of strawberry angular leaf spot in california nursery production. Phytopathology, 107, 1243–1255.

Gong Z Q, Zhong P, Yu Y, Hu W D, Li S T. 2019. A CNN with multiscale convolution and diversified metric for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 57, 3599–3618.

Gu J, Kwon H, Wang D, Ye W, Li M, Chen Y H, Lai L, Chandra V, Pan D Z. 2022. Multi-scale high-resolution vision transformer for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New Orleans, LA, USA. pp. 12084–12093.

Guan Z B, Zhang Y Q, Chai X J, Chai X, Zhang N, Zhang J H, Sun T. 2022. Visual learning graph convolution for multi-grained orange quality grading. Journal of Integrative Agriculture, 22, 279–291.

He K M, Zhang X Y, Ren S Q, Sun J. 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, NV, USA. pp. 770–778.

Howard A, Sandler M, Chu G C, Chen L C, Chen B, Tan M X, Wang W J, Zhu Y K, Pang R, Vasudevan V, Le Q V, Adam H. 2019. Searching for MobileNetV3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Seoul, Korea (South). pp. 1314–1324.

Hu J, Shen L, Sun G. 2018. Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, USA. pp. 7132–7141.

Huang L, Wang W M, Chen J, Wei X Y. 2019. Attention on attention for image captioning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Korea (South). pp. 4633–4642.

Karthik R, Hariharan M, Anand S, Mathikshara P, Johnson A, Menaka R. 2020. Attention embedded residual CNN for disease detection in tomato leaves. Applied Soft Computing, 86, 105–933.

Krizhevsky A, Sutskever I, Hinton G E. 2017. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60, 84–90.

Li M P, Zhou G X, Cai W W, Li J Y, Li M X, He M F, Hu Y H, Li L J. 2022. Multi-scale sparse network with cross-attention mechanism for image-based butterflies fine-grained classification. Applied Soft Computing, 117, 108419–108435.

Li X P, Li S Q. 2022. Transformer help CNN see better: A lightweight hybrid apple disease identification model based on transformers. Agriculture, 12, 884.

Liang T L, Glossner J, Wang L, Shi S B, Zhang X T. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461, 370–403.

Liu B, Zhang Y, He D J, Li Y X. 2018. Identification of apple leaf diseases based on deep convolutional neural networks. Symmetry, 10, 11.

Liu Y J, Hu Y W, Cai W W, Zhou G X, Zhan J L, Li L J. 2022. DCCAM-MRNet: Mixed residual connection network with dilated convolution and coordinate attention mechanism for tomato disease identification. Computational Intelligence and Neuroscience, 2022, 4848425.

Liu Z, Mao H Z, Wu C Y, Feichtenhofer C, Darrell T, Xie S. 2022. A ConvNet for the 2020s. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New Orleans, LA, USA. pp. 11976–11986.

Luo Y Q, Sun J, Shen J F, Wu X H, Wang L, Zhu W D. 2021. Apple leaf disease recognition and sub-class categorization based on improved multi-scale feature fusion network. IEEE Access, 9, 95517–95527.

Lv M J, Zhou G X, He M F, Chen A B, Zhang W Z, Hu Y H. 2020. Maize leaf disease identification based on feature enhancement and DMS-Robust alexnet. IEEE Access, 8, 57952–57966.

Ma N N, Zhang X Y, Zheng H T, Sun J. 2018. ShuffleNet V2: Practical guidelines for efficient CNN architecture design. In: Proceedings of the European Conference on Computer Vision (ECCV). Springer, Cham. pp. 122–138.

Misra D, Nalamada T, Arasanipalai A U, Hou Q B. 2021. Rotate to attend: Convolutional triplet attention module. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, Penang, HI, USA. pp. 3138–3147.

Nasir M I, Bibi A, Shah H J, Khan A M, Sharif M, Iqbal K, Nam Y Y, Kadry S. 2021. Deep learning-based classification of fruit diseases: An application for precision agriculture. Computers, Materials & Continua, 66, 1949–1962.

Perveen K, Kumar S, Kansal S, Soni M, Alshaikh N A, Batool S, Khanam M N, Osei B. 2023. Multidimensional attention-based CNN model for identifying apple leaf disease. Journal of Food Quality, 12, 9504186.

Pradhan P, Kumar B, Mohan S. 2022. Comparison of various deep convolutional neural network models to discriminate apple leaf diseases using transfer learning. Journal of Plant Diseases and Protection, 129, 1461–1473.

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L C. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Salt Lake City, UT, USA, pp. 4510–4520.

Song C Y, Wang D W, Bai H R, Sun W H. 2021. Apple disease recognition based on s-mall-scale data sets. Applied Engineering in Agriculture, 37, 481–490.

Sujatha P K, Sandhya J, Chaitanya J S, Subashini R. 2018. Enhancement of segmentation and feature fusion for apple disease classification. In: Tenth International Conference on Advanced Computing. IEEE, Chennai, India. pp. 175–181.

Sun H N, Xu H W, Liu B, He D J, He J R, Zhang H X, Geng N. 2021. MEAN-SSD: A novel real-time detector for apple leaf diseases using improved light-weight convolutional neural networks. Computers and Electronics in Agriculture, 189, 106379.

Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. 2015. Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Boston, MA, USA. pp. 1–9.

Tan M X, Le Q V. 2019. MixConv: Mixed depthwise convolutional kernels. Computer Research Repository, arXiv, 10.48550.

Trang K, TonThat L, Thao N G M, Thi N T T. 2019. Mango diseases identification by a deep residual network with contrast enhancement and transfer learning. In: Proceedings of the IEEE Conference on Sustainable Utilization and Development in Engineering and Technologies (CSUDET). IEEE, Penang, Malaysia. pp. 138–142.

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł, Polosukhin I. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.

Wang F K, Huang Y Q, Huang Z C, Shen H, Huang C, Qiao X, Qian W Q. 2022. MRUNet: A two-stage segmentation model for small insect targets in complex environments. Journal of Integrative Agriculture. 22, 1117–1130

Wang G M, Jia Q S, Qiao J F, Bi J, Zhou M C. 2021. Deep learning-based model predictive control for continuous stirred-tank reactor system. IEEE Transactions on Neural Networks and Learning Systems, 32, 3643–3652.

Wang P, Niu T, Mao Y R, Zhang Z, Liu B, He D J. 2021. Identification of apple leaf diseases by improved deep convolutional neural networks with an attention mechanism. Frontiers in Plant Science, 12, 723–294.

Wang Y X, Yang H, Bai X X, Qian X M, Ma L, Lu J, Li B, Fan X. 2021. PFAN++: Bi-directional image-text retrieval with position focused attention network. IEEE Transactions on Multimedia, 23, 3362–3376.

Woo S, Park J, Lee J Y, Kweon I S. 2018. CBAM: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV). Springer, Munich, Germany. pp. 3–19.

Xiong P W, Liao J J, Zhou M C, Song A G, Liu P X. 2023. Deeply supervised subspace learning for cross-modal material perception of known and unknown objects. IEEE Transactions on Industrial Informatics, 19, 2259–2268.

Xu Q, Wu M, Khoo E, Chen Z H, Li X L. 2023. A hybrid ensemble deep learning approach for early prediction of battery remaining useful life. IEEE/CAA Journal of Automatica Sinica, 10, 177–187.

Yadav A, Thakur U, Saxena R, Pal V, Bhateja V, Lin J CW. 2022. AFD-Net: Apple foliar disease multi classification using deep learning on plant pathology dataset. Plant and Soil, 477, 595–611.

Yan Q, Yang B H, Wang W Y, Wang B, Chen P, Zhang J. 2020. Apple leaf diseases recognition based on an improved convolutional neural network. Sensors, 20, 3535.

Yang G F, He Y, Yang Y, Xu B B. 2020. Fine-grained image classification for crop disease based on attention mechanism. Frontiers in Plant Science, 11, 600–854.

Yang K, Yi J Z, Chen A B, Liu J Q, Chen W J. 2022a. ConDinet++: Full-scale fusion network based on conditional dilated convolution to extract roads from remote sensing images. IEEE Geoscience and Remote Sensing Letters, 19, 1–5.

Yang K, Yi J Z, Chen A B, Liu J Q, Chen W J, Jin Z. 2022b. ConvPatchTrans: A script identification network with global and local semantics deeply integrated. Engineering Applications of Artificial Intelligence, 113, 104–916.

Yu H J, Son H C, Lee D H. 2020. Apple leaf disease identification through region-of-interest-aware deep convolutional neural network. Journal of Imaging Science and Technology, 64, 10356.

Yu H L, Cheng X H, Chen C C, Heidari A A, Liu J W, Cai Z N, Chen H L. 2022. Apple leaf disease recognition method with improved residual network. Multimedia Tools and Applications, 81, 7759–7782.

Zeng W, Li H, Hu G, Liang D. 2022. Lightweight dense-scale network (LDSNet) for corn leaf disease identification. Computers and Electronics in Agriculture, 197, 106–943.

Zhang C L, Zhang S W, Yang J C, Shi Y C, Chen J. 2017. Apple leaf disease identification using genetic algorithm and correlation based feature selection method. International Journal of Agricultural and Biological Engineering, 10, 74–83.

Zhang W Z, Zhou G X, Chen A B, Hu Y W. 2022. Deep multi-scale dual-channel convolutional neural network for Internet of Things apple disease detection. Computers and Electronics in Agriculture, 194, 106749.

Zhang Y, Yi J Z, Chen A B, Cheng L. 2023. Cardiac arrhythmia classification by time–frequency features inputted to the designed convolutional neural networks. Biomedical Signal Processing and Control, 79, 104224.

Zhao B, Wu X, Feng J S, Peng Q, Yan S C. 2017. Diversified visual attention networks for fine-grained object classification. IEEE Transactions on Multimedia, 19, 1245–1256.

Zhao M H, Xiong G, Zhou M C, Shen Z, Liu S, Han Y J, Wang F Y. 2022. PCUNet: A context-aware deep network for coarse-to-fine point cloud completion. IEEE Sensors Journal, 22, 15098–15110.

Zhong Y, Zhao M. 2020. Research on deep learning in apple leaf disease recognition. Computers and Electronics in Agriculture, 168, 105–146.