Scientia Agricultura Sinica ›› 2024, Vol. 57 ›› Issue (12): 2322-2335.doi: 10.3864/j.issn.0578-1752.2024.12.005

• TILLAGE & CULTIVATION·PHYSIOLOGY & BIOCHEMISTRY·AGRICULTURE INFORMATION TECHNOLOGY • Previous Articles     Next Articles

A Method for Testing Phenotype Parameters of Wheat Grains on Spike Based on Improved Mask R-CNN

WANG YunYun(), LI YiNian(), CHEN YuLun, DING QiShuo, HE RuiYin   

  1. College of Engineering, Nanjing Agricultural University, Nanjing 210031
  • Received:2023-12-08 Accepted:2024-03-02 Online:2024-06-16 Published:2024-06-25
  • Contact: LI YiNian

RichHTML

7

PDF

197

Abstract:

【Objective】 Wheat grain phenotype parameters were tested after grains only must been threshed by combine, this process was time-consuming, laborious and complicate. Therefore, a method to test morphological parameters of wheat grains on spike based on improved Mask R-CNN was proposed in this research.【Method】Two sides front images of three varieties wheat spikes, including Zhenmai 25, Ningmai 13 and Longmai 88 (Early maturity variety), were collected, and then the image enhancement data set was constructed by using Gaussian filter, salt and pepper noise, and vertical flip image enhancement method. A method combined with deep learning and morphological processing for testing phenotype parameters of wheat grains on spike was proposed. Firstly, the improved Mask R-CNN network model for segmenting spike glume was constructed, which was based on feature extraction networks of ResNet and FNP, and the innovative components Coordinate Attention (CA) module, Aggregation module, and Halfconv module were integrated into it. And the glumes on spike image were accurately detected, located, segmented and counted by the improved Mask R-CNN network model. Secondly, five phenotype parameters of the glumes on spike were extracted by using morphologic processing method from the segmented mask image of wheat spike glume, and the linear correlation equations between the phenotype parameters of the glumes and the phenotype parameters of grains were established. Finally, the linear correlation equations between the phenotype parameters of glume and the phenotypic parameters of grain were used to predict the phenotype parameters of grain.【Result】(1) F1 score, average precision (AP) and recall rate of the optimal model for separating spike glume based on the improved Mask R-CNN network model were 91.12%, 94.13% and 88.30%, respectively, and the average consuming time for detecting a single image was 97 milliseconds, so the improved Mask R-CNN network model could quickly and accurately identify the glumes on the single wheat spike. The root-mean-square error and average relative error for segmenting spike grain by the model were 0.94 piece and 0.65%, respectively, so this showed that spike glumes were segmented precisely by the model. (2) The established linear correlation equations for length, thickness, area, circumference, and length-thickness ratio between wheat spike glume and actual grain were y=0.7258x, y=0.5166x, y=0.3748x, y=0.6756x and y=1.4085x, respectively, and the determination coefficient (R2) of the linear correlation equations all were greater than 0.85. (3) The above correlation equations were verified and phenotype parameters of grain were predicted by using the extracted phenotype parameter data of wheat spike glume, the root-mean-square errors and average relative error for length, thickness, area, circumference and length-thickness ratio between predicted values and actual values of wheat grains were 0.17 mm, 0.08 mm, 0.46 mm2, 0.33 mm, 0.12, and 0.02%, 0.02%, 0.02%, 0.03%, respectively. The determination coefficient (R2) for each phenotype parameter between the predicted data and the actual data was above 0.85, research results indicated that the proposed method in this study was feasible【Conclusion】 The method for testing phenotype parameters of wheat grains on spike based on improved Mask R-CNN was proposed in this research, and the phenotype parameters of wheat grain on spike could be predicted accurately and effectively by the phenotype parameters of wheat spike glume. This research provided a new method to extract rapidly and simply wheat grain phenotype parameters on spike.

Key words: wheat spike, grain phenotype, deep learning, Coordinate Attention

Fig. 1

Device for collecting image A: Camera; B: Light source; C: Stage"

Fig. 2

Enhancement images and annotated images a: Original image of wheat spike; b: Enhancement image (Gaussian filter image, salt and pepper noise image, vertical flip image from left to right; c: Annotated image"

Fig. 3

Improved Mask R-CNN model architecture"

Fig. 4

Improved feature extraction network"

Fig. 5

Coordinate attention structure"

Fig. 6

Aggregate structure"

Fig. 7

Local location information structure"

Fig. 8

Halfconv structure"

Fig. 9

Method for counting grain number of wheat spike"

Fig. 10

Testing method for phenotype parameters of wheat spike glume a: Testing method for phenotypic parameter of glume; b: Glume mask; c: Edge detection; d: Ellipse fitting"

Fig. 11

Testing method for grain actual phenotype parameters a: Manually measuring grains number; b: Grain length; c: Grain thickness"

Table 1

Set parameters for network model"

参数Parameter 数值Value
初始学习率Initial learning rate 0.001
锚框Anchor box [32, 64, 128, 256, 512]
最大图像维度Maximum image dimension 512
最小图像维度Minimum image dimension 512
动量因子Momentum 0.9
置信度Confidence 0.7
迭代轮数Epoch 100
批量大小Batch_size 2

Table 2

Segmentation performance comparison of different network models"

模型
Model		 平均精确率
Average Precision AP (%)		 平均检测耗时
Average inspection
time t (ms)
Mask R-CNN 89.26 213
Mask R-CNN_Agg 89.54 198
Mask R-CNN_Agg_CA 94.13 187
Mask R-CNN_Agg_CA_halfconv 94.13 97

Fig. 12

Segmentation results of wheat spike glume"

Fig. 13

Results analysis on counting grain number of spike"

Fig. 14

Correlation fitting curve and equations between glume phenotype parameters and grain actual phenotypic parameters a: Length; b: Thickness; c: Area; d: Circumference; e: Length-thickness ratio"

Fig. 15

Comparing analysis between actual phenotype parameters and model predicting phenotype parameters of wheat grain a: Length; b: Thickness; c: Area; d: Circumference; e: Length-thickness ratio"

[1]
王寒冬, 陈文杰, 张波, 刘宝龙, 王蕾, 张连全, 张怀刚, 刘登才. 人工合成小麦改良品系的种子表型性状分析. 分子植物育种, 2018, 16(18): 6097-6104.
WANG H D, CHEN W J, ZHANG B, LIU B L, WANG L, ZHANG L Q, ZHANG H G, LIU D C. Phenotypic analysis of grain traits in improved lines of synthetic wheat. Molecular Plant Breeding, 2018, 16(18): 6097-6104. (in Chinese)
[2]
JIA M L, LI Y N, WANG Z Y, TAO S, SUN G L, KONG X C, WANG K, YE X G, LIU S S, GENG S F, MAO L, LI A L. TaIAA21 represses TaARF25-mediated expression of TaERFs required for grain size and weight development in wheat. The Plant Journal, 2021, 108(6): 1754-1767.

doi: 10.1111/tpj.15541 pmid: 34643010
[3]
YU K, LIU D C, CHEN Y, WANG D Z, YANG W L, YANG W, YIN L X, ZHANG C, ZHAO S C, SUN J Z, LIU C M, ZHANG A M. Unraveling the genetic architecture of grain size in einkorn wheat through linkage and homology mapping and transcriptomic profiling. Journal of Experimental Botany, 2019, 70(18): 4671-4688.

doi: 10.1093/jxb/erz247 pmid: 31226200
[4]
崔炜楠, 聂志刚, 李广, 王钧. 基于改进的混合蛙跳算法对旱地小麦籽粒生长模型参数的优化. 中国农业科学, 2023, 56(12): 2274-2287. doi: 10.3864/j.issn.0578-1752.2023.12.004.
CUI W N, NIE Z G, LI G, WANG J. Optimization of dryland wheat grain growth model parameters based on an improved shuffled frog leaping algorithm. Scientia Agricultura Sinica, 2023, 56(12): 2274-2287. doi: 10.3864/j.issn.0578-1752.2023.12.004. (in Chinese)
[5]
李栓明, 郭银巧, 王克如, 谢瑞芝, 戴建国, 肖春华, 李静, 李少昆. 小麦籽粒蛋白质光谱特征变量筛选方法研究. 中国农业科学, 2015, 48(12): 2317-2326. doi: 10.3864/j.issn.0578-1752.2015.12.004.
LI S M, GUO Y Q, WANG K R, XIE R Z, DAI J G, XIAO C H, LI J, LI S K. Research on variable selection of wheat kernel protein content with near-infrared spectroscopy. Scientia Agricultura Sinica, 2015, 48(12): 2317-2326. doi: 10.3864/j.issn.0578-1752.2015.12.004. (in Chinese)
[6]
李振海, 徐新刚, 金秀良, 张竞成, 宋晓宇, 宋森楠, 杨贵军, 王纪华. 基于氮素运转原理和GRA-PLS算法的冬小麦籽粒蛋白质含量遥感预测. 中国农业科学, 2014, 47(19): 3780-3790. doi: 10.3864/j.issn.0578-1752.2014.19.006.
LI Z H, XU X G, JIN X L, ZHANG J C, SONG X Y, SONG S N, YANG G J, WANG J H. Remote sensing prediction of winter wheat protein content based on nitrogen translocation and GRA-PLS method. Scientia Agricultura Sinica, 2014, 47(19): 3780-3790. doi: 10.3864/j.issn.0578-1752.2014.19.006. (in Chinese)
[7]
KAYA E, SARITAS İ. Towards a real-time sorting system: Identification of vitreous durum wheat kernels using ANN based on their morphological, colour, wavelet and gaborlet features. Computers and Electronics in Agriculture, 2019, 166: 105016.
[8]
LUO X, JAYAS D S, SYMONS S J. Comparison of statistical and neural network methods for classifying cereal grains using machine vision. Transactions of the ASAE, 1999, 42(2): 413-419.
[9]
张弓, D.S.Jayas, 蒋德云, 张肇鲲. 谷物纹理特征的识别(英文). 农业工程学报, 2001, (01): 149-153.
ZHANG G, JAYAS D S, JIANG D Y, ZHANG Z K. Grain classification with combined texture model. Transactions of the Chinese Society of Agricultural Engineering, 2001, (01): 149-153. (in Chinese)
[10]
董高, 郭建, 王成, 陈子龙, 郑玲, 朱大洲. 基于近红外高光谱成像及信息融合的小麦品种分类研究. 光谱学与光谱分析, 2015, 35(12): 3369-3374.
DONG G, GUO J, WANG C, CHEN Z L, ZHENG L, ZHU D Z. The classification of wheat varieties based on near infrared hyperspectral imaging and information fusion. Spectroscopy and Spectral Analysis, 2015, 35(12): 3369-3374. (in Chinese)
[11]
孟惜, 王克俭, 韩宪忠. 基于改进BP网络的小麦品种识别. 贵州农业科学, 2017, 45(10): 156-160.
MENG X, WANG K J, HAN X Z. Classification of wheat varieties by improved BP neural network. Guizhou Agricultural Sciences, 2017, 45(10): 156-160. (in Chinese)
[12]
谢元澄, 于增源, 姜海燕, 金前, 蔡娜娜, 梁敬东. 小麦麦穗几何表型测量的精准分割方法研究. 南京农业大学学报, 2019, 42(5): 956-966.
XIE Y C, YU Z Y, JIANG H Y, JIN Q, CAI N N, LIANG J D. Study on precise segmentation method for geometric phenotype measurement of wheat ear. Journal of Nanjing Agricultural University, 2019, 42(5): 956-966. (in Chinese)
[13]
章权兵, 胡姗姗, 舒文灿, 程鸿. 基于注意力机制金字塔网络的麦穗检测方法. 农业机械学报, 2021, 52(11): 253-262.
ZHANG Q B, HU S S, SHU W C, CHENG H. Wheat spikes detection method based on pyramidal network of attention mechanism. Transactions of the Chinese Society for Agricultural Machinery, 2021, 52(11): 253-262. (in Chinese)
[14]
GENAEV M A, KOMYSHEV E G, SMIRNOV N V, KRUCHININA Y V, GONCHAROV N P, AFONNIKOV D A. Morphometry of the wheat spike by analyzing 2D images. Agronomy, 2019, 9(7): 390.
[15]
杜颖, 蔡义承, 谭昌伟, 李振海, 杨贵军, 冯海宽, 韩东. 基于超像素分割的田间小麦穗数统计方法. 中国农业科学, 2019, 52(1): 21-33. doi: 10.3864/j.issn.0578-1752.2019.01.003.
DU Y, CAI Y C, TAN C W, LI Z H, YANG G J, FENG H K, HAN D. Field wheat ears counting based on superpixel segmentation method. Scientia Agricultura Sinica, 2019, 52(1): 21-33. doi: 10.3864/j.issn.0578-1752.2019.01.003. (in Chinese)
[16]
谭昌伟, 罗明, 杨昕, 马昌, 严翔, 周健, 杜颖, 王雅楠. 用PLS算法由HJ-1A/1B遥感影像估测区域冬小麦理论产量. 中国农业科学, 2015, 48(20): 4033-4041. doi: 10.3864/j.issn.0578-1752.2015.20.005.
TAN C W, LUO M, YANG X, MA C, YAN X, ZHOU J, DU Y, WANG Y N. Remote sensing estimation of winter wheat theoretical yield on regional scale using partial least squares regression algorithm based on HJ-1A/1B images. Scientia Agricultura Sinica, 2015, 48(20): 4033-4041. doi: 10.3864/j.issn.0578-1752.2015.20.005. (in Chinese)
[17]
李少昆, 索兴梅, 白中英, 祁之力, 刘晓鸿, 高世菊, 赵双宁. 基于BP神经网络的小麦群体图像特征识别. 中国农业科学, 2002, 35(6): 616-620.
LI S K, SUO X M, BAI Z Y, QI Z L, LIU X H, GAO S J, ZHAO S N. The machine recognition for population feature of wheat images based on BP neural network. Scientia Agricultura Sinica, 2002, 35(6): 616-620. (in Chinese)
[18]
姜盼, 张彬, 毕昆. 基于图像处理的小麦穗长测量. 中国传媒大学学报(自然科学版), 2010, 17(4): 69-73.
JIANG P, ZHANG B, BI K. Wheat ear-length measurements based on image processing. Journal of Communication University of China (Science and Technology), 2010, 17(4): 69-73. (in Chinese)
[19]
毕昆, 姜盼, 李磊, 石本义, 王成. 基于形态学图像处理的麦穗形态特征无损测量. 农业工程学报, 2010, 26(12): 212-216.
BI K, JIANG P, LI L, SHI B Y, WANG C. Non-destructive measurement of wheat spike characteristics based on morphological image processing. Transactions of the Chinese Society of Agricultural Engineering, 2010, 26(12): 212-216. (in Chinese)
[20]
李毅念, 杜世伟, 姚敏, 易应武, 杨建峰, 丁启朔, 何瑞银. 基于小麦群体图像的田间麦穗计数及产量预测方法. 农业工程学报, 2018, 34(21): 185-194.
LI Y N, DU S W, YAO M, YI Y W, YANG J F, DING Q S, HE R Y. Method for wheatear counting and yield predicting based on image of wheatear population in field. Transactions of the Chinese Society of Agricultural Engineering, 2018, 34(21): 185-194. (in Chinese)
[21]
刘哲, 袁冬根, 王恩. 基于改进Bayes抠图算法的麦穗小穗自动计数方法. 中国农业科技导报, 2020, 22(8): 75-82.

doi: 10.13304/j.nykjdb.2019.0461
LIU Z, YUAN D G, WANG E. Automatic counting method of wheat grain based on improved bayes matting algorithm. Journal of Agricultural Science and Technology, 2020, 22(8): 75-82. (in Chinese)

doi: 10.13304/j.nykjdb.2019.0461
[22]
路文超, 罗斌, 潘大宇, 赵勇, 于春花, 王成. 基于图像处理的小麦穗长和小穗数同步测量. 中国农机化学报, 2016, 37(6): 210-215.
LU W C, LUO B, PAN D Y, ZHAO Y, YU C H, WANG C. Synchronous measurement of wheat ear length and spikelets number based on image processing. Journal of Chinese Agricultural Mechanization, 2016, 37(6): 210-215. (in Chinese)
[23]
赵越, 卫勇, 单慧勇, 穆志民, 张健欣, 吴海云, 赵辉, 胡建龙. 基于深度学习的高分辨率麦穗图像检测方法. 中国农业科技导报, 2022, 24(9): 96-105.
ZHAO Y, WEI Y, SHAN H Y, MU Z M, ZHANG J X, WU H Y, ZHAO H, HU J L. Wheat ear detection method based on deep learning. Journal of Agricultural Science and Technology, 2022, 24(9): 96-105. (in Chinese)
[24]
黄硕, 周亚男, 王起帆, 张晗, 邱朝阳, 康凯, 罗斌. 改进YOLOv5测量田间小麦单位面积穗数. 农业工程学报, 2022, 38(16): 235-242.
HUANG S, ZHOU Y N, WANG Q F, ZHANG H, QIU C Y, KANG K, LUO B. Measuring the number of wheat spikes per unit area in fields using an improved YOLOv5. Transactions of the Chinese Society of Agricultural Engineering, 2022, 38(16): 235-242. (in Chinese)
[25]
HE K M, GKIOXARI G, DOLLAR P, GIRSHICK R. Mask R-CNN. 16th IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, 2017, (322): 2980-2988.
[26]
HOU Q B, ZHOU D Q, FENG J S. Coordinate attention for efficient mobile network design. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville: IEEE, 2021, (01350): 13708-13717.
[27]
杜世伟, 李毅念, 姚敏, 李玲, 丁启朔, 何瑞银. 基于小麦穗部小穗图像分割的籽粒计数方法. 南京农业大学学报, 2018, 41(4): 742-751.
DU S W, LI Y N, YAO M, LI L, DING Q S, HE R Y. Counting method of grain number based on wheatear spikelet image segmentation. Journal of Nanjing Agricultural University, 2018, 41(4): 742-751. (in Chinese)
[28]
ZHOU H, RICHE A B, HAWKESFORD M J, WHALLEY W R, ATKINSON B S, STURROCK C J, MOONEY S J. Determination of wheat spike and spikelet architecture and grain traits using X-ray Computed Tomography imaging. Plant Methods, 2021, 17(1): 26.

doi: 10.1186/s13007-021-00726-5 pmid: 33750418
[29]
XU X, GENG Q, GAO F, XIONG D, QIAO H B, MA X M. Segmentation and counting of wheat spike grains based on deep learning and textural feature. Plant Methods, 2023, 19(1): 77.

doi: 10.1186/s13007-023-01062-6 pmid: 37528413
[30]
吴婷婷. 单粒小麦种子表型精细化获取方法研究与装备研发[D]. 杨凌: 西北农林科技大学, 2021.
WU T T. Sophisticated phenotyping and equipment development of individual wheat seed[D]. Yangling: Northwest A&F University, 2021. (in Chinese)
[31]
仲晓春, 陈雯, 刘涛, 郝心宁, 李哲敏, 孙成明. 基于图像技术的小麦籽粒三维信息测量. 广东农业科学, 2016, 43(2): 150-155.
ZHONG X C, CHEN W, LIU T, HAO X N, LI Z M, SUN C M. Three-dimensional information measurement of wheat grain based on image technology. Guangdong Agricultural Sciences, 2016, 43(2): 150-155. (in Chinese)
[1] ZHANG JianLong, XING WenWen, YE ShaoBo, ZHANG Chao, ZHENG DeCong. Oat Plant Height Estimation Based on a Dual Output Regression Convolutional Neural Network [J]. Scientia Agricultura Sinica, 2024, 57(20): 3974-3985.
[2] LI MianYan, WANG LiXian, ZHAO FuPing. Research Progress on Machine Learning for Genomic Selection in Animals [J]. Scientia Agricultura Sinica, 2023, 56(18): 3682-3692.
[3] ZHAO XiaoHui,ZHANG YanYan,RONG YaSi,DUAN JianZhao,HE Li,LIU WanDai,GUO TianCai,FENG Wei. Study on Critical Nitrogen Dilution Model of Winter Wheat Spike Organs Under Different Water and Nitrogen Conditions [J]. Scientia Agricultura Sinica, 2022, 55(17): 3321-3333.
[4] SUN Qing,ZHAO YanXia,CHENG JinXin,ZENG TingYu,ZHANG Yi. Fruit Growth Modelling Based on Multi-Methods - A Case Study of Apple in Zhaotong, Yunnan [J]. Scientia Agricultura Sinica, 2021, 54(17): 3737-3751.
[5] LIU QingFei, ZHANG HongLi, WANG YanLing. Real-Time Pixel-Wise Classification of Agricultural Images Based on Depth-Wise Separable Convolution [J]. Scientia Agricultura Sinica, 2018, 51(19): 3673-3682.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
京ICP备09089781号-30
Copyright 2018 © Editorial office of Scientia Agricultura Sinica
Tel: 86-010-82106279    E-mail: zgnykx@mail.caas.net.cn
Supported by Beijing Magtech Co.Ltd