基于ResNet101-ASPP网络的奶牛多尺度关键点检测和运动特征提取

doi:10.1016/j.jia.2024.07.023

Journal of Integrative Agriculture ›› 2026, Vol. 25 ›› Issue (5): 2028-2040.DOI: 10.1016/j.jia.2024.07.023

基于ResNet101-ASPP网络的奶牛多尺度关键点检测和运动特征提取

收稿日期:2024-02-06 修回日期:2024-07-19 接受日期:2024-06-01 出版日期:2026-05-20 发布日期:2026-04-11

Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network

Saisai Wu^{1, 2*}, Shuqing Han^1,
2*, Jing Zhang^{1, 2}, Guodong Cheng^{1, 2}, Yali Wang^1,
2, Kai Zhang^{1, 2}, Mingming Han^{3, 4,} Jianzhai Wu^1,
2#

¹Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing 100081, China
²Key Laboratory of Agricultural Blockchain Application, Ministry of Agriculture and Rural Affairs, Beijing 100081, China
³Zibo Agricultural Science Research Institute, Zibo 255020, China
⁴Zibo Institute for Digital Agriculture and Rural Research, Zibo 255051, China

Received:2024-02-06 Revised:2024-07-19 Accepted:2024-06-01 Online:2026-05-20 Published:2026-04-11
About author:Saisai Wu, E-mail: wusaisai0324@163.com; Shuqing Han, E-mail: hanshuqing@caas.cn; #Correspondence Jianzhai Wu, E-mail: wujianzhai@caas.cn * These authors contributed equally to this study.
Supported by:
The work was supported by the National Natural Science Foundation of China (32102600), the Central Public-interest Scientific Institution Basal Research Fund, China (Y2023XK13, JBYW-AII-2024-28/40, and JBYW-AII-2023-33/37/42), Science and Technology Innovation Project of the Chinese Academy of Agricultural Sciences (CAAS-ASTIP-2021-AII), the Wuhu Science and Technology Bureau Two Strong One Increase Project, China (2023ly12).

摘要/Abstract

摘要：

奶牛关键点检测旨在定位和跟踪身体关节点的运动轨迹，在行为分析和跛行检测等任务中具有至关重要的作用。然而，在真实的奶牛养殖场景中，由于奶牛个体尺度变化较大以及遮挡等问题，可能导致关键点检测效果不佳。因此，本研究在ResNet101的浅层网络中引入了空洞空间金字塔池化（Atrous Spatial Pyramid Pooling, ASPP）模块，旨在提高模型的多尺度特征提取能力。ASPP模块通过在并行空洞卷积层中使用不同的膨胀率来增大模型的感受野，从而增强了对不同维度大小和遮挡关键点的识别鲁棒性。此外，在检测和跟踪不同关键点运动轨迹的基础上，利用特征提取模型同步提取了7种奶牛运动特征，包括步态跟随性、步态对称性、步进姿态差异性、运动速度差异性、头部摆动幅度、头颈部斜率和背部曲率，其中部分特征为本研究首次提出的创新性提取方法和属性。利用包含2385帧图像的数据集对多个骨干网络进行关键点检测任务的训练和测试结果对比，同时对改进算法进行消融实验。实验结果显示，与ResNet50、MobileNet_v2_1.0和EfficientNet-b0骨干网络相比，ResNet101的训练误差和测试误差分别提高了4.04–30.12像素和3.81–28.14像素。因此，采用ResNet101作为基准模型，通过添加ASPP模块进行后续算法改进。与基准模型相比，ResNet101-ASPP网络的训练误差和测试误差分别提高了0.27像素和0.24像素，且在三个不同的奶牛目标尺度数据集上，预测置信度提高了1.65%-2.50%。此外，在不同遮挡条件下的关键点检测效果显著提高，尤其是小尺度关键点，显示了ASPP模块在多尺度特征提取方面的能力。通过分析提取的7个运动特征与奶牛健康、轻度跛行和重度跛行之间的分布关系，结果表明所有运动特征在区分不同程度的跛行方面都发挥了重要作用。本研究的多尺度关键点检测网络也可应用于奶牛或其他动物的行为分类、发情识别等任务，为精准畜牧业智能化管理以及动物信息的实时监测提供理论和技术支持。

Abstract:

Detecting keypoints in dairy cows aims to locate and track the motion trajectories of the body’s joints, which plays a crucial role in behavior analysis and lameness detection. However, real farming scenarios, characterized by occlusions and large variations in object scale may result in poor detection results. Therefore, we introduce the atrous spatial pyramid pooling (ASPP) module into the shallow layers network of ResNet101, designed to improve the multi-scale feature extraction capability of the model. The ASPP module enhances the robustness of recognition for different dimensional sizes and occluded keypoints using different dilatation rates in the parallel atrous convolutional layers to expand the model’s receptive field. Furthermore, seven types of motion features, including tracking up, gait symmetry, step height balance, motion speed variability, head swing amplitude, head-neck slope and back curvature are extracted simultaneously by monitoring and tracking the motion trajectory of distinct keypoints. Several of these features represent innovative extraction models and attributes, first proposed in this study. Multiple models are trained and tested on datasets containing 2,385 frames for ablation experiments. The experiments show that, in comparison with the ResNet50, MobileNet_v2_1.0, and EfficientNet-b0 backbone networks, the training error and test error of ResNet101 are reduced by 4.04–30.12 pixels and 3.81–28.14 pixels. Therefore, ResNet101 is used as the benchmark for subsequent model improvement by adding the ASPP module. The training error and test error of the ResNet101-ASPP network are reduced by 0.27 and 0.24 pixels, respectively, compared to the benchmark network. The prediction confidence improves by 1.65–2.50% at three different dairy cow object scales. In addition, the keypoints under different occlusion conditions improve considerably, especially for small-scale keypoints, demonstrating the capability of the ASPP module for multi-scale feature extraction. By analyzing the distribution of the seven features and health, mild lameness, and severe lameness in dairy cows, it is shown that all the different features play an important role in distinguishing between different levels of lameness.

Key words: dairy cows , multi-scale , keypoints detection , ResNet101-ASPP network , motion features

Saisai Wu, Shuqing Han, Jing Zhang, Guodong Cheng, Yali Wang, Kai Zhang, Mingming Han, Jianzhai Wu. 基于ResNet101-ASPP网络的奶牛多尺度关键点检测和运动特征提取[J]. Journal of Integrative Agriculture, 2026, 25(5): 2028-2040.

Saisai Wu, Shuqing Han, Jing Zhang, Guodong Cheng, Yali Wang, Kai Zhang, Mingming Han, Jianzhai Wu. Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network[J]. Journal of Integrative Agriculture, 2026, 25(5): 2028-2040.

参考文献

Alameer A, Kyriazakis I, Bacardit J. 2020. Automated recognition of postures and drinking behaviour for the detection of compromised health in pigs. Scientific Reports, 10, 13665.

Barney S, Dlay S, Crowe A, Kyriazakis L, Leach M. 2023. Deep learning pose estimation for multi-cattle lameness detection. Scientific Reports, 13, 4499.

Bovo M, Ceccarelli M, Agrusti M, Torreggiani D, Tassinari P. 2024. DAIRY CHAOS: Data driven approach identifying dairy cows affected by HeAt lOad Stress. Computers and Electronics in Agriculture, 218, 108729.

Chen L, Zhang L, Tang J, Tang C, An R, Han R, Zhang Y. 2024. GRMPose: GCN-based real-time dairy goat pose estimation. Computers and Electronics in Agriculture, 218, 108662.

Chen L C, Papandreou G, Schroff F, Adam H. 2017. Rethinking atrous convolution for semantic image segmentation. arXiv, 1706, 05587.

Chen Y, He D, Fu Y, Song H. 2017. Intelligent monitoring method of cow ruminant behavior based on video analysis technology. International Journal of Agricultural and Biological Engineering, 10, 194–202.

Cheng G, Wu J, Xing L, Zhu M, Zhang J, Han, S. 2022. Fine-grained cow’s behavior classification method based on IMU. Journal of China Agricultural University. 27, 179–186. (in Chinese)

Fan Q C, Liu S C, Li S Q, Zhao C J. 2023. Bottom-up cattle pose estimation via concise multi-branch network. Computers and Electronics in Agriculture, 211, 107945.

Fang C, Zhang T M, Zheng H K, Huang J D, Cuan K X. 2021. Pose estimation and behavior classification of broiler chickens based on deep neural networks. Computers and Electronics in Agriculture, 180, 105863.

Flower F, Sanderson D, Weary D. 2005. Hoof pathologies influence kinematic measures of dairy cow gait. Journal of Dairy Science, 88, 3166–3173.

Graving J M, Chae D, Naik H, Li L, Koger B, Costelloe B R, Couzin I D. 2019. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife, 8, e47994.

Guo Y, Zhang Z, He D, Niu J, Tan Y. 2019. Detection of cow mounting behavior using region geometry and optical flow characteristics. Computers and Electronics in Agriculture, 163, 104828.

Gupta D, Artacho B, Savakis A. 2021. VehiPose: A multi-scale framework for vehicle pose estimation. In: Applications of Digital Image Processing XLIV. SPIE (The international society for optics and photonics), United States. pp. 517–523.

Helena R, Rik T, Gert K. 2022. T-LEAP: Occlusion-robust pose estimation of walking cows using temporal information. Computers and Electronics in Agriculture, 192, 106559.

Higaki S, Matsui Y, Miwa M, Yamamura T, Hojo T, Yoshioka K, Vang A, Negreiro A, Dórea J. 2024. Leveraging computer vision-based pose estimation technique in dairy cows for objective mobility analysis and scoring system. Computers and Electronics in Agriculture, 217, 108573.

Hoang Q T, Phung C P K, Bui T N, Chu T P D, Tran D T. 2018. Cow behavior monitoring using a multidimensional acceleration sensor and multiclass SVM. International Journal of Machine Learning and Networked Collaborative Engineering, 2, 110–118.

Insafutdinov E, Pishchulin L, Andres B, Mykhaylo A, Bernt S. 2016. Deepercut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model. Springer International Publishing, The Netherlands. pp. 34–50.

Jiang B, Song H, Wang H, Li C. 2022. Dairy cow lameness detection using a back curvature feature. Computers and Electronics in Agriculture, 194, 106729.

Kang X, Li S, Li Q, Liu G. 2022. Dimension-reduced spatiotemporal network for lameness detection in dairy cows. Computers and Electronics in Agriculture, 197, 106922.

Kang X, Zhang X, Liu G, Ma L. 2019. Hoof location method of lame dairy cows based on machine vision. Transactions of the Chinese Society for Agricultural Machinery, 50, 276–282. (in Chinese)

Li C, Lee G H. 2021. From synthetic to real: Unsupervised domain adaptation for animal pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, pp. 1482–1491.

Mathis A, Mamidanna P, Cury K M, Abe T, Murthy V N, Mathis M W, Bethge M. 2018. DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21, 1281–1289.

Nasiri A, Yoder J, Zhao Y, Hawkins S, Prado M, Gan H. 2022. Pose estimation-based lameness recognition in broiler using CNN-LSTM network. Computers and Electronics in Agriculture, 197, 106931.

Newell A, Yang K, Deng J. 2016. Stacked Hourglass Networks for Human Pose Estimation. Springer International Publishing, The Netherlands. pp. 483–499.

Pereira T D, Aldarondo D E, Willmore L, Kislin M, Wang S S H, Murthy M, Shaevitz J W. 2019. Fast animal pose estimation using deep neural networks. Nature Methods, 16, 117–125.

Poursaberi A, Bahr C, Pluk A, Van N A, Berckmans D. 2010. Real-time automatic lameness detection based on back posture extraction in dairy cattle: Shape analysis of cow with image processing techniques. Computers and Electronics in Agriculture, 74, 110–119.

Shuai L Y, Mu J, Jiang X Q, Chen P, Zhang B D, Li H D, Wang Y C, Li Z Y. 2023. An improved YOLOv5-based method for multi-species tea shoot detection and picking point location in complex backgrounds. Biosystems Engineering, 231, 117–132.

Song H, Jiang B, Wu Q, Li T, He D. 2018. Detection of dairy cow lameness based on fitting line slope feature of head and neck outline. Transactions of the Chinese Society of Agricultural Engineering, 34, 190–199. (in Chinese)

Taneja M, Byabazaire J, Jalodia N, Davy A, Olariu C, Malone P. 2020. Machine learning based fog computing assisted data-driven approach for early lameness detection in dairy cattle. Computers and Electronics in Agriculture, 171, 105286.

Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X. 2020. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43, 3349–3364.

Wu D, Wu Q, Yin X, Jiang B, Wang H, He D, Song H. 2020. Lameness detection of dairy cows based on the YOLOv3 deep learning algorithm and a relative step size characteristic vector. Biosystems Engineering, 189, 150–163.

Zhang H M, Li Y H, Zhou L X, Wang R, Li S Q, Wang H Y. 2022. Multi-objective skeleton extraction method of beef cattle based on improved YOLO v3. Transactions of the Chinese Society for Agricultural Machinery, 53, 285–293. (in Chinese)

Zhang K, Han S, Cheng G, Wu S, Liu J. 2022. Gait phase recognition of dairy cows based on Gaussian Mixture model and Hidden Markov model. Smart Agriculture, 4, 53–63. (in Chinese)

Zhao K, He D. 2014. Real-Time Automatic Classification of Lameness in Dairy Cattle Based on Movement Analysis with Image Processing Technique. American Society of Agricultural and Biological Engineers, Canada.

Zhao K, Zhang M, Ji J, Zhang R, Bewley J. 2023. Automatic lameness scoring of dairy cows based on the analysis of head-and back-hoof linkage features using machine learning methods. Biosystems Engineering, 230, 424-441.

Zimmermann C, Schneider A, Alyahyay M, Brox T, Diester I. 2020. FreiPose: A deep learning framework for precise animal motion capture in 3D spaces. BioRxiv, 27, 967620.

基于ResNet101-ASPP网络的奶牛多尺度关键点检测和运动特征提取

Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

编辑推荐

Metrics