Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network

doi:10.1016/j.jia.2024.07.023

Journal of Integrative Agriculture

2026, Vol. 25

Issue (5): 2028-2040 DOI: 10.1016/j.jia.2024.07.023

Animal Science · Veterinary Medicine

Advanced Online Publication | Current Issue | Archive | Adv Search

Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network

Saisai Wu^{1, 2*}, Shuqing Han^1,
2*, Jing Zhang^{1, 2}, Guodong Cheng^{1, 2}, Yali Wang^1,
2, Kai Zhang^{1, 2}, Mingming Han^{3, 4,} Jianzhai Wu^1,
2#

¹Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing 100081, China

²Key Laboratory of Agricultural Blockchain Application, Ministry of Agriculture and Rural Affairs, Beijing 100081, China

³Zibo Agricultural Science Research Institute, Zibo 255020, China

⁴Zibo Institute for Digital Agriculture and Rural Research, Zibo 255051, China

Highlights
● A ResNet101-ASPP network enhances multi-scale dairy cow keypoints detection in complex farming scenarios.
● Distinct keypoint trajectories were tracked to simultaneously extract seven motion features for gait analysis and lameness characterization in dairy cows.
● The proposed framework improves recognition of occluded keypoints and supports discrimination of health, mild, and severe lameness states.

Abstract
References

Download: PDF in ScienceDirect
Export: BibTeX | EndNote (RIS)

摘要

奶牛关键点检测旨在定位和跟踪身体关节点的运动轨迹，在行为分析和跛行检测等任务中具有至关重要的作用。然而，在真实的奶牛养殖场景中，由于奶牛个体尺度变化较大以及遮挡等问题，可能导致关键点检测效果不佳。因此，本研究在ResNet101的浅层网络中引入了空洞空间金字塔池化（Atrous Spatial Pyramid Pooling, ASPP）模块，旨在提高模型的多尺度特征提取能力。ASPP模块通过在并行空洞卷积层中使用不同的膨胀率来增大模型的感受野，从而增强了对不同维度大小和遮挡关键点的识别鲁棒性。此外，在检测和跟踪不同关键点运动轨迹的基础上，利用特征提取模型同步提取了7种奶牛运动特征，包括步态跟随性、步态对称性、步进姿态差异性、运动速度差异性、头部摆动幅度、头颈部斜率和背部曲率，其中部分特征为本研究首次提出的创新性提取方法和属性。利用包含2385帧图像的数据集对多个骨干网络进行关键点检测任务的训练和测试结果对比，同时对改进算法进行消融实验。实验结果显示，与ResNet50、MobileNet_v2_1.0和EfficientNet-b0骨干网络相比，ResNet101的训练误差和测试误差分别提高了4.04–30.12像素和3.81–28.14像素。因此，采用ResNet101作为基准模型，通过添加ASPP模块进行后续算法改进。与基准模型相比，ResNet101-ASPP网络的训练误差和测试误差分别提高了0.27像素和0.24像素，且在三个不同的奶牛目标尺度数据集上，预测置信度提高了1.65%-2.50%。此外，在不同遮挡条件下的关键点检测效果显著提高，尤其是小尺度关键点，显示了ASPP模块在多尺度特征提取方面的能力。通过分析提取的7个运动特征与奶牛健康、轻度跛行和重度跛行之间的分布关系，结果表明所有运动特征在区分不同程度的跛行方面都发挥了重要作用。本研究的多尺度关键点检测网络也可应用于奶牛或其他动物的行为分类、发情识别等任务，为精准畜牧业智能化管理以及动物信息的实时监测提供理论和技术支持。

Abstract

Detecting keypoints in dairy cows aims to locate and track the motion trajectories of the body’s joints, which plays a crucial role in behavior analysis and lameness detection. However, real farming scenarios, characterized by occlusions and large variations in object scale may result in poor detection results. Therefore, we introduce the atrous spatial pyramid pooling (ASPP) module into the shallow layers network of ResNet101, designed to improve the multi-scale feature extraction capability of the model. The ASPP module enhances the robustness of recognition for different dimensional sizes and occluded keypoints using different dilatation rates in the parallel atrous convolutional layers to expand the model’s receptive field. Furthermore, seven types of motion features, including tracking up, gait symmetry, step height balance, motion speed variability, head swing amplitude, head-neck slope and back curvature are extracted simultaneously by monitoring and tracking the motion trajectory of distinct keypoints. Several of these features represent innovative extraction models and attributes, first proposed in this study. Multiple models are trained and tested on datasets containing 2,385 frames for ablation experiments. The experiments show that, in comparison with the ResNet50, MobileNet_v2_1.0, and EfficientNet-b0 backbone networks, the training error and test error of ResNet101 are reduced by 4.04–30.12 pixels and 3.81–28.14 pixels. Therefore, ResNet101 is used as the benchmark for subsequent model improvement by adding the ASPP module. The training error and test error of the ResNet101-ASPP network are reduced by 0.27 and 0.24 pixels, respectively, compared to the benchmark network. The prediction confidence improves by 1.65–2.50% at three different dairy cow object scales. In addition, the keypoints under different occlusion conditions improve considerably, especially for small-scale keypoints, demonstrating the capability of the ASPP module for multi-scale feature extraction. By analyzing the distribution of the seven features and health, mild lameness, and severe lameness in dairy cows, it is shown that all the different features play an important role in distinguishing between different levels of lameness.

Keywords: dairy cows multi-scale keypoints detection ResNet101-ASPP network motion features

Received: 06 February 2024 Accepted: 01 June 2024 Online: 19 July 2024

Fund:

The work was supported by the National Natural Science Foundation of China (32102600), the Central Public-interest Scientific Institution Basal Research Fund, China (Y2023XK13, JBYW-AII-2024-28/40, and JBYW-AII-2023-33/37/42), Science and Technology Innovation Project of the Chinese Academy of Agricultural Sciences (CAAS-ASTIP-2021-AII), the Wuhu Science and Technology Bureau Two Strong One Increase Project, China (2023ly12).

About author: Saisai Wu, E-mail: wusaisai0324@163.com; Shuqing Han, E-mail: hanshuqing@caas.cn; #Correspondence Jianzhai Wu, E-mail: wujianzhai@caas.cn * These authors contributed equally to this study.

	Service
	E-mail this article
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors

Cite this article:

Saisai Wu, Shuqing Han, Jing Zhang, Guodong Cheng, Yali Wang, Kai Zhang, Mingming Han, Jianzhai Wu. 2026. Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network. Journal of Integrative Agriculture, 25(5): 2028-2040.

Alameer A, Kyriazakis I, Bacardit J. 2020. Automated recognition of postures and drinking behaviour for the detection of compromised health in pigs. Scientific Reports, 10, 13665.

Barney S, Dlay S, Crowe A, Kyriazakis L, Leach M. 2023. Deep learning pose estimation for multi-cattle lameness detection. Scientific Reports, 13, 4499.

Bovo M, Ceccarelli M, Agrusti M, Torreggiani D, Tassinari P. 2024. DAIRY CHAOS: Data driven approach identifying dairy cows affected by HeAt lOad Stress. Computers and Electronics in Agriculture, 218, 108729.

Chen L, Zhang L, Tang J, Tang C, An R, Han R, Zhang Y. 2024. GRMPose: GCN-based real-time dairy goat pose estimation. Computers and Electronics in Agriculture, 218, 108662.

Chen L C, Papandreou G, Schroff F, Adam H. 2017. Rethinking atrous convolution for semantic image segmentation. arXiv, 1706, 05587.

Chen Y, He D, Fu Y, Song H. 2017. Intelligent monitoring method of cow ruminant behavior based on video analysis technology. International Journal of Agricultural and Biological Engineering, 10, 194–202.

Cheng G, Wu J, Xing L, Zhu M, Zhang J, Han, S. 2022. Fine-grained cow’s behavior classification method based on IMU. Journal of China Agricultural University. 27, 179–186. (in Chinese)

Fan Q C, Liu S C, Li S Q, Zhao C J. 2023. Bottom-up cattle pose estimation via concise multi-branch network. Computers and Electronics in Agriculture, 211, 107945.

Fang C, Zhang T M, Zheng H K, Huang J D, Cuan K X. 2021. Pose estimation and behavior classification of broiler chickens based on deep neural networks. Computers and Electronics in Agriculture, 180, 105863.

Flower F, Sanderson D, Weary D. 2005. Hoof pathologies influence kinematic measures of dairy cow gait. Journal of Dairy Science, 88, 3166–3173.

Graving J M, Chae D, Naik H, Li L, Koger B, Costelloe B R, Couzin I D. 2019. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife, 8, e47994.

Guo Y, Zhang Z, He D, Niu J, Tan Y. 2019. Detection of cow mounting behavior using region geometry and optical flow characteristics. Computers and Electronics in Agriculture, 163, 104828.

Gupta D, Artacho B, Savakis A. 2021. VehiPose: A multi-scale framework for vehicle pose estimation. In: Applications of Digital Image Processing XLIV. SPIE (The international society for optics and photonics), United States. pp. 517–523.

Helena R, Rik T, Gert K. 2022. T-LEAP: Occlusion-robust pose estimation of walking cows using temporal information. Computers and Electronics in Agriculture, 192, 106559.

Higaki S, Matsui Y, Miwa M, Yamamura T, Hojo T, Yoshioka K, Vang A, Negreiro A, Dórea J. 2024. Leveraging computer vision-based pose estimation technique in dairy cows for objective mobility analysis and scoring system. Computers and Electronics in Agriculture, 217, 108573.

Hoang Q T, Phung C P K, Bui T N, Chu T P D, Tran D T. 2018. Cow behavior monitoring using a multidimensional acceleration sensor and multiclass SVM. International Journal of Machine Learning and Networked Collaborative Engineering, 2, 110–118.

Insafutdinov E, Pishchulin L, Andres B, Mykhaylo A, Bernt S. 2016. Deepercut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model. Springer International Publishing, The Netherlands. pp. 34–50.

Jiang B, Song H, Wang H, Li C. 2022. Dairy cow lameness detection using a back curvature feature. Computers and Electronics in Agriculture, 194, 106729.

Kang X, Li S, Li Q, Liu G. 2022. Dimension-reduced spatiotemporal network for lameness detection in dairy cows. Computers and Electronics in Agriculture, 197, 106922.

Kang X, Zhang X, Liu G, Ma L. 2019. Hoof location method of lame dairy cows based on machine vision. Transactions of the Chinese Society for Agricultural Machinery, 50, 276–282. (in Chinese)

Li C, Lee G H. 2021. From synthetic to real: Unsupervised domain adaptation for animal pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, pp. 1482–1491.

Mathis A, Mamidanna P, Cury K M, Abe T, Murthy V N, Mathis M W, Bethge M. 2018. DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21, 1281–1289.

Nasiri A, Yoder J, Zhao Y, Hawkins S, Prado M, Gan H. 2022. Pose estimation-based lameness recognition in broiler using CNN-LSTM network. Computers and Electronics in Agriculture, 197, 106931.

Newell A, Yang K, Deng J. 2016. Stacked Hourglass Networks for Human Pose Estimation. Springer International Publishing, The Netherlands. pp. 483–499.

Pereira T D, Aldarondo D E, Willmore L, Kislin M, Wang S S H, Murthy M, Shaevitz J W. 2019. Fast animal pose estimation using deep neural networks. Nature Methods, 16, 117–125.

Poursaberi A, Bahr C, Pluk A, Van N A, Berckmans D. 2010. Real-time automatic lameness detection based on back posture extraction in dairy cattle: Shape analysis of cow with image processing techniques. Computers and Electronics in Agriculture, 74, 110–119.

Shuai L Y, Mu J, Jiang X Q, Chen P, Zhang B D, Li H D, Wang Y C, Li Z Y. 2023. An improved YOLOv5-based method for multi-species tea shoot detection and picking point location in complex backgrounds. Biosystems Engineering, 231, 117–132.

Song H, Jiang B, Wu Q, Li T, He D. 2018. Detection of dairy cow lameness based on fitting line slope feature of head and neck outline. Transactions of the Chinese Society of Agricultural Engineering, 34, 190–199. (in Chinese)

Taneja M, Byabazaire J, Jalodia N, Davy A, Olariu C, Malone P. 2020. Machine learning based fog computing assisted data-driven approach for early lameness detection in dairy cattle. Computers and Electronics in Agriculture, 171, 105286.

Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X. 2020. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43, 3349–3364.

Wu D, Wu Q, Yin X, Jiang B, Wang H, He D, Song H. 2020. Lameness detection of dairy cows based on the YOLOv3 deep learning algorithm and a relative step size characteristic vector. Biosystems Engineering, 189, 150–163.

Zhang H M, Li Y H, Zhou L X, Wang R, Li S Q, Wang H Y. 2022. Multi-objective skeleton extraction method of beef cattle based on improved YOLO v3. Transactions of the Chinese Society for Agricultural Machinery, 53, 285–293. (in Chinese)

Zhang K, Han S, Cheng G, Wu S, Liu J. 2022. Gait phase recognition of dairy cows based on Gaussian Mixture model and Hidden Markov model. Smart Agriculture, 4, 53–63. (in Chinese)

Zhao K, He D. 2014. Real-Time Automatic Classification of Lameness in Dairy Cattle Based on Movement Analysis with Image Processing Technique. American Society of Agricultural and Biological Engineers, Canada.

Zhao K, Zhang M, Ji J, Zhang R, Bewley J. 2023. Automatic lameness scoring of dairy cows based on the analysis of head-and back-hoof linkage features using machine learning methods. Biosystems Engineering, 230, 424-441.

Zimmermann C, Schneider A, Alyahyay M, Brox T, Diester I. 2020. FreiPose: A deep learning framework for precise animal motion capture in 3D spaces. BioRxiv, 27, 967620.

No related articles found!

No Suggested Reading articles found!

Viewed

Full text

Abstract

Cited

Shared

Discussed

Cite this article:

About JIA

Editorial board

For authors