Grain number is crucial for analysis of yield components and assessment of effects of cultivation measures.  The grain number per spike and thousand-grain weight can be measured by counting grains manually, but it is time-consuming, tedious and error-prone.  Previous image processing algorithms cannot work well with different backgrounds and different sizes.  This study used deep learning methods to resolve the limitations of traditional image processing algorithms.  Wheat grain image datasets were collected in the scenarios of three varieties, six background and two image acquisition devices with different heights, angles and grain numbers, 1 748 images in total.  All images were processed through color space conversion, image flipping and rotation.  The grain was manually annotated, and the datasets were divided into training set, validation set and test set.  We used the TensorFlow framework to construct the Faster Region-based Convolutional Neural Network Model.  Using the transfer learning method, we optimized the wheat grain detection and enumeration model.  The total loss of the model was less than 0.5 and the mean average precision was 0.91.  Compared with previous grain counting algorithms, the grain counting error rate of this model was less than 3% and the running time was less than 2 s.  The model can be effectively applied under a variety of backgrounds, image sizes, grain sizes, shooting angles, and shooting heights, as well as different levels of grain crowding.  It constitutes an effective detection and enumeration tool for wheat grain.  This study provides a reference for further grain testing and enumeration applications.