Soil microbes play essential roles in the biogeochemical processes of organic carbon and nutrient cycling.  Many studies have reported various short-term effects of fertilization on soil microbes.  However, less is known about the effects of long-term fertilization regimes on the rhizosphere.  Therefore, the objective of this study was to explore how the soil microbial communities in the rhizosphere respond to different long-term fertilization strategies.  Based on a 21-year field treatment experiment in Guizhou, China, we extracted phospholipid fatty acids (PLFAs) to determine the microbial community structure in both the non-rhizosphere (NR) and rhizosphere (R).  Six treatments were included: no fertilizer (CK), mineral nitrogen fertilizer (N), N with potassium (NK), phosphorus with K (PK), NPK, and NPK combined with manure (MNPK).  The results showed that total PLFAs under unbalanced mineral fertilization (N, NK and PK) were decreased by 45% on average in the NR compared with CK, whereas MNPK increased fungi and G–bacteria abundance significantly in both the NR (by 33 and 23%) and R (by 15 and 20%), respectively.  In addition, all microbial groups in the R under these treatments (N, NK and PK) were significantly increased relative to those in the NR, except for the ratio of F/B and G+/G–, which might be due to the high nutrient availability in the R.  Soil pH and SOC significantly regulated the soil microbial community and structure, explaining 51 and 20% of the variation in the NR, respectively.  However, the rhizosphere microbial community structure was only significantly affected by soil pH (31%).  We concluded that the soil microbial community in the NR was more strongly affected by long-term fertilization than that in the R due to the rhizosphere effect in the agricultural ecosystem.  Rhizosphere nutrient conditions and buffering capacity could help microbial communities resist the change from the long-term fertilization.