The migration trajectory of agricultural surface source pollution in watersheds and the factors influencing it are of great theoretical and practical significance. Further study of these factors would be beneficial in the prevention and control of surface source pollution in watersheds. An analysis of panel data on agricultural surface source pollution loads within the Yuanjiang River Basin in Hunan Province from 2013 to 2022 was analyzed to investigate the spatial and temporal characteristics of this phenomenon. The results demonstrate that the total nitrogen and total phosphorus pollutant loads in the Yuanjiang River Basin of Hunan Province from 2013 to 2022 exhibit a general decreasing trend, with a spatial pattern of “more in the east and less in the west”. Additionally, the intensity of pollutant loads demonstrates a decreasing trend, with a spatial decrease from a northeast-southwest line to both sides. The intensity of pollution loads is also observed to be decreasing, exhibiting a pattern of decline from the northeast to the southwest. Additionally, the migration trajectory of the center of gravity of the pollution loads is identified to be generally V-shaped, with an eastward migration.