The annual average carbon emissions of prefabricated building in the physical and chemical phase account for the largest proportion in their entire life cycle. In order to effectively promote the green and sustainable development of prefabricated building, the key factors that affect the carbon emissions of prefabricated building in the physical and chemical phase from the perspective of carbon footprint and carbon sources were comprehensively identified, a system dynamics model to quantify the impact of these factors on carbon emissions was established, and the effectiveness of the model was empirically analyzed and verified. The carbon emission of prefabricated building in China was predicted and analyzed through simulation. The results indicate that material utilization rate, vehicle fuel consumption, and prefabrication rate are the main factors affecting carbon emissions, providing reference for policy formulation to promote sustainable development of the construction industry in the context of “dual carbon”.