Sulfur doping of silicon beyond the solubility limit by femtosecond laser irradiation leads to near-unity broadband absorption of visible and infrared light and the realization of silicon-based infrared photodetectors. The nature of the infrared absorption is not yet well understood. Here we present a study on the reduction of infrared absorptance after various anneals of different temperatures and durations for three chalcogens (sulfur, selenium, and tellurium) dissolved into silicon by femtosecond laser irradiation. For sulfur doping, we irradiate silicon in SF6 gas; for selenium and tellurium, we evaporate a film onto the silicon and irradiate in N2 gas; lastly, as a control, we irradiated untreated silicon in N2 gas. Our analysis shows that the deactivation of infrared absorption after thermal annealing is likely caused by dopant diffusion. We observe that a characteristic diffusion lengthcommon to all three dopantsleads to the reduction of infrared absorption. Using diffusion theory, we suggest a model in which grain size of the re-solidiﬁed surface layer can account for this characteristic diffusion length, indicating that deactivation of infrared absorptance may be caused by precipitation of the dopant at the grain boundaries.