Solid-state direct metal laser sintering (S-DMLS) builds structures by using laser energy to sinter powder particles in a layer-by-layer manner. Powder size distribution (PSD) is an important parameter governing the densification of powders and the overall quality of as-built S-DMLS parts. Therefore, this work aims to reveal the underlying mechanism of layer-by-layer powder compact densification during the S-DMLS with different PSDs via a multi-scale computational framework: 1) a powder-based 3D heat transfer simulation is conducted to predict the thermal response at the macroscale during laser heating; 2) the obtained thermal information is input to a non-isothermal phase-field model to simulate the sintering behavior of powder particles in a layer-by-layer manner at the mesoscale. Using stainless steel 316 L as an example, a narrow PSD presents a small volume of gap between powders with an elevated effective thermal conductivity, causing a deep laser-induced heating zone that promotes full grain coalescence and reduces porosity during the S-DMLS. Furthermore, a bimodal powder mixture with an optimized size ratio can also effectively reduce the porosity of as-built S-DMLS parts. Moreover, the effect of layer-wise manufacturing on the densification is comprehensively explored. Finally, the influences of laser beam size and scanning speed are discussed.