A large number of studies have examined the association between depressive symptoms and sleep quality, however, the psychological mechanism underlying the association remains nebulous. Using moderated mediation analysis, the present study aimed to examine to what extent the association was mediated by rumination and whether the mediation effect was moderated by self-compassion. Self-reported measures on depressive symptoms, rumination, self-compassion, and sleep quality were collected from 564 college students. The results showed that (a) rumination mediated the association between depressive symptoms and sleep quality, and (b) self-compassion moderated the mediation effect. Specifically, the mediating effect of rumination was stronger for students with low self-compassion than those with high self-compassion. These findings suggest that self-compassion may be a useful intervention target for health care practitioners to evaluate and improve sleep quality for individuals suffering from depressive symptoms.