(1) Background: Sleep, a physiological necessity, has strong inflammatory underpinnings. Diet is a strong moderator of systemic inflammation. This study explored the associations between the Dietary Inflammatory Index (DII®) and sleep duration, timing, and quality from the Energy Balance Study (EBS). (2) Methods: The EBS (n = 427) prospectively explored energy intake, expenditure, and body composition. Sleep was measured using BodyMedia’s SenseWear® armband. DII scores were calculated from three unannounced dietary recalls (baseline, 1-, 2-, and 3-years). The DII was analyzed continuously and categorically (very anti-, moderately anti-, neutral, and pro-inflammatory). Linear mixed-effects models estimated the DII score impact on sleep parameters. (3) Results: Compared with the very anti-inflammatory category, the pro-inflammatory category was more likely to be female (58% vs. 39%, p = 0.02) and African American (27% vs. 3%, p < 0.01). For every one-unit increase in the change in DII score (i.e., diets became more pro-inflammatory), wake-after-sleep-onset (WASO) increased (βChange = 1.00, p = 0.01), sleep efficiency decreased (βChange = −0.16, p < 0.05), and bedtime (βChange = 1.86, p = 0.04) and waketime became later (βChange = 1.90, p < 0.05). Associations between bedtime and the DII were stronger among African Americans (βChange = 6.05, p < 0.01) than European Americans (βChange = 0.52, p = 0.64). (4) Conclusions: Future studies should address worsening sleep quality from inflammatory diets, leading to negative health outcomes, and explore potential demographic differences.