Computer-mediated communication (CMC) is an increasingly dominant communication modality, and there has been a subsequent increase in the number of reported cases of individuals with impaired CMC. A literature review reveals evidence such deficits can occur in individuals who are otherwise linguistically intact, and that existing assessment tools may not reliably detect such impairments. SpeechBot, a prototype chatbot-based rapid assessment tool, was designed to meet this need. By conducting the assessment and data collection using participants’ personal devices, each individual’s digital idiolect (their hardware, software, input method, and other preferences) can be accommodated more naturalistically. Results from an initial exploratory study at a large acute-care hospital indicate that SpeechBot can identify CMC impairment in individuals who otherwise demonstrated no linguistic deficits. The tool was able to discriminate between severe and mild/moderate impairment, and initial results suggest sensitivity to several different CMC impairment profiles. This study indicates that further research into CMC assessment is urgently needed, and that dynamic chatbot-based tools like SpeechBot may provide a solution to many of the confounds that have previously challenged such research.