As humans continue to incorporate automation into professional and personal settings, researchers are forced to reconsider how humans make judgments in a world where we are increasingly reliant on automated systems to meet our goals. 80 participants completed a visual search task. Critically, over 280 trials, participants alternated between performing the task themselves, and watching automation perform the same task. Task difficulty, a central cue to difficulty, was varied across four dimensions (i.e., clicks, feedback, set size, timing) and changed every five trials. Target identification, or whether a target was identified on a given trial, served as a peripheral cue to difficulty. After each trial, participants made a Judgment of Difficulty (JOD) by indicating if the trial was "easier" or "harder" than before. A multi-level logistic regression revealed significant differences in peripheral cue use (specifically weighting of misses) between participants who performed first and those who observed first (p < .001). Participants who performed first alternated to observing and began to continually down weight peripheral cues to difficulty for automation. Despite an identical task, these participants thought the task was easier for automation than it was for themselves. Participants who observed automation first reported depressed JODs which never reached the level of those who performed first. This suggests that initially observing automation can cause perceptions of ease to persist in one's own judgments of a task. Together, these results suggest that prior task experience serves as an important moderator in how humans make JODs for automated systems.