Analytical rubrics that use several categories, for example, ‘Use of grammar’ or ‘Punctuation’, are commonly employed. Nevertheless, rubric validity is a common concern. A rubric used in a coordinated writing program at a large university in Western Japan was investigated for validity. Five raters evaluated first and second drafts of student essays (N = 14). The raters’ scores were analyzed with MultiFacets Rasch software. They were then asked to explain ratings in interviews. Rasch analysis showed that raters’ scores as not being very consistent with the overall model. Of the five raters, two had acceptable in-fit mean squares (1.05 and 0.79), and three did not (0.55, 1.37, and 0.35). Additionally, the categories of ‘Topic Sentence’ and ‘Language / Format’ showed rater bias in the case of three out of the five raters (18 and 26 instances, respectively). In the interviews, the raters expressed diverging ideas of what constituted an effective topic sentence and of what degree of lexical / grammatical error was still intelligible. The Program Coordinators had been somewhat doubtful about this rubric. Due to these doubts, reinforced by this study’s findings, they agreed to work towards the creation of a new rubric.