Methods for Professional Dissertation Proofreading and Editing

The researcher reasoned in advance that the fairly commonplace term, "characteristic of writing," might be more familiar to the raters than the more technical term, "factor," and so the former was used in all discussions.

During this initial group interview as well as throughout the day, each rater was encouraged to express any difficulties she encountered due to the phrasing of a rubric, the contrast of a factor-specific focus with the holistic scoring paradigm to which she was accustomed, or any other circumstance.

To examine in greater detail whether the raters were following processes that focused implicitly on the features analyzed by e-rater, the researcher asked each rater to think out loud while scoring essays.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
essays, applying each rubric in turn.

Intermittently, she was stopped by the researcher and asked to report retrospectively on which cognitive processes she believed she had engaged as she performed the immediately preceding scoring task. The ETS expert in think-aloud procedures examined the set of rater instructions and the setting where the think-aloud sessions would be conducted. He also observed portions of the sessions on the first day.

Next, each rater participated in what Ericsson and Simon (1993) labeled a "social verbalization" exercise, in which the rater scores essays interactively with the researcher and, in a second step, with another rater as well. Social verbalization differs from think-aloud with retrospective reporting in that social verbalization permits the participant to offer more complete explanations of thoughts as they occur, rather than mere utterances of singular, perhaps disjointed thoughts.

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Each mock scoring session concluded with a debriefing session, preceded by a short normalization period of silent, independent scoring intended to get each rater "into the groove" prior to the debriefing.

In the debriefing session, the raters were asked to revisit their first impressions of the rubrics, recount their experiences with using each rubric that day, and render a summative, "last impression" of each rubric.