Caryl Lyons and Sandra Bolton, ACT, Inc.

What are Rater Comments?

In our presentation at TESOL, 1999, we discussed our experience in using Rater Comments in large-scale writing assessment and gave the audience an opportunity to practice actually using some Rater Comments. Finally, we made some suggestions for additional uses for Rater Comments. Rater Comments are comments about writing that raters can choose from a list of available comments. The purpose of these Rater Comments is to give students more information on the results of their writing assessment than simple numerical scores. The Comments are not given to justify the student scores but rather are designed to give additional narrative feedback to students that they and their teachers can use to build student confidence and to improve writing skills in the classroom.

For the past two years, ACT has been working on a contract with the Benesse Corporation of Japan to produce an English language test to be used for Japanese secondary students who are learning English as a Foreign Language. The purpose of the Benesse English Test is to test English — basically American English — in an authentic and communicative way. The Benesse test contains sections on listening, reading, and writing. The listening portion is based on authentic American speech patterns, and the reading portion is based on authentic reading materials. The direct writing assessment asks the students to write two short essays. Students have 10 to 12 minutes to write each essay. Since all three sections of the test are designed to fit into two class periods in Japanese junior and senior high schools, the writing section must be completed in a total of 20 or 25 minutes, depending on which level of test a student is taking.

Each essay is scored by two native speakers of American English using a six-point rubric. Scoring is holistic but based on the traits listed within the rubric, namely fluency or development, organization, vocabulary, and language elements. In addition to giving a score of 1 to 6 points, the first rater of each essay gives it four comments selected from the list of 56 comments.

What are the underlying assumptions?

Rater Comments have been used previously for several of ACT’s other writing assessments such as the 8th and 10th grade assessments and the portfolio assessment, though the list was modified and added to for this EFL writing assessment. Some of the main assumptions that underlie the Rater Comments are:

1. Raters try to convey perceptions of achievement and areas needing improvement specific to each paper. When giving comments to a paper, Raters are asked to think of what they would like to say to a student if they were able to have a one-to-one conference with that student — what they might say about the strengths of the writing and what they might say that would help the student revise this paper or improve the next one.

2. The tone of the response is supportive and facilitative. We want to build the student’s confidence, not undermine it.

3. Responses are as explicit as possible and are related to information shared in the classroom as much as possible.

4. Raters are of necessity responding to “final” drafts, but, with their choice of comments, they can encourage revision and different approaches and behaviors in future work. On the Benesse test, for example, students will get their papers back with these comments so they can be used for revision purposes in the classroom.

5. Raters use four comments for each paper. They are asked to use at least one positive and one improvement comment. The other two comments are left to the raters’ discretion. We’ve learned through feedback on other projects that all students, no matter how experienced, need positive encouragement on their writing. Then, too, even the best student writers want to be shown how they can improve, and, of course the less experienced writers need suggestions for improvement.

How are Rater Comments used?

The Rater Comments come from the same four areas that are measured by the rubric: fluency and development, organization, vocabulary, and language elements. Notice that mechanics and spelling are not included among these comments. The rubric considers these things only to the extent that they interfere with meaning—to the extent that the errors are global rather than local errors, to use terms used in TheGooficon by Burt and Kiparsky.

After ACT receives the student papers from Japan, the next task is to prepare raters, most of whom are accustomed to working with native speaker writing. Raters learn what to look for (or NOT to look for) when rating papers written in a second language We give the raters a crash course in ESL/EFL with lots of examples from student writing, prior to actually training them on sets of papers representing the various score points.

The main point of our training is to teach raters how to look for meaning, to try to understand what has been written, even when it is confusing to them in some way. For example, it is an easy thing to train raters to read words that are spelled with “r”s and “l”s that are reversed, and they easily learn to read “green grass” even though the paper says “green glass,” or to see that sometimes Japanese students will write the extra, epenthesized vowel that they often say when pronouncing an English word with a consonant cluster. One of our favorite words is “stores.” It looks like the English word “stores,” but when you see the context you can tell that it means “stress.”

Raters also need to learn that students may not be accustomed to English punctuation and that they may write sentences in a “listing” format rather than in a paragraph format. We mention the difficulties that nearly all speakers and writers of English as a second or foreign language have with verb forms, pronouns, prepositions, and articles. We point out a few specific words that Japanese students may use in their essays, such as the word “shinkansen” which means “bullet train.” We discuss how hard it is to use the appropriate tone and register in a language that is not your L1, and we show how “organization” is often evidenced by the use of simple coordinating or subordinating conjunctions or perhaps by simple conjunctive adverbs such as “first, “ “second,” and “third”.

With this background, we then introduce raters to actual student papers, practice scoring using the rubric, and then practice giving comments to the papers. The comments fall under the four categories mentioned before. Within each category are two kinds of comments: praise and suggestions for improvement. Since at least one of the four comments must be praise and at least one must be a suggestion for improvement, the list of comments includes a variety of comments. This means, for example, that the list must include some praise comments that could be used for a low-level paper with many problems as well as suggestions for improvement that could be given to a high level paper with few problems.

What do Rater Comments look like?

Below are just a few examples of the kinds of comments that can be used, listed in the four categories mentioned above:

Fluency and Development
        Praise
               You use clear reasons to support your opinions.
        Suggestions for Improvement
              Your writing would be more interesting if you added related
             details about what you see, hear, taste, or smell.

Vocabulary
       Praise
              The words you use are usually accurate.
       Suggestions for Improvement
              Greater variety in your word choice would make your writing more                interesting.

Organization
       Praise
              The transitions in your writing show clear connections among ideas.
       Suggestions for Improvement
              Following your ideas was sometimes difficult.

Language Elements
       Praise
             Your use of a variety of sentence structures helps make your writing              interesting.        
Suggestions for Improvement
             Your sentences often use the same simple sentence structure.

Additional Uses for Rater Comments

We use the Rater Comments in large-scale assessment, but we believe that the practice of giving comments from a list is also adaptable to various kinds of writing response. For example, comments could be useful for writing that is done for university, community college, or IEP placement testing so that a student, as well as the teacher, would receive some feedback from the writing based on what the teachers in the program feel is important. In a classroom setting, a teacher could devise his or her own list of comments based on what that teacher would like to focus on. Or a teacher could develop a comprehensive list of comments but choose a particular group of comments to focus on for a particular type of writing or a particular assignment. One advantage for the teacher is the time-saving factor of writing down a number rather than a prose comment. After working with a list of comments for a while, one begins to know, without looking, which number goes with which comment. An advantage for students is that teachers can discuss this focused list of comments with students so that students understand what each comment means. One size never fits all, but research indicates that ESL students sometimes don’t understand the comments that their teachers write in the margins of their papers, or that sometimes they can’t read the teacher’s handwriting. A list of comments could also be used for peer feedback among students. So using a list of comments to give at least part of the response to ESL/EFL writing in various settings is something to consider.

Reference
Burt, M.K., &Kiparsky, C. (1972). The Gooficon: A Repair Manual for English. Rowley, MA: Newbury House Publishers, Inc.

MIDTESOL Application

MIDTESOL Board

Job Information

Conference Index

Newsletter Index

*******************

ARTICLES

Teaching Reading Skills: Tips From The Trenches

Beat the Clock!

Bringing Novels to Life