《大規模英語考試作文評分信度與網上閱卷實證研究》紹有關大規模英語考試作文網上閱卷的研究成果，內容涉及作文網上閱卷過程的各個環節、與作文網上閱卷有關的閱卷員心理決策過程、作文評分過程中的認知作用、網上閱卷模型、計算機自動評分以及網上閱卷繫統的設計開發與試用。實證研究發現，網上閱卷能確保隨機分配作文卷，能加強閱卷過程的質量監控，而且閱卷員之間的評分一致性、閱卷員個人一致性和評分正確性都較高。研究結果證明，網上閱卷有利於提高閱卷信度和閱卷效率。

Chapter 1 Introduction
1.1 Rationale for the study
1.2 Objectives of the study
1.3 Organization of the thesis
1.4 Definition of terms
1.4.1 Online
1.4.2 Marking
1.4.3 Online marking
1.4.4 Online Marking System （OMS）
1.4.5 Local Area Network （LAN）

Chapter 2 Research Questions and Methodology of the Study
Chapter 3 Issues in the Direct Testing of EFL/ESL
Writing Ability
3.1 Introduction
3.2 What is a direct writing test?.
3.3 EFL/ESL writing ability: What shall we test?
3.4 Issues in validity
3.4.1 What is validity?
3.4.2 Types of validity
3.5 Issues in reliability
3.5.1 What is reliability?
3.5.2 Methods of judging reliability of writing assessments
3.6 The relationship between validity and reliability
3.7 Four components of a direct writing test
3.7.1 The task
3.7.2 The writer
3.7.3 The scoring procedure
3.7.4 The rater
3.8 Washback
3.8.1 Washback in general
3.8.2 Washback of direct tests of writing
3.9 Practicality
3.10 Summary

Chapter 4 The CET Writing Test
4.1 Introduction
4.2 The writing test required by the CET
4.2.1 A direct test
4.2.2 Positive washback
4.3 The scoring of CET compositions
4.3.1 The scoring approach currently adopted
4.3.2 Procedures involved in scoring CET essays
4.3.2.1 Scoring Principles and Marking Scheme
4.3.2.2 Range-finders and sample essays
4.3.2.3 Rater training
4.3.2.4 Rating process
4.3.2.5 Monitoring raters' scoring during the scoring sessions
4.3.2.6 Recording essay scores
4.4 Computer-aided adjustment of writing scores
4.5 Discussion

Chapter 5 The First Experimental Study
5.1 Introduction
5，2 Compositions
5.3 Participants
5.4 Data collection procedure
5.5 The introspection and retrospection studies
5.5.1 Introduction
5.5.2 Data elicitation
5.5.3 Tape transcription
5.5.4 Data analysis
5.6 The questionnaire studies
5.6.1 Design of the questionnaires
5.6.2 Analysis of questionnaire responses
5.7 Findings from the introspection， retrospection and questionnaire studies
5.7.1 Issues and problems in rating CET essays online
5.7.2 Decision-making behaviors while rating CET-4 essays.
5.7.3 Summary of comments made by the raters on essays
5.7.3.1 Overall summary
5.7.3.2 Variations in raters' comments
5.7.4 Essay elements' influences on raters' decision-making.
5.7.5 Elements of good CET essays in the raters' eyes
5.8 Analysis of writing scores
5.9 Summary and discussion
5.9.1 About the issues and problems involved
5.9.2 About the raters' scoring decisions
5.9.3 About the writing scores

Chapter 6 The Second Experimental Study
6.1 Introduction
6.2 Compositions
6.3 Participants
6.4 Data collection procedure
6.5 Problems encountered
6.6 Data analysis
6.7 Results
6.8 Summary

Chapter 7 Design of the OMS
7.1 Introduction
7.2 Literature review on online marking of compositions
7.2.1 Automated scoring of essays
7.2.1.l Overview of four major automated scoring methods
7.2.1.2 Analysis of the four major automated scoring methods
7.2.1.3 Summary
7.2.2 Online scoring of essays by human raters
7.2.2.1 Overview of online scoring of essays by human raters
7.2.2.2 Empirical research on online scoring of essays by human raters
7.2.2.3 Summary
7.3 A preliminary model of marking essays online
7.4 Overview of the CET Online Marking System （OMS）
7.4.1 The data management module
7.4.l.1 Basic information management
7.4.1.2 Essay management
7.4.1.3 Search and report
7.4.2 The training module
7.4.3 The rating module
7.4.4 The monitoring module
7.5 Operation of the OMS and the rater interface
7.5.1 Overview of the operation of the OMS
7.5.2 The OMS rater interface
7.6 Main features of the CEI- OMS
7.6.1 Random distribution of scripts
7.6.2 Efficient score recording
7.6.3 Online real-time monitoring of scoring
7.6.4 Quality control of raters
7.6.4.1 Adherence to the CET Scoring Principles and Marking Scheme
7.6.4.2 Rater training
7.6.4.2.1 Compulsory training
7.6.4.2.2 Individual rater's self training
7.6.4.2.3 Forced training
7.6.4.3 Online discussion
7.6.4.4 Back-reading and score revising
7.6.4.5 Time control
7.7 Advantages of the CE-I OMS
7.7.1 Real and efficient random distribution of scripts at the national level
7.7.2 Real-time online monitoring of raters
7.7.3 Assured quality control of scoring
7.7.4 Overall efficiency
7.7.5 Efficient and economical storage of scripts
7.7.6 Express retrieval of scripts and scores
7.7.7 Efficient management and potential utilization of test data for research
7.8 Limitations of online scoring and solutions
7.9 Summary

Chapter 8 The Third Experimental Study
8.1 Context of the experiment
8.2 Participants
8.3 Compositions
8.4 Data collection
8.4.1 Step1: Online marking
8.4.1.l The first round ordme marking
8.4.1.2 The second round onlLne markLng
8.4.2 Step 2: Conference marking
8.5 Data analysis
8.6 Results
8.7 Summary and discussion

Chapter 9 Data Analysis Using FACETS
9.1 FACETS and method
9.2 The first approach: comparison of rater severity and consistency from the online setting and the conference setting
9.2.1 Rater severity and consistency: the online setting
9.2.1.1 Rater severity: the online setting
9.2.1.2 Rater consistency: the online setting
9.2.2 Rater severity and consistency: the conference setting
9.2.2.1 Rater severity: the conference setting
9.2.2.2 Rater consistency: the conference setting
9.2.3 Comparison of rater severity and consistency in two settings
9.2.4 Comparison of rater severity change between two settings
9.3 The second approach: bias analysis
9.3.1 Bias analysis: rater by essay interactions
9.3.2 Bias analysis: rater by setting interactions
9.4 Conclusion
9.5 Discussion

Chapter 10 Summaries， Discussions， Implications and Recommendations
10.1 A refined model of online scoring of CET essays and its main features
10.2 Benefits proceeding from online scoring
10.3 Practicality
10.4 Scoring quality
10.5 Raters' comments
10.6 Suggestions for the improvement of the Online Marking System
10.7 Implications for other writing tests
10.8 Suggestions and recommendations for future research
10.8.1 Suggestions for future research in online marking of compositions
10.8.2 Recommendations for future research in EFL writingassessment
10.9 Theoretical and practical significance of the study
References
Appendices
後記

《大規模英語考試作文評分信度與網上閱卷實證研究》：
Bachman and Palmer (1996: 35) point out that the impact of a test should be evaluated with reference to the contextual variables of society's goals and values, the educational system in which the test is used, and the potential outcomes ofits use. They state that the impact of a test may operate at two levels: the micro level (i.e., the effect of the test on individual students and teachers) and the macro level (the impact on
society and its educational systems).
Yang and Gui (2007) point out that large-scale high-stakes tests have social impact and they should be made to measure test takers' English proficiency accurately, objectively and fairly. A good test should have positive washback on teaching and beneficial impact on society. They argue that the negative washback effect of a public test is mainly caused by the social weight it carries, and has little to do with the item types and
the test content. Therefore, all the test stake-holders should share their responsibility for avoiding any misuse ofthe test results.
Bailey (1996: 264) proposes a model of washback based on a framework consisting of participants, process and product. Participants include learners and teachers, administrators, material developers,and publishers (because they publish tests written by people in the language teaching field and/or textbooks that are designed explicitly as test preparation materials). Process refers to any action taken by the participants that may contribute to the process oflearning. Such action may include materials development, syllabus design, changes in teaching methods and/or content, learning and/or test-taking strategies. Product is what learners have learned through the process and the quality oflearning.
Alderson and Wall (1993: 120-121) proposes fifteen possible hypotheses that they hope will better define the washback construct in empirical investigations. Among these hypotheses, five directly address learner washback, five directly address teacher washback, and three refer to both teaching and learning. The importance oflearners and teachers in washback processes is emphasized.
However, although language learners are the key participants whose lives are most directly influenced by washback, there is little research into
how tests actually influence ESL/EFL learners' behavior and attitudes before and after tests. In contrast, there have been much more studies of teachers as participants in the washback process. When considering the varied research about washback and language teachers, Bailey (1999) notes that "teachers' classroom behavior can either support or override the intended positive washback effect of new or revised tests" (p.23). There have also been differences observed between experienced and novice teachers with respect to washback. The former "turned to the test as their main source of guidance for teaching oral language and used only material to be included in the test" (Shohamy et al., 1996: 301), while the latter used "a variety of additional activities in the teaching of oral language"(ibid.). Also, teachers have been observed to change their teaching contents, but not their teaching methods as a result of examination changes (Wall, 1996: 348).
……