On Rater Agreement and Rater Training


  •  Binhong Wang    

Abstract

This paper first analyzed two studies on rater factors and rating criteria to raise the problem of rater agreement. After that the author reveals the causes of discrepencies in rating administration by discussing rater variability and rater bias. The author argues that rater bias can not be eliminated completely, we can only reduce the error to a cetain degree by training raters. The study on rater factors can help us better understand rater variability and rater bias, train raters more effectively and find out ways to modify the scores given by raters. The author suggests that rater files which contain rater information including each rater bias tendency should be established and kept so that information can be retrieved about the selection of raters, the interpretation as well as the modification of the scores given by raters. Raters need to receive pre-service training, on-service training and pilot -on-task training.



This work is licensed under a Creative Commons Attribution 4.0 License.
  • ISSN(Print): 1916-4742
  • ISSN(Online): 1916-4750
  • Started: 2008
  • Frequency: monthly

Journal Metrics

Citation Metrics (for informational purposes only; sourced from Google Scholar as of May 2026):

1. Citations (May 2026): 112462

2. h-index (May 2026): 145

3. i10-index (February 2025): 1811

*Note: Citation metrics vary significantly across indexing platforms. These figures are provided transparently and should not be used for cross-database comparisons. For methodological details, please refer to the official  Google Scholar Metrics documentation.

Contact