Exploring Computerized Lexical Analysis to Predict Calibrated Peer Review Ratings of Student Writing in Chemistry

TitleExploring Computerized Lexical Analysis to Predict Calibrated Peer Review Ratings of Student Writing in Chemistry
Publication TypeConference Paper
Year of Publication2013
AuthorsHaudek, KC, Russell AA, Urban-Lurain M
Refereed DesignationRefereed
Conference NameNational Association on Research in Science Teaching
Date Published04/2013
Conference LocationRio Grande, Puerto Rico
KeywordsAACR, AACR-pub, buffers, calibrated peer review, chemistry, Computerized scoring, Lexical analysis
AbstractWriting in undergraduate science courses represents an authentic scientific task and allows insight into student thinking, but is often limited in large enrollment courses due to resource constraints. We are investigating the combination of two approaches for evaluating student writing to overcome these constraints: using Calibrated Peer Review (CPR) and computerized text analysis. We investigate the possibility of computerized analysis of long (approximately 1700 characters), highly-structured essays, which have been scored by multiple trained, student peer reviewers. We extended and revised resources created in previous lexical projects for an assignment about buffer systems given in a general chemistry course. Our analysis revealed that students used many ideas in their writing. Over 100 lexical categories were created to capture the ideas in student writing, with each student’s response placed into about 40 categories. We used these lexical categories as independent variables in statistical models to predict peer ratings. Also, we investigated the addition of limited semantic information (word proximity) and results of writing-quality rubric criteria as independent variables. Addition of these variables resulted in a better performing statistical model, with R2=0.551. This scoring model used 24 of the independent variables: 20 lexical categories, 2 rubric criteria, one word proximity pair and response length. The lexical categories selected by the scoring model align well with the scoring rubric used by the peer reviewers, providing face validity for the lexical analysis. In addition, both rubric criteria included as variables were selected by the statistical model, indicating that writing quality was an important consideration of peer review.
Your rating: None