AAAI Publications, Twenty-Third International FLAIRS Conference

Font Size: 
Enhancing Protocol Evaluation Through Semantic Modification of Local Benchmarks
Shravan Mylavarapu, Irwin Levinstein, Chutima Boonthum, Joseph Magliano, Keith Millis

Last modified: 2010-05-06


A computer-based evaluation is essential to allow us to assess students' protocols without human intervention in R-SAT (Reading Strategy Assessment Tool). Word-matching and Latent Semantic Analysis (LSA) approaches were explored for use in the evaluation of protocols and. experimentally it was found that word-matching alone outperformed LSA or LSA in combination with word-matching. Why does LSA not work well with the R-SAT protocol evaluation? This paper demonstrates that modifying the benchmarks on which it is based can indeed benefit its performance. A variety of local benchmark modifications are investigated and results are compared against human expert. The preliminary results show that modified local benchmarks improve the R-SAT protocol evaluation on local bridging strategy: 0.1 increase in correlation and 8% increase in percent agreement between original benchmarks and modified benchmarks.


Protocol Evaluation; Latent Semanticcs Analysis (LSA); Benchmark Modification; Reading Strategy Assessment Tool (R-SAT)

Full Text: PDF