#365: AI-Feedback System: Integrating Human-in-the-Loop for Customized Support

This study presents an iterative human-AI collaboration design process for developing an AI-Feedback system aligned with learning theories, providing timely, customized feedback to support teachers and students in developing usable knowledge. The process involved collaboration among teachers, content experts, and learning scientists to refine rubrics and validate AI models for analyzing elementary students’ written responses on complex science tasks—often vague or inconsistent—to identify weaknesses and uncertainty in students’ understanding. Using a multi-agent human-in-the-loop system, we improved analytical accuracy, quantified uncertainty, and generated analytical rationales to guide customized feedback. Analyses of 90 student responses demonstrates an overall accuracy 88%, showing that integrating Generative AI with continuous expert guidance can provide reliable and meaningful feedback, even for lower elementary student writing.