We develop the AI layer of an XR copilot that understands worker behavior, retrieves relevant task knowledge, and provides grounded feedback for construction component fabrication and installation.
In construction and manufacturing workflows, much of expert skill is not written in manuals. It appears in where experts look, how they hold parts, when they slow down, and how they inspect the result. MLCS studies how to capture these signals and turn them into guidance that workers can use through an XR interface.
Counterfactual Skill Feedback
The copilot compares the worker's current motion with expert demonstrations and identifies the action gap. The goal is not only to detect that an action is different, but to suggest a small, concrete change that can improve the task.
This feedback can highlight a hand pose, wrist angle, gaze target, or inspection step, helping novice workers understand what to adjust without relying only on text instructions.
The system compares the current hand pose with an expert trajectory and highlights the action gap that should be corrected.
Agentic XR Copilot with RAG and MCP
The copilot also needs to reason over task knowledge. Given a worker's question and XR context, the LLM retrieves relevant manuals and process documents through RAG and uses MCP to call structured tools that query BIM models, schedules, sensor APIs, or perception modules.
By combining retrieved evidence with tool results, the system can generate guidance grounded in the current work situation rather than generic text.
The LLM copilot retrieves document evidence through RAG and calls construction-specific tools through MCP before generating XR guidance.
Research Timeline
Year 1
We build pipelines for capturing first-person demonstrations, including gaze, hand pose, voice, and task context. The focus is to organize expert non-verbal behavior into reusable data for skill modeling.
Year 2
We develop counterfactual feedback models and connect the LLM copilot with task knowledge and tools through RAG and MCP. The system begins to provide personalized, context-aware guidance.
Year 3
We evaluate the copilot in realistic construction workflows, refine the XR interaction loop, and measure whether the system helps users understand tasks, reduce mistakes, and improve their actions.
What Students May Work On
- Multimodal behavior data collection from first-person sensors
- Gaze, hand pose, voice, and task-context modeling
- Counterfactual feedback for skill improvement
- LLM agents grounded by RAG and MCP tool use
- XR interaction design for human-centered guidance
- Field-oriented evaluation and user studies