IITP Project · MLCS Research Focus

XR Copilot for Construction

We develop the AI layer of an XR copilot that understands worker behavior, retrieves relevant task knowledge, and provides grounded feedback for construction component fabrication and installation.

XR Copilot Non-verbal Behavior Multimodal Learning RAG MCP Human-centered AI

In construction and manufacturing workflows, much of expert skill is not written in manuals. It appears in where experts look, how they hold parts, when they slow down, and how they inspect the result. MLCS studies how to capture these signals and turn them into guidance that workers can use through an XR interface.

Official project title: Development of XR Copilot Technology Based on Non-verbal Behavior for Quality and Safety Assurance in Construction Component Fabrication and Installation.

Counterfactual Skill Feedback

The copilot compares the worker's current motion with expert demonstrations and identifies the action gap. The goal is not only to detect that an action is different, but to suggest a small, concrete change that can improve the task.

This feedback can highlight a hand pose, wrist angle, gaze target, or inspection step, helping novice workers understand what to adjust without relying only on text instructions.

Counterfactual feedback compares current hand pose with an expert pose. — The system compares the current hand pose with an expert trajectory and highlights the action gap that should be corrected.

Agentic XR Copilot with RAG and MCP

The copilot also needs to reason over task knowledge. Given a worker's question and XR context, the LLM retrieves relevant manuals and process documents through RAG and uses MCP to call structured tools that query BIM models, schedules, sensor APIs, or perception modules.

By combining retrieved evidence with tool results, the system can generate guidance grounded in the current work situation rather than generic text.

LLM copilot agent retrieves documents through RAG and queries tools through MCP. — The LLM copilot retrieves document evidence through RAG and calls construction-specific tools through MCP before generating XR guidance.

Research Timeline

Year 1

We build pipelines for capturing first-person demonstrations, including gaze, hand pose, voice, and task context. The focus is to organize expert non-verbal behavior into reusable data for skill modeling.

Year 2

We develop counterfactual feedback models and connect the LLM copilot with task knowledge and tools through RAG and MCP. The system begins to provide personalized, context-aware guidance.

Year 3

We evaluate the copilot in realistic construction workflows, refine the XR interaction loop, and measure whether the system helps users understand tasks, reduce mistakes, and improve their actions.

What Students May Work On

Multimodal behavior data collection from first-person sensors

Gaze, hand pose, voice, and task-context modeling

Counterfactual feedback for skill improvement

LLM agents grounded by RAG and MCP tool use

XR interaction design for human-centered guidance

Field-oriented evaluation and user studies

Interested in joining this project?

We are looking for students interested in multimodal AI, robot learning, XR interfaces, and human-centered intelligent systems.

See how to apply →