Design and Implementation of an LLM-Agent Framework for Synthetic Data Generation in Virtual Homes

Carlos Marín. (2026). Design and Implementation of an LLM-Agent Framework for Synthetic Data Generation in Virtual Homes. Trabajo Fin de Titulación (TFM). Universidad Politécnica de Madrid, ETSI Telecomunicación.

Abstract:
Research in Human Activity Recognition (HAR) for smart homes requires extensive, fully annotated datasets describing how residents interact with domestic spaces, household objects, and daily routines. However, collecting activity data in physical living spaces poses severe operational barriers, including prohibitive deployment costs, manual annotation overhead, and critical privacy and data protection concerns. This creates a structural data bottleneck that hinders the development of Ambient Assisted Living (AAL) solutions, particularly in the context of Europe’s rapidly ageing demographic. The primary objective of this master’s thesis is to design and implement an autonomous software framework capable of generating high-fidelity, auditable synthetic HAR datasets within a 3D domestic simulation environment without manual script authoring. To achieve this, the work introduces a deterministic compilation framework that couples the gen- erative reasoning of Large Language Models (LLMs) with a robust physical validation and execution engine built on the VirtualHome platform, which models domestic environments as three-dimensional semantic scene graphs. The proposed system operates as a multi-stage progressive-constraining pipeline: (1) an LLM synthesis module generates resident persona profiles, weekly activity schedules, and granular daily routine descriptions; (2) a schema validation and semantic cleaning module standardises vocabulary aliases and enforces strict Pydantic JSON contracts; (3) a scene graph grounder maps textual entity names to concrete 3D node identifiers and verifies phys- ical affordance constraints; (4) a Finite State Machine (FSM) compiler automatically injects implicit spatial navigation and posture preconditions; (5) a simulation engine executes com- piled scripts in VirtualHome over TCP sockets; and (6) a hybrid repair engine combines deterministic rules with LLM-guided diagnostic feedback loops to resolve execution failures. Experimental results demonstrate the effectiveness and robustness of the proposed frame- work. The system achieves a 100% complete simulation and rendering success rate across the comprehensive multi-environment benchmark (127/127 activity blocks across 10 scenarios and all seven VirtualHome apartment scenes; 1,422 executed actions without error), substantially outperforming the AgentSense-inspired baseline (5.51% block success; 17.72% at step level), semantic retrieval with Word2Vec (3.15% block success; 9.38% at step level), and random baseline sampling (0.00% block success; 0.94% at step level). Across the primary 3-day longi- tudinal run in Apartment 1, the system additionally achieves 100% execution success across 38/38 blocks (637 actions vs. 52.63%, 5.26%, and 0.00%). The grounding engine applies 1,785 safe compilation interventions to prevent physics engine crashes and invalid interactions. The hybrid repair module achieves a 100% repair success rate on evaluated edge cases. Further- more, the exported synthetic dataset passes 24/24 structural invariant integrity checks and enables a downstream Naive Bayes classifier to achieve 95.91% accuracy in room location prediction, confirming its practical utility for machine learning research.