Abstract:
Research in Human Activity Recognition (HAR) for smart homes requires extensive, fully
annotated datasets describing how residents interact with domestic spaces, household objects,
and daily routines. However, collecting activity data in physical living spaces poses severe
operational barriers, including prohibitive deployment costs, manual annotation overhead,
and critical privacy and data protection concerns. This creates a structural data bottleneck
that hinders the development of Ambient Assisted Living (AAL) solutions, particularly in the
context of Europe’s rapidly ageing demographic.
The primary objective of this master’s thesis is to design and implement an autonomous
software framework capable of generating high-fidelity, auditable synthetic HAR datasets
within a 3D domestic simulation environment without manual script authoring. To achieve
this, the work introduces a deterministic compilation framework that couples the gen-
erative reasoning of Large Language Models (LLMs) with a robust physical validation and
execution engine built on the VirtualHome platform, which models domestic environments as
three-dimensional semantic scene graphs.
The proposed system operates as a multi-stage progressive-constraining pipeline: (1) an
LLM synthesis module generates resident persona profiles, weekly activity schedules, and
granular daily routine descriptions; (2) a schema validation and semantic cleaning module
standardises vocabulary aliases and enforces strict Pydantic JSON contracts; (3) a scene
graph grounder maps textual entity names to concrete 3D node identifiers and verifies phys-
ical affordance constraints; (4) a Finite State Machine (FSM) compiler automatically injects
implicit spatial navigation and posture preconditions; (5) a simulation engine executes com-
piled scripts in VirtualHome over TCP sockets; and (6) a hybrid repair engine combines
deterministic rules with LLM-guided diagnostic feedback loops to resolve execution failures.
Experimental results demonstrate the effectiveness and robustness of the proposed frame-
work. The system achieves a 100% complete simulation and rendering success rate across the
comprehensive multi-environment benchmark (127/127 activity blocks across 10 scenarios and
all seven VirtualHome apartment scenes; 1,422 executed actions without error), substantially
outperforming the AgentSense-inspired baseline (5.51% block success; 17.72% at step level),
semantic retrieval with Word2Vec (3.15% block success; 9.38% at step level), and random
baseline sampling (0.00% block success; 0.94% at step level). Across the primary 3-day longi-
tudinal run in Apartment 1, the system additionally achieves 100% execution success across
38/38 blocks (637 actions vs. 52.63%, 5.26%, and 0.00%). The grounding engine applies 1,785
safe compilation interventions to prevent physics engine crashes and invalid interactions. The
hybrid repair module achieves a 100% repair success rate on evaluated edge cases. Further-
more, the exported synthetic dataset passes 24/24 structural invariant integrity checks and
enables a downstream Naive Bayes classifier to achieve 95.91% accuracy in room location
prediction, confirming its practical utility for machine learning research.