From Single-Task Imitation to Real-World Generalization: A Hierarchical Reason-Act Robot System

Labour shortages, an aging population, and worker burnout have created urgent demand for robotic systems across household, healthcare, and elder care settings. The world is projected to face a shortage of over 50 million workers by 2030, and vital sectors like healthcare are already understaffed(Fig1-2-3).

Today, with AI breakthroughs demonstrating reasoning and understanding like never before, robotics is at a tipping point.

But existing systems face four key challenges.

Generalization: Most robotic projects today finetune a single VLA model (ex. GR00T) on a collected dataset and report overfitted performance (ie, around 99%). This is a well-known fundamental problem in machine learning: the model has memorized its training data, not learned to generalize. Once these systems encounter any unseen tasks, performance collapses. The single most important criterion of machine learning is generalization to unseen tasks.

Hallucination: When robots misbehave or hallucinate, there are no principled methods to trace failures back to specific gaps in the training data or model, making targeted correction practically infeasible.

Progression: Robotic systems lack a structured evaluation framework to systematically improve from real-world errors and validate performance improvement without regression; critical for continuous self-improvement over time.

Transparency: Most robot interfaces demand technical expertise and operate as black boxes, raising decision-making and safety concerns in sensitive environments.

This project addresses all four challenges through:

A hierarchical Reason-Act architecture that generalizes to unseen tasks

A model introspection pipeline that diagnoses and resolves hallucination

A structured evaluation harness for systematic, regression-free improvement

A voice-driven transparent interface for trustworthy human-robot interaction

Overfitting is the biggest enemy of machine learning: a model trained and evaluated on the same tasks memorizes its training data rather than learning to generalize. Most robotic projects today produce impressive overfitted performance on paper but have no capability when faced with the unseen tasks in real-world environments. This project is designed to address this.

Inspired by Kahneman's Nobel Prize-winning System 1/System 2 framework, the system(Fig1) is built around a two-tier architecture(Fig2) that separates high-level reasoning from physical execution. A customized VLM serves as the reasoning layer, decomposing multi-step unseen tasks into executable subtasks. A fine-tuned VLA serves as the execution layer, translating those subtasks into precise physical movements. An orchestrator coordinates both in real time, managing state, dispatching tasks, and maintaining context.

Training data was collected through teleoperation, capturing camera feeds and joint positions at 30 fps across 800+ demonstrations(Fig6).

Two continuous improvement pipelines drive systematic improvement over time:

Progression: Inspired by test-driven development, every session is automatically recorded and labelled: successes added to the evaluation dataset, errors flagged for correction. Every modification is backtested across the full evaluation history, ensuring fixes do not silently break previously working behaviours; allowing the system to continuously learn from its own real-world experience(Fig3).

Hallucination: Following a first-principles approach, the VLA's internal mechanism(Fig4) was mathematically unpacked(Fig5), isolating the key-value cache as the primary variable driving misbehaviour; enabling statistically targeted correction.

Real-Life Application

Speech-to-text and text-to-speech models translate between voice commands and the robot system. The system's planning, monitoring, and replanning outputs are parsed from the reasoning model's responses and surfaced in a live web interface, giving users transparent, real-time visibility into the system's decision-making process.

Robot hardware was deliberately kept low-cost(Fig6), redirecting resources towards robot system development(data collection, model training, system-engineering). The same architecture transfers to humanoid, drones, and other forms.

Single-Step Task Performance

Out of the box, the VLA model produced no meaningful task performance; domain-specific fine-tuning was required. Through systematic experimentation, three data diversity dimensions were identified as critical performance drivers: task variety (+18%), scene variation (+35%), and spatial arrangement variation (+47%). Scaling to 800+ demonstrations across all three dimensions contributed towards 97% accuracy on single-step tasks(Fig1), with predicted joint trajectories closely matching ground truth, confirming high execution precision.

Multi-Step and Unseen Task Performance

Performance on multi-step unseen tasks was evaluated through an ablation study, isolating each component's contribution(Fig2):

VLA alone: 0% on multi-step unseen tasks. A fine-tuned VLA executes memorized movements reliably within its training dataset, but presents no meaningful capability on unseen tasks.

VLA + VLM reasoning layer: 70%. A 70 percentage point jump directly demonstrates the generalization capability of the Reason-Act architecture. Structured task decomposition and planning enable the system to handle tasks it was never trained on

+ Iterative monitoring and replanning: 85% (+15%). The orchestrator detects mid-task failures and autonomously replans and retries, allowing the system to recover without human intervention.

+ Human-in-the-loop: 90% (+5%). Through the user interface, users can instruct through voice to help robot get through edge cases it cannot yet handle.

The system achieves 90% on multi-step unseen tasks using low-cost hardware; acheiving SOTA performance comparable to π0.5 (79%) and OpenVLA (83%)(Fig3). These results are a direct reflection of the architecture's generalization capability.

Systems built around a single fine-tuned VLA are typically evaluated on tasks drawn from their own training dataset. High accuracy under those conditions reflects memorization and imitation, not true intelligence.

Generalization to unseen tasks is the defining property of robotic intelligence that determines whether it is usable in the real world, not superficial high-accuracy on paper. The 90% result reported here is measured entirely on tasks and configurations the system had never encountered during training, which works as the proxy to show the system's performance in real-life application.

Evaluation and Improvement Systems

The structured evaluation harness automates the testing and improvement of the VLM: every modification to the prompt was autonomously backtested against the evaluation system before deployment, ensuring targeted fixes did not introduce regressions across previously working behaviours, while allowing the system to evolve from past mistakes.

The introspection pipeline mathematically unpacked the VLA's internal mechanism, isolating the KV cache as the key variable driving misbehaviour. By statistically analyzing the KV cache, defects in the training dataset were identified and precisely located; enabling targeted data collection that resolved misbehaviours such as hallucination directly at their source(Fig4&5).

Together, both pipelines support steady, measurable performance improvement throughout development, converting what would otherwise be unprincipled trial-and-error into a controlled and repeatable improvement process.

Robot User Interface

The voice-driven interface allows users to issue vocal commands and receive spoken responses conversationally, without technical knowledge. A live decision-making display surfaces every reasoning, planning, and replanning step in real time(Fig6); validated as an effective mechanism for building user trust and enabling informed human-in-the-loop oversight by testers.

General-purpose assistive robotics has long been constrained by projects that memorize rather than generalize, operate without transparency, and improve through brute-force data scaling rather than self-improvement-oriented engineering.

The hierarchical Reason-Act architecture achieves 97% on single-step tasks and 90% on multi-step unseen tasks, achieving state-of-the-art performance, which is ready for real-world deployment(Fig1). The system is hardware-agnostic and sector-agnostic: transferable to humanoid robots and other platforms, and extendable to healthcare, elder care, household settings, and more, through fine-tuning.

Deployment in new environments will inevitably surface new failures and misbehaviours. The evaluation harness and introspection pipeline ensure the system can diagnose those failures, collect targeted data, and continuously improve from its own experience to directly address the labour and caregiving challenges that motivated this work.

The voice-driven transparent interface ensures this capability is accessible to anyone without technical expertise. In healthcare, elder care, and other sensitive environments, users can interact conversationally through voice and see exactly what the robot is doing and why, building the trust necessary for real-world application.

Finally, collected datasets and developed robotics tools (Fig2-3-4) have been open-sourced and published on GitHub (github.com/JasmineW6214/robotagent_project) and Hugging Face (huggingface.co/datasets/jasmine314342/picknplace-bimanual-680) to contribute directly to the robotics research community.

Full project documentation, experimental methodology, and additional results are available on the project website (robotagent-project.vercel.app), alongside a library of video demonstrations showcasing the system performing real-world tasks across a range of unseen configurations at "robotagent-project.vercel.app/demos".

The immediate next step is sector-agnostic testing across healthcare, elder care, and household settings(Fig1). This includes identifying sector-specific misbehaviours, collecting targeted data, and fine-tuning accordingly.

In parallel, the architecture will be evaluated across different hardware platforms, including humanoid robots and other embodiments, to demonstrate hardware-agnostic transferability. This includes migrating the robot system from a desktop GPU setup to embedded platforms such as Raspberry Pi with GPU acceleration, enabling self-contained on-site deployment(Fig2).

On the research side, key directions include joint fine-tuning of reasoning and execution models, decomposition of demonstrations into movement primitives, and simulation to improve generalization from limited training data(Fig3-4).

There are several people I would like to acknowledge for their support throughout this project.

First, I would like to thank my parents for their encouragement and support through the many long hours of work this project required, from late nights debugging to weekends spent collecting data.

I would also like to thank BASEF mentors for their guidance and feedback during the preparation of this project, and the BASEF and CWSF organizations for facilitating this opportunity.

Finally, I would like to acknowledge the broader open-source robotics and machine learning community, whose shared knowledge and tools have been an invaluable resource throughout this work.

[1] D. Kahneman, Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux, 2011.

[2] J. Huang, "GTC 2024 Keynote: The Next Wave of AI," presented at NVIDIA GTC Conference, Mar. 2024.

[3] A. Brohan et al., "RT-1: Robotics Transformer for Real-World Control at Scale," arXiv:2212.06817, Dec. 2022.

[4] A. Brohan et al., "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control," arXiv:2307.15818, Jul. 2023.

[5] M. Ahn et al., "Do As I Can, Not As I Say: Grounding Language in Robotic Affordances," arXiv:2204.01691, Apr. 2022.

[6] M. J. Kim et al., "OpenVLA: An Open-Source Vision-Language-Action Model," arXiv:2406.09246, Jun. 2024.

[7] Octo Model Team, "Octo: An Open-Source Generalist Robot Policy," arXiv:2405.12213, May 2024.

[8] B. Y. Lin et al., "Data Scaling Laws in Imitation Learning for Robotic Manipulation," arXiv:2410.18647,

Oct. 2024.

[9] B. Wen et al., "X-VLA: Cross-Embodiment Vision-Language-Action Model with Soft-Prompted Transformers," Under Review, 2024.

[10] R. Allard et al., "SmolVLA: Efficient Vision-Language-Action Models for Embodied AI," Hugging Face Research, 2025.

[11] Team Physical Intelligence, "Scaling Up Foundation Model Techniques for Robotics: Proven Methods for VLA Robot Learning," Blog post, 2025.

[12] J. Moss et al., "Practical VLA Robot Learning for SO-ARM," LeRobot Community, 2025.

[13] J. He et al., "Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting,"

Under Review, 2024.

[14] J. Duan et al., "AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation," arXiv:2410.00371, Oct. 2024.

[15] A. Wang et al., "MolmoAct: Learning Robot Policies from Vision-Language Models," Under Review, 2025.

[16] V. Saxena et al., "What Matters in Learning from Large-Scale Datasets for Robot Manipulation," arXiv:2506.13536, 2025.

[17] A. Khazatsky et al., "DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset," arXiv:2403.12945, Mar. 2024.

[18] K. Black et al., "π0.5: A Vision-Language-Action Model with Open-World Generalization," arXiv:2504.16054, Apr. 2025.

[19] NVIDIA, "GR00T N1.5: Advancing Open Foundation Models for Generalist Humanoid Robots," 2025. [Online]. Available: https://research.nvidia.com/labs/gear/gr00t-n1_5/

[20] W. Khaksar et al., "Robotics in Elderly Healthcare: A Review of 20 Recent Research Projects," arXiv:2302.04478, Feb. 2023.

[21] ResearchGate, "Robotics in Different Fields," Figure 1 in Research Publication. [Online]. Available: https://www.researchgate.net/figure/Robotics-in-Different-Fields_fig1_388470544

[22] N. Janzen, C. Fan, and N. Powell, "Proof Point: Canada's Labour Shortages Will Outlive a Recession," RBC Economics, Jul. 2022. [Online]. Available: https://www.rbc.com/en/economics/canadian-analysis/featured-analysis/insights/proof-point-canadas-labour-shortages-will-outlive-a-recession/

[23] NVIDIA Omniverse, "Training a Robot from Scratch in Simulation, from URDF to OpenUSD," YouTube,

Apr. 2025. [Online]. Available: https://www.youtube.com/watch?v=_HMk7I-vSBQ

[24] G. Wang et al., "XLeRobot: A Practical Low-Cost Household Dual-Arm Mobile Robot Design for General Manipulation," GitHub, 2025. [Online]. Available: https://github.com/Vector-Wangel/XLeRobot

[25] T. Zhao et al., "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware," arXiv:2304.13705, Apr. 2023.

[26] C. Chi et al., "Diffusion Policy: Visuomotor Policy Learning via Action Diffusion," arXiv:2303.04137, Mar. 2023.

[27] R. Cadene et al., "LeRobot: An Open-Source Library for End-to-End Robot Learning," Hugging Face / ICLR 2026, 2024. [Online]. Available: https://github.com/huggingface/lerobot

[28] Open X-Embodiment Collaboration, "Open X-Embodiment: Robotic Learning Datasets and RT-X Models," arXiv:2310.08864, Oct. 2023.

[29] T. Buamanee et al., "Bi-ACT: Bilateral Control-Based Imitation Learning via Action Chunking with Transformer," arXiv:2401.17698, Jan. 2024.