Imitation learning has emerged as a powerful paradigm in robot manipulation, yet its generalization capability remains constrained by object-specific dependencies in limited expert demonstrations. To address this challenge, we propose knowledge-driven imitation learning, a framework that lever ages external structural semantic knowledge to abstract object representations within the same category. We introduce a novel semantic keypoint graph as a knowledge template and develop a coarse-to-fine template-matching algorithm that optimizes both structural consistency and semantic similarity. Evaluated on three real-world robotic manipulation tasks, our method achieves superior performance, surpassing image-based diffusion policies with only one-quarter of the expert demonstrations. Extensive experiments further demonstrate its robustness across novel objects, backgrounds, and lighting conditions. This work pioneers a knowledge-driven approach to data-efficient robotic learning in real-world settings.
Given an RGB-D image as input, the system generates a knowledge template for a specific object. This template is then matched to the demonstration, and the policy is learned based on the matching results. When encountering novel objects, the learned policy can be transferred, enabling generalization to new scenarios.
We present videos below that showcase our method's performance on different tasks, including examples of both successes and failures. These videos are played back at 5x speed. The gripper opening and closing sections were not included
@inproceedings{miao2025knowledge,
title={Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions},
author={Miao, Zhuochen and Lv, Jun and Fang, Hongjie and Jin, Yang and Lu, Cewu},
booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year={2025},
organization={IEEE}
}