Real Time Open-Vocabulary Referential Gaze Control for Listening Robots
4Amazon Robotics • 5MIT EECS
Abstract
While listening, a robot must infer from live streaming speech and visual context whether to shift gaze, what to look at, and where the referent is, within the timing constraints of natural interaction. We propose an approach which combines a multimodal foundation model and an object segmentation model for real time open vocabulary referential gaze control. To achieve fast inference without sacrificing performance, we distill a slow but capable teacher into a fast student using a novel privileged information distillation method with human egocentric videos. Across held-out egocentric data and a new out-of-distribution panoramic benchmark, our model significantly improves performance over the base student and matches that of the teacher. We measure inference time across edge-to-cloud hardware and observe that our trained model has significantly lower inference time and memory requirements. In a video-based human-subject study, our approach improves perceived message understanding, behavioral interdependence, referential attention, and competence. These results demonstrate a practical path toward real-time, open-vocabulary referential gaze control.
Overview & Contributions
Multimodal Foundation Model & Segmentation
Combines streaming speech understanding (e.g., Gemma-4) and open-vocabulary object segmentation (e.g., SAM3) to anticipate referential gaze shifts toward indirect and direct targets.
Privileged Information Distillation
Distills a slow but high-capacity teacher into a fast student network using egocentric video supervision, matching teacher accuracy while dramatically slashing compute latency.
Real-Time Edge Deployment
Optimized for edge-to-cloud hardware to ensure latency meets natural interaction constraints (< ~400ms after noun onset) on physical robotic platforms.
Human-Subject Validation
Demonstrates statistically significant improvements in perceived message understanding, behavioral interdependence, referential attention, and robot competence in user studies.
BibTeX
@inproceedings{lee2027realtime,
title={Real Time Open-Vocabulary Referential Gaze Control for Listening Robots},
author={Lee, Dong Won and Oh, Junseok and Gillet, Sarah and Ahuja, Chaitanya and Tagliabue, Andrea and Kim, Yoon and Morency, Louis-Philippe and Breazeal, Cynthia and Park, Hae Won},
booktitle={In Submission},
year={2027}
}