Disability and rehabilitation. Assistive technology · 2025
Re-imagining sensory substitution through gestural control: Point-To-Tell 2.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Ruan Ligao, Hamilton-Fletcher Giles, Beheshti Mahya, Hudson Todd E, Porfiri Maurizio, Rizzo John-Ross
Аннотация
AIM: Sensory substitution devices (SSDs) can convert environmental information into an accessible format for people who are blind or have low vision (pBLV). Yet, current SSDs often passively deliver all of the information available with limited user control, potentially leading to confusion and/or cognitive overload. To address this issue, this work proposes a selective, gesture-controlled system intended to improve information relevance and reduce cognitive overload.
METHODS: We present Point-To-Tell 2, a system that enables pBLV to privately and efficiently select which information to convey through simple pointing-based gestural control. By integrating a monocular camera with AI-driven pipelines for depth estimation, hand pose tracking, and object detection/segmentation, the system identifies the users' 3D pointing direction and announces the names and distances of objects as they are pointed at, thereby connecting an object's spatial position and identity through hand proprioception.
RESULTS: Validation tests in controlled indoor environments show high hand pose tracking accuracy, ensuring reliable ray-casting and object selection despite declining object recognition at longer distances. Distance estimates are stable at close range, though a systematic bias is present.
CONCLUSION: This work introduces and technically validates an assistive system designed to improve the usability of assistive technologies by focusing system feedback-potentially reducing users' cognitive load and enhancing their spatial comprehension by leveraging concurrent hand proprioception. Future work will involve user testing and expanding system features to further enhance its practicality across more diverse scenarios.
METHODS: We present Point-To-Tell 2, a system that enables pBLV to privately and efficiently select which information to convey through simple pointing-based gestural control. By integrating a monocular camera with AI-driven pipelines for depth estimation, hand pose tracking, and object detection/segmentation, the system identifies the users' 3D pointing direction and announces the names and distances of objects as they are pointed at, thereby connecting an object's spatial position and identity through hand proprioception.
RESULTS: Validation tests in controlled indoor environments show high hand pose tracking accuracy, ensuring reliable ray-casting and object selection despite declining object recognition at longer distances. Distance estimates are stable at close range, though a systematic bias is present.
CONCLUSION: This work introduces and technically validates an assistive system designed to improve the usability of assistive technologies by focusing system feedback-potentially reducing users' cognitive load and enhancing their spatial comprehension by leveraging concurrent hand proprioception. Future work will involve user testing and expanding system features to further enhance its practicality across more diverse scenarios.