Sign Language Video Retrieval with
Free-Form Textual Queries

Overview

Systems that can efficiently search collections of sign language videos have been highlighted as a useful application of sign language technology. However, the problem of searching videos beyond individual keywords has received limited attention in the literature. To address this gap, in this work we introduce the task of sign language retrieval with free-form textual queries: given a written query (e.g., a sentence) and a large collection of sign language videos, the objective is to find the signing video in the collection that best matches the written query.
We propose to tackle this task by learning cross-modal embeddings on the recently introduced large-scale How2Sign dataset of American Sign Language (ASL). We identify that a key bottleneck in the performance of the system is the quality of the sign video embedding which suffers from a scarcity of labeled training data. We, therefore, propose SPOT-ALIGN, a framework for interleaving iterative rounds of sign spotting and feature alignment to expand the scope and scale of available training data. We validate the effectiveness of SPOT-ALIGN for learning a robust sign video embedding through improvements in both sign recognition and the proposed video retrieval task.

Video Summary

Citation

Acknowledgements

The authors would like to thank M. Fischetti, C. Marsh and M. Dippold for their work on data annotation and also A. Dutta, A. Thandavan, J. Pujal, L. Ventura, E. Vincent, L. Tarres, P. Cabot, C. Punti and Y. Kalantidis for their help and valuable feedback.
This work was supported by the project PID2020-117142GB-I00, funded by MCIN/ AEI /10.13039/501100011033, ANR project CorVis ANR-21-CE23-0003-01, and gifts from Google and Adobe. Amanda Duarte received support from la Caixa Foundation (ID 100010434), fellowship code LCF/BQ/IN18/11660029. Samuel Albanie thanks Z. Novak and N. Novak for enabling his contribution.

The website template was borrowed from Jon Barron .