The last decade has witnessed a tremendous increase in the digital multimedia data including images, audio, videos as well as News, entertainment and sports channels. Such enormous collections of videos have opened up a whole new world of challenges to develop smart content retrieval systems allowing users an efficient and effective retrieval of desired content. Traditional video search engines rely on matching the provided query words with user-assigned tags for retrieval purposes. Such systems do not take into account the actual content of video which itself can serve as a semantic index for robust and efficient retrieval. Content-based search engines, on the other hand, exploit the rich visual, audio and textual content to index and subsequently retrieve videos. Among these, the focus of our current product lies on the textual (News tickers, scorecards, credits etc.) and audio (spoken words) content. More specifically, we have developed a comprehensive video retrieval system that allows users to input (query) keywords and retrieve all videos where the keyword has appeared (either in speech on in the form of caption text). State-of-the-art deep learning-based algorithms are applied for text detection and recognition while an off-the-shelf API is employed for speech recognition. Taking into account the local needs, we have targeted text and audio in two languages, Urdu and English but the system can easily be extended to other languages as well. Once the videos are indexed, users can provide query keyword and all the videos (and frames) where the keyword has appeared are returned to the user. The system is developed as a desktop application while retrieval can also be supported through web if required.

Funding Body: IGNITE, National Technology Fund.
Partner: Associated Press of Pakistan (APP)