login
Home / Papers / App-Based Task Shortcuts for Virtual Assistants

App-Based Task Shortcuts for Virtual Assistants

15 Citations2021
Deniz Arsan, Ali Zaidi, Aravind Sagar
The 34th Annual ACM Symposium on User Interface Software and Technology

Svant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps by mapping text and semantic information contained within UIs to identify relevant screens, is presented.

Abstract

Virtual assistants like Google Assistant and Siri often interface with external apps when they cannot directly perform a task. Currently, developers must manually expose the capabilities of their apps to virtual assistants, using App Actions on Android or Shortcuts on iOS. This paper presents savant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps. For a given natural language task (e.g., “send money to Joe”), savant leverages text and semantic information contained within UIs to identify relevant screens, and intent modeling to parse and map entities (e.g., “Joe”) to required UI inputs. Therefore, savant allows virtual assistants to interface with apps and handle new tasks without requiring any developer effort. To evaluate savant, we performed a user study to identify common tasks users perform with virtual assistants. We then demonstrate that savant can find relevant app screens for those tasks and autocomplete the UI inputs.