Erik Gabrielsen, Ian Johnson, Tyler Giallanza, Elena Sharp, Vianka Barbosa, Kristofor Horst
Mobile phones and other devices with embedded sensors are becoming increasingly ubiquitous. Audio and motion sensor data may be able to detect information that we did not think possible. Some researchers have created models that can predict computer keyboard typing from a nearby mobile device; however, certain limitations to their experiment setup and methods compelled us to be skeptical of the models’ realistic prediction capability. We investigate the possibility of understanding natural keyboard typing from mobile phones by performing a well-designed data collection experiment that encourages natural typing and interactions. This data collection helps capture realistic vulnerabilities of the security of typed data.
This thesis presents an implementation and analysis of a data collection experiment from twenty participants that systematically controls for keyboard type, ambient audio noise, and table position while collecting sensor data from eight mobile phones. We found these variables to be the most important to control because they may greatly affect result capabilities. Additionally, we allow participants to type and interact normally, so we can generalize our model to realistic scenarios. We use multimodal convolutional neural networks to show that mobile phones have some capability at predicting natural keyboard typing in various evaluation scenarios.
Computer Science and Engineering
Machine Learning, Mobile Sensing
Number of Pages
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial 4.0 License
Siems, Travis, "Understanding Natural Keyboard Typing Using Convolutional Neural Networks on Mobile Sensor Data" (2018). Computer Science and Engineering Theses and Dissertations. 3.
Analysis Commons, Artificial Intelligence and Robotics Commons, Categorical Data Analysis Commons, Information Security Commons, Numerical Analysis and Scientific Computing Commons, Other Computer Sciences Commons, Software Engineering Commons, Theory and Algorithms Commons