Erik Gabrielsen, Ian Johnson, Tyler Giallanza, Elena Sharp, Vianka Barbosa, Kristofor Horst

Subject Area

Computer Science


Mobile phones and other devices with embedded sensors are becoming increasingly ubiquitous. Audio and motion sensor data may be able to detect information that we did not think possible. Some researchers have created models that can predict computer keyboard typing from a nearby mobile device; however, certain limitations to their experiment setup and methods compelled us to be skeptical of the models’ realistic prediction capability. We investigate the possibility of understanding natural keyboard typing from mobile phones by performing a well-designed data collection experiment that encourages natural typing and interactions. This data collection helps capture realistic vulnerabilities of the security of typed data.

This thesis presents an implementation and analysis of a data collection experiment from twenty participants that systematically controls for keyboard type, ambient audio noise, and table position while collecting sensor data from eight mobile phones. We found these variables to be the most important to control because they may greatly affect result capabilities. Additionally, we allow participants to type and interact normally, so we can generalize our model to realistic scenarios. We use multimodal convolutional neural networks to show that mobile phones have some capability at predicting natural keyboard typing in various evaluation scenarios.

Degree Date

Spring 2018

Document Type


Degree Name



Computer Science and Engineering


Eric C. Larson

Second Advisor

Mitchell Thornton

Third Advisor

Fred Chang


Machine Learning, Mobile Sensing, Convolutional Neural Networks, Deep Learning, Keyboard typing detection using embedded sensors on mobile phones in various evaluation scenarios, combine sensor data from multiple phones, accelerometer, gyroscope, microphone, audio, motion, analysis, useful metrics in categorical analysis in the presence of an imbalanced distribution, macro averaged precision, recall, F1 score, overall accuracy, test generalizability across different keyboard types, device and seating positions, in the presence of significant audio noise, twenty participants using a natural typing style

Number of Pages




Creative Commons License

Creative Commons Attribution-Noncommercial 4.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial 4.0 License