Contributor
Erik Gabrielsen, Ian Johnson, Tyler Giallanza, Elena Sharp, Vianka Barbosa, Kristofor Horst
Subject Area
Computer Science
Abstract
Mobile phones and other devices with embedded sensors are becoming increasingly ubiquitous. Audio and motion sensor data may be able to detect information that we did not think possible. Some researchers have created models that can predict computer keyboard typing from a nearby mobile device; however, certain limitations to their experiment setup and methods compelled us to be skeptical of the models’ realistic prediction capability. We investigate the possibility of understanding natural keyboard typing from mobile phones by performing a well-designed data collection experiment that encourages natural typing and interactions. This data collection helps capture realistic vulnerabilities of the security of typed data.
This thesis presents an implementation and analysis of a data collection experiment from twenty participants that systematically controls for keyboard type, ambient audio noise, and table position while collecting sensor data from eight mobile phones. We found these variables to be the most important to control because they may greatly affect result capabilities. Additionally, we allow participants to type and interact normally, so we can generalize our model to realistic scenarios. We use multimodal convolutional neural networks to show that mobile phones have some capability at predicting natural keyboard typing in various evaluation scenarios.
Degree Date
Spring 2018
Document Type
Thesis
Degree Name
M.S.
Department
Computer Science and Engineering
Advisor
Eric C. Larson
Second Advisor
Mitchell Thornton
Third Advisor
Fred Chang
Number of Pages
110
Format
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial 4.0 License
Recommended Citation
Siems, Travis, "Understanding Natural Keyboard Typing Using Convolutional Neural Networks on Mobile Sensor Data" (2018). Computer Science and Engineering Theses and Dissertations. 3.
https://scholar.smu.edu/engineering_compsci_etds/3
Included in
Analysis Commons, Artificial Intelligence and Robotics Commons, Categorical Data Analysis Commons, Information Security Commons, Numerical Analysis and Scientific Computing Commons, Other Computer Sciences Commons, Software Engineering Commons, Theory and Algorithms Commons
Notes
Machine Learning, Mobile Sensing, Convolutional Neural Networks, Deep Learning, Keyboard typing detection using embedded sensors on mobile phones in various evaluation scenarios, combine sensor data from multiple phones, accelerometer, gyroscope, microphone, audio, motion, analysis, useful metrics in categorical analysis in the presence of an imbalanced distribution, macro averaged precision, recall, F1 score, overall accuracy, test generalizability across different keyboard types, device and seating positions, in the presence of significant audio noise, twenty participants using a natural typing style