Much progress has been made in text analysis, specifically within the statistical domain of Term Frequency (TF) and Inverse Document Frequency (IDF). However, there is much room for improvement especially within the area of discovering Emerging Trends. Emerging Trend Detection Systems (ETDS) depend on ingesting a collection of textual data and TF/IDF to identify new or up-trending topics within the Corpus. However, the tremendous rate of change and the amount of digital information presents a challenge that makes it almost impossible for a human expert to spot emerging trends without relying on an automated ETD system. Since the U.S. Government (USG), one of the largest purchasers of products and services, is using the Request for Proposals (RFP) to award contracts for various Information Technology and other services, this project will use Natural Language Processing (NLP) to mine the wealth of textual data that is embedded within the RFPs to develop an ETDS to identify emerging technology trends. The preliminary results show good promise that NLP may well be leveraged to mine text data and create an accurate ETDS.
Beason, Sterling; Hinton, William; Salamah, Yousri A.; and Salsman, Jordan
"Automated Analysis of RFPs using Natural Language Processing (NLP) for the Technology Domain,"
SMU Data Science Review: Vol. 5
, Article 1.
Available at: https://scholar.smu.edu/datasciencereview/vol5/iss1/1
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial 4.0 License