Mining Archive.org’s Twitter Stream Grab for Pharmacovigilance Research Gold

  • Ramya Tekumalla Georgia State University
  • Javad Rafiei Asl Georgia State University
  • Juan M. Banda Georgia State University

Abstract

In the last few years, Twitter has become an important resource for the identification of Adverse Drug Reactions (ADRs), monitoring flu trends, and other pharmacovigilance and general research applications. Most researchers spend their time crawling Twitter, buying expensive pre-mined datasets, or tediously and slowly building datasets using the limited Twitter API. However, there are a large number of datasets that are publicly available to researchers that are underutilized or unused. In this work, we demonstrate how we mined over 9.4 billion Tweets from archive.org’s Twitter stream grab using a drug-term dictionary and plenty of computing power. Knowing that not everything that shines is gold, we used pre-existing drug-related datasets to build machine learning models to filter our findings for relevance. In this work, we present our methodology and the 3,346,758 identified tweets for public use in future research.

Published
2020-05-26
How to Cite
Tekumalla, R., Rafiei Asl, J., & Banda, J. M. (2020). Mining Archive.org’s Twitter Stream Grab for Pharmacovigilance Research Gold. Proceedings of the International AAAI Conference on Web and Social Media, 14(1), 909-917. Retrieved from https://www.aaai.org/ojs/index.php/ICWSM/article/view/7357