AAAI Publications, Second AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Testing Pre-Annotation to Help Non-Experts Identify Drug-Drug Interactions Mentioned in Drug Product Labeling
Andres M. Hernandez, Harry S. Hochheiser, John R. Horn, Rebecca S. Crowley, Richard D. Boyce

Last modified: 2014-10-14


In this study, a system for allowing combination of textmining and crowdsourcing of annotation approaches for detection of DDIs from drug package inserts is presented. An annotation study was designed to evaluate expert versus non-expert curation performance, and the impact of NLP pre-annotation on precision and recall on both groups. The design and development of the system and annotation study, consisted of three stages. First, our existing NLP pipeline for DDI extraction was improved, and it was used to preannotate 208 drug product labels with drug mentions and DDIs. Secondly, a DDI machine readable representation scheme was created using the Annotation Ontolgy. This model allowed us to load the NLP preannotated drug label sections into our plugin for human curation created using the Annotation tool DOMEO. Finally, the annotation study was performed along with usability questionnaires for collecting qualitative feedback. To our knowledge, this is the first study in comparing experts and non-experts for pharmacokinetic DDI annotation. Results showed lower performance on non-experts compared with expert annotation without the use of NLP,and an improvement of non-expert annotation performance using the NER module of the NLP assistance. Simplification of the workflow for NLP assisted annotation is necessary for scaling ourapproach.


Becker, M. L.; Kallewaard, M.; Caspers, P. W.; Visser, L. E.; Leufkens, H. G.; and Stricker, B. H. 2007. Hospitalisations and emergency department visits due to drugdrug interactions: a literature review. Pharmacoepidemiol Drug Saf 16(6):641–51.Boyce, R. D.; Gardner, G.; and Harkema, H. 2012. Using natural language processing to identify pharmacokinetic drug-drug interactions described in drug package inserts. 2391151 206-213.Ciccarese, P.; Ocana, M.; Garcia Castro, L. J.; Das, S.; and Clark, T. 2011. An open annotation ontology for science on web 3.0. J Biomed Semantics 2 Suppl 2:S4.Ciccarese, P.; Ocana, M.; and Clark, T. 2012. Domeo: a web-based tool for semantic annotation of online documents. Bio-Ontologies 2011.Percha, B., and Altman, R. B. 2013. Informatics confronts drug drug interactions. Trends in pharmacological sciences 34(3):178–184.

Full Text: PDF