Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse
Por:
Kummervold PE, Martin S, Dada S, Kilich E, Denny C, Paterson P and Larson HJ
Publicada:
1 oct 2021
Ahead of Print:
8 oct 2021
Resumen:
Background: Social media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful insights to inform future interventions. However, owing to the volume of web-based posts, manual annotation and analysis are difficult and time consuming. Automated processes for this type of analysis, such as natural language processing, have faced challenges in extracting complex stances such as attitudes toward vaccination from large amounts of text. Objective: The aim of this study is to build upon recent advances in transposer-based machine learning methods and test whether transformer-based machine learning could be used as a tool to assess the stance expressed in social media posts toward vaccination during pregnancy. Methods: A total of 16,604 tweets posted between November 1, 2018, and April 30, 2019, were selected using keyword searches related to maternal vaccination. After excluding irrelevant tweets, the remaining tweets were coded by 3 individual researchers into the categories Promotional, Discouraging, Ambiguous, and Neutral or No Stance. After creating a final data set of 2722 unique tweets, multiple machine learning techniques were trained on a part of this data set and then tested and compared with the human annotators. Results: We found the accuracy of the machine learning techniques to be 81.8% (F score=0.78) compared with the agreed score among the 3 annotators. For comparison, the accuracies of the individual annotators compared with the final score were 83.3%, 77.9%, and 77.5%. Conclusions: This study demonstrates that we are able to achieve close to the same accuracy in categorizing tweets using our machine learning models as could be expected from a single human coder. The potential to use this automated process, which is reliable and accurate, could free valuable time and resources for conducting this analysis, in addition to informing potentially effective and necessary interventions. (JMIR Med Inform 2021;9(10):e29584) doi: 10.2196/29584
Filiaciones:
:
Vaccine Research Department, FISABIO-Public Health, Valencia, Spain
Martin S:
Centre for Clinical Vaccinology and Tropical Medicine, University of Oxford, Oxford, United Kingdom
Rapid Research Evaluation and Appraisal Lab, Departament of Targeted Intervention, University College London, London, United Kingdom
Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, United Kingdom
Ethox Centre, Nuffield Department of Population Health, Big Data Institute, University of Oxford, Oxford, United Kingdom
Dada S:
Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, United Kingdom
UCD Centre for Interdisciplinary Research, Education and Innovation in Health Systems, School of Nursing, Midwifery and Health Systems, University College Dublin, Dublin, Ireland
Kilich E:
Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, United Kingdom
Denny C:
Faculty of Science, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
Paterson P:
Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, United Kingdom
NIHR Health Protection Research Unit, London, United Kingdom
Larson HJ:
Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, United Kingdom
NIHR Health Protection Research Unit, London, United Kingdom
Institute of Health Metrics and Evaluation, University of Washington, Seattle, WA, United States
Chatham House Centre on Global Health Security, The Royal Institute of International Affairs, London, United Kingdom
gold, Green Published, Green Submitted, Green Accepted
FULL TEXT
 |
Submitted Version |
|
| No Accesible |
|