Accuracy of health‐related information regarding COVID‐19 on Twitter during a global pandemic

INTRODUCTION

In December, 2019, a novel coronavirus was identified as the cause of pneumonia in a cluster of patients in Wuhan, China (Zhu et al., 2020). Since this initial outbreak, the identified virus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has spread across the globe with the World Health Organization (WHO) declaring a global pandemic on March 11, 2020 (World Health Organization, 2020a). As of April 13, 2021 there were over 136 million cases and 2.9 million deaths reported worldwide (World Health Organization, 2021). WHO control of this pandemic has involved public health measures including quarantine of exposed individuals, isolation of infected individuals, and social distancing with a heavy reliance on the public to follow recommendations from national and international experts and governments. Subsequently, the dissemination of information has become an important component of regulating behavior and guiding the worldwide response to this disease.

One of the many means of transmitting health information to the public is via social media. Twitter use rose 23% in 2020, largely due to the influence of social distancing and individuals seeking information regarding COVID-19 (Kafka, 2020). As Twitter use increased, questions have arisen about the spread of COVID-19-related misinformation on this platform (Frenkel et al., 2020). During previous infectious disease outbreaks (2012 Middle East respiratory syndrome coronavirus (MERS-CoV), 2013 H7N9, 2014 Ebola, and 2016 Zika), social media was used positively as an avenue for identifying and tracking infections, gauging a community’s response to those outbreaks, and providing information to at-risk populations (Al-Surimi et al. 2017; Fung et al., 2016, 2013; Rudra et al., 2018) However, the spread of inaccurate information during past disease outbreaks occurred in 10%–60% of Ebola-related, 61% of yellow fever-related, 25% of diphtheria-related, and 5%–10% of H1NI-related tweets (Chew & Eysenbach, 2010; Ortiz-Martinez & Jimenez-Arcia, 2017; Oyeyemi et al., 2014; Pandey et al., 2010; Porat et al., 2019; Sell et al., 2020) More recently, tweets related to COVID-19 have contained misinformation in 25%–39% of cases and unverifiable information in 17% of tweets (Brennan et al. 2020; Kouzy et al., 2020).

The International Fact-Checking Network (IFCN) has labeled COVID-19 as “the biggest challenge fact-checkers have ever faced” (Suarez, 2020). Twitter is one of the most common means of spreading public health information with nearly 123 million COVID-19-related tweets posted between January 21 and May 15, 2020, the majority of which were posted in English (Chen et al., 2020). The WHO states that in addition to a global pandemic related to COVID-19, there is a COVID-19 infodemic or an “excess amount of information (misinformation) about a problem which makes it difficult to identify a solution” (Department of Global Communications, 2020) According to the WHO, an infodemic allows for the “spread of misinformation, disinformation, and rumors during a health emergency,” hampers an effective public health response, and has the potential to create confusion and distrust among populations (Department of Global Communications, 2020).

Because Twitter is so widely used a public source of communication and news, the accuracy, relevancy, and timeliness of information is important. For this reason, we chose to study health-related information regarding Twitter during this COVID-19 global pandemic. Our primary goal was to determine the accuracy of health-related tweets related to COVID-19. A secondary goal was to determine features associated with accurate tweets.

METHODS

Between March 24 and April 5, 2020, five new Twitter accounts were established with no followers, no accounts followed, default of English language, people (anyone), location (anywhere), and highly sensitive content turned off. Before performing the study, two physician study authors created five sets of search terms to be used. Search terms were chosen to elicit tweets that addressed common issues surrounding the management of COVID-19. These included “COVID-19” plus either “cure,” “treat or treatment,” “prevent or prevention,” “emergency or emergency room,” or “supplements or vitamins.” Tweets were categorized as having an authoritative source if they were written by a government, physician/physician group, or hospital/hospital system. Tweets were defined as medical or health-related if they contained medical or health information presented as a fact, a recommendation, a statement, or an opinion. Tweets were categorized as news if they self-reported that the author was a news organization or worked for a news organization. Tweets that were primarily political, memes or jokes, or religious without health or medical claims were excluded.

Before initiating the study, a 2-h training session took place with all study authors (data abstractors) that emphasized definitions, uniform tweet reviews, and coding of information to be placed into an Excel spreadsheet. Two weeks before initiating this study, using “influenza” as a practice search term, all data abstractors simultaneously analyzed 10 non-COVID-19-related tweets to ensure uniform reviews and information collection. During the study, data abstraction, data entry, and coding rules were rereviewed with abstractors by the principal investigator after each 25 tweets. The principal investigator arbitrated all data collection and coding questions on an ongoing basis.

On three separate dates (Friday, April 17; Wednesday, April 22; Saturday, April 25, 2020) between 4:00 and 8:00 p.m. Eastern Standard Time the searches (queries) were performed on Twitter using previously selected terms. Each term was assigned to an individual study author/abstractor during this time period such that only one individual performed each search for only their assigned terms on each given day. To perform their search, five study authors logged into their new/native Twitter account and entered “COVID-19” into the search box within the Twitter search box plus either “cure,” “treat or treatment,” “prevent or prevention,” “emergency or emergency room,” or “supplements or vitamins” such that there was no overlap of searches. This time of day was selected to coincide with the peak time for retweets and click-throughs on Twitter across the United States (Sailer, 2019). On each date, the top 25 retained tweets for each search with any health or medical-related information were copied into a spreadsheet. Twenty-five tweets were chosen as the search limit since it was estimated that each tweet and accompanying links/figures/pictures could be read within one minute and the average half-life of a tweet is 18–24 min with engagement and retweets tapering rapidly after this period (Wilson, 2016). For each tweet, the following information was recorded: The tweet, any tweet hyperlinks, the author, author’s country listed on their public profile, author’s credentials, author’s Twitter verification status, the number of followers, number of retweets, number of likes, and the tweet’s rank order within that day’s search. Duplicate tweets were excluded.

An online tool termed a Botometer developed by the Observatory on Social Media and the Network Science Institute at Indiana University was used to generate a score that estimated whether or not authors of Tweets exhibited bot-like activity (https://botometer.osome.iu.edu/; Botometer, 2021). Bots are automated programs that generate messages, follow accounts, reply to or share hashtags via automation or machine learning. The Botometer uses machine learning to characterize tweet authors based on user data, temporal features/patterns, content, friends/retweets, networking/links, and sentiment-related features. According to the Pew Research Center, a Botometer score > 0.43 on a 0–1 scale (>2.15 on a 5-point scale) is the optimum cutoff for classifying a Tweet as more likely to be written by a bot than a human (Wojcik et al., 2018).

RESULTS

There were 375 tweets collected during the study period with 17 duplicates deleted leaving 358 evaluable tweets. Two hundred and sixty-seven tweets (74.6%, 95% CI: 69.8–78.8) were graded as accurate. Tweets with the search terms “COVID-19” plus “cure” were more likely to be inaccurate compared to other tweets (Table 1).

Authoritative authors wrote 69 tweets (31 government, 25 physicians, and 13 hospital tweets) and nonauthoritative authors wrote 289 tweets. A total of 67 of 69 authoritative tweets were accurate compared to 200 of 289 nonauthoritative tweets (97.1% vs. 69.2%, 27.9% difference, 95% CI: 19.2–33.8). The inaccurate tweets were written by authoritative authors comprised two physician tweets claiming COVID-19 was cured with vitamin C (Table 2). Of the subset of authoritative authors listed as government or hospitals/hospital systems, 41 (100%, 95% CI: 91–100) tweets were accurate. A total of 64 accurate and 25 inaccurate tweets were authored by self-reported news organizations (24% vs. 27.5%, −3.5% difference, 95% CI: −14.5 to 6.3).

Overall, accurate tweets were significantly more likely to be written by authoritative sources (25.1% vs. 2.2%, 22.9% difference, 95% CI: 15.6–28.6) and authors verified by Twitter (49.8% vs. 20.9%, 28.9% difference, 95% CI: 17.7–38.2). Authors of accurate tweets had significantly more followers than authors of inaccurate tweets (19,491 vs. 7346; 3446 difference, 95% CI: 234–14,054). The number of likes, tweets authored by news organizations, Botometer scores, retweets, tweet length, and Flesch–Kincaid grade level did not differ between accurate and inaccurate tweets (Table 3). North America was the most common author location, N = 171 (47.8%), with the United States comprising the source for 153 (42.7%) of all tweets (Table 3).

The median overall rank order of retained tweets was 37 (95% CI: 32–46) with a range of 1–219. The rank order of tweets was not associated with tweet accuracy (Spearman’s rho = −0.0164, 95% CI: −0.12 to 0.087).

The interrater reliability for the physicians assessing tweet accuracy was substantial (kappa = 0.77, 95% CI: 0.69–0.84).

DISCUSSION

Our study found 74.6% of COVID-19 health-related tweets to be accurate. Accurate tweets were more likely to be written by authoritative authors (physicians, hospitals, or government agencies), authors verified by Twitter, and authors with more followers compared to inaccurate tweets. We found no association between tweet accuracy and number of retweets (shares), number of likes, tweet length (number of characters), or Flesch–Kincaid Grade Level. These findings indicate that medical advice related to COVID-19 on Twitter should not be trusted when written by nonauthoritative sources. The high number of inaccurate tweets indicate that efforts to improve health-related information on Twitter regarding global outbreaks are needed.

The most common inaccurate tweets in our study comprised recommendations for using unproven prescription medicines, hydroxychloroquine or chloroquine, to treat or prevent COVID-19 (e.g., “#Hydroxychloroquine with Zn supplement cures #COVID-19”). The United States Food and Drug Administration (FDA) approves medicines after their effects have been reviewed by the Center for Drug Evaluation and Research and their benefits are found to outweigh known and potential risks for an intended population (Food and Drug Administration, 2019a). Since neither is approved for use in COVID-19, tweeted recommendations to use hydroxychloroquine or chloroquine for this disease would be classified as unapproved or off-label by the FDA. Off-label drug use is allowed by the FDA when there is no FDA-approved drug to treat a condition or there is insufficient supply of FDA-approved drugs for a particular condition (Food and Drug Administration, 2019b). Importantly, recent FDA attempts to limit off-label drug use have been constrained by court rulings that support a pharmaceutical company’s right to “free speech” when promoting such use “as long as their statements are not false or misleading” (Kim & Kapcynski, 2017). In a similar manner, the FDA does not have the authority to regulate the free speech (i.e., tweets) of individuals especially if they have no commercial interest in a product (Kim & Kapcynski, 2017).

Unproven herbs, vitamins, and supplements were another common recommendation within inaccurate tweets (e.g. “there are medicinal plants in Madagascar such as Artemisia, which can cure COVID-19”) These products, collectively termed dietary supplements, are characterized by containing at least one identified dietary ingredients such as a vitamin, mineral, herb, botanical, amino acid, enzyme, or metabolite. Dietary supplements are not approved for use by the FDA and can be brought to market without having been proven safe or effective (Harris, 2000). Product labeling, although regulated in the United States by the FDA (under the 1994 Dietary Supplement and Health Act), is much less stringently regulated. While specific disease claims are prohibited, the FDA allows for claims regarding the structure and function of these supplements with a required label that these products are “not intended to diagnose, treat, prevent, or cure any disease” (Harris, 2000; Owens et al., 2014). A 2014 study of 1300 dietary supplement retail and nonretail websites found that 20%–38% of websites made disease-related claims and only 8% of retail websites studied included the required FDA disclaimer regarding disease claims (Owens et al., 2014). This study cited a lack of FDA manpower and resources to adequately enforce labeling requirements for dietary supplements. Based on its limited ability to monitor and regulate both drugs and dietary supplements, it is likely the FDA would need more funding and more regulatory authority before it could meaningfully impact misinformation on social media platforms like Twitter.

The Federal Communications Commission (FCC) regulates interstate and international communications with authority over communications law, regulation, and technological innovation (Federal Communications Commission, 2021). While not directly tasked with verifying the accuracy of the information on social media sites, the FCC has the legal authority to interpret important aspects of communication laws regarding websites and social media companies. Section 230 of the Communications Act of 1934 and an amendment (the Telecommunications Act of 1996) state that “no provider or user of an interactive computer service shall be held liable on account of… any action voluntarily taken in good faith to restrict access to or availability of material that the provider or user considers to be obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable, whether or not such material is constitutionally protected” (Johnson, 2020). In addition to protection from criminal lawsuits, experts have interpreted Section 230 as conferring protection to social media companies from civil lawsuits due to harm caused by third-party content (i.e., harmful or inaccurate information; Barriott & Wilkens, 2020). As such, social media companies have protection from liability when either illegal or inaccurate information is posted as long as those companies make good faith efforts to remove this material. While there may be bipartisan Congressional support to amend or replace Section 230 such that companies would be required to remove or moderate inaccurate or harmful content, it is undetermined if proposed changes would meaningfully improve the accuracy or validity of health-related content on social media (Reardon, 2020).

As in our study, others found that unverified Twitter accounts contain more misinformation compared to verified accounts (authors authenticated by Twitter (Kouzy et al., 2020). The 25% rate of inaccurate information in our study is consistent with these prior studies of epidemic-related tweets (Brennan et al., 2020; Kouzy et al., 2020). These findings indicate that misinformation during disease outbreaks continues to be a common and important problem.

While most tweets listed individual authors and contained no hyperlinks, 44% of inaccurate and 36.7% of accurate tweets were authored by or had links to self-reported news sites. Knowledge regarding the accuracy of news sites is especially important since up to 67% of Americans get their news from social media (Shao et al., 2018). Inaccurate tweets found in our study included those reported by an affiliate of major news organizations (e.g., NEWS4SanAntonia/NBC promoting melatonin for COVID-19), international news organizations (e.g., abs-cbn, “virgin coconut oil could be COVID-19 cure”), entertainment news, and news sites that appeared to mix advocacy with news (e.g., vaxxter.com, “China cures coronavirus with vitamin C”). Identifying whether or not a reported news story is real and verifying the legitimacy of a news source can be difficult. While there is no single technique for verifying whether or not a story is real (vs. fake), experts recommend reading reputable news sources, reading original studies or sources, looking for verification of stories on multiple sites, ensuring there are author attributions for stories, and using fact check tools (i.e., www.snopes.com, www.factcheck.org, www.politifact.com. www.punditfact.com; Spector, 2020). An international not-for-profit organization, Health on the Net/HON, exists for verifying the accuracy and legitimacy of health and medical websites (Boyer et al., 2016) However, HON requires that individual websites request a review. The review provided by HON only analyzes eight principles of information authority (e.g., confidentiality, authorship, source attribution, supporting information, transparency, advertising, objectiveness, and financial disclosure) and not the accuracy of the information within websites (Boyer et al., 2016). It is unlikely that a similar review process could be performed on Twitter which relies on a rapid dissemination of information during news cycles with immediate commentary and feedback by users of this platform. Due to the speed of information dissemination and high volumes of information on social media, automated algorithms (i.e., artificial intelligence) are one method for identifying real versus fake news currently being studied (Lara-Navarra et al., 2020).

Social media bots or automated programs used to engage social media have been accused of spreading misleading information on Twitter (Shi et al., 2020). Botometer scores were high for both accurate and inaccurate authors of Tweets in our study indicating that a substantial amount of COVID-19-related information on Twitter might be spread by these automated programs. Tweet accuracy did not appear to be associated with bot scores. Not all bots are malicious, and many bots can perform legitimate functions. Bots that appropriately follow Twitter rules can include those that update real-time news and weather. Human Twitter authors also can automate a portion of their account by forwarding Facebook posts, sending Real Simple Syndication feeds, or tweeting/retweeting in the absence of the human user. These bot functions would tend not to disseminate misinformation. Importantly, Botometer scores fluctuate over time leading some experts to question their reliability, validity, and reproducibility (Rauchfleisch & Kaiser, 2020). A recent study using the Botometer also incorrectly misclassified humans as bots in 41%–76% of political tweets (Rauchfleisch & Kaiser, 2020). Subsequently, Botometer scores should be interpreted with caution.

In addition to relaying inaccurate medical information, Twitter has been identified as a source for unfounded conspiracy theories during disease outbreaks. Government conspiracies to hide information, control populations, or hide treatments were rumored during Ebola, Zika, influenza-H1N1, and MERS outbreaks (Sell et al., 2020; Smallman, 2015; Vijaykumar et al., 2018; Yang & Lee, 2020). With COVID-19, an unfounded rumor linking the SARS-CoV-2 virus to 5G (fifth-generation mobile phone) networks began in late January 2020 and spread rapidly leading to widespread misinformation and the burning of 5G towers in the United Kindgom (Ahmed et al., 2020). Other COVID-19 conspiracies include the deliberate release of the virus as a bioweapon and pharmaceutical companies blocking known treatments to boost their own drugs and vaccines (Neil & Campbell, 2020). Experts have noted misinformation and conspiracy theories were amplified and retweeted more than accurate tweets during the Zika outbreak (Brennan et al., 2020; Cinelli et al., 2020; Vosoughi et al., 2018). Tweets describing conspiracies in our study included five linking pharmaceutical companies with attempts to suppress available cures, two stating governments were purposefully hiding cases, and one linking death panels to COVID-19 mortality in the United States.

Multiple organizations (i.e., WHO, United Nations, Centers for Disease Control, and IFCN) and experts have provided recommendations for combating misinformation on social media (Centers for Disease Control and Prevention, 2020; International Fact-Checking Network, 2020; United Nations, 2020; World Health Organization, 2020b). Proposals include a sustained coordinated effort by independent fact-checkers like the IFCN, an independent (nonbiased) news media, tracking of misinformation plus dissemination of accurate, easy to read, information by public health and government authorities/agencies, and censorship by social media companies (Brennan et al., 2020; Garrett, 2020; Limaye et al., 2020; LLewyllen, 2020; Sell et al. 2020; Yang & Lee, 2020). Our study was conducted after  Twitter implemented a policy stating they would delete tweets that run the risk of causing harm by spreading misinformation about COVID-19. Thus, self-censorship alone may be an ineffective screen for information accuracy within Twitter (Gadde & Derella, 2020; Hern, 2020).

CONCLUSION

We found over one-quarter of health-related COVID-19 tweets to be inaccurate. Authoritative authors of tweets, especially government entities and hospitals/hospitals systems, were more likely to post accurate tweets. These findings suggest the public be wary of COVID-19 health information posted on Twitter.

Ideally, Section 230 of the amended Communications Act should be updated by Congress to hold social media companies responsible for harm from inaccurate health-related information on their sites if comprehensive attempts are not made to identify and remove this information. This action would incentivize those companies to ensure information is fact-checked and removed if potentially harmful or misleading. Increased funding for the FDA would allow enhanced review of supplement companies for removal of improper disease and medical claims within advertisements and on their websites. Legislation that requires supplements to be proven safe with oversight and approval by the FDA would potentially decrease adverse events related to supplements. While not directly addressing misinformation, such legislation would increase surveillance of these companies and potentially improve their adherence to regulations disallowing health and medical claims for their products. Other measures to improve the accuracy of health-related information on social media include enhanced public/private industry oversight, independent fact-checking, and effective automated, artificial intelligence algorithms. It is doubtful that any single approach will resolve the “infodemic” of COVID-19 misinformation on Twitter and a multi-faceted approach encompassing each of these potential solutions is needed to improve the accuracy of health-related information on social media.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests as per the ICJME guidelines.

AUTHOR CONTRIBUTIONS

All authors (Sarah B. Swetland, Ava N. Rothrock, Halle Andris, Bennett Davis, Linh Nguyen, Phil Davis, Steven G. Rothrock) were involved in the conception of study, design of the study, data collection and abstraction, drafting and revision of the manuscript. All authors analyzed the data and Sarah B. Swetland/Ava N. Rothrock/Steven G. Rothrock performed statistical analyses. All authors take responsibility for the paper as a whole.

ETHICS STATEMENT

The data, models, and methodology used in this research are not proprietary.