Measuring the accuracy and learnability of tools in the ...

Measuring the Accuracy and Learnability of Tools in the Struggle Against Misinformation in Social Media Applications Alexandre Pinheiro

Ademar Aguiar

Faculdade de Engenharia da Universidade do Porto Porto, Portugal [email protected]

Faculdade de Engenharia da Universidade do Porto Porto, Portugal [email protected]

Claudia Cappelli

Cristiano Maciel

Universidade Federal do Estado do Rio de Janeiro Rio de Janeiro, Brazil [email protected]

Universidade Federal do Mato Grosso Mato Grosso, Brazil [email protected]

Abstract — Misinformation became pervasive on social media applications. The companies behind this kind of system have launched tools to avoid the problem, but some issues regarding the user behavior and proper software quality still need a forceful approach. First attempts to mitigate misinformation did not take into account user behavior and softwares requirements like learnability and accuracy, furthermore the characteristics of actors and artifacts from social media applications ecosystem has not been explored. This research aims to evaluate the usability of available tools made to combat the spread of misinformation and to verify the interrelationship between actors and artifacts from social media applications ecosystem for suggesting improvements on development of these tools. Keywords – Misinformation; Social Media Ecosystem; Usability; Accuracy; Learnability.

I. INTRODUCTION The scale-out model of social media software and the unprecedented statistics regarding the continuous adherence of new individuals to these applications' users bases, allows such artifacts of information and communication technology (ICT) to provide a portrait of the society by emulating several human behaviors in an online environment. Among human behaviors, some are good and others are harmful to the community. One harmful behavior that can be detected on social media applications is the spread of misinformation. Since human activities can potentially lead to a hostile state of reality [1] and people rely in a cognitive path when they access information [2], social media applications with its capacity to build relationships and emergent aspect as informational sources, need to be prepared to deal with the human being characteristics. In this way, this work is based on evaluation of users’ behavior and actions when they face the product of misinformation spreaded on social media applications and their ability to use the tools made to mitigate this problem.

Additionally, we have a scenario to analyze the accurateness of these tools cited above and explore the possible contributions of artifacts and actors from social media software ecosystem in their struggle against the misinformation flow. II. RESEARCH PROBLEM When people are misinformed, they may make decisions that are not in their interest with serious consequences. If a group of individuals believes in something that is incorrect, the misinformation may support decisions that run against to the community wellness [3]. Despite the subject of misinformation spread on Internet being a trend topic of discussions and research nowadays, this phenomena is not exclusive to the digital environment. From the popularization of technologies that supported mass media arise, people were already exposed to the problem of misinformation. A notorious example of the consequences of biased information dissemination is the radio broadcast of “The War of the Worlds” made in 1938 by Orson Welles. The Welles' fiction novel was transmitted with journalistic narrative, deceiving those who heard the stream. Just as radio, the television also had issues such as dubious advertisements followed by various propaganda techniques relatively unrestricted [4]. After major events of dangerous behavior, some actions against misinformation on mass media were adopted. The creation of regulatory marks, state control and ethics approaches were measures taken to avoid the problem. Unlike radio and television, the reach of consumers to the Internet was faster and surpassed by the reach of social media applications. We must consider that, in the case of social media applications, the users themselves generate content and they are increasingly lead to do it. From this outlook, emerges the question: How to fight the misinformation spread on social media applications that encourage the sharing of information, considering the huge number of users inserted into information “bubbles” shaped to their own preferences, with limited knowledge of system features or even without the will to use it to mitigate the problem?

Analyzing their responsibility and under pressure by the growing cases of rumors, fake news and hoaxes shared on their systems, the companies behind the social media applications have launched tools and took serious actions to avoid the spread of misinformation. However, these new tools and applications features have not shown accuracy. Among examples of this gap of accuracy are the case of Twitter suffering with automated “bot” accounts generating misinformation even after hardened the policy to prevent the creation of this kind of account [5] and Sina Weibo has, at least, one rumor widely spread per day even having its own rumor-busting service [6]. The Facebook became the center of attention after the last US presidential elections due to a massive exposure of users to fake news regarding political affairs, mainly attempts to influence voter's opinion [7]. The Facebook's system of promoted publications was used in a malicious way to spread misinformation. While it is a hard task to develop tools with accuracy to prevent misinformation spread [8] and ensure the users learnability on how to use these tools, the companies responsible for the social media applications have not well explored the capacity of the actors and other artifacts from social media ecosystem to help in this challenge. Some artifacts (fact-check websites, trusted information repositories) and relevant actors from social media applications (companies, brands, government agencies, public figures) can help the developers of tools against misinformation by following guidelines of mutual cooperation, using the structure of the ecosystem in which they are inserted. III. OUTLINE OF OBJECTIVES The aim of the study is to analyze if the currently available tools made to mitigate misinformation on social media applications can fulfill the purpose for which it were developed and if the users can learn how to use these tools. Furthermore, we propose improvements for these tools considering the role of other stakeholders (beyond the average users) in the social media applications ecosystem and previous research on auditability of information in artifacts from this environment. Among the complementary objectives, we have: x

To do a literature review on spread of misinformation from earlier effects on mass media to the new ICT;

x

To map the stakeholders of the social media ecosystem, their influences and how they are affected by the misinformation or how they can help to combat it;

x

To develop a prototype of social media application containing key features to test the research issues with a sample of users statistically significant, since we cannot implement our tools improvements suggestions in a real one industry artifact;

x

Improve and expand for a wide range of social media applications a catalog of information auditability [28] previously made only for Online Social Networks domain.

IV. INVESTIGATION TOPICS To set a research background we need to review the literature about few subjects to support our work. Below this line, we present the topics covered by this study.

A. Misinformation An information assumed to be true but that need to be corrected in some way is considered as misinformation [9]. The term misinformation is designed for information that is accidentally false due a mistake [10]. Despite the existence of a suggestion of taxonomy that distinguish types of bad information, we will follow other authors using the term misinformation for classifications such as rumors, hoaxes, fake news, conspiracy theories [3] [11]. Misinformation can be related to subjects like politics, medicine, science, theology, race, gender, and others. Common topics containing misinformation on Internet are related to bad practices like instigate hatred, racism, theological intolerance, self-made medical diagnosis and crimes [12]. Misinformation is one of the main problems to our society and it became pervasive, especially on social media applications [13]. Since people are inclined to belief an information someway consistent with their system of judgment, this makes misinformation difficult to be discovered and resistant to correction due to cognitive factors [14]. The exchange of information between people became easier due the use of social media applications, thus the potential to generate or share misinformation in this way is considerable. Information sources surrounded by a social context are more likely to have pieces of misinformation [15]. B. Social Media Ecosystem According to Jansen et al. [16], a software ecosystem is defined by the relationships between software and services working together and its interaction in a shared market. A well done reuse of resources and the exchange of information between stakeholders under software ecosystem are important to improve the quality of social media applications, mainly because this kind of software provide interrelationship products such as Application Programming Interfaces (APIs). The dynamism of the today’s world makes it important to have the competence to know the ecosystem of social media. Several platforms have been emerging and providing new ways of interaction on Internet [17] and the development of tools for social media applications must be in line with the complexity established by the exchanges between stakeholders of these technologies [18]. Since the vertices of social media applications can be associated to a wide range of actors, content and metadata [19] it is possible to combine this characteristic with other artifacts of the ecosystem to perform hard tasks such as mitigation of misinformation. As example, the social media applications can consult fact-check websites to filter fake news. C. Software Quality Quality can be defined as the set of characteristics present in a commodity and it is based on the positive value that these characteristics represent for the users of this commodity (software) [20]. Ensure productivity and acceptance of software are some of the major challenges for developers. Furthermore, a good implementation of usability became an imperative requirement for nowadays applications, so the responsibility of developers increases, especially for social media applications considering the variety of profiles and the huge amount of people using these softwares.

The International Standards Organization (ISO) developed standards that can be used in software development to ensure usability of the products. Two of these standards can give guidance to create accurate and easy-to-learn software and respective tools. To check the existence of characteristics showing the usage of the ISO standards on the application development will help to measure its accuracy and learnability. The ISO/IEC 9126 [21] has guidelines focusing the end-user perception of quality, so if the developers follows these guidelines the end-user will perform tasks in an efficient, fast and understandable way facilitating the software learnability. The ISO 9241-11 [22] describes the software context of use and the guidance for software characteristics measurements in an explicit manner [23]. Under the measurements described by the standard, we can find orientation to evaluate software (and tools) accuracy. V. RELATED WORKS Some academic works address issues about misinformation spread in social media applications, the actions did by the companies that support these applications to reduce this problem and the user behavior dealing with unreliable sources of information. Research on software features that present lack of usability and effectiveness are also considered. A survey of Zubiaga et al. [10] provides guidelines on detecting rumors in social media and clarifies the phenomena of misinformation in such applications. This work details the challenge of combating the spread of rumors, analyzes tools and artifacts made by industry and academic community as an effort to mitigate the misinformation. Ngai et al. [24] describes a systematic review of social media research made in the last years. The importance of this study is the report of frameworks and constructs used in these researches, since that support new ideas of work like the one described in this paper and concerning on user behavior toward social media. According to Pennycook and Rand [25] a Facebook's tool that allow the users to tag fake articles and then send it for analysis of “third party fact-checkers” has problems with “implied truth” effect and was considered not enough to undermine belief in fake news. The implied truth effect is based in the observation that only a fraction of misinformation will be marked with the tool since it is easier to create misinformation than verify its reliability. Basically, the tool is doomed to deal with a lot of information that is hard to assess. VI.

METHODOLOGY

Our methodological approach will follow specifications of measurement of software usability. Using ISO/IEC 9126 [21] and ISO 9241-11 [22] international standards for software evaluation we expect to check if available tools made to mitigate misinformation on social media applications can successfully complete its target tasks. Our suggested tools with improvements will be submitted to this assessment too. This kind of classification is mentioned on ISO 9126 standards as Accuracy under the functionality requirements of external and internal software quality.

To test the accuracy of tools by emulating a social media application environment, we will need the support of a dataset misinformation threads. This data will be collected by programmatically crawling content from fact-check websites. These sites keep records of misinformation (mainly fake news) found circulating on Internet according to the spread relevance in respective locations. After collect, we will manually filter data choosing long stand misinformation items. This kind of misinformation is characterized by running on the Internet for a while. We also will need to collect reliable information from trusted sources, preferably with the same topics of the misinformation cited above. It will be manually collected. The valid information will be mixed up with the misinformation on our repository to support our study questions. The accuracy of tools can be validated by checking its capacity of successfully distinguish misinformation between trusted information and show it to the user [26]. Learnability is a characteristic of usability defined by the capacity of the software product to be learned and understood by the user [23], but some problems like spread of misinformation in social media applications are associated to incomprehensible software features. It is not enough to provide tools to avoid the problem, but to ensure that users learn to use these tools. To analyze the learnability of software by users we will use a summative evaluation. This method aims to check if the system tools meets the requirements which it were developed and compare the tools to other made for the same purpose available in similar applications [27]. The emulated social media application used in accuracy tests will also support the measure of learnability. We will apply several metrics such as task performance (e.g. tasks completed by users without any help), command usage (e.g. commands that user already knows) and metrics based on user feedback. Due to the difference of dimensions between measuring the research issues on a prototype application and to do the same on a real one application, a sample size of users representing the target population (social media application users) will be considered for the tests. VII. EXPECTED O UTCOMES This research has it core basis in works about auditability of information in social networks and considerations on software development and its usability to prevent user mistakes or misunderstandings that increase misinformation spread. The approach of the research sequence presented in this paper regards that misinformation affects other applications of social media beyond online social networks. Furthermore, new efforts made by social media application developers to provide tools to mitigate misinformation, although necessary, are not successful. There are problems in these tools, mainly regarding usability and lack of utilization of information that can be retrieved from other artifacts of the social media applications ecosystem to help in the struggle against misinformation.

Our first work based on this broader approach [29] deals with improvements and more transparency in the Facebook's account verification process. The social media applications must have a well-designed and accessible tool to validate key user accounts, company, brands and governmental pages, so the information generated by these stakeholders can be reliably identified and malicious impersonations avoided. Currently the research is in the evaluation stage of social media application tools related to mitigation of misinformation. This evaluation is allowing us to collect characteristics of these tools that need to be present in our prototype for the usability tests cited in objectives section. The usability testing project and a systematic review on aspects derived from a catalog of transparency of information in social networks are take into account in our agenda, since this catalog will be extended to other social media applications and it was part of our previous research. Our proposal for the doctoral symposium in CISTI 2018 is to present the issues regarding misinformation spread in social media applications and discuss the role of its ecosystem's actors into the efforts to mitigate this problem. Another highlight for discussion is the user's learnability and interest in tools developed to combat misinformation and the methodologies to measure this characteristic. ACKNOWLEDGMENT This research is funded by Coordination for the Improvement of Higher Education Personnel (CAPES), Brazil and developed in a Doctoral Mobility under an International Cooperation Agreement between the Faculty of Engineering of the University of Porto (FEUP) and the Federal University of the State of Rio de Janeiro (UNIRIO). REFERENCES [1]

[2]

[3]

[4]

[5]

[6]

[7] [8]

A. A. Anderson, D. Brossard, D. A. Scheufele, M. A. Xenos, and P. Ladwig, “The nasty effect: online incivility and risk perceptions of emerging technologies,” Journal of Computer-Mediated Communication, vol. 19, no. 3, 2014, pp. 373–387. D. A. Scheufele and B. V. Lewenstein, “The public and nanotechnology: How citizens make sense of emerging technologies,” Journal of Nanoparticle Research, vol. 7, no. 6, 2005, pp. 659–667. S. Lewandowsky, U. K. Ecker, C. M. Seifert, N. Schwarz, and J. Cook, “Misinformation and its correction: Continued influence and successful debiasing,” Psychological Science in the Public Interest, vol. 13, no. 3,pp. 106–131, 2012. D. Rohatyn, “The (mis) information society: an analysis of the role of propaganda in shaping consciousness,” Bulletin of Science, Technology& Society, vol. 10, no. 2, pp. 77–85, 1990. S. Haustein, T. D. Bowman, K. Holmberg, A. Tsou, C. R. Sugimoto, and V. Larivière, “Tweets as impact indicators: Examining the implications of automated bot accounts on twitter,” Journal of the Association for Information Science and Technology, vol. 67, no. 1, pp. 232–238, 2016. F. Yang, Y. Liu, X. Yu, and M. Yang, “Automatic detection of rumor on Sina Weibo,” in Proceedings of the ACM SIGKDD Workshop on Mining Data Semantics. ACM, 2012, p. 13. H. Allcott and M. Gentzkow, “Social media and fake news in the 2016 election,” National Bureau of Economic Research, Tech. Rep., 2017. G. L. Ciampaglia, “Fighting fake news: a role for computational social science in the fight against digital misinformation,” Journal of Computational Social Science, pp. 1–7, 2017.

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18] [19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

S. Lewandowsky, U. K. Ecker, and J. Cook, “Beyond misinformation: Understanding and coping with the post-truth era,” Journal of Applied Research in Memory and Cognition, vol. 6, no. 4, pp. 353–369, 2017. A. Zubiaga, A. Aker, K. Bontcheva, M. Liakata, and R. Procter, “Detection and resolution of rumours in social media: A survey,” arXivpreprint arXiv:1704.00656, 2017. C. Shao, G. L. Ciampaglia, A. Flammini, and F. Menczer, “Hoaxy: A platform for tracking online misinformation,” in Proceedings of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, 2016, pp. 745–750. F. Safieddine, W. Masri, and P. Pourghomi, “Corporate responsibility in combating online misinformation,” International Journal of Advanced Computer Science and Applications, vol. 7, no. 2, pp. 126–132, 2016. WEF, “Global risks report 2013,” Ed. Lee Howell – World Economic Forum, Cologny/Geneva, Switzerland, Tech. Rep. Eighth Edition, 2013. [Online]. Available: http://reports.weforum.org/global-risks- 2013/riskcase-1/digital-wildfires-in-a-hyperconnected-world/ A. Bessi, M. Coletto, G. A. Davidescu, A. Scala, G. Caldarelli, and W. Quattrociocchi, “Science vs conspiracy: Collective narratives in the age of misinformation,” PloS One, vol. 10, no. 2, p. e0118093, 2015. F. Gabbert, A. Memon, K. Allan, and D. B. Wright, “Say it to my face: Examining the effects of socially encountered misinformation,” Legal and Criminological Psychology, vol. 9, no. 2, pp. 215–227, 2004. S. Jansen, A. Finkelstein, and S. Brinkkemper, “A sense of community: A research agenda for software ecosystems,” in Software EngineeringCompanion. ICSE-Companion 2009. 31st International Conference on IEEE, 2009, pp. 187–190. R. Hanna, A. Rohm, and V. L. Crittenden, “Were all connected: The power of the social media ecosystem,” Business horizons, vol. 54, no. 3,pp. 265–273, 2011. K. Larson and R. Watson, “The value of social media: toward measuring social media strategies,” in ICIS 2011 Proceedings, vol. 10, 2011. S. Papadopoulos, Y. Kompatsiaris, A. Vakali, and P. Spyridonos, “Community detection in social media,” Data Mining and Knowledge Discovery, vol. 24, no. 3, pp. 515–554, 2012. G. Tassey, “The economic impacts of inadequate infrastructure for software testing,” National Institute of Standards and Technology, RTI Project, vol. 7007, no. 11, 2002. “ISO/IEC 9126 – Software Product Evaluation – Quality Characteristics and Guidelines for the User,” International Organization for Standardization, Geneva, Switzerland, Standard, 2001. “ISO 9241-11 – Ergonomic requirements for office work with visual display terminals (VDT’s), Part 11: Guidance on Usability specification and measures,” International Organization for Standardization, Geneva, Switzerland, Standard, 2008. A. Abran, A. Khelifi, W. Suryn, and A. Seffah, “Usability meanings and interpretations in ISO standards,” Software quality journal, vol. 11, no. 4, pp. 325–338, 2003. E. W. Ngai, S. S. Tao, and K. K. Moon, “Social media research: Theories, constructs, and conceptual frameworks,” International Journal of Information Management, vol. 35, no. 1, pp. 33–44, 2015. G. Pennycook and D. G. Rand, “The implied truth effect: Attaching warnings to a subset of fake news stories increases perceived accuracy of stories without warnings,” SSRN Digital Library, 2017. M. Mendoza, B. Poblete, and C. Castillo, “Twitter under crisis: Can we trust what we rt?” in Proceedings of the first workshop on social media analytics. ACM, 2010, pp. 71–79. T. Grossman, G. Fitzmaurice, and R. Attar, “A survey of software learnability: metrics, methodologies and guidelines,” in Proceedings of the SIGCHI Conference, vol. 10, 2011 A. Pinheiro, “Projetando Auditabilidade de Informações em Softwares de Redes Sociais,” Master’s thesis, Universidade Federal do Estado do Rio de Janeiro - UNIRIO, Rio de Janeiro, Brasil, 2015. A. Pinheiro, C. Cappelli, and C. Maciel, “Building up a verified page on Facebook using information transparency guidelines,” in International Conference on Social Computing and Social Media. Springer, 2017, pp. 125–137.