User training from the perspective of information

User training from the perspective of information literacy: the case of the Capes Portal of E-Journals Fernanda Gomes Almeida Doctoral Student at the Graduate Program in Information Science School of Information Science Federal University of Minas Gerais, Brazil +55 31 3409-4627 [email protected]

Beatriz Valadares Cendon Full Professor, Graduate Program in Information Science School of Information Science Federal University of Minas Gerais, Brazil +55 31 3409-6118 [email protected] ABSTRACT This paper presents a qualitative research study that aimed at determining the impact of user training in the information literacy level of users of the Capes Portal of E-Journals, a governmental initiative in Brazil. The paper starts with an account on the Capes Portal of E-Journals and the actions promoted by the Federal University of Minas Gerais (UFMG) Library to promote access to and use of the Portal information resources. Next, the article presents a review about information literacy, information searching and construction of search strategies. The methodology was mostly qualitative and used tests, interviews and observation. Data collection was conducted in two stages: pre-instruction and post-instruction. 14 students who enrolled voluntarily in a training program offered by the university library to the UFMG community formed the sample in the first stage - preinstruction. After the pre-instruction test, the students went through an instructional session. In the second stage of the research, post-instruction test and interviews, seven of the 14 students present at the initial stage participated. Data collection was conducted through observation with computer screen recording in the pre-instruction test and, two weeks later, in the post-instruction test. In the second stage, data collection also used interviews. The second standard of the ACRL Information Literacy Competency Standards for Higher Education guided data collection and analysis. Results led to the conclusion that participation in training sessions positively influenced the development and achievement of information literacy. The methodology developed for the study is a contribution of the research, providing both objective and subjective assessment of information literacy of university students.

Keywords Capes Portal of E-Journals; information literacy; assessment; higher education; university library; user training; qualitative methods. 1. INTRODUCTION This paper reports on a research study that aimed at verifying the impact of user training sections in the level of students’ information literacy. The study was conducted at the campus of the Federal University of Minas Gerais (UFMG), Brazil with students who participated in a training session on the use of the Capes Portal of E-Journals, an important information resource in the scenario of Brazilian universities. In this study, conducted within the Graduate Program in Information Science at UFMG, impact was assessed by the identification of a positive relationship between the student participation in an instructional session and the acquisition and development of information competencies related to the second standard document of the Association of College and Research Library (ACRL) - Information Literacy Competency Standards for Higher Education (ACRL, 2000). According to Standard Two - "The information literate student accesses needed information effectively and efficiently. The next session presents the Capes Portal of E-Journals and the actions promoted by the Federal University of Minas Gerais (UFMG) Library to promote access to and use of the Portal information resources. Follows a review about information literacy, information searching and construction of search strategies. The remainder of the paper discusses the methodology, results and conclusions. 2. PORTAL CAPES OF E-JOURNALS Launched in 2000, the Portal Capes of E- Journals is perhaps the largest digital library of scientific journals in Brazil (Cendon, Ribeiro and Souza, 2011), and in Latin America. Financed entirely by the Brazilian government, it is a unique model of consortium of its kind (CAPES, 2015). The history of the Capes Portal of E-Journals starts in 1990 when the Brazilian Ministry of Education (MEC) established a program for university libraries in order to strengthen higher education in Brazil. Five years later the Support Program for Journal Acquisition (Programa de Apoio à Aquisição de Periódicos - PAAP), was created. This was the origin of the current Capes Portal of E-Journals of the Institute for Development of Higher Education Personnel (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Capes) (CAPES, 2015). After 15 years of its inception, in November 2015, the Capes Portal is considered an essential tool for teaching and research in Brazil. Offering easy access to the best scientific publications in the world and to updated knowledge; permitting democratic access to information in the entire Brazilian territory and international integration of scientific knowledge it is a priceless resource for the scholarly community.

The Capes Portal has, at the time of writing, 424 participating institutions, distributed throughout the national territory. Currently, the Portal provides access to about 37,000 journal titles and 126 reference databases in all areas of knowledge, as well as books, patents, technical standards and other materials. Since its launch, the Capes Portal has changed over time, both in regards to the amount of the available content as well as in regards to the search interface. The Portal introduced a major change in 2009 with the inclusion of a federated search engine, MetaLib, by the company Ex Libris. In 2012, the search engine was updated to Metalib Plus, which uses a central index (Primo Central Index). Metalib Plus is a web discovery service based on cloud computing, which centralizes the index of (almost) all the resources offered by the Portal, enabling faster information retrieval around a single search interface. In July 2014, the Capes Portal started to present its current interface keeping all existing services and features in the previous version – boxes for searches by subject, journals, books or databases and the possibility of user accounts for customized access (CAPES, 2015). There are two approaches for searching: in simple search, the search will be conducted in the Central Index of metadata collected previously by the Primo Central Index, and, in the advanced search, the search will be conducted in real time in selected databases (federated search) (Almeida and Cendon, 2014). 3. LITERATURE REVIEW This research used as theoretical framework studies on user information searching behavior, particularly studies on users of bibliographic databases. Undertaking a search, a process where a user interacts with a system, is not an easy or simplistic endeavor but a cyclical process where best terms are selected, an effective strategy is conceived, options are evaluated, selected, tested and reviewed constantly, so that searchers can assess the impact of the search and change their routes and processes (Debowski, 2001; Almeida and Cendon, 2014). Also fundamental to this study was the development and growing importance of the concept of information literacy. The consolidation of the understanding that students in higher education should develop skills in the use of information led to belief on the need for standards for information literacy for libraries and for library users. 3.1 Information Literacy Paul Zurkowski introduced the concept of information literacy in 1974, in a context of technological change. In 1989 the American Library Association (ALA) published a final report defining the concept of information literacy and its importance for education, citizenship and the workforce in the age of information (Rader, 2006). According to the Presidential Committee on Information Literacy: Final Report, all people should have access to information that can improve their

lives helping them meet a wide range of personal and business needs. To meet their information needs, people should be information literate and be able to "[...] to recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information." (ALA, 1989). In Brazil, the information literacy movement comes slowly in the early twenty-first century as part of the field of Library and Information Science (LIS). Nevertheless, discussions on this matter need to be intensified both in the basic education and in the higher education fields (Gasque, 2012). In 2000, the Association of College & Research Libraries, a division of the ALA, published the Information Literacy Competency Standards for Higher Education, thus elevating the visibility of the issue of information literacy (Rader, 2006). This document has five standards, 22 performance indicators and 87 expected results. (ACRL, 2000). The current research focused only on Standard Two - "The information literate student accesses needed information effectively and efficiently." (ACRL, 2000). Table 1. Information Literacy Competency Standards for Higher Education Standard The information literate student One Defines and articulates the need for information Two Accesses needed information effectively and efficiently Evaluates information and its sources critically and Three incorporates selected information into his or her knowledge base and value system Individually or as a member of a group, uses information Four effectively to accomplish a specific purpose Understands many of the economic, legal, and social Five issues surrounding the use of information and accesses and uses information ethically and legally 3.2 Information Retrieval Information retrieval can be seen as the process of searching a set of documents to identify those about a subject (Lancaster, 1979) within an information retrieval system (IRS). Souza (2006) points out that the information retrieval systems organize and enable access to information items having functions of representation, storage, physical management and retrieval of information. Users resort to information systems with the intention of satisfying their information needs. Needs can be conceptualized as a mental or psychological state of the individual, or a dissatisfaction regarding information. To meet the need for information, the user materializes their need in a search query. The IRS compares the search query with the representation of documents in the system, retrieving those wherein the representation match the search query (Pérez Gutierrez, 2000). Some authors point out that the IRS does not inform or change the user's knowledge on the subject researched, but only informs about the existence of documents concerning the search query (Lewis and Jones, 1996).

In the interaction with the system, the user must identify the best terms to describe the need for information and then develop an effective strategy search strategy - to retrieve the necessary items (Debowski, 2001). 3.2.1 Search Strategies The search strategy can be defined as the set of actions and decisions taken during the search that affects outcomes of the IRS in terms of items retrieved or not retrieved (Hartley et al, 1990). It is considered a medium to communicate an information need to the system (Kowalski and Maybury, 2000). According to Lancaster (1979) the preparation of the search strategy involves the analysis and translation of the terms. Through the interfaces of IRSs users translate their needs into questions or keywords (Souza, 2006). The retrieval of unstructured textual information can be performed according to classical models – Boolean, vector or probabilistic (BaezaYates and Ribeiro-Neto, 2011). When a search is performed, the system compares the search query to records in the system to find out those which have the search terms. (Hartley et al, 1990; Rowley, 2002). The current study focuses only on the Boolean model where the search expression consists of terms connected by Boolean operators - AND, OR and NOT - and in response to the search, the system presents documents that satisfy the search expression (Ferneda, 2003). 4. METHODOLOGY The research adopted a qualitative approach. The study was conducted in the campus of the Federal University of Minas Gerais – UFMG. The nonprobabilistic, accidental sample was formed by graduate and undergraduate students enrolled voluntarily in the training sessions on searching / using the Capes Portal offered by the University Library to the UFMG community. The survey was conducted in two stages. In the first stage, pre-instruction, 14 students went through a test where they received a search guide with tasks. Of these, 13 were undergraduates and one was a graduate student. In the second phase, post-instruction, 7 undergraduate students participated in the test also going through the search guide. No participant of the second stage of the research had attended previous training session or had had any guidance on searching the Capes Portal, other than the training session they received in this study, which is important for assessing the impact of the instruction to which they were subjected. The research used participant and direct observation and screen recording in both phases of research as methods for data collection, as well as interview and tests. Screen recording was performed using the free software AutoScreenRecorder that allows screen recording directly from the computer, capturing images and mouse movement. The observations and screen recording were guided by the aspects covered in the second standard

and its indicators as detailed in the document Standards for Information Literacy in Higher Education - Information Literacy Competency Standards for Higher Education - developed by ACRL (Association of College and Research Libraries). According to the ACRL document: “To be information literate, a person must be able to recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information.” (ALA, 1989). The second stage of the research also included a semi-structured interview, recorded by recorder, in addition to observations and screen recording of the participants. The study focused only on the second standard of the ACRL (2000) Information Literacy Competency Standards for Higher Education. This standard states that “The information literate student accesses needed information effectively and efficiently”. This document contains five standards, 22 performance indicators and 87 expected outcomes. For the current research just a few indicators and outcomes belonging to standard two which were considered pertinent to the objectives of the study, the research object and the time available for evaluation. The expected outcomes and indicators selected for the study will be presented in the Table 2. Two search guides were developed to test the participants in their use of the Portal Capes, one for the pre-instruction test and another for the postinstruction test. The topics addressed in the guides were based first, in the second standard of the Information Literacy Competency Standards for Higher Education, its indicators and expected outcomes selected for the study (Table 2) and, second, in the search options available in the Capes Portal E-Journals interface. Table 2. Performance Indicators and selected Outcomes for research Performance Indicators and selected Outcomes 2.1.D – Selects efficient and effective approaches for accessing the information needed from the investigative method or information retrieval system 2.2.B - Identifies keywords, synonyms and related terms for the information needed 2.2.D - Constructs a search strategy using appropriate commands for the information retrieval system selected 2.3.A - Uses various search systems to retrieve information in a variety of formats 2.4.A - Assesses the quantity, quality, and relevance of the search results to determine whether alternative information retrieval systems or investigative methods should be utilized 2.4.B - Identifies gaps in the information retrieved and determines if the search strategy should be revised 2.4.C - Repeats the search using the revised

Tasks in the preinstruction test

Tasks in the postinstruction test

T1, T3, T4

T1, T2, T3

T1, T4

T1, T3

T1, T4

T1, T3

T1, T3, T4

T1, T2, T3

T1, T4

T1, T3

T1, T4

T1, T3

T1, T4

T1, T3

strategy as necessary 2.5.A - Selects among various technologies the most appropriate one for the task of extracting the needed information 2.2.E - Implements the search strategy in various information retrieval systems using different user interfaces and search engines, with different command languages, protocols, and search parameters

T2, T3

T1, T2, T3

T4

T3

In the first stage, the students enrolled for the training sessions were invited to participate voluntarily in the study. Participants received the search guide for pre-instruction test as prepared by the researcher and were again informed about the screen recording. Participants had about 30 minutes to perform the tasks indicated in the search guide and respond the other questions. After testing, the students participated in a training session, lasting 2 hours and 30 minutes. In the second stage, a letter of invitation for the next meeting was sent via email to the 14 participants of the first stage (pre-instruction test). Among the 14 students who participated in the initial stage, only 7 took part in the second. All were undergraduate. As students from any area of knowledge, graduate or undergraduate, at UFMG, potentially could enroll in the training session, both search guides presented to the participants, as the first search topic, subjects of general interest and easy acceptance such as social networks (in the pre-instruction search guide) and scientific communication (in the post-instruction search guide). The first set of tasks (Task 1) in both search guides – corresponding to outcomes 2.1.D, 2.2.B, 2.2.D, 2.3.A, 2.4.A, 2.4.B, 2.4.C in Table 1- was to find, in the Capes Portal, information on the topic proposed. While the tasks presented in the pre and post-instruction search guides were different, the underlying concepts of information literacy remained the same. The set of tasks 2 (Task 2) proposed in the pre-instruction search guide – corresponding to outcome 2.5.A in Table 1 - was to select and save an article using the features found in the «Search by subject» option (Task 1). In the post-instruction search guide this task was incorporated into Task 1. In Task 3 of the pre-instruction search guide which corresponded to Task 2 in the post-instruction search guide the participant was asked to search for a specific journal and save an article of that journal on the computer. These tasks covered the performance indicators and outcomes 2.1.D, 2.3.A and 2.5.A. Task 4 in the pre-instruction search guide which corresponded to Task 3 in the post-instruction search guide – and to outcome 2.2.E - allowed to assess whether the participants could perform a search in similar information retrieval systems with different search interfaces, applying the information literacy concepts in other databases. For that end, reference or full text databases which had search interfaces similar to that of the Portal were identified and listed in these questions.

After the completion of the post-instruction test participants were interviewed following an interview guide previously developed. The interviews aimed to verify the perception of students related to the training session, to the use of Capes Portal and to their search skills after participating in the training session. Data collected in the video recordings were analyzed based on the second standard of the Standards for Information Literacy in Higher Education (ACRL, 2000), the selected indicators and the expected outcomes for the search. To analyze the data collected in the interviews, categories were created which were based on the mixed model proposed by Laville, Dione and Siman (1999). These authors prescribe that the construction of categories can be performed a priori based on the theoretical framework and on the interview guide, and also allow the inclusion and modification of categories from the interpretation of the data collected. 5. PRESENTATION AND ANALYSIS OF RESULTS Performance of participants in the pre-instruction and post-instruction test was compared to verify if there was improvement after the training session. For expected outcome 2.2.B, the identification of keywords remained at the same level. The identification of synonyms and related terms improved in the post-training test, particularly the identification of corresponding terms in other languages. In the post-instruction test, only one participant did not identify terms related to research topic or corresponding terms in other languages. For expected outcome 2.2.D there was improvement especially regarding the understanding and use of Boolean operators and quotes for the construction of the search strategy. Only one participant, however, used truncation. None of the participants used parenthesis in the construction of the search strategy. For expected outcome 2.2.E, results show that participation of students in the training session contributed to the implementation of similar search strategies in other systems, such as other databases available via the Portal Capes. These databases have search interfaces which differ from the Portal’s. Results show that the skills developed in the training session were applied in situations other than the ones experienced in the instruction session. For expected outcomes 2.1.D and 2.3.A, when the task involved the selection of a database for the research, noticeable progress was made in the selection and use of the most appropriate search option (2.1.D) or most appropriate system for the research (2.3.A). While the participants in the pre-instruction test showed difficulty to identify the most appropriate search option in the Portal interface – Database searching - in the post-instruction test, this choice was more direct and conscious. No improvement was noticed in the correct selection of the search option – Subject searching -, while for the search option – Journal searching - only one participant showed improvement.

For the expected outcome 2.4.A – the user assesses the quantity, quality, and relevance of the search results – analysis led to the conclusion that after training participants remained at the same level in relation to the results obtained in the pre-instruction test. For expected outcomes 2.4.B and 2.4.C - identifies gaps in the information retrieved and determines if the search strategy should be revised and repeats the search using the revised strategy as necessary – results suggest that students modified the initial search strategy more often after participation in the training session, but it is not possible to demonstrate a direct connection of this behavior with participation in the training session. For expected outcome 2.5.A – selects among various technologies the most appropriate one for the task of extracting the needed information – results show that there was a slight improvement in article extraction (i.e. saving the article and obtaining the full text) from search options – Journal searching and Database searching. There was no significant change in article extraction from search option - Subject searching. In general, the article extraction was not a problem for participants in both tests. However, participants had difficulties in the selection of the article from the search results, a step previous to the article extraction. This happened for various reasons: lack of knowledge about the system interface or about the subject proposed for the search, or even errors that happened because the full text databases were not available. Being unable to select the article,the students could not perform the next step of extracting the text. 6. CONCLUSIONS The methodology proposed allowed the evaluation of the information literacy level related to the second standard of the Information Literacy Competency Standards for Higher Education (ACRL, 2000) of students participating in training sessions on use of the Capes Portal. Results indicate progress was made in the student’s information literacy level after participating in training activities on the Journal Portal promoted by the University Library. The methodology developed for the study stands out as a positive contribution of the research, since the objective assessment of information literacy of university students has not been object of much discussion and research in Brazil. In this study the evaluation of the development of the information literacy level of the students was conducted in an objective way - through direct and participant observation and screen recordings - and also subjectively - through interviews, based on the opinions of the participants. Some limitations of the methodology should also be pointed out: the lack of a control group in the study; the awareness about the post-instruction test by the participants; the short time between the completion of the pre-instruction test and the post-instruction (only two weeks); the use of different search guides in each test with also different search topics; and proposed activities unrelated to real needs for the student research. Despite the limitations of the study, results showed improvement by students regarding the development of information literacy after the training session. The highest positive results were observed in outcomes 2.2.B -

Identification of keywords, synonyms and related terms; 2.2.D - Use of Boolean operators (AND, OR, NOT), truncation, quotes and parentheses; and 2.2.E - Implementation of the search strategy in various information retrieval systems. 7. REFERENCES Almeida, Fernanda Gomes and Cendon, Beatriz Valadares. 2014. Information literacy in the Capes Portal of E-Journals: the impact of user training programs. QQML Journal, 4 (Dec., 2014), 857–863. http://www.qqml.net/papers/December_2014_Issue/347QQML_Journal_2014 _AlmeidaandValadaresCendon_Dec-857-863.pdf American Library Association. 1989. Presidential Committee on Information Literacy: Final Report. Chicago. http://www.ala.org/acrl/publications/whitepapers/presidential Association of College & Research Libraries. 2000. Information literacy competency standards for higher education. Chicago. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf Baeza-Yates, R. and Ribeiro-Neto, B. 2011. Modern information retrieval: the concepts and technology behind search. 2nd ed. Addison Wesley, Harlow. Cendon, Beatriz Valadares, Souza, Juliana Lopes and Ribeiro, Nádia Ameno. 2011. Satisfação dos Usuários do Portal da Capes: Um Estudo sobre a Obtenção de Sucesso no Uso do Sistema. Perspectivas em Ciência da Informação, 16, 2, 67-100. http://portaldeperiodicos.eci.ufmg.br/index.php/pci/article/view/1236 Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. 2015. Portal de Periódicos da CAPES. http://www.periodicos.capes.gov.br Debowski, Shelda. 2001. Wrong way: Go Back! An Exploration of Novice Search Behaviours while Conducting an Information Search. The Electronic Library, 19, 6, 371- 382. Ferneda, Edberto. 2003. Recuperação de Informação: Análise sobre a Contribuição da Ciência da Computação para a Ciência da Informação. Doctoral Thesis. Escola de Comunicação e Artes, Universidade de São Paulo. Gasque, Kelley Cristine Gonçalves Dias. 2012. Letramento informacional: pesquisa, reflexão e aprendizagem. Faculdade de Ciência da Informação/Universidade de Brasília, Brasília. Hartley, R.J. et al. 1990. Online searching: principle and practice. London: Bowker-Saur, London. Kowalski, Gerald J and Maybury, Mark T. 2000. Information storage and retrieval systems: theory and implementation. 2nd ed. Kluwer Academic Publishers, Norwell. Lancaster, F. W. 1979. Information retrieval systems: characteristics, testing and evaluation. 2nd ed. John Willey, New York. Laville, Christian, Dionne, Jean; Siman, Lana Mara de Castro (1999). A construção do saber: manual de metodologia da pesquisa em ciências humanas. Artmed, Porto Alegre. Lewis, David D. and Jones, Karen Sparck. 1996. Natural language processing for information retrieval. Communications of the ACM, 39, 1, 92-101 Pérez Gutiérrez, Mario. 2000. El lenguaje de interrogación: una gramática formal para la recuperación de la información. Revista Española de Documentación Científica, 23, 3, 247-266. Rader, Hannelore. Foreword. In Neely, Teresa Y. Information literacy assessment: standards-based tools and assignment. ALA, Chicago. Rowley, Jennifer. 2002. A biblioteca eletrônica. 2 ed. Briquet de Lemos, Brasília.

Souza, R. R. 1996. Sistemas de recuperação de informações e mecanismos de busca na web: panorama atual e tendências. Perspectivas em Ciência da Informação, 11, 2 (May./Aug. 2006), 161-173.