Transparency as a learning strategy to teach Software Engineering Elizabeth Suescún Monsalve1, Paola Vallejo1, Raúl Mazo1,2 and Daniel Correa3 1
2
Universidad EAFIT- Grupo GIDITIC, Medellín, Colombia Université Panthéon Sorbonne - Centre de Recherche en Informatique (CRI), Paris, France 3 Universidad Nacional de Colombia – Facultad de Minas, Medellín, Colombia
[email protected],
[email protected],
[email protected],
[email protected]
Abstract. Background: Transparency is anchored in the principle of disclosure information. In pedagogy, transparency emerges as an important issue that proposes to aware students about the educational processes and contents. Here, we investigate the viability of using the transparency concept through the SimulES game as a pedagogical support to teach an introductory course in Software Engineering (SE). Objective: This paper presents SimulES games for conducting an experience with students in introductory course in SE using transparency approach, and it is illustrated and evaluated by quasi-experiment in class. Method: To support this proposal two experiences were designed, one played with a preliminary teacher’s guidance about the rules of the game. The other one used a traditional expository lecture, both experiences followed by a feedback. Briefly, we aim at evaluating to what extent the use of pedagogical transparency in the context of game-based learning helps students to better understand the basic concepts of SE. To assess the impact of pedagogical transparency in the context of game-based SE learning. Results: We examine the experiences and outcomes from seven groups, of which two are published. From these, we note there were not significant differences. Both groups acquired a similar quantity of knowledge about SE concepts. Conclusions: We identify our challenges as improving the experiment design with a large quantity of subjects, and selecting a different SE topic to involve higher level students in the experiment; performing better experiments, improving data analysis, and minimizing validation threats. Keywords: Software Engineering, Games Based Learning, Transparency.
1
Introduction
According to Somerville, “Software engineering is an engineering discipline that is concerned with all aspects of software production from the early stages of system specification to maintaining the system after it has gone into use” [1]. Thus, SE is not just concerned with the technical processes of software development but also with activities such as software project management and with the development of tools, methods and theories to support software production. In this context, we recognize the importance that each of these aspects has for students. And it is quite challenging to be able to teach all these aspects in traditional classes. That is why we are looking for alternatives that stimulate the work within all these aspects during the lecture time just like if the students were working in real environments. There are studies intending to improve the teaching methods, like the ones based on the concept of pedagogical transparency in the context of game-based learning (GBL)
2
[2, 3]. According to Monsalve [3], transparency is a pedagogical strategy that aims to improve the quality of teaching, and the relationship between students, teachers and teaching methods. This teaching approach uses a vision anchored in the principle of transparency as information disclosure [4]. “Transparency in pedagogy emerges as an important issue which aims at making the learner aware about his teaching-learning process and content production” [2]. In addition, concepts which are explained in a practical way are better suited to promote the learning of increasingly complex new concepts, which are considered important for the development of thought [5]. The remainder of this paper is structured as follows: Section 2 presents the requirements for teaching SE in a more engaging manner. Section 3 describes the experiment performed and the evaluation of our approach. Section 4 presents important aspects from this experience. And finally, Section 5 rounds out the paper with some conclusions and an outlook on future work.
2
Using SimulES to teach Software Engineering transparently
The advantage of using games in learning processes are manifold; for instance, GBL is designed to balance subject matter with gameplay and the ability of the player to retain, and apply the acquired knowledge to the real world [6]. In addition, Ebner and Holzingerb [7] suggest that there is empirical evidence showing that the learning results of using games is at least equivalent to the results from learning using traditional methods. Besides, GBL also allows learners have access to the concepts in dynamic and operational way. The effectiveness of GBL is not only evident in its informativeness but also for application concepts in a practical way. Monsalve and Leite [8] present a study which shows how GBL is one of the most preferred modes of learning from the student’s perspective. Other important evidence is related to long-term learning, the experiment's result presented in [9] using GBL revealed that using the situation-problem strategy allows the learners to deduce and apply concepts learned in real situations. Therefore, in this paper we want to study to what extent the use of GBL together with the concept of transparent teaching can offer to the students a motivating environment that allows them to learn the fundamentals of SE. First, GBL is comparable to the traditional form of teaching concerning immediate knowledge gains. GBL has a significant medium positive effect size regarding retention. Then, it does not disturb the process, on the contrary, the student can gain in motivation and reasoning about SE. Second, GBL is not detrimental to information transmitted in the expository lecture but not strengthened by the game. Then, GBL combined with transparency would give us the opportunity to present study tools that are revealing for students, with open processes, feedback and with active participation of students.We justify our proposal in the fact that the concept of transparency is already being used in other social, governmental and software contexts [10]. The society has a growing interest for transparency as information disclosed [4]. It is a right that allows citizens to have free access to information, enabling them to be aware of acts, actions and conditions that an organization offers for services or products, so that they are sustained and justified by organizations for the citizen. This fundamental definition is recently being used in the context of pedagogy adopting much of its principles.
3
We claim that transparency brings on the evolution of teaching which is undergoing transformations, with students who seek to be more participatory, compare and exercise their rights. SE is our context of interest therefore we believe that the use of games as tools combined with pedagogical foundation and focused on the concepts of transparency can bring benefits in the process and in the final result. Our motivation is to use the concept of transparency in a learning process. For that reason in this paper we show how, through the SimulES game, we incorporate that concept to teach SE. In particular, we use SimulES as relatively simple game that does not requiere special skills to be played, just a few minutes for training and rules description and that challenge players with clearly definited goals reacheable within session class to play. SimulES is an educational board and cards game [11] which is an evolution of the ideas of the PnP game [9]. Different from PnP, SimulES does not have any specific development process and the development process can be explored pedagogically during the game; for instance, one player can use an agile approach whereas other one can follow a waterfall process. Exploring different development processes can be organized by the instructor, such as an after-game discussion about development approaches. The main components of the SimulES game are: project cards, engineer cards, problem cards and concept cards as presented in Fig. 1. Further information about SimulES can be found in [12].
Fig. 1. Project card, Software Engineer card, Problem card and Concept card.
3
The quasi-experiment design
During lasts years, industrial and research communities have used empirical studies with students (ESWS) to gain insight into new or existing techniques and methods [13]. ESWSs are useful to obtain preliminary evidence in support of or against research hypothesis. ESWSs should be conducted in an adequate way, addressing appropriate goals, do not overstate the generalizability of the results and considering threats to internal and external validity. There are different types of ESWS like: controlled experiments, quasi-experiments, correlation studies, case studies, or surveys. We select a quasi-experiment study because randomization was not possible with the students of the undergraduate courses. In particular, we carried-out a quasi-experiment in order to (i) obtain preliminary evidence about how SimulES works, and (ii) evaluate the following hypothesis (being: H0 the null hypothesis, H1 the alternative hypothesis). H0 (null): Providing novice SE learners with SimulES, and through a transparent teaching strategy, increases their knowledge of SE concepts in a similar way that traditional learning.
4
H1: Providing novice SE learners with SimulES, and through a transparent teaching strategy, increases their knowledge of SE concepts in greater proportion than with a traditional learning. At this point it is important to highlight that the intention was not only to obtain preliminary evidence about the “transparency concept” but to compare the acquisition of knowledge through a traditional learning versus a transparent game-based learning (specifically using the SimulES game). The quasi-experiment was divided into five different phases which are represented in Fig. 2 and described thereafter.
Fig. 2. Quasi-experiment phases.
A. Participants’ selection: We decided to select 40 undergraduate students from the Computer Science program at the EAFIT University. They were part of the first semester of this career, and they attended to a mandatory course named Principles in Software Development. This course introduces issues related to SE. B. Pre-questionnaire: The second phase of the quasi-experiment was to hand out a pre-questionnaire to the previous students. The questionnaires were designed using a Likert scale [14]. For all the questionnaires in this quasi-experiment (both pre- and post-), the Likert items had a five-point format: (1) strongly disagree, (2) somewhat disagree, (3) neither agree nor disagree, (4) somewhat agree, and (5) strongly agree. The pre-experiment questionnaire was used to confirm the students’ lack of knowledge about SE concepts, and to carry out the group formation (which is described in the next phase). It was very simple, only two questions. We had out this prequestionnaire to the students one week before the first day of class, they had to attend to some introductory sessions and they completed the pre-questionnaire in one of these sessions. The pre-questionnaire is detailed in Appendix A [15] and the answers are detailed in Appendix B [15]. Group formation: Based on the previous results we discarded some students, then, we divided the students in two groups; each group with 10 subjects. Groups were not randomly constituted; on the contrary, the assignment of the students to G1 or G2 was based on the results of the pre-questionnaire (see Section 4). Group1 (G1) or baseline: Its subjects learnt some SE concepts through a 2 hours lecture session (traditional learning). Group2 (G2) or experimental: Its subjects learnt some SE concepts through SimulES game (game-based learning). C. Treatments: In the fourth phase, the students were submitted to a treatment, where each group was introduced to their own experiment environment. Treatment A: this treatment was only applied to G1 (i.e., baseline). The experiment was conducted in a classroom with a 2 hours lecture session about SE concepts.
5
Treatment B: this treatment only applied for G2 (i.e., experimental). The experiment was conducted in a classroom with the SimulES game. We divided the 10 subjects into 3 groups (3-3-4 subjects per group, respectively), each group played with its own SimulES game. This session also took 2 hours. D. Post-questionnaire: At the end of the quasi-experiment, the students of both groups were submitted to the post-questionnaire (Appendix C [15]); this phase only took 15 minutes. This test allowed verifying whether students acquired or not new knowledge and it also contained some questions related with the experiment environment (satisfaction and external factors). The main post-questionnaire questions contained six true/false statements. The answers from all the participants are detailed in Appendix D [15]. Results about the post-questionnaire are discussed in the next section.
4
Results and validation threats
As previously described, the subjects were submitted to a pre-questionnaire to carry out the group formation and to screen out possible background and basic skills deviations, and at the end of the quasi-experiment, a post-questionnaire collected further data. To provide statistical relevance in the analysis of the questionnaires items, the results are interpreted as described next. Let the null hypothesis be denoted as H0, the alternative hypothesis as H1, the baseline group as G1, the experimental group as G2 and p the probability estimator of wrongly rejecting the null hypothesis. Then, the alternative hypotheses are either: (i) H1:G2≠G1, the experimental group differs from the baseline, (ii) H1:G2G1, the measure in the experimental group is greater than the baseline. The outcomes of the two treatments were compared for every answer using the nonparametric, two-sample, rank-sum Wilcoxon-Mann-Whitney [16] test, with n1=10 and n2=10, being n1 and n2 the number of subjects of G1 and G2. The significance level for all tests was set to 5%, so probability values of p ≤ 0.05 are considered significant, and p ≤ 0.01 considered highly significant. The corresponding alternative hypothesis are further detailed for each question, and a summary of the base statistics and corresponding test values can be found in appendices B and D [15]. A. Group formation and background: This section rejects subjective difference amongst the students with respect to their basic skills. Pre-questionnaire responses do not show significant difference (Q1 p = 0,651; Q2 p = 1,000) between both groups. Q1 responses show that students did not have ac-quaintance about SE Q2 responses show that students did not have participated in SE courses. This was a mandatory condition to the effective prosecution of the quasi-experiment goals.
6 Table 1. Summary of pre-questionnaire results between the Baseline Group (G1) and Experimental Group (G2), including the values of the non-parametric significance Wilcoxon-MannWhitney test. Question 1
x (G2) 1,70
σ (G2) 0,48
x (G1) 1,80
σ (G1) 0,42
H1 ≠
p-value 0,651
Question 2
1,00
0,00
1,00
0,00
≠
1,000
B. External Factors: The quasi-experiment environment was an important concern. In a common working place, there are aspects out of control (inter-participants interaction, disturbances, noise, etc.), so the main idea was to discard possible validation threatening environmental factors. External factor questions responses do not show significant difference (Q1 p = 0,485; Q2 p = 0,490) between both groups. Q1 responses show that students did not find the experience intimidating. Q2 responses show that students did not feel distracted by other students. Both questions serve to discard validation treats related with external factors. Table 2. Results of the external factors between the baseline group (G1) and experimental group (G2), including the values of the non-parametric significance Wilcoxon-Mann-Whitney test. Question 1
x (G2) 2,00
σ (G2) 1,15
x (G1) 1,70
σ (G1) 1,06
H1 ≠
p-value 0,485
Question 2
1,70
0,48
2,20
1,32
≠
0,490
C. Overall satisfaction: This collection of questions was intended to obtain subjective information about the satisfaction that the students had with the different learning methodologies, by questioning subjects on their performance, comfort and feel for the presented learning environment and material. Overall satisfaction questions responses do not show significant difference (Q1: p = 0,679; Q2: p = 0,427; Q3: p = 0,348) between both groups. Q1 responses show that students find adequate the time to develop the experiment. Q2 responses show that students find the experience fun. Q3 responses show that students find the experience conducive to learning. Both groups had similar results. This could make sense, because this was the first college experience of both groups of students which could be taken as a fun first experience. Table 3. Overall satisfaction results between the baseline group (G1) and experimental group (G2), including the values of the non-parametric significance Wilcoxon-Mann-Whitney test. Question 1
x (G2) 4,30
σ (G2) 0,48
x (G1) 3,90
σ (G1) 1,20
H1 ≠
p-value 0,679
Question 2
4,20
0,79
4,50
0,53
≠
0,427
Question 3
4,30
0,48
4,50
0,71
≠
0,348
7
D. Software engineering questions: To measure the increase in SE knowledge, a set of 6 questions (appendix C [15]) were presented to the students at the end of the experiment. These questions intended to ascertain how much correct information about SE the students had acquired. The relevance of an item-to-item analysis of the scores is not so much important as the total amount of knowledge the students acquired. Thus, the results are shown aggregated and processed as the total knowledge acquired. Table 4. Questions and answers about software engineering. Question
1
2
3
4
5
6
Correct answer
T
F
F
T
F
T
Table 5. Software engineering questions group statistics. Question
N
Mean
Standard Deviation
Standard Error Mean
G1 - Baseline
6
4,100
0,738
0,3300
G2 - Experimental
6
3,500
0,707
0,3162
Table 5 shows subjects from baseline group G1 had an average of 4.1 correct answers of 6 questions; G2 had an average of 3.5/10. The results of the “software engineering questions” responses are very similar, so we cannot obtain any preliminary evidence to falsify the null hypothesis and therefore, confirm the alternative hypothesis of this quasi-experiment.
5
Conclusion and future work
This article shows that the use of SimulES through a transparent learning experience can be used as an option to teach the basic concepts of SE in undergraduate courses. SimulES was designed based on the transparency concept, a pedagogical strategy that aims to improve the quality of teaching, and the relationship between students, teachers and teaching methods. SimulES was compared with a traditional learning methodology (a lecture session) and a quasi-experiment was designed and executed. The main idea was to obtain preliminary evidence about how SimulES works and compare how students acquire knowledge using this game-based learning experience and compared with a lecture session. The results show that there is no significant difference between a group of students who learnt with SimulES and a group of students who learnt with a traditional lecture session. Both groups acquired a similar quantity of knowledge about SE concepts. Therefore, the students were submitted to different questions related with their background and the experiment environment; and there were no significant differences between the results of both groups. We are working on a new experiment in which the use of transparency as a teaching strategy will not depend of the use of SimulES and will not depend of a particular topic of SE. This modification of the experiment will permit us to better assess the
8
impact of transparency in teaching SE. In addition, we intend to replicate that experiment in several universities and courses to better test the new hypotheses related, exclusively, to the use of transparency in teaching software engineering.
References [1] Sommerville, I.: Software Engineering. 8th edn. Pearson Education, p. 7. England (2007). [2] Monsalve, E., do Prado Leite, J. and Werneck, V. :Transparently teaching in the context of game-based learning: the case of simulES-W. In: ICSE, 2015 IEEE/ACM 37th IEEE International Conference, vol. 2, pp. 343-352. IEEE (2015). [3] Monsalve, E.: Uma Abordagem para Transparência Pedagógica usando Aprendizagem Baseada em Jogos (Doctoral dissertation, PUC-Rio) (2014). [4] Leite, J. and Cappelli, C.: Software Transparency, Business & Information Systems Engineering, vol. 2, nº 3, pp. 127-139 (2010). [5] Resende, A. and Valdés, H.: Galperin: implicações educacionais da teoria de formação das ações mentais por estágios, Educação & Sociedade, vol. 27, nº 97, pp. 1205-1232 (2006). [6] EdTechReview. What is GBL (Game-Based Learning)?. http://edtechreview.in/dictionary/298-what-is-game-based-learning, last accessed 2017/05/20. [7] Ebner, M. and Holzinger, A.: Successful implementation of user-centered game based learning in higher education: an example from civil engineering, Computers and Education, vol. 49, nº 3, pp. 873–890 (2007). [8] Monsalve, E. and Leite, J.: Using i* for Transparent Pedagogy. In: 7th Internacional i* Workshop, Valencia (2013). [9] Baker, A. and Navarro, E. and van der Hoek, A.: Problems and Programmers: an educational software engineering card game. In: Proceedings 25th Intern. Conference on Software Engineering, IEEE Computer Society Press, Portland (2003). [10] Brom, C. and Preuss, M. and Klement, D.: Are educational computer micro-games engaging and effective for knowledge acquisition at high-schools? A quasi-experimental study. Computers & Education 57.3 (2011). [11] Figueiredo, E. M. and Lobato, C. A. and Dias, K. L. and Leite, J. C. and Lucena, C. J. P.: Um Jogo para o Ensino de Engenharia de Software Centrado na Perspectiva de Evolução. In: Workshop sobre Educação em Computação (WEI – 2007), Rio de Janeiro (2007). [12] Monsalve, E. and Werneck, V. and Leite, J. C. S. P.: Teaching Software Engineering with SimulES-W. In: Proceedings of XXIV Conference on Software Engineering Education and Training (CSEE&T), Hawaii (2011). [13] Carver, J. and Jaccheri, L. and Morasca, S. and Shull, F.: A checklist for integrating student empirical studies with research and teaching goals. Empirical Software Engineering, 15(1), 35-59 (2010). [14] Flores, N.: Patterns and Tools for Improving Framework Understanding: a Collaborative Approach (Doctoral dissertation, University of Porto) (2012). [15] Quasi-experiment Appendix A, B, C and D. Available at: https://docs.google.com/document/d/1SloqKt_e8E_IMKHdCn29rQL374DCeWfKNQkR6ZthZ Eg/edit?usp=sharing, last accessed 2017/05/28.
[16] Hollander, M. and Wolfe, D. A.: Nonparametric Statistical Methods: By Myles Hollander, Douglas A. Wolfe. J. Wiley (1999).
