EXPERIMENT DESIGN FOR PILOT PROGRAMS IN TEACHING COMPUTER SCIENCE Ashok Kumar, Mike Unuakhalu, James Carigan Kentucky State University Frankfort, KY 502-597-6384 phone, 502-597-6179 fax
[email protected] [email protected] [email protected]
ABSTRACT We often conceive of new teaching ideas, which we may implement. But we are often at a loss as to how we measure the success of the innovation. This is particularly a problem when we deal with small student groups and few instances of like groups. In this paper, we will examine elements, such as threats to validity, a common enemy of experiment design, and such as numbers constraints that cause us to deal with small samples. We will consider some possible solutions, and discuss these in light of a recent case and a proposed case. We will use our observations to make recommendations, particularly to recommend a compendium of checks for good experiment design.
INTRODUCTION We have two high order teaching objectives at Kentucky State University (KSU) in the Computer and Technical Sciences Division – to imbue student learning with a solid grasp of problem solving skills and to engage our students in the learning process by innovative teaching. In the first instance, we want to climb the hierarchy of learning with our students to the point of critical thinking at an appropriate level for each course. In the second, we want to make the transition results-driven rather than process-driven. In either case, we will wish to assess the effectiveness of the approach. We believe that a good assessment can be based on proper experiment design and refined requirements. In this paper, we will examine problems, such as threats to validity, a common enemy of experiment design, lack of specificity in design requirements, and numbers constraints that cause us to deal with small samples. We will consider some possible solutions, and discuss these in light of a recent case and a proposed case. We will use our observations to make recommendations, particularly to recommend a compendium of checks for good experiment design. The work is a continuing collaboration among the authors, and as such is being reported here in the context of a “work in progress.” THREATS TO VALIDITY All experiments must be constructed to avoid internal threats to the validity of the results and conclusions based on those results. There follows an overview of the threats we may find in our environment. •Selection – We have a wide variety of subjects at the highly diverse KSU. Differences of gender, age, ethnicity, and related experience must be gauged for their impact on the 122
variables. For example, our program attracts a higher percentage of non-traditional students who have already begun careers in computer areas. •History – In recent years the population of students has changed in respect to career goals and interests. The connected multi-user environment likely appeals to different types than the mainframe and midrange installations of the past. Security issues have affected the career choices. In fact, we have made curriculum changes responsive to these changes, thus altering our active and passive recruiting face. •Maturation – We will be testing at different times in most cases. What will our subjects learn about our testing? •Repeated Testing – Can we use consistent measuring processes from one observation to the next? •Instrumentation – If possible we need to use objective, deterministic measuring devices, but if we must use subjective means, such as human observation or open-ended questions, the instrumentation must be true to the nature of the inquiry. •Regression to the Mean – The experimental design must consider zones of reasonableness so that cases of outlying values returning to the mean do not lead us to conclude that this is part of a normal progression. •Experimental Mortality – In small populations, dropout of subjects may have a disproportionate effect on our results or even cause an abortive halt. We need plans to recognize and reorganize. •Selection-Maturation Interaction – There can be an unforeseen interaction between these two threats, compounding their threats’ effects. •Experimenter Bias – Faculty members, who want their innovations to succeed, for both the benefits to students and the advancement of the art, may minimize negatives and maximize positives. We need to stabilize this effect through experiment design which includes third-party analysis and peer review.
NUMBERS CONSTRAINTS Because our institution has a low student to faculty ratio, which is culturally consistent with our mission, we face number constraints in trying to measure the effects of pedagogical initiatives. The natures of some of these are: •Generalizations – It may be difficult to transfer our findings from one setting to another. •Studies – Small numbers of classes taught by a small faculty corps will perforce limit the number of studies that we can conduct. •Insufficient Statistics – The numbers constraints may act to render necessary measurement activities incomplete or even irrelevant. •Biased Statistics – Will the population size problems mask biases? •Measurement Errors – Will measurement errors be disproportionately large relative to population sizes? •Correlation and Causality – We must take extra precautions not to mistake correlation for causality. 123
•Too few subjects – any one group of students taking one of our courses may be small. •Too few instances – most of our courses may meet with only one section, once an academic year. REQUIREMENTS REFINEMENT In computer science, we frequently must engage in systems analysis, which activity relies heavily on a firm grasp of requirements. Coming up with a good experiment design should respond to a good requirements specification process, which proceeds from elicitation, analysis, and refinement. •Requirements elicitation – This phase should produce a relatively complete list of roughly stated requirements, reviewed for feasibility, reviewed for leanness. •Requirements analysis – Here we would use various tools which would include numeric, graphic, and descriptive modeling •Requirements refinement – A stepwise refinement process should eliminate indeterminacy, ambiguity, redundancy, and incompleteness, while isolating risk. Risk should be weighed for elimination, tolerance, and mitigation. Some of the techniques to consider for refinement should be: •Divide and Conquer – Iteratively separate the unclear portion of a requirement statement from the clear portion. Recursively treat the unclear portion until an exit condition is encountered. •Verifiability & Determinacy – Clarity should be synonymous with verifiability, i.e. when will we know that the requirement has been met? By what indications? Determinacy is a condition where one state determines the next; are “next” states known? •Repeatability – Will repeats produce the same results? •Classification – Do we understand the properties and their ranges of values that will define an instance of a requirement met? •Risk Evaluation – What effect will indeterminacy have on our results and can it be reasonably reduced, nullified or tolerated? REJECTION SAMPLING One of the most likely solutions for numbers constraints will be a methodology that allows for early rejection of poor hypotheses. A constrained observation may give insight on false indications without resorting to full statistical treatment. In a previous professional role, one of our authors had to optimize the use of time and other resources, while finding the most germane results. Practically this involved excluding (or rejecting) populations that were not likely going to produce salient findings. The problem domain was sampling insurance policies for correct application of pricing rules and for unfair discrimination among policyholders. If we took a correct sub-sample, and operated on the idea that problems would be likely to surface in that sub-sample, then we could confidently dismiss the whole population if very few problems were seen. By the same process, if we found problems, then we had a preview as to what types of problems for which to watch. 124
Techniques to be tried: •Observations from a distribution – We can watch for the early emergence of a normal or abnormal distribution. Samples that are done in incremental sub-samples can provide an emerging view of distribution. •Probabilistic rejection -- The null hypothesis is often used to prove an opposite, but if we merely need to accept the probability of an opposite, we may have an earlier exit condition. For example, we may hypothesize that our innovation in teaching is doing harm, but determine quickly that there is no real likelihood of that. •Rejection sampling generates sampling values from some distribution function that is subsumed in the population’s full sample distribution function. •If a sub-sample indicates a significant observation, then we can examine a disjoint subsample to see if the remarkable is repeated. This can be reiterated until a confident acceptance or rejection of the significance can be made. •Importance sampling – We may reliably use a previous population to make assumptions about important elements to sample for in a subsequent population. This is a variance reduction approach. •Slice sampling – If we can hypothesize the distribution function of the population then we can sample at equal intervals in the region under the graph of the function. •Stochastic processes – If there is a known, then a number of probable next states can be predicted. If a conclusion can fall in the zone of reasonable variance, then the conclusion has a reasonable proximity to the result we may select to report. •Other techniques – validation of the above techniques can likely lead us to other similar techniques. APPLICATION OF PROPOSALS TO ACTUAL CASES CASE 1 HYPOTHESIS: Specific complementary training in problem solving techniques will enhance teaching and learning in programming courses. PROBLEM STATEMENT: This has been a subject of a continuing formal empirical analysis at our school, but have we recognized all threats to validity which may impact our results? PROPOSED SOLUTION: The use of formal experiment design and development processes on an iterative basis will increase the reliability of analysis. Effect of Computer Programming on the Problem Solving Capabilities of Introductory College-Level Students LITERATURE REVIEW During the last several years, numerous studies have investigated the effect of microcomputer use in schools at all levels from kindergarten to college. The aim of most of the studies was to examine the effectiveness of computer programming and applications on students’ academic performance, achievement, and problem solving. The findings of the problem solving research are mixed. Unfortunately, most evidence 125
supporting the cognitive effects of programming have been anecdotal, based on small sample of students, and often non-empirical in nature (Dalbey & Linn,1985). While some studies, notably, Palumbo (1990); Soloway, Lockhead, and Clement (1982); Palumbo and Reed (1991);Choi and Repman (1993); Norris and Jackson (1992); Lee and Thompson (1997); Cardelle-Elawar (1992), report positive effects, some empirical studies do not support the use of computer programming as a means of teaching logical thinking and problem solving skills (Liao & Bright, 1991; Clements & Gullo, 1985; Pea & Kurland, 1984; Overbaugh, 1993). One contributing factor for the conflicting results could be the lack of many of today’s college students in using cognitive training strategies effectively in solving problems (Chipman & Segal, 1985; Goldenson, 1996; Lee & Thompson, 1997). There is very little empirical research available that tests the effects of object-oriented programming, such as, Visual BASIC using an embedded problem transfer instructional strategy to enhance students’ problem solving skills. Thus, the purpose of this present study was to examine the effectiveness of an object-oriented programming instruction embedded with transfer training activities on undergraduate college-level students’ problem solving skills. The effect of the language used on the mastery of the programming design concepts was also determined. While problem solving transfer to general problem solving domains might be difficult to achieve when students are exposed to programming language instruction, findings indicate that transfer of specific problem solving skills can occur under certain conditions (Palumbo & Reed, 1988; Palumbo & Reed, 1991). In an everyday problem task activity such as the game of chess, expert players do not readily demonstrate transfer of their problem solving abilities to skills in politics and/or economics. Terry and Ziegler (1987) research explored the relationship between programming language instruction and problem solving skills in sixty gifted elementary school children. A comparison of the three treatment groups revealed that the computer problem-solving group scored significantly higher in computer literacy and problem solving than did the creative problem solving group, and the traditional gifted curriculum group. This research addresses a specific skill and, thus, the extent to which programming instruction allows specific transfer of training from the programming environment to the students’ specific problem solving skills (Palumbo & Palumbo, 1993). SUMMARY OF REVIEW Several generalizations about the computer programming effect on college students problem solving capability can be derived from the literature review on programming and problem solving. Most notably, it was simply demonstrated that in most cases, students who are given training in an area and an opportunity to practice the skill will do better than uninstructed students. In many of the studies reviewed, students appear to need guided or mediated instruction to insure they acquire the concepts and strategies underlying computing programming. When researchers study real children in real classrooms, the results indicate a need for more guided and mediated instructional methods. While it may not seem possible to achieve expertise in programming after a 126
semester or even a year of programming exposure, this does not limit the possibility of some type of problem solving transfer. The growing list of successful problem solving transfer studies indicates that learning a programming language can have effects on thinking in nonprogramming domains. However, in most cases the greatest amount of transfer is found for tests that are most similar to the original programming tasks. Conditions under which students learn to program appear to be related to the process involved in learning to think. These conditions are initial mastery of the syntax of the language, the development of appropriate mental models of the system, and the acquisition of strategies for how to solve procedural problems. Studies that did not show positive transfer to nonprogramming domains can be attributed to several theoretical and statistical problems, but in studies in which many of these problems are not evident, the problem solving transfer effect is still unimpressive. A starting point in establishing the relationship between programming language instruction and transfer of training should focus on the specific transfer of those skills acquired in the programming environment to other problem solving environments. After this relationship is established, then longer, more intense exposure should be provided to investigate whether generalized transfer will occur. The duration of programming effect on problem solving studies should also be adequate and programming mastery must be tested. Findings of most studies seem to suggest that the use of randomly formed control and experimental groups needs to be improved for future studies. It is evident that the instrument that has been used most frequently by evaluators to measure university level problem solving is the Watson-Glaser Critical Thinking Appraisal. This dependent measure, when employed in studies, did not always result in a significant gain in problem solving skills. As Burton & Magliaro (1988) point out, the dependent measures are crucial when trying to determine effects or relationships. If researchers are looking for gains or relationships but use problem solving measures that are not representative of the kinds of problem solving skill the participants engage in when programming, part of the design might be faulty. Thus, the lack of gains may be due to use of assessment instruments (such as Watson-Glaser Critical Thinking Appraisal) that may not have paralleled the kinds of problem solving skill the subjects were engaged in when given the programming treatment. This amplifies the need for careful selection of dependent measures to match program goals. SUMMARY OF STUDY Subjects in the study were forty students enrolled in either of two sections of a semester long college level computer science programming course titled “Programming in Visual BASIC” at a southeastern university. Participants were largely freshmen, who had reported little or no knowledge of the subject matter. Thirty-two students had no computer programming experience, and eight had some limited experience in one other structured programming language. Twenty-four males and sixteen females were included in the study. The mean age was 20. The researcher instructed the two-sections of the Visual BASIC course in order to minimize the possibility of confounding resulting from instructional effect.
127
The impact of programming on high order thinking skills such as problem solving has been extensively investigated. In an effort to capitalize on the lack of research on appropriate learning tools and instructional methods that can facilitate the development of student cognitive monitoring process, the present study attempted to investigate the effects of an object-oriented programming instruction embedded with transfer training activities on the problem solving processes of undergraduate college students. The study was done to meet the ever increasing societal needs to teach transfer training strategies to college students using Visual BASIC, an object-oriented language, and adults in order to help them become efficient thinkers and problem solvers in varied situations. The study employed the use of an effective learning tool, Visual BASIC, and methodology comprising of five major activities for cognitive monitoring (decomposing, planning, executing, error identification, and debugging) that will improve students’ higher-order thinking and problem solving skills. The two treatments were: (a) programming instruction plus transfer strategies, and (b) programming instruction only. Specifically, this study addressed the impact of a completed computer programming course on four problem solving skills – Deduction, Interpretation, Analysis of Relevant and Irrelevant Information, and Analysis of Attributes as measured by a problem solving instrument. Another outcome measure was a programming competence test. Results indicated that there were no significant differences between the programming plus transfer and programming-only groups in the development of problem solving skills. Significant improvement in performance favoring the programming plus transfer group versus programming-only group was identified. An unexpected but significant difference between the programming plus transfer and programming-only groups on the content knowledge test was also identified. DATA COLLECTION AND ANALYSIS The pre- and posttest and demographic data were scored and stored using SPSS software. Posttest composite and subscale scores on the problem solving instrument were compared to pretest composite and subscale scores on the problem solving instrument in order to determine effect of instructional design on problem solving performance. Likewise, for analysis of the content knowledge, posttest composite and subscale scores on the programming content exam were compared to pretest composite and subscale scores on the programming content exam in order to determine effect of instructional design on programming competence performance. A two-factor (Treatment by Time Exposure) analysis of variance with repeated measures on one factor (Time Exposure) was used to investigate the relationship between instructional intervention and composite and subscale scores on each of the two instruments: programming content exam and problem solving instrument. INSTRUMENTS Two sections from the Ross Test of Higher Cognitive Processes (Analysis of Relevant and Irrelevant Information, and Analysis of Attributes) and two sections from WatsonGlaser Critical Thinking Appraisal (Deduction, and Interpretation) were combined in a 61-item problem solving instrument used as a pre- and posttest. The instrument is 128
composed of questions that measure the students’ ability to draw valid conclusions, to evaluate generalizations, and to analyze data and identify critical information. The scores from this dependent measure were collected to be analyzed. Reliability for the Ross Test range from .92 to .94 (Ross & Ross, 1976), and Watson-Glaser split-half reliability coefficients range from .69 to .85 (Watson & Glaser, 1980). The pre- and posttest of the programming content knowledge measures contained forty-seven questions each, that were taken from the content of the instructional module. Both tests contained the same type of questions (twenty true or false, twenty multiple choice, six short answer questions, and one hands-on programming). Multiple raters, who were experienced computer science faculty, scored the Short Answer segment of the posttest. For interrater reliability, a Pearson correlation was conducted using the three rater’s scores for each subject. As illustrated in Table 5, the results indicated all raters were significantly correlated. From this observation, any threat caused by grading inconsistency can be assumed to be not serious. An SPSS program run by the researcher showed that the Cronbach alpha reliabilities are 0.43 and 0.73 for the pre and posttest respectively. CASE 2 HYPOTHESIS: The use of ALICE will enhance teaching and learning in preprogramming courses introducing logic, problem solving, and design. PROBLEM STATEMENT: This is a new, untested teaching initiative, but we do not yet know how we will analyze the outcomes. Are we ready for full formal design? PROPOSED SOLUTION: The use of experiment design guidelines will help us to fit the pilot project to the existing environment, and to make a decision about the degree of formality required. We have a freshman level course, Logic, Problem Solving, and Design, which is intended to provide new computer science students with some fundamental tools to face the depth and complexity of their future studies, but without the rigors of learning an advanced programming language (C++) at the same time. We have found, however, that without some “hands on” way of demonstrating concepts, the course can be difficult for the students in finding engagement. Nothing seems to replace the wonder of that first HelloWorld.prg. On attending a conference, one of us was introduced to the idea of using Alice, to provide a learning environment. We have decided to experiment with this idea as a means for testing our experiment design framework, and as importantly, for assessing the outcomes of the experiment. LESSONS LEARNED We have seen that the first requirements elicited include: •a framework – we need to develop a framework for specifying the requirements of an experiment relative to specificity, accuracy, efficiency, and effectiveness. •risk management – risks must be recognized in a precise and thorough way, and criteria for acceptance of risk must be predetermined. •elimination or mitigation of threats – threats should be compensated for or neutralized. 129
•elimination or modification of hypotheses – there should be guidelines for identifying inappropriate hypotheses, and there should be bases for rejecting or correcting them. •piloting vs full scale formal experiment – feasibility of any proposed experiment needs to be ascertained, often to be addressed alternatively through a scaled back pilot study. •prototyping – simulations and walk-throughs can be liberally applied to aid in experiment design •complete testing – a comprehensive test of every component of an experiment should precede an integration test which covers the extent of the experiment, which should precede an environmental test of the experiment in situ. FUTURE WORK In the months preceding our Fall term 2008, we will: •complete a feasibility analysis of the Alice experiment •write the requirements for our experiment or pilot study •review the literature on using “teaching” programming languages (PL), specifically the Alice PL.
REFERENCES Anderson, & B. Hunter (Eds.), Computer Literacy: Issues and Directions for 1985 (pp. 171-185). New York: Academic Press. Anthony, J. (2003). Design of Experiments for Engineers and Scientists. ButterworthHeinemann, Burlington, MA. Athabasca University (2007), Centre for psychology. Concept Definition: Internal Validity. Alberta, Canada: http://psych.athabascau.ca/html/Validity/concept.shtml. October 2007. Burton, J. K. & Magliaro, S. (1987). Computer Programming and Generalized problemsolving skills: In Search of Direction. Computers in the Schools, 4(3), 63-87. Calafiore, G. and Dabbene, F. (eds) (2006). Probabilistic and randomized methods for design under uncertainty. Springer-Verlag, London. Cardelle–Elawar, M. (1992). Effects of teaching metacognitive skills to students with low mathematics ability. Teaching and Teacher Education 8(3), 109-121. Choi, W. S., & Repman, J., (1993). Effects of Pascal and FORTRAN Programming on the Problem-Solving Abilities of College Students. Journal of Research on Computing in Education ,25(3), 291-302. Clements, D.H., & Gullo, D. F., (1984). Effects of Programming on Young Childern’s Cognition. Journal of Educational Psychology, 76(6), 1051-1058. Dalbey, J. & Linn, M.C. (1986). Demands & Requirements of Computer Programming. Journal of Research on Computing in Education , 1, 253-274. Gaddis, T. (2007). Starting out with Alice: a visual introduction to programming. Addison Wesley. Gersting, J. (2007). Programming with Alice: A new strategy for introductory computer science courses. Chautauqua, Univ of Dayton, June 2007. Dayton, OH.
130
Goldenson, D. (1996). Why teach computer programming? Some evidence about generalization and transfer, Proceedings of NECC 1996, National Educational Computing Conference. Minneapolis, June 1996. Lee, M. C. and Thompson, A. (1997), Guided instruction in Logo programming and the development of cognitive monitoring strategies among college students, Journal of Educational Computing Research, 16 (2), 125-144. Liao, Y., & , Bright (1991). Effects of Computer Programming on Cognitive Outcomes: A Meta Analysis, Journal of Educational Computing Research, 7(3), 252-268. Norris, C., Jackson, L., & Poirot, J. (1992), The Effect of Computer Science Instruction on Critical Thinking and Mental Alertness, Journal of Research on Computing in Education, 24 (3) 329-337. Overbaugh, R. C., (1993). A BASIC Programming Curriculum for Enhancing Problem Solving Ability (Report #IR016017), Convention of the Assoc. of Educational Communication and Technology, (ERIC ED 355921). Palumbo, D.B., (1990). Programming Language/Problem Solving Research: A Review of Relevant Issues. Paper Presented at the Annual Conference of Southwestern Educational Research Association, Austin, TX. Palumbo, D.B., & Reed, W.M., (1987/88). Intensity of Treatment and its Relationship to Programming/Problem Solving Research. Computers in the Schools, 4(3/4), 119128. Palumbo, D.B., (1989). Problem Solving Skills and Programming Language Instruction: The Effect of Extended Period of Treatment. Review of Educational Research, 60, 65-89. Palumbo, D. L. & Palumbo, D. B. (1993). A Comparison of the Effects of Lego TC Logo and Problem Solving Software on Elementary Students’ Problem Solving Skills. Journal of Computing in Childhood Education, 4(4), 307-323 Palumbo, D.B. & Reed, W. M. (1990). The Effect of BASIC Programming Language Instruction on High School Students’ Problem Solving Ability and Computer Anxiety. Paper presented at the Annual Conference of the Eastern Educational Research Association, ClearWater Florida Pea, R. D., & Kurland, D. M (1984). On the Cognitive Effects of Learning Computer Programming, New Ideas in Psychology, 2(2), 137-168. Soloway, E., Lochhead, J., & Clement, J. (1982) Does Computer Programming Enhance Problem Solving Ability? Some Positive Evidence on Algebra Word Problems. In R. J. Seidel, R.E. Watson, G. & Glaser, E.M. (1980). Watson-Glaser Critical Thinking Appraisal Manual. San Antonio: The Psychological Corporation.
131
