36 CHAPTER 3 RESEARCH DESIGN AND METHODOLOGY ...

36 CHAPTER 3 RESEARCH DESIGN AND METHODOLOGY Introduction The review of literature has produced reoccurring themes emphasizing the importance of technological literacy for citizens in the 21st Century (Garfinkel, 2003; Hall, 2001; Lemke, 2003; Murray, 2003; NAE, 2002; Partnership for 21st Century Skills, 2003; Rose & Dugger, 2003; Zhao & Alexander, 2002; U.S. Department of Education, 2004; Technology Counts, 2005). Education is a critical component in preparing students for a knowledge-based, digital society. According to Hall (2001), available technologies, our perceptions of those technologies, and how they are used will determine the shape of our world. Citizens of the future will face challenges that depend on the development and application of technology. Are we preparing students, the citizens of tomorrow, for these challenges? Purpose of the Study This study developed and implemented a faculty survey and a student assessment. The purpose of the faculty survey was to determine what basic computer skills are needed by undergraduate students for academic success in post-secondary education. This phase of the study examined the data collected for trends and differences between the independent variables of subject/content area, institution, gender, and years of faculty experience. The purpose of the student assessment was to evaluate the computer competencies of students entering a post-secondary education. This phase of the study examined the data collected for trends and differences between the independent variables

37 of home state, number of high school computer courses taken, gender, and major field of study? Data collection and analysis assisted in determining if students possess the necessary computer/technology skills entering a post-secondary institution or if a need exists for a general education course to teach computer literacy/skills to the undergraduate student population. This study also provided valuable information in regards to the content of such a course. Research Questions 1. What technology skills do post-secondary faculty members deem important for all students to possess at the college level? 2. Are there differences between the student technology skills post-secondary faculty members deem important when grouped by subject/content area, institution/stratum, gender, years of faculty experience? 3. What technology skills can students demonstrate proficiently upon entering a post-secondary institution? 4. Are there differences between the proficiency level of students’ technology skills when grouped by home state, number of high school computer courses, gender, or major field of study? 5. Are students technologically ready entering post-secondary education or does a need exist for a computer literacy/skills course for all undergraduate students? Instrumentation Two instruments were employed for data collection in this research study: a faculty survey and a student assessment. A faculty survey was designed by the researcher

38 to help identify technology/computer skills deemed important for undergraduate students to possess in order to be successful in their post-secondary endeavors. A survey research design was applied to investigate the research questions. A second instrument was developed and implemented to assess technology skills of freshmen undergraduate students who had not yet taken a post-secondary computer literacy/skills course. A description of the two instruments used in this study follows. Faculty Survey Introduction According to Leedy and Ormrod (2001), “Research is a viable approach to a problem only when there are data to support it” (p. 94). Nesbary (2000) defines survey research as “the process of collecting representative sample data from a larger population and using the sample to infer attributes of the population” (p. 10). The main purpose of a survey is to estimate, with significant precision, the percentage of population that has a specific attribute by collecting data from a small portion of the total population (Dillman, 2000; Wallen & Fraenkel, 2001). The researcher wanted to find out from members of the population their view on one or more variables. As noted by Borg and Gall (1989), studies involving surveys comprise a significant amount of the research done in the education field. Data are ever-changing and survey research portrays a brief moment in time to enhance our understanding of the present (Leedy & Ormrod, 2001). Educational surveys are often used to assist in planning and decision making, as well as to evaluate the effectiveness of an implemented program (McNamara, 1994; Borg & Gall, 1989).

39 An online faculty survey was conducted to identify computer literacy skills that faculty members deem important for an undergraduate student to possess in order to be academically successful at the post-secondary level. Population and Sample The population for this faculty survey consisted of post-secondary faculty members at four-year public institutions in the state of Missouri. Four-year public institutions were determined by visiting the Missouri Department of Higher Education Web site at http://www.cbhe.state.mo.us/Institutions/pubinst.htm. Private or independent institutions and community colleges were not included in the population. Thus the sampling frame consisted of all faculty members at thirteen institutions in Missouri, as summarized in Table 1. Table 1 Summary of 4-Year Public Institutions in Missouri Central Missouri State University Harris-Stowe State College Lincoln University Missouri Southern State University Missouri Western State College Northwest Missouri State University Southeast Missouri State University Southwest Missouri State University Truman State University University of Missouri-Columbia University of Missouri-Kansas City University of Missouri-Rolla University of Missouri-St. Louis

A sample population was drawn from the sampling frame. A sampling frame includes the actual list of individuals included in the population (Nesbary, 2000) which was approximately 4821 faculty members. According to Patten (2004), the quality of the

40 sample affects the quality of the research generalizations. Nesbary (2000), suggests the larger the sample size, the greater the probability the sample will reflect the general population. However, sample size alone does not constitute the ability to generalize. Patten (2004), states that obtaining an unbiased sample is the main criterion when evaluating the adequacy of a sample. Patten also identifies an unbiased sample as one in which every member of a population has an equal opportunity of being selected in the sample. Therefore, random sampling was used in this study to help ensure an unbiased sample population. Because random sampling may introduce sampling errors, efforts were made to reduce sampling errors, and thus increasing precision, by increasing the sample size and by using stratified random sampling. To obtain a stratified random sample, the population was divided into strata according to institutions as shown in Table 2. Typically, for stratified random sampling, the same percentage of participants, not the same number of participants, are drawn from each stratum (Patten, 2004). Table 2 Strata (subgroups) for Stratified Random Sampling Instructors and professors at Central Missouri State University Instructors and professors at Harris-Stowe State College Instructors and professors at Lincoln University Instructors and professors at Missouri Southern State University Instructors and professors at Missouri Western State College Instructors and professors at Northwest Missouri State University Instructors and professors at Southeast Missouri State University Instructors and professors at Southwest Missouri State University Instructors and professors at Truman State University Instructors and professors at University of Missouri-Columbia Instructors and professors at University of Missouri-Kansas City Instructors and professors at University of Missouri-Rolla Instructors and professors at University of Missouri-St. Louis

41 Patten (2004) suggests that a researcher should first consider obtaining an unbiased sample and then seek a relatively large number of participants. Patten (2004) provides a table of recommended sample sizes. A table of recommended sample sizes (n) for populations (N) with finite sizes, developed by Krejcie and Morgan and adapted by Patten (2004), was used to determine estimated sample size. According to the table, and for purposes of this study, the researcher used an estimated population size N = 4821 and thus a sample size goal of n = 357. Survey Procedures In 1998, according to Nesbary (2000), Web surveys were almost non-existent in the public sector. Nesbary decided to test the waters and conduct three surveys to compare response rate and response time of Web surveys to regular mail surveys. Survey results and respondent feedback of all three surveys indicated that Web surveys were more cost effective, easier to use, had quicker response rates, and greater responses. One of Nesbary’s Web surveys was distributed to selected universities. Of those surveyed, respondents indicated a strong preference for use of technology to take advantage of speed and convenience. The researcher used a Web-based survey for the faculty survey portion of this study. UNL IRB approval was obtained (Appendix A). Two approvals for change of protocol were also obtained, one for changing the title of the study (Appendix B) and the other for changing the survey format (Appendix C). From the original IRB request, the survey was condensed to reduce the number of items, shortening the survey to increase response rate.

42 Ethical Issues McNamara (1994) identifies five ethical concerns to be considered when conducting survey research. These guidelines deal with voluntary participation, no harm to respondents, anonymity and confidentiality, identifying purpose and sponsor, and analysis and reporting. Each guideline will be addressed individually with explanations to help eliminate or control any ethical concerns. First, researchers need to make sure that participation is completely voluntary. However, voluntary participation can sometimes conflict with the need to have a high response rate. Low return rates can introduce response bias (McNamara, 1994). In order to encourage a high response rate, Dillman (2000) suggests multiple contacts. For this study, up to five contacts were made per potential participant. The first email contact (Appendix D) was sent a few days preceding the survey to not only verify email addresses, but also to inform possible participants of the importance and justification for the study. The second email contact (Appendix E) was the actual email cover letter explaining the study objectives in more depth. This email consisted of a link to the Webbased survey and a password to enter. By clicking on the link provided and logging into the secure site, the participant indicated agreement to participate in the research study. The third email contact (Appendix F) was sent a week later reminding those that had not responded. The fourth email contact (Appendix G) was sent two weeks after the actual survey email reemphasizing the importance of faculty expertise in providing input to the study. The fifth and final email contact (Appendix H) was sent three weeks after the

43 actual survey email to inform faculty that the study was drawing to a close and that their input was valuable to the results of the study. McNamara’s (1994) second ethical guideline is to avoid possible harm to respondents. This could include embarrassment or feeling uncomfortable about questions. This study did not include sensitive questions that could cause embarrassment or uncomfortable feelings. Harm could also arise in data analysis or in the survey results. Solutions to these harms will be discussed under confidentiality and report writing guidelines. A third ethical guideline is to protect a respondent’s identity. This can be accomplished by exercising anonymity and confidentiality. A survey is anonymous when a respondent cannot be identified on the basis of a response. A survey is confidential when a response can be identified with a subject, but the researcher promises not to disclose the individual’s identity (McNamara, 1994). To avoid confusion, the cover email clearly identified the survey as being confidential in regards to responses and the reporting of results. Participant identification was kept confidential and was only used in determining who had not responded for follow-up purposes. McNamara’s (1994) fourth ethical guideline is to let all prospective respondents know the purpose of the survey and the organization that is sponsoring it. The purpose of the study was provided in the cover email indicating a need to identify technology skills necessary for students to be successful in their academic coursework and to determine if a general education computer literacy/skills course should be required of all undergraduate

44 students. The cover email also explained that the results of the study would be used in a dissertation as partial fulfillment for a Doctoral degree. The fifth ethical guideline, as described by McNamara (1994), is to accurately report both the methods and the results of the surveys to professional colleagues in the educational community. Because advancements in academic fields come through honesty and openness, the researcher assumes the responsibility to report problems and weaknesses experienced as well as the positive results of the study. Validity and Reliability Issues An instrument is valid if it measures what it is intended to measure and accurately achieves the purpose for which it was designed (Patten, 2004; Wallen & Fraenkel, 2001). Patten (2004) emphasizes that validity is a matter of degree and discussion should focus on how valid a test is, not whether it is valid or not. According to Patten (2004), no test instrument is perfectly valid. The researcher needs some kind of assurance that the instrument being used will result in accurate conclusions (Wallen & Fraenkel, 2001). Validity involves the appropriateness, meaningfulness, and usefulness of inferences made by the researcher on the basis of the data collected (Wallen & Fraenkel, 2001). Validity can often be thought of as judgmental. According to Patten (2004), content validity is determined by judgments on the appropriateness of the instrument’s content. Patten (2004) identifies three principles to improve content validity: 1) use a broad sample of content rather than a narrow one, 2) emphasize important material, and 3) write questions to measure the appropriate skill. These three principles were addressed when writing the survey items. To provide additional content validity of the survey

45 instrument, the researcher formed a focus group of five to ten experts in the field of computer literacy who provided input and suggestive feedback on survey items. Members of the focus group were educators at the college and/or high school level who have taught or are currently teaching computer literacy skills. Comments from the focus group indicated that the skills listed in the survey were basic/intermediate skills and were appropriate for all college students to know and be able to do. Some members of the focus group suggested that the survey might be a bit long and that skills could be generalized and consolidated for a more concise survey. The researcher categorized application skills and condensed the application component items from 20 per application to eight items per application. The computer concepts component was reduced from 22 items to eight items. According to Patten (2004), “. . . validity is more important than reliability” (p. 71). However, reliability does need to be addressed. Reliability relates to the consistency of the data collected (Wallen & Fraenkel, 2001). Cronbach’s coefficient alpha was used to determine the internal reliability of the instrument. The faculty survey instrument was tested in its entirety, and the subscales of the instrument were tested independently. Data Collection An informal pilot study was conducted with a small group of faculty members at the researcher’s home institution. Conducting a local pilot study allowed the researcher to ask participants for suggestive feedback on the survey and also helped eliminate author bias. Once the pilot survey had been modified as per the educational expert’s feedback, the survey was administered online to the stratified, random sample population.

46 Participants of the study were contacted by email explaining the research objective and asking them to participate. The objective of the research was to gather information about technology skills, in particular, what technology skills should students possess to be successful during their post-secondary courses. The email also contained a link to the Web-based faculty survey and a password to enter the survey. Follow-up email contacts were sent to increase response rate. Upon completion of the survey, each respondent was directed to a Web page thanking them for their response and offering them a copy of the study results if they were interested. Screen shots of the Web-based faculty survey are presented in Appendix I. The Web-based survey was conducted using surveymonkey.com, a survey software program offered online. For a small fee, the program offered many features including unlimited number of survey questions, ability to add a personalized logo, custom redirects, result filtering, and the capability to export data for statistical analysis. The program provided a list management tool where responses could be tracked by their email address which proved to be very useful for follow-up emails. The program also provided security including the option to turn on SSL (Secure Sockets Layers) to utilize data encryption and provide data protection. Responses to the survey were recorded, exported in a spreadsheet, and transferred to a statistical software package for in-depth analysis. Descriptive statistics were calculated and data relationships were analyzed.

47 Variables and Measures Variables used in the survey have been summarized in Table 3. The variables consisted of seven independent variables that grouped respondents by common characteristics and five dependent variables that grouped responses by content categories. The independent variables included professional title, institution, department/content area, school size, gender, number of years at current institution, and total number of years in education. The dependent variables included word processing, spreadsheet, presentation, database, and computer concepts. Table 3 Summary of Dependent and Independent Variables in the Faculty Survey Independent Variables (n = 7) Dependent Variables (n = 5) Professional title Word processing Institution Spreadsheet Department Presentation School size Database Gender Computer concepts Years as faculty member at current institution Years in education

Data Analysis Plan To begin the data analysis process, descriptive statistics were calculated on the independent variables to summarize and describe the data collected. Survey results were measured by category. There were five categories (subscales), representing the five dependent variables. Reponses to the survey items were coded from 1 to 4 depending on the importance of each skill. One represented ‘not important’, two represented ‘somewhat important’, three represented ‘important’, and four represented ‘very important’. The code for all survey items in the same category were summed together for a composite

48 score per category. This category composite score was used for statistical analysis. Item analysis was conducted to determine the internal consistency and reliability of each individual item as well as each subscale. Cronbach’s Alpha test was also used to test internal reliability. Inferential statistics were used to reach conclusions and make generalizations about the characteristics of populations based on data collected from the sample. Frequencies and/or percentages were used to identify computer skills that faculty members deem important for all students to possess. Independent t-tests and/or simple analysis of variance (ANOVA) were used to look for significant differences between the student technology skills faculty members deem important when grouped by department/content area, institution/stratum, gender, or years of faculty experience. The type of tests that were used to answer specific research questions are summarized in Table 4. A statistical software program, SPSS (Statistical Package for Social Sciences) was used for in-depth data analyses. Table 4 Summary of Data Sources, Types and Measures Applied by Research Question Research Question # 1 2

Data Source Faculty Survey Responses Faculty Survey Responses

Response Type Likert Scale Likert Scale

Data Type Nominal Nominal

Analysis Plan f, % t test, ANOVA

Student Assessment Introduction To assist in evaluating the technology skills of students, a technology assessment was conducted to determine computer literacy and performance skills of students entering a post-secondary institution prior to taking a computer course at the post-secondary level.

49 Population and Sample The population for the student assessment consisted of college freshmen from a small mid-western university enrolled in a computer literacy course. Permission from students to use their scores in the study was requested through informed consent forms. This resulted in a sample size of 164 students. Student Assessment Procedures The purpose of the student assessment was to describe specifically what a typical student entering post-secondary education knows about computer operations and concepts as well as the computer skills they can demonstrate proficiently. An additional Northwest Missouri State University IRB form (Appendix J) was obtained and student consent forms (Appendix K) were collected from participants. A series of assessments were given to all students during the first few weeks of a computer literacy course to determine the computer skills students possess prior to taking a computer literacy course. Measurement Instrument The student assessment consisted of a few demographic questions, and two major components: 1) computer concepts and 2) computer application skills. The computer concepts component of the assessment covered six different modules. Module one questions covered computer and information literacy, introduction to application software, word processing concepts, and inside the system. Module two questions covered understanding the Internet, email, system software, and exploring the Web. Module three questions covered spreadsheet concepts, current issues, emerging

50 technologies, and data storage. Module four covered presentation packages, special purpose programs, multimedia/virtual reality, and input/output. Module five questions covered database concepts, telecommunications, and networks. Module six questions covered creating a Web page, ethics, and security. The assessment for the computer concepts component of the study consisted of 150 questions, 25 questions randomly selected from each of six module test banks. The number of questions in each module test bank ranged from 143 to 214 questions. This portion of the study was administered using an online program called QMark (Question Mark). The assessment was automatically graded and scores were recorded on a server at Northwest Missouri State University. The computer application skills component was assessed using a commercial software program called SAM (Skills Assessment Manager). SAM is a unique performance-based testing software program that utilizes realistic, powerful simulations. The software package works just like the actual Microsoft Word, Excel, Access, and PowerPoint applications, but without the need for preinstalled Microsoft Office software. Course Technology, the publisher of SAM software, provided the researcher with a site license for use in this research. The application skills assessed for this study include word processing skills, spreadsheet skills, presentation skills, and database skills. Validity and Reliability Issues Patten’s (2004) three principles to improve content validity: 1) use a broad sample of content rather than a narrow one, 2) emphasize important material, and 3) write questions to measure the appropriate skill, were addressed when developing assessment items.

51 In 1998 Course Technology introduced SAM 1997 and has continued to update the product through SAM 2000, SAM 2003, and now SAM XP. SAM is a unique performance-based testing software program that utilizes realistic, powerful simulations. The software package works just like the actual Microsoft Word, Excel, Access, and PowerPoint applications, but without the need for preinstalled Office software. According to Course Technology, SAM is the most powerful testing and reporting tool available. According to Course Technology (2002), SAM is becoming the provider of the most widely-used and effective technologybased assessment product line for Microsoft Office used in educational institutions today. SAM is used at high schools, colleges, career colleges, MBA programs, and in the workplace as a screening tool for placing people in the right training courses, a ‘test-out’ tool to determine students’ proficiency before they take a course, and a seamless, in-course assessment tool to allow students to demonstrate their proficiency as they go through a course. Proficiency skills on the assessment matched categories on the faculty survey so results could be compared. Data Collection All students enrolled in the course completed the assessments during the first few weeks of class using the SAM and QMark software to determine their computer literacy/skill proficiency level prior to taking the computer literacy course. The assessments were graded online and the results were immediate. Prior to taking the assessment, the participants provided data on a few demographic questions. A list of the

52 demographic questions can be found in Appendix L. Computer concepts questions were randomly selected from a test bank of questions. Sample screen shots can be found in Appendix M. A list of proficiency skills for the computer applications component of the student assessment can be found in Appendix N. Student names were kept confidential to ensure individual privacy. Variables and Measures Variables used in the student assessment have been summarized in Table 5. The variables consisted of five independent variables that group participants by common characteristics and five dependent variables that group participants by content categories. The independent variables included student home state, size of high school graduating class, number of high school computer courses taken, gender, and post-secondary major field of study. The dependent variables included computer skills grouped in five categories including word processing, spreadsheet, presentation, database, and computer concepts. Table 5 Summary of Dependent and Independent Variables in the Student Assessment Independent Variables (n = 5) Dependent Variables (n = 5) Home state Word processing High school size Spreadsheet Number of high school computer courses Presentation Gender Database Major Computer concepts Data Analysis Plan Results of the student assessment were recorded in a spreadsheet and transferred to SPSS for statistical analysis. Descriptive statistics and data relationships were calculated. Independent t-tests and simple analysis of variance (ANOVA) were used to

53 look for significant differences between the proficiency level of students’ technology skills when grouped by home state, number of high school computer courses taken, gender, and major field of study. A statistical software program, SPSS (Statistical Package for Social Sciences) was used for in-depth data analyses. The type of tests used to answer specific research questions are summarized in Table 6. Table 6 Summary of Data Sources, Types and Measures Applied by Research Question Research Question # 3 4 5

Data Source Title Student assessment score Student assessment score Student assessment score

Response Type Percentage Percentage Percentage

Data Type Interval Interval Interval

Analysis Plan f, % t test, ANOVA f, %