Educational Researcher

1 downloads 0 Views 269KB Size Report
Feb 3, 2012 - of wiki usage: (a) trial wikis and teacher resource-sharing sites (40%), ..... We measured wiki quality at 7, 14, 30, 60, 100, and 400 days.
Educational Researcher http://er.aera.net

The State of Wiki Usage in U.S. K−12 Schools: Leveraging Web 2.0 Data Warehouses to Assess Quality and Equity in Online Learning Environments Justin Reich, Richard Murnane and John Willett EDUCATIONAL RESEARCHER 2012 41: 7 DOI: 10.3102/0013189X11427083 The online version of this article can be found at: http://edr.sagepub.com/content/41/1/7

Published on behalf of

American Educational Research Association

and http://www.sagepublications.com

Additional services and information for Educational Researcher can be found at: Email Alerts: http://er.aera.net/alerts Subscriptions: http://er.aera.net/subscriptions Reprints: http://www.aera.net/reprints Permissions: http://www.aera.net/permissions

>> Version of Record - Feb 3, 2012 What is This?

Downloaded from http://er.aera.net by guest on October 11, 2013

Feature Articles The State of Wiki Usage in U.S. K–12 Schools: Leveraging Web 2.0 Data Warehouses to Assess Quality and Equity in Online Learning Environments Justin Reich1, Richard Murnane1, and John Willett1 To document wiki usage in U.S. K–12 settings, this study examined a representative sample drawn from a population of nearly 180,000 wikis.The authors measured the opportunities wikis provide for students to develop 21st-century skills such as expert thinking, complex communication, and new media literacy.The authors found four types of wiki usage: (a) trial wikis and teacher resource-sharing sites (40%), (b) teacher content-delivery sites (34%), (c) individual student assignments and portfolios (25%), and (d) collaborative student presentations and workspaces (1%). Wikis created in schools serving low-income students have fewer opportunities for 21st-century skill development and shorter lifetimes than wikis from schools serving affluent students.This study illustrates the exciting potential that Web 2.0 data warehouses offer for educational research. Keywords: computers and learning; content analysis; descriptive analysis; equity; instructional technologies; measurements; regression analyses; research methodology; technology

W

eb 2.0 tools—online platforms that allow nonprogrammers to contribute content to the World Wide Web (O’Reilly, 2005)—are transforming our society. Comment forums turn newspaper columns into conversations, marketers use blogs to get real-time feedback from consumers (Li & Bernoff, 2008), and dictators have fallen as dissidents organize online (Zuckerman, 2010). For good and for ill, Facebook and MySpace have reshaped notions of community, friendship, and identity for their users (boyd, 2008; Turkle, 2011). As over 2 billion Internet users share words, images, and videos through Web 2.0 tools, global changes have followed (Reich, 2008). Web 2.0 tools have made inroads into the U.S. education system as well. In a 2009 Fast Response System Survey (FRSS) conducted by the National Center for Education Statistics, 38% of public school teachers reported using blogs or wikis for class preparation and instruction, and 21% reported that they required Educational Researcher,Vol. 41, No. 1, pp. 7–15 DOI: 10.3102/0013189X11427083 © 2012 AERA. http://er.aera.net

their students to contribute to blogs or wikis (Gray, Thomas, & Lewis, 2010). Given that these technologies have been widely available for only a decade, this represents a striking pattern of growth. How then, can these new technologies be used in classrooms? Consider the Flat Classroom Project (www.flatclassroomproject. org), an international wiki project started by Vicki Davis of the Westwood Schools in Georgia and Julie Lindsay of the Beijing International School. Now in its sixth year, the Flat Classroom Project has engaged dozens of classrooms around the world in producing wiki pages devoted to explaining the 10 “world-flatteners” described in Thomas Friedman’s (2005) The World Is Flat. Students work in international teams to collaboratively create a multimedia presentation about their topics. Project guidelines ensure that students work closely with their foreign peers; for instance, each team’s wiki page includes videos produced with raw film recorded in one country that has been “outsourced” for editing. Thus, a video about social networks might be shot in Shanghai and edited in Vienna. Each final project presents a global student perspective on the future of technology and society. Through these experiences, students have the chance to exercise their skills in communication, creativity, collaboration, and critical thinking in a global context. The Flat Classroom Project stands out as a proof of concept that Web 2.0 learning environments can facilitate rich educational experiences that prepare students to thrive in a networked world. But is this student-centered, global, collaborative project representative of typical wiki usage, or is it a relatively rare exception? Are such exemplary projects found in diverse settings, or do they exist primarily in schools serving affluent students? In this article, we address these key questions of quality and equity by adopting a novel research approach made possible by the data records of Web 2.0 platforms. Every time a user makes a change to a blog, wiki, or content management system (e.g., Blackboard or Moodle), the data warehouse supporting that online environment records the change. In aggregating these continuously recorded actions, researchers can examine global patterns of online teaching and learning at gradations of nearly infinite granularity. We can conduct focused studies Harvard Graduate School of Education, Cambridge, MA

1

January/February 2012

7

of a student’s activity over a period of minutes, or we can compare hundreds of thousands of learning communities over years. In this study, we leverage these new data sets to understand patterns of wiki usage in U.S. K–12 settings. We focus on wikis because they are emblematic of Web 2.0: They are collaborative, multimedia spaces where any community member can edit any page at any time (Larusson & Alterman, 2009; Reich & Daccord, 2008; Richardson, 2008). We analyzed hundreds of wikis randomly drawn from a population of nearly 180,000 publicly viewable, education-related wikis. We measured wiki quality through examining the detailed edit histories of each of our sampled wikis, using a quality rating rubric that we created. We assessed equity by examining how quality differed across wikis created in schools serving different socioeconomic populations. Our findings suggest that wikis do support emerging models of innovative, online pedagogies that can foster the development of essential competencies for a networked age. We also find that two stubborn challenges of education technology persist into the Web 2.0 era: (a) the difficulty of supporting teachers in using new tools for innovation rather than gaining efficiencies in established routines and (b) the disturbing trend of innovations taking root primarily in already advantaged school settings. From a methodological perspective, we demonstrate that new forms of educational data allow detailed characterizations of classroom technology and pedagogy on a national scale. Background and Context In their article “Web 2.0 and Classroom Research: What Path Should We Take Now?” Greenhow, Robelia, and Hughes (2009) argued in this journal that researchers should focus on what learners do with Web 2.0 tools and issues of equity in networked learning environments. We concur that learner activity and equity are central themes for inquiry into a medium defined by its capacity for broadening participation in knowledge production. In this section, we describe two theories that have informed our inquiry into wiki activity and equity: 21st-century skills and the digital divide.

21st-Century Skills and Measuring Wiki Quality Although education technology is often used to generate efficiencies in content delivery and testing, many educational technologists focus on developing higher order thinking skills and allowing students to rehearse for future performances in a technology-rich workforce and civic sphere. Papert’s (1980) work on computers and constructivism represents one important intellectual wellspring of this tradition. Scardamalia and Bereiter’s (1993) work on knowledge-building communities, especially their wiki-like Knowledge Forum platform, represents another vein of theory that explains how learning technologies can prepare students for the challenges of life rather than for achievement tests. Although, as we shall demonstrate, wikis can be used to facilitate content delivery, we are keenly interested in uses of wikis, like the Flat Classroom Project, that incorporate the collaborative, multimedia affordances of wikis to allow students to create sophisticated performances of understanding (Wiske, Franz, & Breit, 2005). Recent research into the skill demands of America’s rapidly changing labor market has clarified the kinds of higher order thinking skills that today’s students will need to thrive in the future. Levy and Murnane (2004) provide the empirical foundation for the 8

educational Researcher

movement to reorient schools toward the teaching of 21st-century skills. They argue that computers have taken over many of the repetitive tasks that dominated the 20th-century economy. Therefore, education should focus on developing skills where humans have a comparative advantage over computers in the 21stcentury labor market. Levy and Murnane identify expert thinking (ill-structured problem solving) and complex communication as the most important of these skill domains. Jenkins (2009) has proposed another compelling dimension of 21st-century skills: new media literacy, defined as the capacity to critically consume and produce technology-rich media in a social context. Although other scholars have provided their own lists of 21st-century skills, Dede (2010) found that expert thinking, complex communication, and technology literacy are the key domains anchoring the prominent compilations. This scholarship on 21st-century skills provides a theoretical framework for considering how best to measure the quality of online learning environments where higher order thinking skills are emphasized. Numerous studies have investigated the use of online environments to develop particular dimensions of higher order thinking, such as cognitive engagement (Oriogun, Ravenscroft, & Cook, 2005), collaboration (Cortez, Nussbaum, Woywood, & Aravena, 2009; Trentin, 2009), or knowledge building (Moskaliuk, Kimmerle, & Cress, 2009). Most of these studies have been conducted within a single-subject domain, such as algebra (Chiu, 2008) or business ethics (Jeong, 2003). Although these focused studies spotlight selected dimensions of online learning, in this study we attempt to build upon this research by assessing wikibased 21st-century learning broadly and at scale.

Web 2.0 and the Digital Divide In recent decades, the profound impact of technology on the workforce and civic sphere has given rise to serious concerns about the digital divide: inequities in technology-rich educational opportunities. Most early investigations of the digital divide focused on issues of access to computing technology and raised questions about the number of computers or speed of networks in schools serving communities with differing income levels (Warschauer & Matuchniak, 2010). Attewell (2001), however, argued that issues of access were secondary to inequities in technology usage. Several studies that predate the Web 2.0 revolution discovered that students from economically advantaged families were more likely to use technology for higher order thinking when there was more adult involvement (Attewell, 2003; Attewell & Battle, 1999; Wenglinsky, 1998). In contrast, students from disadvantaged families tended to use computers for unsupervised drill-and-practice routines. Attewell (2001) characterized this division as the second digital divide, the gap between how learners use technologies in different communities. For instance, in the 2009 FRSS study cited previously, there were essentially no differences in teachers’ adoption of blogs and wikis across schools serving different populations. As we shall see, however, when we examine wikis in terms of the opportunities they provide for students to develop 21st-century skills, important divisions become apparent.

Research Questions In this study, we define wiki quality in terms of the opportunities that wikis provide for students to develop 21st-century skills such

Table 1 Summary Questions of Coding Categories Used in the Wiki Quality Instrument Category and Subcategory Information Consumption   Course materials   Information gateway Student Participation  Contribution   Individual pages   Shared pages  Ownership Expert Thinking   Academic knowledge   Information organization  Metacognition  Crediting   Teacher feedback New Media Literacy  Formatting  Links  Hyperlinks  Images  Uploads  Multimedia Complex Communication  Concatenation  Copyediting  Co-construction  Commenting  Discussion  Scheduling  Planning

Summary Question Do students come to the wiki to access academic materials? Do students come to the wiki to access links to other Web sites? Does at least one student contribute, in any form, to the wiki? Does at least one student own his or her own page on the wiki? Does at least one pair (or group) of students own their own wiki page? Do student(s) serve as primary facilitator and content creator of the wiki? Does at least one student complete a task requiring academic knowledge (as opposed to simply writing about hobbies or one’s family)? Does at least one student complete a task requiring information organization, rather than routine information retrieval? Does at least one student reflect on his or her work product or process? Does at least one student credit the sources of his or her work? Do teachers provide feedback on student work? Does at least one student use formatting elements beyond plain text? Does at least one student post a link to another page or document? Does at least one student create links rendered as simple text or images? Does at least one student embed an image into a page? Does at least one student upload a document? Does at least one student embed a multimedia element into a page? Do multiple students add discrete sections of text to the same page? Does at least one student copyedit text created by another student? Does at least one student substantively edit text created by another student? Does at least one student comment upon another student’s work on the wiki? Do students respond to each other’s comments for at least four conversational turns? Do students schedule meetings or tasks? Do students plan for future work?

as expert thinking, complex communication, and new media literacy. We measure wiki quality through a detailed analysis of the edit histories of a representative sample of wikis created in U.S. K–12 public schools. Using these quality measurements, we address two questions: (a) To what extent do wikis created in U.S. public schools provide opportunities for the development of 21st-century skills? (b) Do wikis created in schools that serve more affluent populations provide more opportunities for the development of 21st-century skills than wikis created in schools serving less affluent children? Research Design For this study, we drew samples from the population of all 179,851 publicly viewable, education-related wikis hosted by PBworks.com from the founding of the site in June 2005 through August 2008. PBworks is one of the three most visited sites that offer free wiki hosting (Alexa, 2010).These wikis are used from elementary school through graduate schools across the world in nearly every academic subject. For each of these wikis, PBworks preserves every revision of every page. We summarize our methods here, and further details are available in the online supplemental document (available on the journal website).

From the population of 179,851 wikis, we drew a 1% random sample of 1,799 wikis and separated out the 255 wikis identifiably associated with a specific U.S. K–12 public school. Our sample includes wikis created in schools from 41 of the 50 states.1 To evaluate the degree to which wikis provide opportunities for students to develop 21st-century skills, we applied a newly developed instrument called the Wiki Quality Instrument (WQI). We developed the WQI over a two-year period after 68 interviews with wiki-using teachers, 40 student focus groups, observations in 19 classrooms, and several rounds of pilot testing and revision (Reich, Murnane, & Willett, 2010). The WQI has 24 items in five subsections: (a) Information Consumption (2 items) (b) Student Participation (4 items), (c) Expert Thinking (5 items), (d) New Media Literacy (6 items), and (e) Complex Communication (7 items). In each section, coders assess whether students participate in activities that provide opportunities to develop 21st-century skills: activities such as co-creating a shared page, reflecting on a work product, or embedding multimedia frames into a page. In Table 1, we present a summary list of questions posed by the WQI. (The WQI instrument and documentation describing its development and use are available from the first author and at www.edtechresearcher.org.) January/February 2012

9

The WQI poses dichotomous questions about the presence or absence of activities that can provide students with opportunities to develop 21st-century skills. We do not compare the quality or the frequency of activities between wikis, as the learning environments in our sample are too diverse for scalar comparisons (such studies would be fruitful within narrower domains, such as 7thgrade earth science wikis.). Nor do we measure actual student development, as we cannot assess baseline competencies or changes in student competencies resulting from wiki-based learning (such studies would be fruitful if students could be assessed outside the wiki). Instead, the WQI measures “opportunities for 21st-century skill development,” a set of behaviors that are preconditions for 21st-century skill development. Each of our 255 wikis was coded by two research assistants and then reconciled by a third senior research assistant. Our coders identified several key features of each wiki: its academic subject area(s), student grade level(s), and host school. From these school names, we obtained each school’s Title I eligibility and percentage of students eligible for free and reduced priced lunches (FRPL) as indicators of school-level socioeconomic status (SES; National Center for Education Statistics, 2007–2008). To create a final outcome measure, we summed the values of our 24 dichotomous WQI items to form a 0- to 24-point composite wiki quality scale. Interrater agreement across our 24 items averaged .92. We measured wiki quality at 7, 14, 30, 60, 100, and 400 days. We found that wiki quality trajectories tended to be logarithmic; typically, wiki quality rises quickly within the first two weeks and then the quality trajectories level off. As a result, we use wiki quality at Day 14 as a summary statistic that permits consistent comparison across wiki learning communities. To address our first research question concerning overall quality levels in U.S. K–12 wikis, we present the distribution of composite wiki quality scores in our sample as well as descriptions of wikis at various quality levels. To test whether quality differs in wikis created in socioeconomically different schools, we use Poisson regression because our wiki quality scores have features in common with count data (we count the number of identifiable wiki behaviors that can promote 21st-century skill development). We fit a Poisson regression model with the composite wiki quality score as the outcome and the percentage of students eligible for FRPL in a wiki’s school as our question predictor. Patterns of Wiki Usage In our analysis of wiki-using classrooms, we found an extraordinary diversity of learning activities. Students used wikis to publish homework assignments, maintain portfolios, peer review writing, post artwork, download music for rehearsals, and review drills for physical education. One sampled wiki began as a teacher-facilitated reading group for middle school girls, and the girls voluntarily continued using the wiki to maintain a detailed table mapping romantic crushes within their grade. This diverse activity occurred throughout the K–12 sector. Of our 255 public school wikis, 25% supported instruction in Grades K–5, 28% in Grades 6–8, and 52% in Grades 9–12 (the sum of these percentages exceeds 100% because some wikis supported multiple grades). Wikis were used not just in computer classes; they supported instruction throughout the curriculum. 10

educational Researcher

Figure 1. Estimated survivor function of wikis created in U.S. public schools (N = 255).

We found that 34% of wikis supported English/language arts instruction, 13% supported social studies, 18% supported science, 13% supported math, 14% supported computer science, and 26% supported another subject or no subject. These findings generally align with the distributions of wiki adoption reported in the aforementioned 2009 FRSS survey. In addition to these cross-sectional data, we measured each wiki’s lifetime from the moment of creation until the final page edit. In Figure 1, we present the Kaplan-Meier estimated survivor function for our wiki sample (Singer & Willett, 2003). We display the time since wiki creation on the x-axis and estimated survival probabilities (the proportion of wikis that remain active beyond each particular time point) on the y-axis. The steep initial drop in the estimated survivor function indicates that many wikis are terminated almost immediately after creation. For instance, the estimated median lifetime (the length of time beyond which 50% of the original wikis survive) of public school wikis is only 13 days, and only one quarter of wikis persist beyond 151 days. These estimates suggest that most wikis that are used at all are used for short-term projects and assignments rather than as long-term course platforms or student portfolios. We also found evidence that wikis created in schools serving predominantly low-income families cease development earlier than wikis created in other schools. In Figure 2, we display estimated survivor functions for wikis created in schools eligible for Title I versus schools that are not eligible. In our sample of 255 public school wikis, 117 were created in Title I eligible schools and 130 were from non-Title I eligible schools (5 were from schools with missing Title I data). On average, wikis created in non–Title I schools persist longer (Wilcoxon χ2 = 11.38, df = 1, p = .0007). The estimated median lifetime for wikis created in non–Title I schools is 32 days compared with 6.5 days for wikis created in Title I schools. Furthermore, 42% of wikis created in Title I schools do not last more than 1 day, compared with 21% of wikis created in non–Title I schools. Longevity is by no means a perfect proxy for wiki quality, but these findings provide one indication of the second digital divide of usage.

four types of educational wikis. Finally, we illustrate this taxonomy with descriptions of typical wikis from each category. In Table 2, we show the distribution of wiki quality scores at Day 14. In the first column, we list values of composite wiki quality scores (0–24), and in the second column we present frequency counts of wikis at each composite score value. In the following five columns, we present average WQI subdomain scores within each composite score value.2 Using the values presented in these five columns, we show how composite wiki quality scores at each value are derived from the five subdomains. In Table 2, notice that most wikis have a composite wiki quality score of 0, 1, or 2. There is then a long tail of wikis with scores between 3 and 13, and finally the three highest scoring wikis with scores of 17 and 19. From our analysis of this table (and from analyses, not presented here, of wiki quality scores at Day 30, 60, 100, and 400), we developed a taxonomy of four types of wikis: (a) failed wikis, trial wikis, and teacher resource sites; (b) teachercentered content delivery devices; (c) individual student presentations and portfolios with limited collaboration; and (d) collaborative student presentations and workspaces. In Table 3, we show the distribution of wikis among these four categories. To explore these wiki categories, it is illuminating to discuss representative wikis of each type. At Day 14, 40% of our wikis received a composite wiki quality score of 0, meaning that students had no identifiable interaction with the wiki at all. A typical wiki in this category would

Figure 2. Estimated survivor functions for wikis hosted by Title I schools (n = 117) and non–Title I schools (n = 133), through day 200.

To what extent do wikis created in U.S. public schools provide opportunities for 21st-century skill development? To present our findings on overall levels of quality in U.S. K–12 wikis, we first display detailed results from our WQI measures. We then summarize these results by presenting a taxonomy of

Table 2 Frequency Counts of Wikis at Each Composite Wiki Quality Score Value and Average Wiki Quality Instrument Subdomain Scores Within Each Composite Wiki Quality Score Value (N = 255) Composite WQI Score (24 items) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

n 102 49 37 6 4 4 5 7 5 9 7 11 3 3 0 0 0 2 0 1 0 0 0 0 0

Information Consumption (2 items)

Student Participation (4 items)

Expert Thinking (5 items)

New Media Literacy (6 items)

Complex Communication (7 items)

0.0 1.0 1.9 0.8 0.0 0.3 0.0 0.7 0.4 0.3 0.4 0.3 1.3 2.0

0.0 0.0 0.1 1.0 2.5 2.5 3.0 2.6 2.6 3.1 3.0 3.1 3.3 3.7

0.0 0.0 0.0 0.0 0.5 1.3 1.2 1.6 2.4 2.6 2.9 2.7 3.7 2.3

0.0 0.0 0.0 0.8 1.0 1.0 1.8 1.7 2.6 3.0 3.4 4.5 3.0 4.7

0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.4 0.0 0.0 0.3 0.5 0.7 0.3

1.0

3.5

3.5

3.5

5.5

2.0

3.0

3.0

4.0

7.0

January/February 2012

11

Table 3 Distribution of Composite Wiki Quality Scores on Day 14 in Wikis Created in U.S. K–12 Public Schools, by Title I Eligibility (N = 255), in Percentages Wiki Quality Score Range 0 1–2 3–15 16–24

Wiki Type Failed wikis, trial wikis, or teacher resource-sharing sites without student audience or participation Teacher-centered content delivery devices Individual student assignment or portfolio, with minimal collaboration Collaborative, multimedia assignment or workspace

Public Schools (N = 255)

Title I Schools (n = 117)

Non–Title I Schools (n = 133)

40

50

30

34

34

35

25

15

35

1

2

1

have no changes or content. The front page might contain the automatically generated text created by PBworks (“Welcome to your new wiki!”) or some simple modification (“Welcome to Ms. Jones’ World History Wiki”). These were trial balloons that failed to take off. A small proportion of wikis scoring 0 on the WQI served meaningful teacher purposes, such as sharing links or resources, but did not involve students as audience or participants. Next, 34% of wikis had composite wiki quality scores of 1 or 2, and these 1 or 2 points came from the Information Consumption subscale. These were teacher-created content delivery devices with students as receivers of information, not content producers. Many of these wikis provided students with syllabi, class policies, teacher contact information, homework calendars, lists of links to resources for research projects, and other features that might commonly be found on a teacher website. Some wikis had newsletters or updates aimed at parents and families. Some content delivery wikis were quite basic and updated infrequently, and others appeared to be a central part of teachers’ communication routines. In the next category, 25% of wikis had scores between 3 and 15, and these were primarily individually created student assignments or portfolios.3 For instance, five wikis had a composite quality score of 6 on Day 14, which means that we identified six behaviors on the wiki that provide an opportunity for students to develop 21st-century skills. All of these wikis have elementary features of an individual portfolio, where students have posted simple material about themselves and their hobbies and created a basic site architecture with links and pages for future material. They have no collaborative behaviors and limited evidence of expert thinking or the use of multimedia features. Returning to Table 2, notice that two WQI subdomains— Expert Thinking and New Media Literacy—are primarily responsible for score differences among wikis with composite quality scores between 3 and 15. The lowest scoring wikis were spaces where students completed simple tasks, such as writing a basic introduction about themselves in plain text. In a few cases, students posted comments or questions on a teacher-created wiki without interacting with other students. On the highest scoring wikis in this category, students published multimedia-infused presentations or portfolios on academic topics requiring information organization and crediting of sources. 12

educational Researcher

This activity, however, rarely involved interaction among students. Only 11 of the 63 wikis within this score range have any form of collaboration, and what we found was quite limited, such as students commenting on each other’s work or students posting individually created content to the same page. This finding coheres with previous research about the difficulty of nurturing collaborative wiki environments. For instance, in the evocatively titled article, “I DON’T CARE DO UR OWN PAGE!” Grant (2009) provides a case study of how students’ strong individual ownership of text prevents a collaborative ethos from developing in a wiki-using U.K. classroom. That said, a handful of wikis do involve richer collaboration among students. In our sample, 1% of wikis score above 15 on the WQI by Day 14, and these were collaboratively created student assignments and workspaces. For instance, the highest scoring wiki was a group presentation about the philosopher Thomas Hobbes. Students used the wiki to collaboratively plan and then co-construct an intellectually rich multimedia presentation about the philosopher. Another of the top-scoring wikis was used by a middle school literature circle. While reading a novel, students shared responses to reading questions and commented on each other’s answers. As a final project, students collaboratively scripted a “movie trailer” for the book and then used the wiki to plan a video shoot and share multimedia resources for the final edit of the trailer. In these rare cases, students take full advantage of the collaborative and technological affordances of wikis. In summary, most U.S. K–12 wikis provide few opportunities for 21st-century skill development. The majority of wikis are abandoned immediately or are teacher-centered, content delivery devices. An important minority of wikis, however, provides multiple opportunities for students to develop 21st-century skills. Most of these wikis are individual productions where students publish assignments or curate portfolios. The few highest quality wikis are collaborative, multimedia presentations and workspaces.

Do wikis created in schools that serve more affluent populations provide more opportunities for 21st-century skill development than wikis created in schools serving less affluent populations? We find that wikis created in schools serving more affluent populations have more opportunities for 21st-century skill

development than wikis created in schools serving less affluent populations. In Table 3, we show the distribution of composite wiki quality scores by the Title I status of the hosting school. Notice that 50% of wikis created in Title I schools have a score of 0, compared with 30% of wikis created in non–Title I schools. Also, 15% of wikis created in Title I schools have scores between 3 and 15—the individual student wikis—compared with 35% of wikis created in non–Title I schools. We can test and quantify these differences using Poisson regression. (We summarize our analysis here, and parameter estimates and fit statistics are presented in the online supplemental document.) From an unconditional Poisson regression model, we can derive the population-expected wiki quality score, which is 2.80. We would expect the typical wiki to show about 3 behaviors that provide opportunities for students to develop 21st-century skills. When we add to this model a variable controlling for the percentage of students eligible for FRPL in the wiki’s school, we find that wikis created in schools with lower proportions of students eligible for FRPL provide more opportunities for 21stcentury skill development (β1 = –1.59, p < .001). Consider two prototypical wikis, one created in a school with 10% of students eligible for FRPL and another created in a school with 90% of students eligible for FRPL. We estimate that the prototypical wiki created in the high-SES school would have a Day 14 composite wiki quality score of 3.82, compared with a score of 1.07 for the prototypical wiki created in the low-SES school. Quality and Equity in U.S. K–12 Wiki Learning Environments We found four primary patterns of classroom wiki usage: (a) wikis that were abandoned without being used or used exclusively among educators (40%), (b) teacher-centered content delivery devices (34%), (c) individual student assignments and portfolios (25%), and (d) collaboratively created multimedia student presentations (1%). Although a minority of all wikis, the wikis in categories c and d show promise as learning environments that can prepare students for publishing and collaborative problem solving in a networked age. From our analysis of wiki equity, we have documented that wikis created in schools serving more affluent students provide more opportunities for 21st-century learning and persist longer than wikis created in schools serving less affluent students. There is a great danger that the promise and potential of free Web 2.0 tools will disproportionately benefit those already advantaged. These findings, to some extent, cohere with key insights of education technology research from the turn of the century. In his influential book Oversold and Underused, Cuban (2001) analyzed technology use in two Silicon Valley high schools. He found that teachers and students used technology infrequently; that few students had student-centered, technology-rich experiences; and that teachers nearly universally used technologies to gain efficiencies in established routines rather than to transform practice. We see these patterns echoed in our own findings, but we note two important differences. First, although most wikis are used to gain efficiencies in teacher dissemination of information, a considerable proportion of wikis do provide students with opportunities to publish their writing in a new media platform, and a smaller number of wikis foster collaborative student work. If Cuban found a “tiny

percentage” of engaging student uses of technology in his landmark research, we find a small but important collection of wikis that do fulfill the potential of networked technologies to support rich learning. Moreover, Cuban argued that low levels of technology use did not justify the high costs of building school technology infrastructure. With free Web 2.0 tools in an age of ubiquitously wired classrooms, the cost–benefit consideration is different. Many wikis are abandoned soon after creation, but the only cost of such failures is the time of the creator. Teachers who use wikis as content delivery devices have gained efficiencies in typical routines with very low marginal cost. In Cuban’s study, disappointing uses of technology came at great financial cost. In our study, disappointing uses of technology come at minimal cost, and they are accompanied by an important minority of wikis that do offer opportunities for students to develop 21st-century skills. In evaluating our findings about wiki learning environments, we also find it useful to consider research about classroom learning opportunities more broadly. One set of studies in 20 Midwestern middle and high schools (Nystrand, Gamoran, & Carbonaro, 1998; Nystrand, Wu, Gamoran, Zeiser, & Long, 2003) found that in social studies and English classes, classroom time was primarily dominated by recitation (approximately 20 minutes per class) and lecture (approximately 7 minutes per class). More open discussion occurred in fewer than 10% of class periods and lasted, on average, less than 1 minute.4 If these findings are representative of schools nationally, then wikis appear to offer opportunities for publishing, communication, and collaboration that are rarely available in U.S. classrooms. Although we believe that a nuanced examination of wiki quality highlights the potential for Web 2.0 tools to support 21stcentury learning, we remain concerned about the persistence of Attewell’s second digital divide of usage. In our representative wiki sample, we observed patterns that Attewell (2003) found in his small ethnographic samples: Wikis created in schools serving high-income families provide more opportunities for 21st-century skill development than those created in schools serving less advantaged students. Moreover, inequities within schools may be as serious as inequities between schools. While interviewing wikiusing teachers to develop the WQI, several teachers informed us that they used wikis more often or more extensively with their higher tracked students, who are disproportionately wealthier than their peers. Education technology development and research that does not specifically account for these inequities is likely to exacerbate the second digital divide. A New Direction for Education Research One contribution of this research is to provide a detailed portrait of wiki usage in U.S. K–12 settings, with particular attention to how wikis support 21st-century skill development and potentially exacerbate digital divides. In crafting this portrait from wiki edit histories, we hope that a second contribution is to present an application for new data sources generated by online learning environments. Without leaving our offices, we made observations from continuously recorded, student–teacher interactions occurring across the United States, and from these observations we characterized wiki activity both in depth and at scale. January/February 2012

13

We suggest several avenues for leveraging these new data. Future research could produce additional large-scale scans examining different wiki providers, Web 2.0 tools, outcome measures, or countries. With this kind of research, researchers could better contextualize the ethnographic and design research that constitutes the core of education technology scholarship. In addition, broad patterns from quantitative content analysis can suggest new avenues for qualitative investigation. For instance, we are intrigued and concerned by our finding that the percentage of wikis created in Title I schools that fail on the first day is twice as high as the percentage of early failures among wikis created in non–Title I schools. Our content analysis will not unravel this puzzle, but teacher interviews and ethnographic approaches could. Data from online learning environments also have great potential for student assessment (Clarke-Midura & Dede, 2010). In our research, we characterized learning opportunities in wiki communities, and the next logical step would be to use similar analytic methods to track individual student learning. There are no multiple-choice tests that will effectively evaluate students’ abilities to solve ill-structured problems or collaborate with peers. Online learning environments, however, collect continuous data about student performance on such tasks. These data could enable the development of time-efficient, valid assessments of higher order thinking skills. We are optimistic that the earliest forays into this field might rival the efficacy of our current testing systems. For instance, we hypothesize that the number of words that a student writes in secondary school—tracked online— would be a better predictor of college persistence than scores from any contemporary standardized writing assessment. If true, then rather than developing measures of 21st-century skills by devising ever more time-consuming testing regimes (Tamayo, 2010), researchers and policy makers should explore strategies for using real-time, online data sources to measure learning as learners go about their daily activities. The challenges of realizing the potential of Web 2.0 tools in education are considerable. However, these challenges are paired with new research and assessment opportunities enabled by emerging online learning platforms. Although our research has only touched the surface of these new opportunities, we believe that the analysis of large-scale data sets from online learning environments is one of the most exciting new frontiers of educational research. Notes

This research was supported in part by a grant from the William and Flora Hewlett Foundation’s Open Education Resources initiative. 1In the online supplemental document, we discuss two potential limitations of our sample. If the wikis created at PBworks differ from the wikis hosted at other sites, such as Wikispaces.com, then we cannot generalize our findings beyond the population of PBworks wikis. We have no reason to believe that such differences exist, but further research is needed to verify this assertion. Similarly, the population of 179,851 wikis from which we drew our analytic samples represented only 70% of the education-related wikis that were created at PBworks during the time period in question; the remaining 30% were “private” wikis and were not viewable publicly. If publicly viewable wikis differ from private wikis, then our findings cannot be generalized to private wikis. 2Because wiki quality scores are counts, they possess Poisson distributions, characterized by long upper tails. The use of Poisson regression 14

educational Researcher

analysis is more appropriate than ordinary least squares linear regression analysis for the modeling of hypothesized relationships between such outcomes and predictors. Similarly, from a strict statistical perspective, the geometric—rather than the arithmetic—average is a more appropriate summary of the central tendency of Poisson-distributed counts. Typically, in Poisson distributions, the arithmetic mean overestimates the center of the distribution, especially when counts are large. Later in our article, when we model such relationships and conduct statistical tests, we make use of Poisson regression analysis. However, in the descriptive presentation of Table 2, we made a decision to list the arithmetic—not geometric—means of the wiki quality subdomain scores. We did this for several reasons. First, we believe readers will find the arithmetic means more intuitive and interpretable. Second, because scores in each of the wiki quality subdomains are typically low, the bias in the arithmetic mean is small or nonexistent. Second, whenever any of the item scores from which the geometric mean is constituted are zero, the corresponding geometric mean must also be zero. Although anticipated, this would have occurred very frequently in Table 2 had we chosen to display geometric means, concealing what we believe is interesting substantive detail, especially at low wiki quality, which is where the bulk of our sampled wikis fall. 3We chose to use the composite wiki quality score of 15 as our cutoff point for this category of individual student wikis, rather than 13 or some other value, after analysis of wiki quality scores at Days 30, 60, 100, and 400. See the online supplemental document for further details. 4Research projects such as the Gates Foundation Measures of Effective Teaching Project (Learning About Teaching, 2010), where researchers have video recorded and are analyzing approximately 20,000 lessons, may provide some answers to these questions in the future, presenting another example of how emerging technologies can foster detailed investigations of learning activity at scale. References

Alexa. (2010). Top sites within categories: Wiki farms. Retrieved from http://www.alexa.com/topsites/category/Top/Computers/Software/ Groupware/Wiki/Wiki_Farms Attewell, P. (2001). Comment: The first and second digital divides. Sociology of Education, 74(3), 252–259. Attewell, P. (2003). Beyond the digital divide. In P. Attewell & N. Seel (Eds.), Disadvantaged teens and computer technologies (pp. 15–34). Munster, Germany: Waxmann. Attewell, P., & Battle, J. (1999). Home computers and school performance. Information Society, 15(1), 1–10. boyd, d. (2008). Taken out of context: American teen sociality in networked contexts (Unpublished doctoral dissertation). University of California, Berkeley. Chiu, M. M. (2008). Flowing toward correct contributions during group problem solving: A statistical discourse analysis. Journal of the Learning Sciences, 17(3), 415–463. Clarke-Midura, J., & Dede, C. (2010). Assessment, technology, and change. Journal of Research on Technology in Education, 42(3), 309–328. Cortez, C., Nussbaum, M., Woywood, G., & Aravena, R. (2009). Learning to collaborate by collaborating: A face-to-face collaborative activity for measuring and learning basics about teamwork. Journal of Computer Assisted Learning, 25(2), 126–142. doi:10.1111/ j.1365-2729.2008.00298.x Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press. Dede, C. (2010). Comparing frameworks for 21st century skills. In J. A. Bellanca & R. S. Brandt (Eds.), 21st century skills : Rethinking how students learn (pp. 51–76). Bloomington, IN: Solution Tree Press.

Friedman, T. (2005). The world is flat: A brief history of the twenty-first century. New York: Farrar, Straus and Giroux. Grant, L. (2009). “I DON’T CARE DO UR OWN PAGE!” A case study of using wikis for collaborative work in a UK secondary school. Learning, Media and Technology, 34(2), 105–117. Gray, L., Thomas, N. T., & Lewis, L. (2010). Teachers’ use of educational technology in U.S. public schools: 2009 (NCES No. 2010040). Washington, DC: National Center for Education Statistics. Greenhow, C., Robelia, B., & Hughes, J. E. (2009). Web 2.0 and classroom research: What path should we take now? Educational Researcher, 38(4), 246–259. Jenkins, H. (2009). Confronting the challenges of participatory culture : Media education for the 21st century. Cambridge, MA: MIT Press. Jeong, A. (2003). The sequential analysis of group interaction and critical thinking in online threaded discussions. American Journal of Distance Education, 17(1), 25–43. Larusson, J., & Alterman, R. (2009). Wikis to support the “collaborative” part of collaborative learning. International Journal of ComputerSupported Collaborative Learning, 4(4), 371–402. Learning about teaching—Initial findings from the measures of effective teaching project. (2010). Seattle, WA: Bill and Melinda Gates Foundation. Levy, F., & Murnane, R. J. (2004). The new division of labor: How computers are creating the next job market. Princeton, NJ: Princeton University Press. Li, C., & Bernoff, J. (2008). Groundswell: Winning in a world transformed by social technologies. Boston: Harvard Business Press. Moskaliuk, J., Kimmerle, J., & Cress, U. (2009). Wiki-supported learning and knowledge building: Effects of incongruity between knowledge and information. Journal of Computer Assisted Learning, 25(6), 549–561. doi:10.1111/j.1365-2729.2009.00331.x National Center for Education Statistics. (2007–2008). Public elementary/secondary school universe survey data [Data file]. Retrieved from http://nces.ed.gov/ccd/ Nystrand, M., Gamoran, A., & Carbonaro, W. (1998). Towards and ecology of learning: The case of classroom discourse and its effects on writing in high school English and social studies (No. 11001). Albany, NY: National Research Center on English Learning and Achievement. Nystrand, M., Wu, L. L., Gamoran, A., Zeiser, S., & Long, D. A. (2003). Questions in time: Investigating the structure and dynamics of unfolding classroom discourse. Discourse Processes, 35(2), 135–198. O’Reilly, T. (2005, September 30). What is Web 2.0: Design patterns and business models for the next generation of software. Retrieved from http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/ what-is-web-20.html Oriogun, P., Ravenscroft, A., & Cook, J. (2005). Validating an approach to examining cognitive engagement within online groups. American Journal of Distance Education, 19(4), 197–214. Papert, S. (1980). Mindstorms : Children, computers, and powerful ideas. New York: Basic Books. Reich, J. (2008). Reworking the Web, reworking the world: How Web 2.0 is changing our society (Beyond Current Horizons Review paper, Challenge No. 2). Bristol, UK: Futurelab. Reich, J., Murnane, R. J., & Willett, J. B. (2010). The Wiki Quality Instrument: A tool to evaluate the degree to which teachers use wiki learning environments to promote deeper learning. Unpublished manuscript. Reich, J., & Daccord, T. (2008). Best ideas for teaching with technology:

A practical guide for teachers, by teachers. Armonk, NY: M.E. Sharpe. Richardson, W. (2008). Blogs, wikis, podcasts, and other powerful Web tools for classrooms (2nd ed.). Thousand Oaks, CA: Corwin Press. Scardamalia, M., & Bereiter, C. (1993). Computer support for knowledge-building communities. Journal of the Learning Sciences, 3(3), 265–283. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press. Tamayo, J. R., Jr. (2010). Assessment 2.0: “Next-generation” comprehensive assessment systems. Washington, DC: Aspen Institute. Trentin, G. (2009). Using a wiki to evaluate individual contribution to a collaborative learning project. Journal of Computer Assisted Learning, 25(1), 43–55. doi:10.1111/j.1365-2729.2008.00276.x Turkle, S. (2011). Alone together: Why we expect more from technology and less from each other. New York: Basic Books. Warschauer, M., & Matuchniak, T. (2010). New technology and digital worlds: Analyzing evidence of equity in access, use, and outcomes. Review of Research in Education, 34(1), 179–225. doi:10.3102/ 0091732X09349791 Wenglinsky, H. (1998). Does it compute? The relationship between education technology and student achievement in mathematics. Princeton, NJ: Educational Testing Services. Wiske, M. S., Franz, K. R., & Breit, L. (2005). Teaching for understanding with technology. San Francisco, CA: Jossey-Bass. Zuckerman, E. (2010). International reporting in the age of participatory media. Daedalus, 139(2), 66–75. doi:10.1162/daed.2010.139.2.66 AUTHORS

Justin Reich is a doctoral student at the Harvard Graduate School of Education, 6 Appian Way, Longfellow 225, Cambridge, MA 02138; [email protected]. His research examines issues of excellence, equity, and analytics in the use of social technologies in typical K-12 settings. Richard Murnane is a professor at the Harvard Graduate School of Education, 6 Appian Way, Gutman 469, Cambridge, MA 02138; [email protected]. His research focuses on how computer-based technological change has affected skill demands in the U.S. economy, the influence of family income inequality on educational opportunities for children from low-income families, and causal inference in educational research. John Willett, a statistician, is a professor at the Harvard Graduate School of Education, 6 Appian Way, Gutman 412, Cambridge, MA 02138; [email protected]. His research focuses on statistical methods for analyzing the timing and occurrence of events; methods for modeling change, learning, and development; and longitudinal research design.

Manuscript received July 15, 2010 Revisions received July 18, 2011, and September 24, 2011 Accepted September 25, 2011

January/February 2012

15