The Rhetoric and Relevance of IS Research Paradigms - SSRN papers

0 downloads 0 Views 142KB Size Report
research and the role of the researcher in the scientific inquiry. ..... interpretive research, this type of research could be expected to be free of prescriptions.
The Rhetoric and Relevance of IS Research Paradigms: Conceptual Foundations and Empirical Evidence1 Deepak Khazanchi, Ph.D. College of Information Science and Technology, The Peter Kiewit Institute University of Nebraska at Omaha Omaha, NE 68182 E-mail: [email protected]

Bjørn Erik Munkvold, Ph.D. Department of Information Systems University of Agder Post Box 422, NO-4604 Kristiansand, Norway E-mail: [email protected]

Some of the initial ideas included in this paper were first presented at the 36th Hawaii Conference on Systems Sciences (HICSS 2003).

1

I. Introduction As in every other scientific discipline, the conduct of Information Systems (IS) research is guided by a research perspective or paradigm, comprising ontological, epistemological and methodological assumptions that together frame the nature of the research and the role of the researcher in the scientific inquiry. However, these perspectives or assumptions may often be held implicitly, in that the governing structures under which the research is produced are not explicitly discussed or reflected upon by the researcher (Orlikowski and Baroudi, 1991). Over the last two decades, the IS field has been characterized by two parallel and partly intertwined debates: the paradigm debate and the debate related to the relevance of IS research. Mirroring the discussion in the philosophy of social science, the paradigm debate in the IS field has dealt with the discourse on the supremacy of one research perspective over others, especially with regards to the 'quality of research or knowledge claims' produced under each research paradigm (Checkland and Howell, 1998; Fitzgerald and Howcroft, 1998; Klein and Myers, 1999; Mingers and Stowell, 1997; Walsham, 1995b). The relevance debate has been framed around the questions of what characterizes relevant research, and who should be the audience for this research (Benbasat and Zmud, 1999; Fitzgerald and Howcroft, 1998; Hirschheim and Klein, 2003). Both within academia itself and from industry, critical voices have questioned the relevance of IS research today (Ciborra, 1998; Davenport, 1997; Moody, 2000). IS research has been accused of being reactive and impractical, resulting in limited relevance of research outcomes and near ignorance by practitioners in the field

(Benbasat and Zmud, 1999; Ciborra, 1998; Davenport, 1997; Galliers, 1997; Hirschheim and Klein, 2003; Lee, 1999; Saunders, 1998). These two debates are further complicated by the discussion on rigor vs. relevance, where relevance claims under the different paradigms have often been contrasted with the concern for academic rigor of the research produced. For example, two strands in this debate have been that research conducted under the positivist paradigm has been stronger on rigor, while interpretivist research potentially has been more relevant to practice due to its 'contextual grounding.’ In general, the relationship between academic rigor and practical relevance has traditionally been presented as negatively correlated, involving a trade-off situation—suggesting that application of IS knowledge claims to practice can only be achieved at the expense of rigor. However, the nature of these debates so far has mainly been at a theoretical level. Certainly, implications for empirical conduct and practical relevance have been central issues in the debate, but this has still been discussed without giving adequate attention to the connection between the meta-level philosophical rhetoric and the relevance of research results. We believe that there is a need for extending the debate one step further in the ’research value chain’, i.e. to evaluate the impact of the products of research inquiries conducted under the different paradigms. Burrell and Morgan (1979) discuss the phenomenon of 'ontological oscillation', which occurs when researchers in their operationalization of research ideas within an empirical context (unintentionally or intentionally) deviate from the basic ontological assumptions underlying their espoused research paradigm.

Thus, in the words of

Argyris and Schön (1978), we need to go beyond the ontological and epistemological assumptions espoused by the researchers to also look at their ontology and

epistemology in use. The question that begs answering is whether ultimately, when it comes down to applicability of our knowledge-claims, do the values ascribed to the rhetoric of a research paradigm actually matter? In other words, is the rhetoric associated with a research paradigm actually followed in communicating knowledgeclaims or do researchers mold their findings to obtain more practical relevance than their research paradigms would in fact ‘allow’? Alternatively, does the way research is conducted and reported in the IS field actually fulfill the potential relevance inherent in a paradigm's rhetoric? Obtaining an answer to the above questions is important for the successful development of the IS field and provides us the opportunity to introspect on some fundamental meta-theoretical issues of importance to the long-term success of our field. This is aptly echoed in Lee’s (1999, p. 32) comment, “[I]t is not enough for senior IS researchers to call for relevance in IS research. We must also call for an empirically grounded and rigorous understanding of relevance in the first place.” In this vein, this paper aims to contribute to the discussion by increasing our understanding of the concept of relevance and the factors that influence it, and empirically evaluating the link between philosophical assumptions of IS researchers and the relevance of their knowledge claims to the target audience. We freely admit that this is an ambitious task and this research effort only represents a first step in this endeavor. In section III of the paper, we provide the initial basis for addressing the questions raised above by developing a conceptual framework for assessing the relationship between the assumptions and rhetoric of key IS paradigms and the degree of relevance as perceived by different stakeholders. Based on the proposed framework we also discuss a set of research propositions. In section IV, we

elaborate on the research method utilized to conduct the empirical portion of this study. After this, in section V we detail the analysis of the conceptual framework and related propositions using survey data collected from subscribers of the ISWorld mailing list. This empirical substantiation allows us to better reflect on the potential impact of research paradigms on claimed and perceived relevance of knowledge claims for different stakeholder groups. The results of our empirical evaluation also allow us to describe avenues for further research in the area of IS research relevance, as summarized in section VI. In the next section, we provide a quick summary of the paradigm debate and the related concerns about relevance of IS research.

II. Background Ever since Kuhn’s (1962) introduction of the term ’paradigm’ in The Structure of Scientific Revolutions, there has been considerable discussion and controversy about this notion. He argued that at a particular time in the history of scientific development in a field, "a particular paradigm acted as a framework that determined the key concepts and methods, the problems that were significant, and so on" (quoted in Phillips, 1987, p. 21). Most philosophers and scientists today agree that researchers could potentially operate under different paradigms but may disagree on the degree of trust or credibility placed in the knowledge claims of various paradigms (Phillips, 1987). Although Kuhn’s “incommensurability of paradigms” thesis has been found to be invalid by many philosophers including Kuhn himself, the basic notion of a paradigm without the various pejorative connotations attached to the term - is still useful in understanding how scientific research within a field is conducted and how knowledge

claims gain credibility. The basic idea being that “a scientist will normally work within a theoretical framework - a paradigm - that determines the problems that are regarded as crucial, the ways these problems are to be conceptualized, the appropriate methods of inquiry, the relevant standards of judgment, etc” (Philips, p. 205). There is some consensus beginning with Kuhn and subsequent philosophy of science authors (see for example Burrell and Morgan, 1979; Hunt, 1994b; Phillips, 1987) that the term paradigm has the following key characteristics: • Ontology, i.e., the theory or study of existence (being). For example, ontological assumptions in the conduct of inquiry within a paradigm might specifically characterize the nature of reality; • Epistemology, i.e., a theory of knowledge that deals with the nature of knowledge, its scope, and provides a set of criteria for evaluating knowledge claims and establishing whether such claims are warranted; and • Methodology, i.e., a procedure by which knowledge is to be generated. Soft

Hard

The paradigm debate in IS research has been primarily focused Research Researchon the dichotomies related to the dimensions illustratedOntological in Figure 1. As shown in the figure, these Relativist

Level

Realist

dimensions constitute a hierarchy, where higher-level assumptions define the possible scope of the assumptions at lower levels. These levels correspond to those presented by Burrell and Morgan (1979) in their scheme for analyzing assumptions about the Interpretivist Positivist Epistemological nature of social science. Figure 1 also lists the major dichotomies in 'soft' vs. 'hard' IS Subjectivist Objectivist Level Emic/Insider/ research, as identified by Fitzgerald and Howcroft (1998). Subjective

Qualitative Exploratory Induction Field Idiographic

Methodological Level

Etic/Outsider Objective

Quantitative Confirmatory Deduction Laboratory Nomothetic

Figure 1. Dimensional hierarchy and major related dichotomies in IS research

In addition, Fitzgerald and Howcroft (op. cit.) include a fourth level in their analysis, termed the axiological level. This level includes the dichotomy between rigor and relevance, normally associated with the hard and soft approaches, respectively. However, as suggested in the introduction section of this paper, several authors have argued that these should not be seen as conflicting goals, and that relevance does not imply that research needs to be carried out in a less rigorous fashion (see e.g., Benbasat and Zmud, 1999; Khazanchi and Munkvold, 2000). Traditionally, scientific criteria developed from the natural sciences have been applied for judging the quality of research, but a growing number of researchers argue for developing new criteria that better reflect the philosophical underpinnings of alternative paradigms (Guba and Lincoln, 1989; Klein and Myers, 1991). Adding to this are three key problems faced by IS researchers: (1) There continues to be a lack of consensus regarding the fundamental units of research or phenomena being investigated in IS. This is probably because the IS discipline draws upon several reference disciplines, such as management science, organization science, psychology, sociology and anthropology, each representing different strategies for research inquiries; (2) There is some disagreement on whether calls for methodological pluralism for research inquiry in the IS field is in correspondence with the fundamental assumptions and rhetoric associated with research paradigms in use (Kaplan and Duchon, 1988; Lee, 1991; Robey, 1996; Mingers 2001); and (3) There continues to be a lack of consensus on the very elements that form a paradigm; some authors even argue that the IS field is actually in a pre-paradigmatic stage (e.g., Cushing, 1990).

III. Conceptual Framework and Research Propositions In this section we present a conceptual framework and related propositions for assessing the rhetoric and relevance of IS research conducted under various paradigms, as illustrated in Figure 2. In this figure, the left hand side shows constructs that have a potential influence on the degree of relevance for consumers of the researchers’ knowledge claims. For example, it is evident that researchers’ philosophical assumptions have to precede any assessment of degree of relevance. Thus, according to this framework, the 'espoused' philosophical framework held by the researcher, the type and context of the research conducted, and the sustainability of

Philosophical Framework Context of Research Sustainability of Knowledge Claims (Research Results)

Degree of Relevance/Impact on Target Audience or Stakeholder

Type of Research

Figure 2: IS Relevance Conceptual Framework

knowledge claims from the research inquiry will together or individually impact the perceived degree of relevance for different audience/stakeholder groups. The

“perceived” degree of relevance of a researcher’s knowledge claims is conceived as a reasonable surrogate of the actual relevance of their research. Each key element in the proposed conceptual framework illustrated in Figure 2 is further discussed in the following sections.

Major Research Paradigms in IS We use the classification of the three major IS research paradigms into positivist, interpretive and critical research (Chua, 1986; Klein and Myers, 1999; Orlikowski and Baroudi, 1991) as a starting point for our discussion in this paper. Briefly stated, a traditionally positivist position ontologically assumes “naive" or “minimal realism” and a belief that only observable things are real and worthy of study. Epistemologically speaking, consistent positivists assert that any knowledge claim or scientific explanation must be arrived at by means of sensory experience. This “verifiability principle” of the early logical positivists was later replaced by the “testability principle” by logical empiricists. For a detailed discussion on the historical development of positivism the reader is referred to Hunt (1994b) and Philips (1987). It should also be noted that contrary to a common misconception, a preference for quantitative versus qualitative methods to verify or test theories is not a dogma subscribed by most positivists. The reader is referred to Philips (1987) for a detailed argument regarding this point. In contrast to positivism, critical theorists use a historical realist ontological perspective in which the world is not a universe of facts that exists independently of the observer. In this sense, critical research adopts relativism as its ontological basis. The epistemological rhetoric of the critical research paradigm suggests that objective

observation is impossible and that all knowledge is generated or justified in the context of the researcher’s framework and assumptions. In this regard there are clear similarities between the critical theory and interpretivist/constructivist research paradigms - they both include the epistemological notion that objective observation is not possible; however intepretivism includes the additional facet that human experience is a process of interpretation of meanings and actions, that social reality is relative to the observer, and that everyday concepts need to be understood and interpreted to create specific knowledge about the social world. Another basic difference between critical and interpretivist research is the transformative nature of the first, implying a focus on changing the status quo (e.g. related to emancipation and empowerment), whereas interpretivist research can be regarded as more 'neutral' and descriptive in this sense. Table 1 provides a comparison of the key rhetoric related to the three paradigms visà-vis the levels outlined earlier in Figure 1. The table also illustrates the differences in the rhetoric associated with the relationship between theory and practice and the role of the researcher in the three philosophical approaches. A discussion of the relationship between the rhetoric associated with the research paradigms and its impact on the relevance of knowledge claims is postponed till a later section when some key propositions are presented. Suffice it to say that it is well accepted by IS researchers in particular and social science researchers in general, that one’s philosophical framework and its implied or espoused rhetoric may have a relationship with the degree of validity, applicability and generalizability of knowledge claims and that the latter can clearly differ depending upon the targeted stakeholders consuming the products of research. For example, Benbasat and Zmud (1999), who are self-proclaimed positivists in their own right, suggest that the notion of relevance

changes according to the paradigm under which the assessment is conducted, thus resulting in different criteria being applied according to different research paradigms.

Table 1. Comparative overview of key rhetoric of major IS research paradigms2 . Ontological Assumptions

Positivism3 "Naive Realism" in which an understandable reality is assumed to exist, driven by immutable natural laws. True nature of reality can only be obtained by testing theories about actual objects, processes or structures in the real world.

Interpretivism Relativist; the social world is produced and reinforced by humans through their action and interaction

Epistemological • Verification of hypothesis • Assumptions through rigorous empirical testing • Search for universal laws or principles • Tight coupling among • explanation, prediction and control

Critical Research Historical realist; social reality is historically constituted; human beings, organizations, and societies are not confined to existing in a particular state

Understanding of the • social world from the participants' perspective, through interpretation of • their meanings and actions Researchers' prior assumptions, beliefs, values, and interests always intervene to shape their investigations

Knowledge is grounded in social and historical practices Knowledge is generated and justified by a critical evaluation of social systems in the context of researchers' theoretical framework adopted to conduct research Relationship • Generative mechanisms • Generalizations point • It is possible to discover between Theory identified for phenomena to regularities of universal laws that govern and Practice in the social sciences process rather than the external world should be viewed as cross-sectional 'tendencies', which are differences valuable in explanations of • Generalization in past data but not wholly critical research predictive for future focuses on the situations "totality" of relationships • There can be no theory-independent collection and interpretation of evidence to conclusively prove or disprove a theory Objective, impartial observer, Interactive; the researcher Transformative; initiating Role of the passive, value-neutral interacts with the human change in social relations Researcher subjects of the enquiry, and practices, helping to changing the perceptions of eliminate the bases of both parties alienation and domination

Type (or taxonomy) of research

2

These characterizations are synthesized from Guba and Lincoln (1994), Hunt (1991), Orlikowski and Baroudi (1991) and Walsham (1995a). 3 The basic characteristics of logical positivism and logical empiricism are included here. Although, clearly there are some fine distinctions between these two research philosophies.

It is argued that IS research can be classified in terms of its objectives and methods (e.g., Nunamaker et al., 1990-91). Nunamaker et al. (op. cit) classify IS research into basic and applied research, scientific and engineering research, evaluative and developmental research, research and development, and formulative and verificational research. In addition, they argue that there are multi-methodological research approaches such as “systems development” that fit into more than one research category (op. cit). Although the authors propose this classification without the broader paradigmatic context, one could extend the applicability of their classification as follows: if one accepts the assertion that (a) researchers operate under a given paradigm (implicit or espoused) and (b) paradigms have methodological and epistemological assumptions, then one would reasonably have to accept that researchers' paradigmatic stance will potentially restrict which of these types of research they conduct. For example, even if a researcher decides to utilize an approach, the potential relevance of their knowledge claims would be limited by the constraints of the epistemology in use. Context of research

The context of research relates to the question of whether the nature of the implications of a research endeavor is perceived by the researcher to be general or contextually embedded in nature (Khazanchi and Munkvold, 2001). For example, cultural differences may result in relevance being assessed differently in different industries, or, on a global scale, in different regions of the world (e.g. North American vs. European practices). One’s philosophical framework may also potentially dictate the “context of research” and in turn this may have an impact on the degree of relevance.

The espoused rhetoric associated with various research paradigms could have a constraining effect not only on our choice of research methodologies and techniques, but also on the ability to extend one’s research outcomes beyond the context in which research is conducted. The context of research can also influence a researcher's espoused philosophical framework. This can potentially occur in the choice of techniques within the methodological level of a researcher's paradigm. It can also occur when a researcher attempts to modify or adjust their paradigmatic stance for capricious or opportunistic reasons. This tendency is captured in the notion of ontological oscillation (Burrell and Morgan, 1979). In fact, Burrell and Morgan provide examples of research that analytically flags a highly subjective stance that denies the existence of social structures and concrete social reality of any form, but that in the empirical operationalization stage "admits a more realist form of ontology through the back door" (ibid, p. 266).

Degree of relevance to stakeholders In an earlier paper, we proposed that the notion of relevance needs to be expanded to incorporate a broader definition of audience/stakeholders of the research in terms of the scope/value of relevant research (Khazanchi and Munkvold, 2001). Clearly, the potential scope, value and character of relevance vary considerably with the nature of the targeted audience/stakeholder group(s). Based on this argument, we suggest an expanded typology of stakeholders to include practitioners, scholars, educators, users, politicians, economists, citizens, society, national, and global. Therefore in Figure 2, we integrate these concepts into a single construct, “Degree of Relevance” to the stakeholder of research and define it in terms of the total perceived

impact of knowledge claims on the various stakeholders listed on the right hand side of Figure 2. Since perceived relevance is clearly subjective in nature, it is inextricably linked to the value system of the actual stakeholders. Thus, the perceived impact or degree of relevance will vary between different stakeholder groups. Rather than viewing relevance as a dichotomous concept (relevant vs. irrelevant), relevance is regarded in our conceptual framework as a matter of degree. This allows for the possibility that knowledge claims from IS research inquiries can be “partially relevant” to some stakeholders (based on Greisdorf, 2000). Benbasat and Zmud’s (1999) notion of relevance in terms of "content" (i.e., interesting, applicable, current) and "style" (i.e. accessible) is subsumed in our notion of degree of relevance of a knowledge-claim.

Sustainability of Knowledge Claims Finally, the sustainability of the research claims generated by a research inquiry within the context of a researcher’s paradigm in use also has an important impact on the degree of relevance. This relates to the question: 'relevant when, and for how long?' Relevance is clearly somewhat transient in nature, especially in the IS field. What is considered highly relevant now may be regarded as less or even not relevant within a relatively short time frame. The sustainability of knowledge claims in the form of theories and models can clearly have an enormous impact on how research is consumed and perceived by stakeholders. As Lewin (1951, p. 169) correctly suggests, “…nothing is more practical than a good theory,” the better our theories about IS phenomena, the greater their chance of having a long-term impact on IS practice. Research propositions

Based upon the conceptual framework elucidated in the previous section and displayed earlier in Figure 2, it is possible to posit the following propositions. Proposition 1: Notwithstanding philosophical rhetoric to the contrary, IS researchers operating under differing research paradigms are basically motivated to produce knowledge-claims that have applicability beyond the context of research. This proposition goes to the notion whether potential differences in relevance implicit in the rhetoric of each paradigm are valid. For example, based on Table 1 and our discussion of Figure 1, it could be inferred that the research conducted under the critical theory paradigm will be most relevant for contributing towards the empowerment of end users. At the heart of the question of differences in degree of relevance implied by different research approaches, lies the issue of generalizability of the research results generated. Baskerville (1996) discusses how the concept of generalizability in IS research has been restricted by a natural science research model, thus ruling out the potential value of research approaches that focus on unique settings, such as ethnographies, action research and interpretive case studies, to be applied to new problem settings. The limited acceptance of generalizability is reflected by the definition of generalizability as "a quality describing a theory that has been tested and confirmed in a variety of situations, whether such testing is conducted through case research, laboratory experiments, statistical experiments or natural experiments" (Lee, 1989, p. 41). This corresponds to a nomothetic generalization (Burrel and Morgan, 1979). Instead, Baskerville (1996) argues for an idiographic form of generalization, comprising a two-stage generalization process. Stage 1 involves the creation of a general case out of a base case, and stage 2 the application of the general case to the goal case, i.e. the

application setting for the research findings. Stage 2 generalizations can thus be seen to complete the mapping process of the practitioner from experience onto new problem settings. Walsham (1995a) also presents four different types of generalization from interpretive case studies: the development of concepts, the generation of theory, the drawing of specific implications, and the contribution of rich insight. According to this perspective, generalizations should be viewed as tendencies rather than predictions, i.e. as “explanations of particular phenomena derived from empirical interpretive research in specific IS settings, which may be valuable in the future in other organizations and contexts” (ibid, p. 79). Based on the 'weak' notion of generalizability implied in interpretive research, this type of research could be expected to be free of prescriptions regarding approaches for practice. However, when looking beyond the rhetoric applied, one actually finds that interpretive research may contain implications of an equally prescriptive nature as positivist research. An example of such prescription can be identified as part of the seminal work of Walsham (1993), often cited as an exemplar interpretivist study. Although framed as 'issues for debate', several of the practical implications presented related to these issues can be seen to be of a more prescriptive nature, as exemplified by the following two statements: “Senior management need to be intimately involved with development activity, in order to signal commitment, learn, and negotiate” (ibid, p. 206) “[Implementation] Responsibility should be taken by a significant subgroup of IS analysts and at least one member of senior management” (ibid, p. 227).

Positivist researchers working on the TAM (Technology Acceptance Model) based on the Theory of Reasoned Action (TRA) have attempted to validate this model in different contexts and with different technologies (Davis, 1989; Davis et al., 1989; Venkatesh and Davis, 1996). However, this model belies one fundamental ontological assumption in positivism—that the phenomenon under investigation must be observable by one’s sense experiences (logical positivism). Cleary, concepts such as ”perceived usefulness”, ”perceived ease of use” and “self-reported system usage” are terms that represent unobservable notions and their measurement is not direct, not unlike the study of signatures of electrons, quarks and other small particles in Quantum Physics.

Proposition 2: Notwithstanding philosophical rhetoric to the contrary, no single research paradigm is implicitly more suited to producing knowledge-claims that have specific applicability to practice. This proposition is supported by the limiting definition of relevance proposed by mainstream IS researchers such as Benbasat and Zmud (1999) who define relevant research as “one that is potentially useful for, as well as accessible by, … IS professionals (including analysts, user representatives, and IS managers) as well as managers with an interest in information technology (IT) deployment and utilization” (p. 4 & 12). Among their recommendations for attaining increased relevance, Benbasat and Zmud list more contextual description and more prescription. The first comment clearly implies a bias for interpretive research, while the call for more prescriptions seems to encourage a positivist approach, as the rhetoric of interpretive research in general implies caution in presenting general prescriptions. However, as discussed earlier under

Proposition 1, interpretive research studies may also contain implications of a prescriptive nature, although applying a different rhetoric (e.g., Walsham, 1993). In the same vein, it is clear that no single research approach has a monopoly on producing results that have direct relevance to practice.

Proposition 3: Notwithstanding paradigmatic assumptions to the contrary, no single research paradigm is implicitly more suited to producing sustainable knowledge claims that have long-term impact on IS practice. This proposition addresses the question of whether the rhetoric of different research paradigms supports the generation of sustainable theories and concepts or only supports the building of ephemeral theoretical notions that are valid in the immediate term. Clearly, there are two sides to this proposition. From the standpoint of some research paradigms (e.g., positivism), it could be argued that the results of research inquiry are more amenable to producing relevant research for practitioners than other philosophical approaches. On the other hand, it is also possible to assert that a specific paradigmatical stance does not have primacy over relevant research and/or that results of research inquiry need not necessarily have any immediate relevance at all as in the case of basic research where research is conducted ‘for its own sake’. Of course, as IS researchers are discovering that multi-method or multi-paradigmatic research is potentially better suited for IS as a discipline, the supremacy of one paradigm over the other becomes debatable. In fact, the use of a plurality of research perspectives and methods increases the possibility of producing more relevant outcomes that could have a long-term impact on IS practice (Mingers, 2001 & 2003).

Finally it should be noted that with the rapid pace of technological development in the IS field, the time window available to impact IS practice is often short and is even getting more compressed. This can be illustrated by the problem with many articles in major IS journals setting out to present 'novel implications' for implementation and use of 'emerging technologies', that actually already have become largely outdated by the time a submission makes it to publication (Gray, 2001a and 2001b). This problem is further magnified by the lengthy publication process in several leading IS journals which, in many cases, spans several years. However, while this can be seen as a call for more short-term research, several authors argue that it is important to also maintain a long-term perspective to be able to contribute to practice in a proactive way (Galliers, 1997).

IV. Research Method To evaluate the propositions and the relationships implied in our conceptual framework we utilized a survey research design. This entailed the following steps. z Instrumentation: Operationalize the concepts described in our conceptual framework z Survey: Conduct a web-based survey of IS academics and researchers who are subscribers of the ISWorld Mailing List4. Instrumentation (Survey Design) The web-based survey form and accompanying cover letter is provided in Appendix I. The design of the web-based survey utilized recommendations based on the research 4

The survey is available at http://www.isqa.unomaha.edu/relevance/.

conducted by Bowker and Dillman (2000)5. Whenever applicable, respondents were provided an option to click on a particular category or term to view an explanatory definition or description. These explanations for the dependent variable, degree of relevance or stakeholders are provided in Appendix II. Table 2 summarizes the key constructs, definitions, excerpted survey questions and the scale utilized to measure each item.

5

Please refer to the URL, http://survey.sesrc.wsu.edu/dillman/, for a collection of research papers on the issue of web-survey design.

Table 2: Instrumentation Construct

Definition

Philosophical framework [PERSPECT]

Espoused research paradigms of respondents

Context of research [NATURE_CONTEXT]

Nature of knowledge claims or "research outcomes" produced from research is contextually embedded (vs. general or applicable to all contexts) Nature of knowledge claims produced from research is intended to have a lasting impact on the field

Sustainability of knowledge claims [NATURE_IMPACT]

Type (Taxonomy) of research [CLASSIFY]

Degree of relevance or impact on target stakeholders (i.e., practitioners, scholars, educators, users, politicians, economists, citizens, society, nation, and global)

Research classified in terms of methods and objectives Perceived impact of knowledge claims on stakeholders

Survey question

Scale

Choose one phrase that best describes your philosophical perspective when conducting and interpreting research [Q#1] In looking back at your own research work in the IS field, please select a rating from "strongly disagree" (1) to "strongly agree" (5) [Q#3]

Ordinal (Positivism =”1”; Interpretivism/Constructivism = “2”; Critical Research/Critical Theory = “3”; Other = “4”)

Interval Scale (strongly disagree = “1” to strongly agree = “5”). Respondents were also provided a “not applicable” choice.

In looking back at your own research work in the IS field, please select a rating from "strongly disagree" (1) to "strongly agree" (5) [Q#3] Select one phrase that best classifies your research [Q#4]

Interval Scale (strongly disagree = “1” to strongly agree = “5”). Respondents were also provided a “not applicable” choice.

In looking back at your own research work in the IS field, for each "consumer of research" listed, please select a rating from "highly irrelevant" (1) to "highly relevant" (5) that best describes the intended impact of your "knowledge claims" on them [Q#2]

Interval Scale (highly irrelevant = “1” to highly relevant = “5”). Respondents were also provided a “not applicable” choice.

Ordinal (Basic Research = “1”; Applied Research = “2”; Both basic and applied research = “3”; Other = “4”)

V. Analysis of Data and Findings Respondent Profile From the web-based survey, we received a total of 120 responses of which 112 where useful. The survey respondents were predominantly male (70%), reflecting the overall academic population. As displayed in Table 3, only 37% of the respondents identified themselves as positivists and nearly 51% identified themselves as subjectivists – either interpretivists (42%) or critical theorists (9%). These proportions represent an interesting contrast to the dominance of the positivist paradigm reported in previous studies (e.g., Alavi and Carlson, 1992; Chen and Hirschheim, 2004; Orlikowski and Baroudi, 1991). The remainder of the respondents identified themselves as approaching research from a variety of different perspectives. These included respondents who utilize a blend of positivism and interpretivism, pragmatism, empiricism, and action research. These results may be taken to indicate that IS research finally has achieved the goal of incorporating a plurality of research perspectives, as advocated by Mingers (2001), Robey (1996), and others. However, analyses of the major IS journals shows that such a pluralism has yet to be reflected in the published articles in these outlets, with the quantity of non-positivist contributions still being marginal compared to the positivist dominance in journals such as MISQ, ISR, JMIS and EJIS (Chen and Hirschheim, 2004, Farhoomand and Drury, 1999). Two factors that have been suggested as important for explaining this dominance of the positivist paradigm is the current tenure and promotion system in many institutions, and the relative dominance of North American positivist IS researchers in the journal

editorial boards (Chen and Hirschheim, 2004; Walsham, 1995b). With the tenure system mainly emphasizing quantity of publications, "it is safer for authors to stick to positivist orthodoxy" (Walsham, 1995b, p. 391) than to undertake interpretivist research which in general is considered more time consuming than survey based research (Chen and Hirschheim, 2004; Walsham, 1995b). Further the positivist dominance in editorial board of journals results in a self-sustaining system where it is difficult for researchers representing alternative research paradigms to gain acceptance for their work. In light of this, a possible explanation of the much higher interpretivist representation among our respondents could be that the influence of the subjectivist paradigms (i.e. interpretivist and critical research/theory) is actually much larger in terms of the espoused paradigm among IS researchers than what is reflected as their paradigm in use in the journal publications. In this sense, the paradigm orientation of our respondents may reflect a form of ontological oscillation (Burrell and Morgan, 1979), where the "conserving" forces presented above resulting in a positivist bias in the journal publishing game6. Table 3: Espoused Philosophical Perspective Perspective

Frequency

Percent

Positivism Interpretivism/Constructivism Critical Research/Critical Theory Other Total

42 47 10 13 112

37.5 42.0 8.9 11.6 100.0

6

We admit that an alternative explanation could of course be that there is a bias in our sample, in that non-positivist researchers per se are more interested in this type of paradigm debate, and thus also more inclined to respond to the survey. And/or that our respondents are more the "grass roots" of IS, which does not necessarily belong to the more "elite" researchers publishing in the major outlets represented in previously reported research. Whichever explanation the reader is willing to accept, we are convinced that that our sample, though limited in its randomness, is quite reflective of the perception of the IS academic population today.

Table 4: Classification of Research Classification of Research Basic Research Applied Research Both basic and applied research Total

Frequency

Percent

9 51 52 112

8.0 45.5 46.4 100.0

Nearly 46% of the survey respondents classified their research as being both applied and basic in nature (refer Table 4). An equal number of respondents considered their research to be purely applied in nature. Interestingly, this meant that only 8% of the respondents classified their work as basic research. These results have some interesting implications for the epistemological and pragmatic development of IS as a discipline. To quote William R. Brody, President of Johns Hopkins University, “historically, innovation in science and technology has been the direct result of investments in basic research and development”. In fact it is clear that continuing development of a discipline is dependent upon the advancement in basic research; since applied research is necessarily limited to the progress in basic research. Since we are assessing perceptions of relevance, we wanted to ensure the credibility of the respondents by asking them two details about their research perspectives –primary choice for publishing research and the relevance criteria used by researchers from different espoused perspectives. The first question asked survey respondents to list their primary choice of a journal or other publication outlet that they considered to be most relevant or suitable for publishing their own research. While survey respondents listed a number of different journals, the top journals identified -MISQ (35), ISR (11), JMIS (8), CACM (5), and ISJ or Information Systems Journal (6) --

were mostly consistent with studies that focused on IS journal ranking (refer http://www.isworld.org/csaunders/rankings.htm). The following table details the relationship of espoused philosophical perspectives and respondents’ primary choice for publishing their research. Using the nonparametric Chi-Square test of proportions, we find that the proportion of respondents with a specific research perspective differed significantly across the various top journals identified in this study. This conclusion is in line with Mingers' (2003) study of MIS journals where he found that “there were significant relationships between individual journals and particular research methods. For instance, MISQ and ISR are dominated by positivist research, with ISR particularly favoring experiments and statistical analysis... In contrast, the European journals JIT and ISJ are both more pluralist in content with a greater level of (although still little) interpretive research.” Similar results were also found by Chen and Hirschheim (2004) in their analysis of eight major IS publication outlets. Of these, five outlets were clearly dominated by a positivist approach (MISQ, ISR, JMIS, ICIS and EJIS), of which all except EJIS are based in North America. The three outlets where interpretive research had a more equal influence (ISJ, Journal of Information Technology, and Information and Organization (former Accounting, Management, and Information Technology)) are all based in Europe. Somewhat contrary to theseconclusions, however, our results displayed in Table 5 indicate that pluralism also prevails with regard to the ranking of IS journals across North America and Europe -- an equal number of respondents from positivist and interpretivist/critical research perspectives have identified North American journals (MISQ, ISR, CACM and JMIS) along with ISJ and EJIS as some of the top journals in the IS field. The high ranking of these journals among the interpretivists as

well is interesting when considering the relatively imited number of non-positivist articles actually having been published in these outlets so far (except for ISJ) (Chen & Hirshheim, 2004). This could be seen to demonstrate how the ranking of top IS journals seemingly has a universal status across paradigm orientation. However, this could also be a result of the positivist-based tenure and promotion system. It should also be noted that among the non-positivist respondents a total of 28 other journals were listed, while the positivist respondents listed only 8 other journals. Table 5: Espoused Research Perspective and Primary Publication Outlet RESEARCH PERSPECTIVE Positivism Interpretivism/Constructivism Critical Research/ Critical Theory Other Total

OUTLET JOURNAL JMIS CACM ISJ 3 2 1 3 2 5

MISQ 17 10

ISR 6 4

3

0

1

0

5 35

1 11

1 8

1 5

EJIS 2 1

Other 8 22

0

0

6

0 6

0 3

5 41

The second related question asked survey respondents to list three key criteria for judging the relevance of their own knowledge claims or research outcomes. Each criterion cited by the respondents was culled into major thematic categories and the top three are provided below by their level of importance. 1. Rigor, i.e. findings are justifiable, valid, well founded, tied to theory, and potentially contribute to theory development. 2. Relevance, i.e. applicability to industry and academia, and useful to stakeholders. 3. Publishable, i.e. ability to produce peer reviewed publications, co-citations, etc.

The previous result derived from descriptive data is consistent and is closely tied with the ratings provided by survey respondents to questions used to measure the generalizability and sustainability of knowledge claims. As shown in Table 6, on the average, regardless of their philosophical orientation, respondents tended to believe that their knowledge claims or research outcomes were relevant beyond the context in which the research is conducted. When respondents were asked about the lasting impact of their research on their target audience, the mean rating was 4.04 suggesting the importance of long-term impact on the field.

Table 6: Other Independent Variables – Generalizability and Sustainability of Knowledge Claims

The nature of the knowledge claims or "research outcomes" produced from my research is contextually embedded (vs. general or applicable to all contexts). (“1” = Strongly disagree and “5” = Strongly agree) The nature of the knowledge claims produced from my research is intended to have a lasting impact on the field (“1” = Strongly disagree and “5” = Strongly agree)

N

Minimum

Maximum

Mean

111

1

5

3.58

Std. Deviation 1.014

112

2

5

4.04

0.821

Analysis of Conceptual Model and Propositions The descriptive statistics for the dependent variable, degree of relevance, are provided in Table 7. The average rating ranged between 4.53 and 2.57 with only two ratings going below 3.00. It is evident from these ratings that survey respondents felt strongly about the intended impact or relevance of their research as it pertained to practitioners, scholars, educators, users, citizens, society, nation, and the world; while respondents felt that their research results were somewhat relevant to Politicians and Economists.

Table 7: Impact of knowledge claims on target audience (Degree of Relevance – “1” = highly irrelevant and “5” = highly relevant) Intended impact of N "knowledge claims" on: Scholars 112 Practitioners 112 Educators 111 Users 108 Society 102 Global 102 Nation 101 Citizens 97 Politicians 97 Economists 99

Minimum

Maximum

Mean

Std. Deviation

1 1 1 1 1 1 1 1 1 1

5 5 5 5 5 5 5 5 5 5

4.53 4.27 3.96 3.82 3.49 3.40 3.15 3.04 2.70 2.56

.816 .880 1.035 .965 1.060 1.213 1.178 1.145 1.235 1.197

In order to analyze the proposed conceptual model and test the propositions described previously in section III of the paper, we used the general linear model (GLM) multivariate procedure in SPSS v 12.0. We used this procedure because of the multiple dependent variables (i.e., the dimensions of relevance), the available sample size, our mixed measurement model -- both ordinal and interval data, and the robustness of the analysis of variance under this model. We checked for multicollinearity in the data by looking at pairwise non-parametric correlations (Kendall’s tau and Spearman’s rho) and found no statistically significant correlations between each of the independent variables. We have provided these correlation statistics in Appendix III. The main effect statistics produced by the GLM multivariate procedure are displayed in Table 8. It is evident from the p-values for both the Pillai's Trace and Wilks' Lambda statistic that all the independent variables are statistically significant at the 0.05 level and explain with a high degree of confidence the mean variability of the dependent variables, i.e., perceived degree of relevance for various stakeholders. In other words, the overall degree of relevance for consumers of research varies significantly based upon the philosophical perspective, the context of the research, sustainability of knowledge claims, and the type of research of the survey respondents. This conclusion provides general empirical support for propositions 1, 2 and 3. Note also, the main effect variables have high observed power values adding to the statistical validity of our conclusions. Further analysis of the significant main effects using post-hoc pairwise multiple comparisons is shown in Table 8. This analysis reveals that on the average survey

respondents who were interpretivists/constructivists were significantly different in their perception of relevance of their knowledge claims for citizens. However, there was no significant difference between interpretivists/constructivists, critical researchers, and positivists with regards to the relevance to other stakeholders (i.e., practitioners, scholars, educators, users, politicians, economists, society, nation, and global). Furthermore, respondents who reported conducting only basic research were significantly different (lower in magnitude) in their perception of the impact of their research on practitioners than those who stated using applied research or a mixed (both applied and basic) research approach. This is in line with our intuitive expectation that applied research would have greater relevance for practitioners. Similarly, respondents who reported conducting only basic research were significantly different (lower in magnitude) in their perception of the impact of their research on educators and users than those who stated using applied research or a mixed (both applied and basic) research approach. All other post-hoc tests did not show any significant differences between the different levels of relevance of knowledge claims. In other words, there was no significant difference between positivists, interpretivists/constructivists, and critical theorists, when it came to the specific relevance of knowledge claims to practitioners, scholars, educators, users, politicians, economists, society, nation, and global issues.

Table 8: Relevance of IS Research – GLM MultivariateTests7 (Main and Interaction Effects) Effect Intercept

Pillai's Trace

perspect

Wilks' Lambda Pillai's Trace

classify

Wilks' Lambda Pillai's Trace

nature_c

Wilks' Lambda Pillai's Trace

nature_i

Wilks' Lambda Pillai's Trace

perspect classify

perspect nature_c

perspect nature_i

classify nature_c

classify nature_i

nature_c nature_i

*

Wilks' Lambda Pillai's Trace

*

Wilks' Lambda Pillai's Trace

*

Wilks' Lambda Pillai's Trace

*

Wilks' Lambda Pillai's Trace

*

Wilks' Lambda Pillai's Trace

*

Wilks' Lambda Hotelling's Trace Pillai's Trace Wilks' Lambda

Value .942

F 58.247

Hypothesis df 10.000

Error df 36.000

Sig. .000

Partial Eta Squared8 .942

Observed Power(a) 1.000

.058

58.247

10.000

36.000

.000

.942

1.000

1.044

2.029

30.000

114.000

.004

.348

.996

.249

2.151

30.000

106.343

.002

.371

.996

.785

2.390

20.000

74.000

.004

.392

.988

.368

2.332

20.000

72.000

.005

.393

.986

1.422

2.152

40.000

156.000

.000

.356

1.000

.139

2.357

40.000

138.363

.000

.389

1.000

.818

1.426

30.000

114.000

.094

.273

.952

.373

1.417

30.000

106.343

.100

.280

.941

.906

1.644

30.000

114.000

.033

.302

.979

.330

1.624

30.000

106.343

.038

.309

.971

1.513

2.373

40.000

156.000

.000

.378

1.000

.122

2.571

40.000

138.363

.000

.410

1.000

1.471

2.268

40.000

156.000

.000

.368

1.000

.136

2.398

40.000

138.363

.000

.393

1.000

1.113

1.503

40.000

156.000

.042

.278

.990

.247

1.540

40.000

138.363

.035

.295

.985

1.233

1.738

40.000

156.000

.009

.308

.997

.200

1.826

40.000

138.363

.006

.331

.996

2.169

1.871

40.000

138.000

.004

.352

.998

1.449

2.215

40.000

156.000

.000

.362

1.000

.144

2.303

40.000

138.363

.000

.383

1.000

7

Design: Intercept+perspect * nature_c+perspect * nature_i+nature_c * nature_i+nature_c * classify+nature_i * classify+perspect+nature_c+nature_i+classify. 8

The eta-squared statistic describes the proportion of total variability attributable to a factor.

Further, although the context of research and sustainability of knowledge claims explained variability in the degree of relevance, there was no difference in the level of this relationship between any of the relevance dimensions. Thus one could potentially argue, for example, that our data does not support the generally held belief that critical theory-based knowledge claims have a monopoly on the empowerment of users or that relevance to practitioners is more likely for interpretivists and less so for positivist researchers. This conclusion is of course subject to the caveat that measuring intended relevance for users is not exactly the same as measuring intended empowerment of users, but at a minimum, a reasonable surrogate of the same. The above results coupled with the descriptive statistics summarized earlier provide additional empirical credence to our claims in propositions 1, 2 and 3 and also provide additional food for thought about the notion of relevance and the nature of our espoused paradigms or paradigms in use. The specific assertion in proposition 1 is that researchers operating under different paradigms are interested in their research having implications beyond the context of research notwithstanding implicit assumptions to the contrary. This assertion can be made with additional confidence because of the following facts derived from our data analysis. •

Survey respondents report average degree of relevance scores in the 2.56 to 4.53 range (refer Table 7). This clearly indicates that regardless of paradigmatic orientation, researchers want their knowledge claims to have a broad impact.



As evidenced by the mean ratings shown in Table 6, survey respondents believe that the nature of knowledge claims produced from their research should have a lasting impact on the field.



The GLM results described earlier show that context of research has a significant influence on the overall degree of relevance. However, post-hoc analysis shows that there is no difference in the levels of relevance between philosophical paradigms. Similarly, the analysis of data also clearly lends substantive credence to our

assertion in proposition 2 -- that no single research paradigm has a monopoly on producing knowledge claims that are of relevance to practitioners. •

The mean rating relating to the impact of knowledge claims on practitioners had the second highest value of all dimensions of relevance (refer Table 7). This clearly indicates that regardless of paradigmatic orientation, researchers want their knowledge claims to have a broad (vs. limited to a context) and sustainable impact on the field. This is also reflected in the high mean ratings for the lasting impact or sustainability of researchers’ knowledge claims on the field.



The GLM results and post-hoc analysis described earlier that though philosophical perspective explained variability in overall relevance of researchers’ knowledge claims, there is no difference between the three main philosophical perspectives with regards to the perceived relevance of knowledge claims to all stakeholders except citizens. Post-hoc analysis also indicated that respondents who conduct basic research feel that their research has a lesser impact (in magnitude), in terms of relevance of knowledge claims on various stakeholders, than applied researchers.

Finally, the analysis of data discussed earlier clearly also corroborates our assertion in proposition 3 – that no single research perspective or paradigm is implicitly better suited to producing sustainable knowledge claims. As pointed out earlier, respondents felt strongly about the importance of the long-term impact on the field produced from their research outcomes. As displayed in Table 9, post-hoc pairwise comparisons indicate that survey respondents who rated that lasting impact of their knowledge claims was very important were particularly likely to have higher ratings for relevance of their research to scholars.

Table 9: Multiple Comparisons (Scheffe Test – significant results only – 0.05 significance level) Dependent Variable

(I) Perspective

(J) Perspective

Citizens

Interpretivism/Constructivism

Positivism

Dependent Variable (I) Classify

(J) Classify

Practitioners

Applied Research

Educators

Both basic and research Applied Research

Users

Applied Research Both basic research

Dependent Variable Scholars

and

(I) Nature _impact 5 (HIGH)

Basic Research applied Basic Research Basic Research Basic Research applied Basic Research

(J) Nature_impact 2 (LOW)

Mean Difference (I-J) .71(*)

Std. Error

Sig.

.220

.024

Mean Difference Std. Error (I-J) 1.30(*) .255

Sig. .000

1.09(*)

.254

.000

1.10(*)

.416

.033

1.20(*)

.387

.010

1.13(*)

.387

.016

Mean Difference (I-J)

Std. Error

Sig.

.51(*)

.169

.038

VI. Summary and Concluding Remarks In this section we summarize the main findings, and point to implications for practice and further research. At the beginning of this paper we set out to address the general question: do the values ascribed to the rhetoric of a research paradigm regarding the intended applicability of our knowledge claims actually matter? Our theoretical arguments and empirical evidence indicate a negative answer to this question. We found that researchers with different espoused philosophical perspectives are ultimately driven in their research approaches by the bottom line impact of their knowledge claims on different stakeholders. To summarize, we have proposed a conceptual framework for assessing the impact of a researcher’s philosophical framework, context of research, sustainability of research and type of research on the purported degree of relevance to various stakeholders. It provides an organized way of thinking about the status of our knowledge claims and their potential impact on the degree of relevance for various stakeholders. In addition, it indirectly raises the issue of whether there is a correspondence between the characteristics of the implicit and explicit rhetoric associated with key IS research paradigms and the actual relevance of the products of research to practice. Due to the difficulty of objectively assessing relevance to practice, the notion of perceived degree of relevance to stakeholder group is proposed as a predictor of actual relevance.

As stated in the introduction, rather than providing answers to these important questions, the goal of the framework and propositions developed in this paper and the initial empirical validation of our conceptual framework is to stimulate further debate and reflection and provide a foundation for conducting further research on the question of how the 'paradigm in use' may impact the degree of relevance for different stakeholder groups and produce sustainable knowledge claims. It should be reiterated here that in this paper we just begin to scratch the surface of how IS research paradigms (espoused or those ‘in use’) impact relevance of our knowledge claims. Thus, the notion of which criteria would be useful in assessing the impact for different stakeholders is not directly addressed and needs to be further developed in terms of the validity and nature/contribution of knowledge claims under each research paradigm. On the other hand, to also be able to study the 'paradigms in use' and the relationship to the espoused paradigms requires an analysis of exemplar research produced under different research paradigms. A general challenge in this endeavor is the wide variety of research perspectives in use in IS research today. In this paper we have applied a broad classification of IS research into three key paradigms. However, in practice there are other research paradigms (e.g., “critical or scientific realism”) and, in many instances, individual researchers seem to utilize a continuum of adaptations of these paradigms (Morgan, 1983). For any analysis to be complete one would need to take all these variations into account. Still, we believe that it is possible to identify researchers and studies that represent the core assumptions of these different paradigms. Through analyzing the products of this type of research effort, it may be possible to generate an insight into IS research that goes beyond the 'espoused paradigms' and related rhetoric,

and instead focuses on current research practice and the resulting impact on relevance for different stakeholder groups. By doing this, we may be able to move on from the abstract discussions characterizing the current debate on relevance, to the important issue of how we as IS researchers can produce research results that are of most value and accessibility to the different consumers or stakeholder groups.

6. References Alavi, M. and Carlson, P. “A Review of MIS Research and Disciplinary Development”, Journal of Management Information Systems, 8, 4 (Spring 1992), 45-62. Argyris, C. and Schön, D. A. Organizational Learning: A Theory of Action Perspective. Addison-Wesley, Reading, MA, 1978. Baskerville, R. "Deferring Generalizability: Four Classes of Generalization in Social Enquiry," Scandinavian Journal of Information Systems (8:2), 1996, pp. 5-28. Benbasat, I. and Zmud, R. W. "Empirical Research in Information Systems: The Practice of Relevance," MIS Quarterly (23:1), March 1999, pp. 3-16. Bowker D. and Dillman, D. A. “An Experimental Evaluation of Left and Right Oriented Screens for Web Questionnaires.” URL: http://survey.sesrc.wsu.edu/dillman/papers/AAPORpaper00.pdf, 2000 (Accessed: 2/10/2005). Burrell, G. and Morgan, G. Sociological Paradigms and Organizational Analysis, Ashgate Publishing Company, Aldershot, 1979. Checkland, P. and Holwell, S. Information, Systems and Information Systems - making sense of the field, Wiley and Sons, Chichester, 1998. Chen, W.S. and Hirschheim, R. "A paradigmatic and methodological examination of information systems research from 1991 to 2001," Information Systems Journal (14), 2004, pp. 197-235. Chua, W. F. "Radical Developments in Accounting Thought," The Accounting Review (61), 1986, pp. 601-632. Ciborra, C. U. "Crisis and foundations: an inquiry into the nature and limits of models and methods in the information systems discipline," Journal of Strategic Information Systems (7), 1998, pp. 5-16.

Cushing, B. E. "Frameworks, Paradigms, and Scientific Research in Management Information Systems," Journal of Information Systems, (4:2), Spring 1990, pp. 38-59. Davenport, T. "Storming the Ivory Tower," CIO Magazine, (April 15, 1997), http://www.cio.com/archive/ 041597_think_content.html [Accessed 29 September 1998]. Davis, F. “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology”, MIS Quarterly, (September), 1989, pp. 319-340. Davis, F., Bagozzi, R.P. and Warshaw, P.R. “User Acceptance of Computer Technology: A Comparison of two Theoretical Models,” Management Science, 35(8), August 1989, pp. 982-1002. Farhoomand, A.F. and Drury, D.H. "A historiographical examination of information systems: the Minnesota experiments," Communications of the Association for Information Systems (1), 1999, pp. 1-27. Fitzgerald, B. and Howcroft, D. "Toward dissolution of the IS research debate: from polarization to polarity," Journal of Information Technology (13), 1998, pp. 313-326. Galliers, R. "Reflections on information systems research: twelve points of debate," in J. Mingers and F. Stowell (eds.), Information Systems: An Emerging Discipline? McGraw Hill, London, 1997, pp. 141-157. Gray, P. (Ed.). Special Issue on Relevance, Communications of the AIS, Volume 6, 2001a. Gray, P. Three Challenges for IS. The DATA BASE for Advances in Information Systems. Vol. 32, No. 1, Winter 2001b, 7-14, Greisdorf, H. "Relevance: An Interdisciplinary and Information Science Perspective," Informing Science (3:2), 2000, pp. 67-71. Guba, E. G. and Lincoln, Y. S. "Competing Paradigms in Qualitative Research," in N. K. Denzin and Y. S. Lincoln (eds.), Handbook of Qualitative Research, Sage Publications, Thousand Oaks, CA, 1994, pp. 105-117. Guba, E. G. and Lincoln, Y. S. Fourth Generation Evaluation. Sage, CA, 1989. Hirschheim, R. and Klein. H.K. "Crisis in the IS Field? A Critical Reflection on the State of the Discipline", Journal of the Association for Information Systems, Vol. 4, No. 5, pp. 237-293. Hunt, S. D. Modern marketing theory: Critical issues on the philosophy of marketing science. Cincinnati, OH: South-Western Publishing, 1991.

Hunt, S.D. “On the Rhetoric of Qualitative Methods: Towards Historically Informed Argumentation in Management Inquiry,” Journal of Management Inquiry, (3:3), September 1994b, pp. 221-234. Kaplan, B. and Duchon, D. "Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study". MIS Quarterly, December 1988, pp. 571-586. Khazanchi, D. and Munkvold, B.E. "Is information systems a science? An inquiry into the nature of the information systems discipline." The DATA BASE for Advances in Information Systems, Volume 31, Issue 3, Summer 2000, pp. 26-44. Khazanchi, D. and Munkvold, B.E., ”Expanding the Notion of Relevance in IS Research: A Proposal and Some Recommendations,” Communications of the AIS, Volume 6, 2001, Article 14, pp. 1-12. Klein, H. K. and Myers, M. D. "A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems," MIS Quarterly, Special Issue on Intensive Research (23:1), 1999, pp. 67-93. Kuhn, T. The structure of scientific revolutions, Chicago, IL: The University of Chicago Press, 1962. Lee, A. S. "Rigor and Relevance in MIS Research: Beyond the Approach of Positivism Alone," MIS Quarterly (23:1), March 1999, pp. 29-34. Lee, A. S. “Integrating positivist and interpretive approaches to organizational research,” Organizational Science (2), 1991, pp. 342-365. Lee, A. S. "A Scientific Methodology for MIS Case Studies". MIS Quarterly, 13, 1989, pp. 33-50. Lewin, K. Field Theory in Social Science, University of Chicago Press, Chicago, 1951. Mingers, J. The paucity of multimethod research: a review of the information systems literature. Information Systems Journal, 13, 3 (2003), 233-249. Mingers, J. Combining IS Research Methods: Towards a Pluralist Methodology. Information Systems Research, 12, 3 (September 2001), 240-259. Mingers, J. and Stowell, F. (eds.) Information Systems: An Emerging Discipline? McGraw Hill, London, 1997. Moody, D. L. "Building Links Between IS Research and Professional Practice: Improving the Relevance and Impact of IS Research," Proceedings of the Twenty-First

International Conference on Information Systems, Brisbane, December 2000, pp. 351360. Morgan, G. Beyond Method. Strategies for Social Research. Sage, Newbury Park, CA, 1983. Nunamaker, J.F., Chen, M. and Purdin, T.D.M. “Systems Development in Information Systems Research,” Journal of Management Information Systems, (7:3), Winter, 199091, pp. 89-106. Orlikowski, W.J. and Baroudi, J.J. (1991). "Studying Information Technology in Organizations: Research Approaches and Assumptions," Information Systems Research (2:1), pp. 1-28. Philips, D. C. Philosophy, Science, and Social Inquiry: Contemporary Methodological Controversies in Social Science and Related Applied Fields of Research, Oxford, England: Pergamon Press, 1987. Robey, D. "Research Commentary: Diversity in Information Systems Research: Threat, Promise, and Responsibility," Information Systems Research (7:4), 1996, pp. 400-408. Saunders, C. (Ed.). "The Role of Business in Information Technology Research," Special 10th Anniversary Issue of the Information Resources Management Journal (11:1), 1998. Venkatesh, V. and Davis, F.D. “A model of the antecedents of perceived ease of use: Development and test,” Decision Sciences, 27(3), Summer 1996, pp. 451-481. Walsham, G. "Interpretive case studies in IS research: Nature and Method." European Journal of Information Systems (4), 1995a, pp. 74-81. Walsham, G. "The Emergence of Interpretivism in IS Research." Information Systems Research (6:4), 1995b, pp. 376-394. Walsham, G. Interpreting Information Systems in Organizations, John Wiley & Sons, Chichester, 1993.

Appendix I: Web-Based Relevance Survey Form (http://www.isqa.unomaha.edu/relevance)

Appendix II: Web-Based Relevance Survey (Description of Relevance Dimensions -- http://www.isqa.unomaha.edu/relevance) Audience/ Stakeholder Group

Examples of Potential Scope, Value

Examples of Related IS Research Areas

Practitioners

Improving existing business practices; Lead the way to new practices; Legitimatize decisions

Strategic alignment, TAM, BPR

Scholars

Theory development; Communication of knowledge-claims to peers; Satisfy researcher's intellectual curiosity; Impact personal motivation

"Philosophical" research in IS; Relevance of research; Methodological Issues in IS

Educators

Utility of knowledge claims; Engender intellectual curiosity; Develop new instructional methods

Research on IT educational aspects such as pedagogy, delivery, and integration of research in teaching

Users

Empowerment; Improvement of quality of work life

Action research, Socio-technical research, "Scandinavian tradition”

Politicians

Legitimatize political decisions; Value for public policy making

The Internet, Privacy issues, Encryption and security, e-government, edemocracy

Economists

Utility through improved understanding of IS phenomena

Evaluation of IS/IT research

Citizens

Encourage general understanding of the IS discipline; Education about the IS field

IS education and pedagogy, IS ethics, Impact of IT on people

Society

Legitimatize public policy making (e.g., privacy issue and related laws); Engender rational discourse on societal issues

Critical research, Impact of IT on society, Privacy and ethical issues of IT

Nation

Education regarding IS issues at national level

Y2K research, National information infrastructures, IT in developing countries

Global

Education regarding IS issues at international/global level; Understanding IS phenomena in the International context

Transborder data flow, Intellectual property issues, Global diffusion of the Internet

[Source: Khazanchi and Munkvold (2001, March), “Expanding the notion of relevance in IS research: A proposal and some recommendations,” Communications of the AIS, Volume 6, Article 14, pp. 1-12].

Appendix III: Testing for Multicollinearity – Nonparametric Correlations

Kendall's tau_b

perspective

nature_context

nature_impact

classify

Spearman's rho

perspective

nature_context

nature_impact

classify

Correlation Coefficient Sig. (2-tailed) N Correlation Coefficient Sig. (2-tailed) N Correlation Coefficient Sig. (2-tailed) N Correlation Coefficient Sig. (2-tailed) N Correlation Coefficient Sig. (2-tailed) N Correlation Coefficient Sig. (2-tailed) N Correlation Coefficient Sig. (2-tailed) N Correlation Coefficient Sig. (2-tailed) N

perspective

nature_context

nature_impact

classify

1.000

-.015

-.081

-.032

. 112

.858 111

.331 112

.714 112

-.015

1.000

.030

-.025

.858 111

. 111

.717 111

.773 111

-.081

.030

1.000

.045

.331 112

.717 111

. 112

.601 112

-.032

-.025

.045

1.000

.714 112

.773 111

.601 112

. 112

1.000

-.031

-.094

-.035

. 112

.746 111

.322 112

.712 112

-.031

1.000

.038

-.027

.746 111

. 111

.691 111

.777 111

-.094

.038

1.000

.050

.322 112

.691 111

. 112

.603 112

-.035

-.027

.050

1.000

.712 112

.777 111

.603 112

. 112