Assessing teachers' pedagogical content knowledge - Springer Link

0 downloads 0 Views 371KB Size Report
Next is a discussion of three issues that arose when attempting to develop a teacher licensure test that includes assessment of pedagogical content knowledge.
Journal of Personnel Evaluation in Education 4:157-173, 1990 (~) 1990 Kluwer Academic Publishers, Manufactured in the United States of America

Assessing Teachers' Pedagogical Content Knowledge: Item Development Issues ROBERT E. C A R L S O N California Commission on Teacher Credentialing, 1812 9th Street, Sacramento, CA 95814-7000

The work of Lee Shulman and his colleagues in the Teacher Assessment Program, supported by the Carnegie Corporation, has led to new conceptions of teaching and teacher assessment (Shulman, 1986, 1987a, 1987b). A key feature of these new conceptions is the proposition that pedagogical content knowledge is an important category of the knowledge base for teaching (Shulman, 1987a). This article describes some issues related to the development of test items to assess pedagogical content knowledge for an elementary teacher licensure test, the Connecticut Elementary Education Certification Examination. Following a brief description of the exam, pedagogical content knowledge is defined. Next is a discussion of three issues that arose when attempting to develop a teacher licensure test that includes assessment of pedagogical content knowledge. Finally, a number of item types that seem to be useful for assessing pedagogical content knowledge are described briefly.

The Connecticut Elementary Education Certification Examination

The Connecticut Elementary Education Certification Examination is one part of a three-tier teacher assessment system under development in Connecticut. During prospective teachers' undergraduate programs, they must pass a basic skills test in reading, writing, and mathematics. The Elementary Education Certification Examination is administered to individuals who have passed the basic skills test and successfully completed an undergraduate degree and a teacher education program. Those who pass receive an initial teaching certificate and can begin teaching. During their first year of teaching, they are further evaluated in a variety of ways, including observation and performance assessments. Those who pass this third tier of assessment are given a provisional teaching certificate. The Elementary Education Certification Examination assesses a prospective teacher's content knowledge in nine areas of the elementary curriculum as well as A t the time the work described in this paper was performed, the author was a Senior Associate at I O X Assessment Associates.

158

R.E. CARLSON

general pedagogical knowledge, which includes topics such as human development, classroom management, and motivation. In addition, pedagogical content knowledge is assessed across content areas. As currently planned, the test will consist of paper-and-pencil, multiple-choice test items as well as items for which videotaped classroom sequences serve as the stimulus. This article focuses on the development of the paper-and-pencil, multiple-choice items designed to assess pedagogical content knowledge. These items were developed by my colleagues and me at IOX Assessment Associates.

Pedagogical content k n o w l e d g e

Shulman (1987a) included pedagogical content knowledge as one category in a hypothetical list of categories of the knowledge base for teaching. He defined it as • . . that special amalgam of content and pedagogy that is uniquely the province of teachers, their own special form of professional understanding . . . . It represents the blending of content and pedagogy into an understanding of how particular topics, problems, or issues are organized, represented, and adapted to the diverse interests and abilities of learners, and presented for instruction. Pedagogical content knowledge is the category most likely to distinguish the understanding of the content specialist from that of the pedagogue (Shulman, 1987a, p. 8).

Elsewhere, Shulman has referred to this type of knowledge as "the applications of pedagogy to specific subject areas" (1987b, p. 41) and as "the intersection of content and pedagogy" (1987a, p. 15). More recently, Haertel (1990) describes pedagogical content knowledge as " a kind of craft knowledge that goes beyond command of subject matter or general pedagogical principles to an understanding of how to impart a particular subject matter to learners" (p. 279). In developing multiple-choice test items to assess pedagogical content knowledge, we had to deal with a number of issues, three of which are discussed in this article. First, the fact that the Connecticut test will be a licensure test, as opposed to a certification test, affected both the selection of objectives to be tested and item content. Second, we had to decide how to blend pedagogical and content knowledge in test items so that we would be assessing an examinee's pedagogical content knowledge rather than simply testing pedagogical knowledge and content knowledge separately. Third, we had to be very concerned about the defensibility of our items because the test will be a high-stakes licensure test. Given that the identification of pedagogical content knowledge is only just beginning, the challenge of creating defensible test items was greater than usual. Each of these three issues is discussed below.

ASSESSING PEDAGOGICAL CONTENT KNOWLEDGE

159

Licensure versus certification A key issue that influenced the development of the Connecticut test was that the test will be used as a licensure test, as opposed to a certification test. As Mehrens (1987) has pointed out, both the legal and psychological professions distinguish between these two test types. A licensure test is administered under the aegis o f a governmental agency. Licensing is a mandatory program designed to protect the public health, safety, and welfare by ascertaining that those who are licensed have at least a minimal degree of competency. A certification test, on the other hand, is sponsored by a nongovernmental association. Certification is usually voluntary, and it grants recognition to those individuals who have met certain predetermined qualifications specified by the sponsoring association. Certification typically entails more than minimum requirements. This distinction has an important implication for test development. In spite of its name, Connecticut's Elementary Education Certification Examination will be a licensure test. It will be one part of a licensure process used to award or deny individuals the opportunity to teach in Connecticut. Therefore, as stipulated in the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1985) and in the Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor, & Department of Justice, 1978), we focused our test development efforts on knowledge and skills that are necessary for satisfactory performance as an elementary teacher. The result was that we were more constrained in both the identification of the knowledge and skills to be tested, as well as in the selection of test-item content, than we would have been had we been creating a certification examination. For example, when Shulman (1986) talks about pedagogical content knowledge, he includes, for the most frequently taught topics, "the most useful forms of representation of those ideas, the most powerful analogies, illustrations, examples, explanations, and demonstrations" (p. 9). Most would agree that such knowledge would be useful, but it is not clear how much of this type of knowledge is necessary for satisfactory job performance, which is the criterion for inclusion on a licensure test.

Blending pedagogical and content knowledge in test items A second issue with which we struggled was how to create test items that actually assess pedagogical content knowledge. We wanted to create items that require something other than a bit of pedagogical knowledge and a bit of content knowledge. Thus, we tried not to develop items in which the correct answer embodied both good pedagogy and correct content knowledge, while the incorrect

160

R.E. CARLSON

answers were instances of (1) bad pedagogy but correct content, (2) good pedagogy but incorrect content, or (3) bad pedagogy and incorrect content. Such items, in our view, do not represent the "blending of content and pedagogy." Each type of knowledge, that is, content knowledge and pedagogical knowledge', could be considered separately when the examinee evaluated each answer choice. Instead, we attempted to create test items that require the application of pedagogical knowledge to specific content areas. For example, it is a general pedagogical principle that concrete experiences should precede abstract experiences. A test item in mathematics that requires examinees to choose the most appropriate sequence in which to present three measurement activities to students requires not only knowledge of the general principle but also mathematical knowledge about measurement. The examinee will have to use the mathematical knowledge to analyze each measurement activity in light of the general pedagogical principle of concrete before abstract. In other words, the examinee will have to have knowledge of mathematics in order to recognize the correct application of the pedagogical principle. This type of item taps pedagogical content knowledge. Such items were not easy to write, however. We found variations across content areas in the difficulty of identifying appropriate applications of pedagogical knowledge to content areas. Such applications had to be clearly correct, and, as discussed above, the knowledge of them had to be regarded as necessary for satisfactory job performance. In the area of reading, it was relatively easy to find suitable applications, probably because reading is a highly pedagogical content area--that is, the field of reading includes much information about the learning and teaching of reading. Mathematics was also a relatively easy subject area in which to create test items assessing pedagogical content knowledge. Perhaps because of the hierarchical nature of much o f mathematics, there are agreed-upon teaching methods and sequences that can be used in test items. In the subject areas of social studies and science, however, it was very difficult to identify justifiable and necessary instances of pedagogy applied to content. There is much less consensus in these fields about the best ways to organize and present content to students.

The defensibility of the test items

A third issue, related to the previous two, that affected test-item development was a concern for the defensibility of the correct answer choices. As in all testdevelopment efforts, but particularly for high-stakes tests like the Connecticut Elementary Education Certification Examination, test items had to be created with correct answers that were virtually unassailable. For test items that assess content knowledge, this is rarely difficult. It was much more of challenge, however, for items assessing pedagogical content knowledge. When writing items, we used two criteria to justify the correctness of the test

ASSESSI NG P E D A G O G I C A L C O N T E N T K N O W L E D G E

161

items: empirical support and professional consensus. If a specific application of pedagogical knowledge to a content area could be justified based on findings from educational or psychological research, or because it reflected a consensus in the field, then we used that application as the basis for one or more test items. We found, however, that much of what we considered to be pedagogical content knowledge failed to meet either one of these criteria. Educators are only just now beginning to explore pedagogical content knowledge. Much of it is not yet an established part of the knowledge base for teaching. As Shulman (1987a) pointed out, the knowledge base for teaching is not fixed and final, and " m u c h , if not most, of the proposed knowledge base remains to be discovered, invented, and refined" (p. 12). Clearly, those elements of pedagogical content knowledge still to be discovered, invented, or refined cannot be used on a teacher licensure examination. Because in teaching there is rarely a single correct way to do something, and because of the heightened need for defensible test items, we almost invariably developed best-answer rather than correct-answer items. For example, m a n y items require the examinee to identify among four choices the most or least appropriate course of action in a specified context. It is much easier to justify that a particular course of action is more appropriate than others in a specified context than it is to defend a particular course of action as being the only correct one. Our use of best-answer items created a challenge for item writers. We had to select incorrect answer choices very carefully. If they were too plausible, then it would become difficult to assert the superiority of the " c o r r e c t " answer choice, undermining the defensibility of the item. On the other hand, if the wrong answer choices were too inappropriate, then the correct answer would be obvious. This is a c o m m o n difficulty in test development, but it was exacerbated in this case by the lack of a well-grounded, comprehensive knowledge base for teaching, especially with regard to pedagogical content knowledge.

Useful item types In spite of these difficulties, we were able to develop test items that met our criteria. That is, we created items that we believe test pedagogical content knowledge that is required for satisfactory job performance as an elementary teacher, and that have nonobvious correct answers that are either supported empirically or by a consensus in the field. We developed a number of item types that seem to be useful for assessing prospective teachers' pedagogical content knowledge. Many of these items are what we've called "materials-based items" in which the examinee is provided with materials that teachers are likely to use on the job, such as worksheets (both with and without student responses), lesson plans, individualized education programs, report cards, excerpts from textbooks, descriptions of learning activities, maps, and

162

R.E. CARLSON

test reports. Incorporating such materials into test items can enhance the jobrelatedness of the test. Some examples of item types that seem to be appropriate for assessing pedagogical content knowledge are listed below. 1. Given a lesson plan, the examinee must identify (a) the best addition to the plan or (b) the most appropriate behavioral objective for the lesson. 2. Given four worksheets or descriptions of student learning activities, the examinee must identify (a) the most appropriate sequence in which to present the worksheets or activities or (b) the worksheet or activity that most closely matches a specified objective. 3. Given a piece of student work, such as a completed worksheet or a writing sample, the examinee must select (a) the most appropriate evaluation of the work or (b) the most appropriate feedback, explanation, or remedial activity that should be provided to the student. 4. Given a description of a student learning activity, the examinee must (a) identify the skill(s) a student would need in order to be able to complete the activity successfully or (b) identify the skill least useful for the activity. 5. Given the purpose of a lesson, the examinee must select the most or least appropriate materials to include in the lesson.

Conclusion Pedagogical content knowledge represents a fertile area for expanding the knowledge base for teaching. Clearly, however, much work still has to be done to delineate that knowledge. We've had a significant challenge in attempting to develop multiple-choice test items that assess an examinee's pedagogical content knowledge in the context of a teacher licensure test. This article has presented some of the issues involved, as well as some item types that appear to be useful. We believe we've had some success in creating items assessing pedagogical content knowledge. We hope that our efforts will contribute to a developing understanding of the nature of pedagogical content knowledge and how to assess it.

References American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1985). Standards f o r educational and psychological testing. Washington, DC: American Psychological Association. Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor, & Department of Justice. (1978, August). Uniform guidelines on employeeselection procedures. Federal Register, 43(166), 38290-38315. Haertel, E.H. (1990). Performance tests, simulations, and other methods. In J. Millman & L. DarlingHammond (Eds.), The new handbook o f teacher evaluation: Assessing elementary and secondary teachers. Newbury Park, CA: Sage.

ASSESSING PEDAGOGICAL CONTENT KNOWLEDGE

163

Mehrens, W.A. (1987). Validity issues in teacher licensure. Journal o f Personnel Evaluation in Education, 1, 195-229. Shulman, L.S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15, 4-14. Shulman, L.S. (1987a). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 54(1), 1-22. Shulman, L.S. (1987b). Assessment for teaching: An initiative for the profession. Phi Delta Kappan, September, 38-44.