1Department of Economics ²Department of Economics

Campus Box 2276 Mary Washington College

Elon College, NC 27244 Fredericksburg, VA 22401

deloach@elon.edu sgreenla@mwc.edu

The central question of this research is whether the use of electronic discussion groups is potentially a good pedagogical tool for helping to develop critical thinking skills in upper-level economics courses. The first problem to overcome in our attempt to measure this activity is that the critical-thinking literature assumes a monologue. Electronic discussions, however, clearly constitute a dialogue. Borrowing heavily on this literature (Perry's (1970), Nelson (1989), McDaniel and Lawrence (1990), Thoma (1993) and Peterson and Bean (1998)) we construct our own taxonomy of critical thinking skills relevant to upper-level economics courses. Though reliable analysis is still forthcoming, preliminary results suggest that these dialogues may contribute to the development of higher levels of critical thinking.

One of the products of an undergraduate education, especially at liberal arts institutions, is the ability to think critically. This should be particularly true for economics students who, after all, are taught our discipline's version of critical thinking--the economic way of thinking. In practice, however, critical thinking does not come easily. As Peterson and Bean (1998) observe, novice/expert theory says that "confronted with a welter of unfamiliar information, novices, unlike experts, cannot separate meaningful from non-meaningful data because they do not know what questions or issues organize a discipline."

A number of authors have developed taxonomies of critical thinking. The most prominent of these is Perry's (1970) model of intellectual development during the college years. More recently, Nelson (1989) has provided a simpler variation of Perry, shown in Table 1.

Source: Thoma (1993)

Nelson identifies three important, but difficult transitions in critical thinking skills, two of which we think are relevant for upper-division courses in economics. The first transition, which we assume students in 300-level courses have already passed is, from Dualism to Multiplicity. At the Multiplicity stage, they believe that there are "no absolutes". They are able to recognize different opinions or theories as legitimate. They are able to compare and contrast different arguments, but when they are asked to make a choice between alternatives, they either cite simplistic reasons or simply ignore one alternative altogether.

Thus, the two transitions that may take place during a 300-400 level college course are (1) from multiplicity to contextual relativism, where students learn to discriminate between good points of view and bad ones, and (2) from contextual relativism to contextually appropriate decisions, where, using disciplinary appropriate normative judgements, the ability to select a best alternative is developed. We think getting the students into Mode 3 is an appropriate goal for most upper-level economics courses. Thoma (1993) says that in this mode students learn at least the mechanics of how different disciplines use various criteria as critical standards to make choices among competing views and theories. We also think that attempting to guide students to Mode 4 is a reasonable, though ambitious goal for senior-level seminars. But how can instructors accomplish these goals?

We believe that electronic discussion provides an effective medium for teaching critical thinking in undergraduate economic students. To date, however, most studies aimed at assessing the effectiveness of electronic discussion have been descriptive and anecdotal. Building on critical-thinking literature, we have developed a new methodology to quantify the degree to which electronic discussion enhances critical thinking in economics. This paper presents some preliminary results of that analysis.

In recent years, increasing numbers of faculty have experimented with electronic discussion in their teaching, using electronic mail, list-servers, web-based discussion lists, and groupware products. There are a number of benefits to thoughtfully prepared electronic discussions. They can extend the learning environment outside of the classroom, structuring the work students do between class meetings. They change the focus of the learning from instructor to students. They replace the single view of the instructor with a diversity of views from the different students. Often students take the views of their peers more seriously than those of the instructor. Unlike an in-class discussion, every one has the opportunity to be heard. In fact, a common finding of studies is that students who never previously distinguished themselves, take an active role in electronic discussion. In sum, electronic discussion can capture the best of both oral discussion (i.e. the dynamics by which the whole is greater than the sum of the parts) and writing assignments (i.e. where students have the time to reflect and research before responding).

Electronic discussion can also serve as a catalyst for accelerating development of critical thinking skills. It also provides a relatively easy means of observing and measuring it. Two students, working independently, come up with a different answer to the assigned problem. Assessing their work on the Nelson/Perry schema, we conclude that they are each at Level 1, Dualism. When the two students post their different views electronically, the discussion has progressed to Level 2, Multiplicity. And all participants see this. Unlike a single author who must discover and recognize the different views on his own, the electronic discussion provides a structure by which that happens almost automatically.

The discussion provides a natural structure for teaching critical thinking to a group, better than more traditional writing assignments, and better than oral discussions. The visible nature of the argument contributes to this in several ways. The electronic format provides a more lasting record than an oral discussion. This, in conjunction with the asynchronous nature of the discussion, allows participants the time to reflect on what others have said and how they wish to respond. Students have an incentive to put more care into their comments, since the discussion will leave a record of them. It is relatively easier for the instructor to point out the difference between a weaker posting and a stronger one. Students tend to be resistant to instructor criticism of their writing (i.e. in traditional writing assignments). But they seem less able to deny the shortcomings of their own work when they see a superior argument by their peers.

Another way to understand this is to note the positive spillovers inherent in electronic discussions. Typically, an individual student becomes a more complex thinker only after many written assignments, where the instructor comments on their need for coherent argument and supporting evidence. But in an electronic discussion, even a student who enters later in the discussion sees the "standard" which he/she must meet in order to contribute.

It is hard to overestimate the learning effects of students seeing their colleagues' postings. While it is true that like the internet more generally, there can be a lot of chaff in electronic discussion, the jewels truly stand out, and students learn from the powerful modeling effects of their peers.

It is a long way from Perry's quite abstract schema to being able to rate student postings on an electronic discussion of macroeconomics. There are at least two major hurdles to overcome: first, modifying the existing schemas to work in the context of electronic discussions of macroeconomics, and second, dealing with the very significant difficulties involved in deriving reliable and valid measures of critical thinking. Peterson & Bean (1998) observe, "Assessing the effectiveness of critical thinking programs is difficult, especially in disciplinary courses where goals for critical thinking and writing are blended with specific content goals. Similarly, Thoma (1993) notes, "Clearly research faces formidable problems of measurement and interpretation because the results involve qualitative changes in thinking and complex interactions between thinking and value formation."

Synthesizing the critical thinking literature, we have developed a scale for measuring the levels of intellectual activity in an electronic discussion. Our scheme borrows heavily from Nelson's interpretation of Perry (N/P), discussed above, and McDaniel and Lawrence (1990) (M&L). Additionally, to a large extent the organizing framework has been determined by the field of study - in this case, macroeconomics.

Our taxonomy focuses on evaluating how students argue. Students can take a position or make an assertion with differing degrees of evidence, including none; logical argument, though lacking in depth supported by simple evidence; an appeal to economic theory; empirical evidence; and finally, bringing in value judgements based on appropriate disciplinary criteria, such as equity and efficiency. The taxonomy can be seen in summary form in Table 2 of the appendix.

In the rest of this section, we define the six levels, and explain them with examples of student postings, complete with misspellings and bad grammar, from a discussion held during the Fall of 1998. The topic of the discussion was "What was Reaganomics? What is supply-side economics all about?" The students were instructed to begin by developing a consensus to address the first two questions given by the professor, and then to argue whether or not it worked.

LEVEL 1 - Unilateral Descriptions

This is analogous to M&L's Unilateral Descriptions, or N/P's Dualism. The posting defines terms, or repeats information already posted; for example, it may paraphrase the question, posing as an answer. It makes simple "good" or "bad" statements.

These postings are analogous to M&L's Simplistic Alternatives. They offer simple explanations, citing simple rules, or "laws" as proof. They do not address conflicts with opposing views or do not explore them. For example, they may present a definition from the text or lecture notes; often times the language is recognizably not theirs. The fact that it is from the text or notes renders it authoritative in their view. As such (i.e. perceived as factual), they usually don't identify the source.

Examples:

This is analogous to M&L's Emergent Complexity, or N/P's Multiplicity. The posting may appeal to a recognized (appropriate) authority. It may include casual empiricism. It often includes one or more reasons, but as a listing, rather than as a logical argument building to a conclusion. Most importantly, no clear choice is made between alternatives. When pressed for the "best" explanation, students at this level respond that all are equally valid.

Example:

This is comparable to M&L's Broad Interpretations, or N/P's Contextual Relativism. The postings include logical statements based on the discipline's accepted model/school(s) of thought. They may identify assumptions in previous postings.

Example:

Example:

Note: Earlier in the semester, the students were given a partial Excel database which they later used in problem sets and papers. In this case, the student had real "national savings", which she converted into the national savings rate to make her argument.

This is analogous to N/P's Contextually Appropriate Decisions. They may argue that while there is (positive) evidence to validate the use of a particular policy, there are other (normative) consequences that must be considered. They may select a particular policy on some normative basis, when choosing among alternatives, each of which has positive evidence to support it.

Unfortunately, we have not identified any examples in the discussions that we have evaluated, yet.

During the spring 1998 semester, we ran our first electronic discussion including both sections of students - what we refer to as a regional discussion. During the following fall semester, each section participated in one "local" discussion during the first half of the semester and a "regional" one that concluded just before the final exam period. The purpose of the local discussions is to acquaint the students with the concept and mechanics of electronic discussions.

The key element in our analysis is the rating of each posting using our taxonomy of critical thinking. Since the electronic comments are often quite brief, it was decided that each posting would be scored as "the highest level in our taxonomy that had been demonstrated." This methodology borrows from the literature on holistic evaluation of writing.

Our hypothesis is that, if the electronic discussion is an effective tool for developing critical thinking skills, successive postings should, on average, attain higher levels of analysis. In other words, we expect to find increasing proportions of the higher-level postings as the discussion progresses. Since each posting is scored numerically, formal hypothesis testing is possible.

A time series of the "Reaganomics" discussion is shown in Figure 1. At first glance, it does not appear that the preliminary results from this discussion are consistent with the hypothesis.

Overall, progress was made over the entire course of the discussion. The highest-scoring postings came later on in the discussion. However, the students became sidetracked for a substantial period of time. For example, two entire threads of discussion did not pertain even remotely to the topic (postings 29-41 and 43-51 all received a score of "0").

Even so, there seems to be some reason to remain optimistic about our hypothesis. It did not take long before they moved from making simple postings with "unilateral descriptions" to making well-constructed arguments based on appropriate theoretical inference. Eventually, they did in fact attempt to use empirical data to evaluate the claims that they deemed to be central to the supply-side argument (as put into practice by the Reagan administration). If we look at the series of scores taken from the postings that pertained to the subject at hand (Figure 2), there is, in fact, a significant upward trend in scores.

The question, then, is how to treat the "noise" caused by threads that are entirely "off-track." In the instructions, the students were told to address the questions of what Reaganomics was and whether or not it worked. However, they were given freedom to explore any relevant issue associated with this topic. A student engaged in a topical conversation may, from time to time, make completely irrelevant remarks. Since we are attempting to score critical thinking, we should document those stochastic errors of temperament, too. Of more concern to our methodology, however, are the entire threads that are, arguably, intentionally off the subject.

The first thread of these "off-subject" postings was precipitated by a message titled "Krugman," which began with the statement that "so far Krugman's analysis of economics of the US has been a bit confusing." The student went on to argue that "someone (i.e., leading economists) should step up and explain what in the hell is going on today with the existing statistics and what ideas could lead to policies that would promote growth and prosperity." The second round of "off-subject" postings was entitled "Global Economy" and lamented the recent problems that were developing overseas.

In evaluating students' levels of critical thinking, it is not obvious how to properly handle this sort of message. Is it really the result of "poor judgement" in attempting to bring in all the relevant factors pertinent to the topic at hand? Or, it is just a function of the medium that appeals to the students' impulses to discuss whatever they feel like?

5. Summary and Directions for Future Research

In this paper we have attempted to lay the foundation for future research into assessing the effectiveness of electronic discussions as a tool for developing critical thinking skills in economics courses. Building on the work of Perry (1970), Nelson (1989), McDaniel and Lawrence (1990) and Petersen and Bean (1998), we have constructed a framework for evaluating skill levels that appears to be appropriate to our task. Though only preliminary results are available at this time, we are optimistic about the use of this pedagogical tool.

In order to analyze the students' postings, there are two key issues that must be properly addressed, validity and reliability. While we have argued the validity of our scheme in this paper, reliability has yet to be addressed. To do this, it will be necessary to have each posting scored by at least two reviewers. Each will score the postings independently. As long as the ratings are within one level, the scores can be averaged. However, if they are further apart, the posting should be evaluated by a third party or discussed together to reach a consensus. After the scoring is completed, it will be possible to estimate via regression the extent to which the "learning-curve" exists. In addition, a longitudinal regression would allow us to account for individual effects. Finally, in the future we would like to use this approach with a richer medium, like groupware.

Greenlaw, S. A. 1995. "Using e-mail, gopher, and other internet tools to enhance your teaching," Telecommunications in Education News 6 (Fall/Winter) 10-11.

Greenlaw, S.A. 1999. Using Groupware to Enhance Teaching and Learning in Undergraduate Economics," Journal of Economic Education, Vol. 30, No. 1, Winter, pp. 33-42.

Manning, L. 1996. "Economics and the internet: Electronic mail in the classroom," Journal of Economic Education 27 (Summer) 201-204.

McDaniel, E. and C. Lawrence. 1990. Levels of Cognitive Complexity: An Approach to the Measurement of Thinking. New York: Springer-Verlag.

Nelson, C. 1989. Skewered on the Eunuch's Horn: The Illusion of Tragic Tradeoff Between Content and Critical Thinking in the Teaching of Science," in Enhancing Critical Thinking in the Sciences, ed. L. Crow. Washington, D.C.: Society of College Science Teachers.

Perry, W. 1970. Forms of Intellectual and Ethical Development in the College Years: A Scheme. New York: Holt, Rinehart & Winston.

Peterson, D. and J. C. Bean, 1998. "Using a Conceptual Matrix to Organize a Course in the History of Economic Thought," Journal of Economic Education, Summer, Vol.29, No. 3, pp. 262-273.

Thoma, G.A. 1993. The Perry Framework and Tactics for Teaching Critical Thinking in Economics. Journal of Economic Education, Spring, Vol. 24, No. 2, pp.128-136.

White, E.M. 1999. Teaching and Assessing Writing, San Francisco: Jossey-Bass.

• They paraphrase information, they repeat and restate the question
• M&L's Unilateral Descriptions, N/P's Dualism
• Defining terms
• Simply repeat information
• Simple "good" or "bad" statements
• Adds little or nothing new to the issue or question

• they take a side, they do not explore other alternatives, they make unsupported assertions, they make simplistic arguments
• M&L's Simplistic Alternatives
• Simple explanations
• Cites simple rules, "laws" as proof
• Do not address conflicts with opposing views or does not explore them

• There is a serious attempt made to analyze competing arguments and evaluate them with evidence
• M&L's Emergent Complexity, N/P's Multiplicity
• Appeal to a recognized (appropriate) authority
• Casual observation
• Often lists numerous factors as evidence, but does not integrate them within a logical framework
• No clear choice is made
• When pressed for the "best" explanation, they respond that both are equally valid

• They employ economic theory to make a cohesive argument
• M&L level 4 , N/P's Contextual Relativism
• Logical statements based on the discipline's accepted model/school(s) of thought
• Identify assumptions

• They add to the level of sophistication by introducing empirical evidence to strengthen their theoretical argument
• Looking to appropriate, historical data to "test" the validity of an argument

• They are able to move beyond objective analysis to incorporate subjective interests
• N/P's Contextually Appropriate Decisions
• They may argue that while there is (positive) evidence to validate the use of a particular policy, there are other (normative) consequences that must be considered
• They may select a particular policy on some normative basis, when choosing among alternatives, each of which has positive evidence to support it.