International Education Studies www.ccsenet.org/ies

42

The Role for an Evaluator: A Fundamental Issue for Evaluation

of Education and Social Programs Heng Luo

Department of Instructional Design, Development and Evaluation, Syracuse University 330 Huntington Hall, Syracuse University, Syracuse, NY, 13244, USA

E-mail: [email protected] Abstract This paper discusses one of the fundamental issues in education and social program evaluation: the proper role for an evaluator. Based on respective and comparative analysis of five theorists’ positions on this fundamental issue, this paper reveals how different perspectives on other fundamental issues in evaluation such as value, methods, use and purposes can result in different roles for evaluators, and how such difference can affect evaluators’ responsibilities in different stages of an evaluation. Then the paper proposes its own resolution of the issue of evaluator’s role and discusses its implication and limitations. Keywords: Role for an evaluator, Program evaluation, Fundamental issue 1. Introduction Fundamental issues in evaluation include the purpose of evaluation, the nature of evaluation, the best methods, strategies and tools of conducting evaluation, the practical concerns such as politics, clients and resources and their influence on evaluation, as well as the roles, ethics and responsibilities of evaluators. Fundamental issues are defined as “those underlying concerns, problems, or choices that continually resurface in different guises throughout our evaluation work.” (Smith, 2008, p.2) Although those fundamental issues will resurface periodically in the field of evaluation in new forms and cannot really be solved once and for all; the awareness of the recurring nature of such fundamental issues can help one view the current problems in evaluation from a better historical perspective. By identifying and examining such fundamental issues, evaluators can have a deeper understanding of their importance, constraints and alternative solutions, thus propose a more effective, yet still impermanent resolution for the existing problems. 1.1 Evaluator’s Role as a Fundamental Issue The fundamental issue discussed in this paper is the role for an evaluator. Over the years, many evaluation theorists have proposed different roles for evaluators. For example, Scriven sees evaluator as a “judge” who justifies the value of an evaluand and offers his summative judgment in the final report; while Stake believes an evaluator should be a “program facilitator” who works with different stakeholders and assists them to “discover ideas, answers, and solutions within their own mind”. Campbell prefers a “methodologist” role for an evaluator, advocating rigorous experiment design that yields strong causal inferences; but Wholey believes an evaluator should be an “educator”, whose role is to infuse useful information to the potential users of the evaluation. The emphasis resides not only in the immediate outcome of a program, but also in the inputs, implementation and long-term outcome of the program. However, terms such as “judge”, “methodologist” and “educator” are just metaphors to facilitate understanding and cannot always accurately describe the role an evaluator plays during an evaluation. In fact, evaluators often play different roles in different phases of an evaluation. For example, an evaluator can be a judge during the phase of selecting criteria of merit, a methodologist when collecting data, a program facilitator during the program implementation, and an educator during the results dissemination. The roles an evaluator takes during an evaluation reflect his or her beliefs in other fundamental issues such as theories, values, methods, practice and use, etc. Other issues such as education background, previous working experiences, the nature and setting of social programs might also contribute to shaping the proper roles for an evaluator. 1.2 Importance of Evaluator’s Role as a Fundamental Issue Just like any other fundamental issue in evaluation, there is no final resolution to defining the proper role for an evaluator. However, studying the different roles an evaluator can play, as proposed by different theorists, is still quite important for evaluators, clients and evaluation as a profession. Turning first to evaluators, studying the different roles for an evaluator is actually studying the different approaches of conducting evaluation. An evaluator’s role is not self-claimed; rather it is defined by the things an

International Education Studies Vol. 3, No. 2; May 2010

43

evaluator does during the evaluation. For instance, we wouldn’t use the metaphor “judge” to describe the evaluator’s role in Scriven’s theory if evaluators’ job doesn’t include determining the criteria of merits, setting comparative standards and giving a final summative judgment. Or we wouldn’t compare the role of evaluator in Weiss’s theory to an educator if providing “enlightenment” (Note 1) is not a primary task for her evaluators. The familiarity with different roles an evaluator can play allows one to take a more flexible approach to conduct evaluation according to specific context, the nature of social program, available resources, and different client expectations. As for the clients of an evaluation, the awareness of the different roles of an evaluator can play will help them select the right candidate according to their specific needs; and reach an agreement with the selected evaluator about his/her job responsibilities as well as the obligations clients shall make in order to facilitate the evaluation process. For example, for a program that does not welcome intrusion, an evaluator who prefers doing an experiment might not be the best candidate. For an evaluator who prefers the role as a “program facilitator”, program administrators should anticipate frequent meetings with the evaluator and make incremental changes according to his/her feedback. The basic knowledge about evaluator’s roles is especially important in today’s world of globalization, where evaluation in a different nation or culture becomes more common. Clarifying the role an evaluator should play beforehand is a good way to avoid surprise, misunderstanding, and conflict later on. Finally, evaluation as a profession will benefit from a deeper understanding of the roles of evaluators. How is an evaluator different from a social scientist? Can a methodologist be hired to do the job of an evaluator? What is the difference between evaluation and research? What are the competencies that are unique for an evaluator? Those questions are raised due to the lack of distinction between evaluation and other social science professions. Explicating the proper roles for a professional evaluator will be a good approach to address such distinction and solidify the status of evaluation as a profession. 2. Theorists’ Positions on the Fundamental Issue of Evaluator’s Role Many theorists in the field of evaluation have different opinions regarding the proper roles for an evaluator. Their opinions on this issue reflect their overall philosophy about doing evaluation as well as their stances on other fundamental issues in evaluation. This section will first discuss the resolutions proposed by different theorists regarding the role of evaluator, analyzing the strength and weakness of each resolution. Then a comparative analysis will be conducted to study the positions across those theorists. 2.1 Scriven Scriven believes that an evaluator’s role is to investigate and justify the value of an evaluand. Such investigation and justification shall be supported with joining empirical facts and probative reasoning. “Bad is bad and good is good and it is the job of evaluators to decide which is which” (Scriven, 1986, p.19). He rejects the notion that an evaluator’s role is simply to provide information to decision-makers and claims that “the arguments for keeping science value free are in general extremely bad” (Scriven, 1969, p.36). According to Scriven, an evaluator’s responsibilities during an evaluation include:

• Determining criteria of merit from needs assessment. Criteria of merit of an evaluand should be its capacity to meet needs. Although an evaluator can use the results of needs assessment conducted by a program developer, sometimes he/she should do an independent needs-analysis. To avoid bias, Scriven advises evaluators to conduct “goal-free” evaluation and formulate questions by ignoring the program goals and looking for all possible effects an evaluand could have.

• Setting comparative evaluation standards. A set of standards should be created by evaluators to assess the program performance. Such standards are used for comparison, either comparison with a set level of performance, or with alternative programs. The latter comparison is preferred by Scriven since he believes that an evaluator will usually make decisions about choosing among alternatives.

• Assess program performance. An evaluator will need to answer both the evaluative and non-evaluative questions. Evaluative questions focus on the effects of the program and should be given top priority. The evaluator should acquire the skills to collect and analyze both experimental and non-experimental data.

• Offering a final evaluative judgment. An evaluator should synthesize his findings into a final report and offer his/her summative judgment. Strength and Weakness of Scriven’s Position: Scriven differentiates evaluators from researchers or social scientists by emphasizing that the value judgment is an integral part of an evaluator’s role and grounds such role

International Education Studies www.ccsenet.org/ies

44

in the logic of evaluation. His “goal-free” evaluation allows evaluators to identify possible side effects of the evaluand and address the concerns of underrepresented stakeholders. However, besides giving evaluators higher authority over different stakeholders in value judgment, Scriven fails to provide a solution to eliminate personal biases of evaluators. Metaevaluation proposed by Scriven is a good attempt but still it is highly subjective and requires years of experiences and expertise for an evaluator to make a non-biased judgment. For the novice evaluator, the decision of whose needs should be considered and which merit should take higher priority can still be very arbitrary. Besides, a complete goal-free evaluation is also highly unfeasible when an evaluator is hired by his/her clients and has an obligation to answer their specific inquiries. 2.2 Campbell Campbell believes that evaluators should play a role of methodologist during the program evaluation (Shadish, 1991, p.141). Evaluators should use scientific methodologies to design evaluative research that eliminate biases and establish a causal inference about a program and its hypothesized effects. This role of methodologist advocated by Campbell requires evaluators to employ a strong research design such as randomized experiment or good quasi-experiment to determine the causal effectiveness of the program. (Shadish, 1991, p.129) An evaluator should also distance him/herself from the program stakeholders and work independently to find out the facts about the program. As for the dissemination of the evaluation findings, an evaluator should “write honest reports for peers even if they cannot do so for funders or the public.” (Shadish, 1991, p.162) Last but not least, it is also the obligation of evaluators to play an active role in scrutinizing, replicating, and debating the evaluation results. Campbell’s emphasis on methods of measuring the program outcome makes him less concerned about assigning values to the program or facilitating the use of evaluation. As a result, he believes an evaluator is not responsible for doing the following:

• An evaluator is not obligated to assign value to the program being evaluated. Valuing of evaluation results should be left to the political process, not researchers. (Shadish, 1991, p.160).

• An evaluator shouldn’t promote use of her evaluation results actively “since this detracts from the credibility of the more factlike findings.” (Shadish, 1991, p.162).

• It is up to the policy maker, stakeholders to decide how to interpret, disseminate and use the evaluation results.

• An evaluator is not obligated to generate a different or modified program worth testing. Her job is simply testing the efficacy of existing programs.

• An evaluator should avoid evaluating institutions, social organizations, or persons due to the almost inevitable corruption pressure. (Campbell, 1984, p.41). Strength and Weakness of Campbell’s Position: The methodologist role Campbell assigns to evaluators is echoed with the proposal for conducting “scientifically based evaluation” as advocated by the Department of Education. The role of a methodologist as defined by Campbell focuses on the internal validity of the causal inference while is less concerned about the prescribing values and utility of the evaluation findings therefore is quite suitable for an external evaluation regarding program outcome. Such role for an evaluator will also greatly enhance the scientific nature of evaluation as a profession. Nevertheless, the weaknesses are also quite obvious for such role of an evaluator. First of all, it is hard to distinguish evaluation from other social science research if one sees an evaluator merely as a research methodologist. Apparently, not every social scientist can do a good evaluation. Secondly, doing a rigorous experiment design is preferable but not always feasible. The cost and time for doing a randomized control experiment, as well as its intrusion into program might result in fewer and fewer evaluation being done due to the reluctance from program administrators. Last but not least, the methodologist role restricts evaluators to study only the outcome of the program while missing other key information such as how the program is implemented, or which element of the program works and doesn’t work. As a result, an evaluator couldn’t give advice about how to improve the program or adapt the program to fit other contexts. 2.3 Stake Stake believes an evaluator should play a facilitator role during the evaluation. The evaluator should assist different stakeholders to “discover ideas, answers, and solutions within their own mind” by conducting

International Education Studies Vol. 3, No. 2; May 2010

45

responsive evaluation. (Stake & Trumbull, 1982, p.1) According to Stake, the responsibilities of an evaluator include:

• Identifying the stakeholders for whom the evaluation will be used: The evaluator should have a good sense of whom he is working for and their concerns. (Stake, 1975, as cited in Shadish 1991, p.273). Minority stakeholder groups should also be included to ensure justice and fairness.

• Spending more time observing the program and providing accurate portrayals of the program using case studies. Because case studies reflect the complexities of the reality, they help readers to form their own opinions and judgments about the case and they can be “useful in theory building”. (Stake, 1978, as cited in Shadish 1991, p.289)

• Conducting responsive evaluation which allows evaluation questions and methods to emerge from observing the program. In this approach, evaluators will orient evaluation directly to program activities than to the program goals and respond promptly to audience information requests.

• Presenting his evaluation findings in the “natural ways in which people assimilate information and arrive at understandings” so that the writings can reach maximal comprehensibility. (Stake, 1980, p.83) Stake doesn’t believe an evaluator should make a summative value judgment since there is “no single true value” for all the stakeholders of a program. (Stake, 1975, as cited in Shadish 1991, p.274) As a result, evaluators shouldn’t blindly accept state and federal standards and impose treatments on local programs since such standards are not pluralistic and might not be in the best interest of local people. (Shadish, 1991, 279) Stake also believes the responsibility of synthesizing and interpreting case studies lies in the readers rather than evaluators and it is up to the readers to resolve any conflicting arguments. (Shadish 1991, p.293) Strength and Weakness of Stake’s Position: the facilitator role for an evaluator, as suggested by Stake, has two major strengths. First, it indicates the interest shift among evaluators from giving a summative judgment, whether it is a value judgment or an effect judgment, to generating useful information that can be used to improve the program. Secondly, it justifies new ways to conduct an evaluation (e.g. responsive evaluation, case study) and report its findings (e.g. narrative portrayal). However, Stake fails to take into account clients’ expectations about the proper role for an evaluator. Will clients accept the case study as the only approach of investigation? Will clients allow evaluators to start evaluations without preordinate questions? Is it appropriate for an evaluator to completely ignore the state or federal standards when evaluating local programs? All those doubts regarding the feasibility and validity of case studies or responsive evaluation will also undermine the social acceptance of the evaluator role proposed by Stake. 2.4 Weiss Weiss emphasizes the evaluator’s special role in promoting the use of his/her evaluation results, especially in the policy-making process. She is frustrated about the fact that “evaluation results have generally not exerted significant influence on program decisions”, and she argues that evaluation should start out with use in mind and evaluators shouldn’t leave the use of evaluation to the natural processes of dissemination and application. (Weiss, 1972, as cited in Shadish 1991, p.182-183) Weiss claims that evaluation “should be continuing education for program managers, planners and policy makers”. (Weiss, 1988, p.18) As a result, it seems that she sees the role of evaluator more as an educator, who conducts evaluation not for giving an explicit solution to a social problem, but for providing useful information to its potential users, policy-makers in particular. She urges evaluators to look beyond the instrumental use of evaluation results and conduct “enlightenment” research that “provides evidence that can be used by men and women of judgment in their efforts to research solutions” (Weiss, 1978, p.76) so as to maximize the utility of evaluation results. By doing evaluation this way, an evaluator should

• Assess the likelihood that evaluation results might be used. (Shadish, 1991, p.198)

• Ask questions that can “provide an intellectual setting of concepts, propositions, orientations, and empirical generalizations” for policy making. (Weiss, 1978, as cited in Shadish 1991, p.202)

• Use well designed qualitative and quantitative methods to conduct evaluation study with emphasis not only on the immediate outcome of a program, but also on the inputs, implementation and long-term outcome of the program. (Shadish, 1991, p.205)

International Education Studies www.ccsenet.org/ies

46

• Draw policy implications from evaluation research by compiling separate summaries to multiple stakeholders with knowledge and information that best interest them. Make recommendations for future programs from the data of evaluation results. (Shadish, 1991, p.205-206) Strength and Weakness of Weiss’ Position: Weiss further differentiates the role of evaluator from the role of a researcher by addressing the complex political context that besets social programs. She warns evaluators against political naivety and urges them to do evaluation that can be used in policy-making, in the form of “enlightenment” rather than “instrumental use”. The educator role she assigns to evaluators reflects her pragmatic view of evaluation and suggests a new mode that evaluation can be used. However, the role of evaluator proposed by Weiss has some intrinsic flaws. First, such role fails to take into account of the variety of different contexts (ironically). For instance, the decision for the state or federal government to hire an educator is often not for the purpose of “being educated”, but to get concrete data regarding the program effect. The proposal to conduct “scientifically based evaluation” made by the Department of Education is a good example of that. As a result, an evaluator who uses case studies to describe the program input, implementation and long-term effect might not by appreciated by policy-makers in this context. Secondly, her emphasis on providing information to policy-makers poses the danger of evaluators becoming the servant of that particular stakeholder group. What should be the role of an evaluator when the interests of different stakeholder groups conflict with each other and speaking for the underrepresented group might limit the use of evaluation results in the policy-making process? 2.5 Rossi Rossi didn’t give an explicit definition about the role of evaluators. Rather, the roles an evaluator shall play might vary according to different stages of evaluation. For example, in the program conceptualization stage, an evaluator sometimes takes the role of a social scientist, incorporating social science theories into the development of an intervention model. (Shadish, 1991, p.389-391) In the stage of program implementation, an evaluator works as a program administrator, making sure the program is implemented as expected so as to “rule out faulty implementation as a culprit in poor program outcome”. (Shadish, 1991, p.381) Besides, the operational data collected this way can also be useful for the future dissemination of the program. When determining the program utility, an evaluator will take the roles of a methodologist and a project manager, who selects and applies appropriate research methods to assess the impact of program intervention as well as conducts efficiency analysis about the program such as cost-benefit and cost-effectiveness analysis. (Rossi & Freeman, 1985, p.327-328) The different types of social programs will also affect the roles an evaluator plays during the evaluation. Rossi has categorized social programs into three types: innovative, established, and fine-tuning programs. For instance, when evaluating innovative programs, much emphasis is given to the conceptualization of the program. (Shadish, 1991, p. 404) An evaluator’s responsibility will include setting program objectives and constructing an impact model between program objectives and activities, which should be based on not only the stakeholder’s views, but also the results of needs assessment and social science theories. However, conceptualization is rarely the focus of evaluation of established programs since their conceptual frameworks already exist and are less likely to change. (Shadish, 1991, p. 404) Instead, an evaluator will take a more summative approach and much of his/her responsibility falls into judging the program accountability. The role of an evaluator is less summative in fine-tuning programs, with emphasis on identifying the needs for change and formative modifications. Rossi’s attempt to integrate the works of various theorists into one theoretical framework also helps to shape his position on the issue of the proper role for an evaluator. Rossi appreciates the strengths of different roles proposed by other theorists and assign such roles to the contexts that best fit them. The “good enough rule” of doing evaluation proposed by Rossi also frees evaluators from the everlasting debate of internal validity vs. external validity, quantitative methods vs. qualitative methods, descriptive value vs. prescriptive value. Instead, it allows evaluators to choose the best possible design through assessing all kinds of “trade-offs”. Strength and Weakness of Rossi’s position: In my opinion, Rossi’s stance on the role of evaluator is closest to the reality, since evaluation by nature is highly context-based. The nature of different social programs, clients’ expectations, evaluators’ backgrounds and expertise, the employment status, available resources, and restraints, as well as the influence of culture and politics can all result in quite different approaches towards doing an evaluation. Even for the same evaluation, activities in different stages will require various competencies from an evaluator. As a result, there shouldn’t be only one proper role for an evaluator and Rossi’s attempt to integrate different roles into one theoretical framework is plausible. However, Rossi is less explicit when linking a specific context to a specific role and linking the specific role to a set of responsibilities that relates to such role.

International Education Studies Vol. 3, No. 2; May 2010

47

2.6 A Comparative Analysis of Different Theorists’ Positions Looking beyond the different metaphors used by theorists to express their opinions on the roles of evaluator, a comparative analysis will be done in this section to dissect such roles into specific behaviors that an evaluator should do in different phases of an evaluation. Those phases are listed in the table below as: program selection, criteria selection, data collecting, evaluation findings, evaluation use and dissemination. (See table 1) As we can see from the table, different roles an evaluator takes can result in quite different approaches of doing evaluation in certain phases while still share a lot of similarity in the other phases. All the citations in the following table are from the book Foundations of Program Evaluation. (Shadish et al, 1991) Insert Table 1 Here 3. Proposed Resolution of the Fundamental Issue The current debate among evaluation theorists regarding the proper roles for an evaluator reflects their different stances on other fundamental issues such as the value of evaluation (descriptive vs. prescriptive), the methods of evaluation (quantitative vs. qualitative), the use of evaluation (instrumental vs. enlightenment), the purpose of evaluation (summative vs. formative). My own resolution on the issue of evaluator’s role will also be based on my understanding of those fundamental issues. Value: an evaluator should prioritize the values from different stakeholder groups when selecting the criteria of merit for evaluands. Not only does the descriptive values approach reflect the concept of a plural democracy, it also orients the evaluation questions towards the concerns of its stakeholders thus makes it more likely that the evaluation finings will be used by them. By taking into account different opinions regarding the values of the program, an evaluator can also reduce the personal bias, which is hard to avoid when the evaluator is the one who assigns values. However, in case different stakeholders can’t reach an agreement on the issue of value, an evaluator should take measures to make his/her judgment about the program value. For example, the evaluator can conduct a needs-assessment to identify the primary stakeholder group who will be affected most by the program and prioritize their values when selecting the criteria of merit for the evaluation. Methods: an evaluator should be familiar with quantitative and qualitative methods and accept them both as available methods for conducting evaluation. However, an evaluator should meet with clients before evaluation and get their opinions on the suggested method. If the clients have a strong opinion to get some quantitative data and have some nice charts in the final report, then the case study is not an ideal method. If the clients want the program to suffer from minimum intrusion from the evaluation, then experiment or quasi-experiment design shouldn’t be considered as first options. However, this is not to say that evaluators should discard the methods they consider best for the evaluation immediately if such methods are not accepted by clients; but it is essential for evaluators to convince their clients of the proposed methods for evaluation and reach an agreement before applying any method. Use: an evaluator should emphasize the instrumental use of his/her evaluation findings and actively promote the dissemination of the evaluation results. It is hard to imagine that an evaluation will be initiated with no intent to know the effect of a social program, especially when such program costs a lot of tax-payers’ money and impacts a large population. In my opinion, ignoring the instrumental use of evaluation is highly irresponsible and has detrimental impact on the profession of evaluation: why should I hire someone to evaluate a program if he/she cannot tell me if the program is working or not? The enlightenment use of evaluation sounds promising but has its limitation in actual practice. First, it is hard to determine the scope of data collecting. With potential users of evaluation findings in obscurity, it is hard to know what specific data will be useful for them. Second, it is hard to enlighten people without telling them what works in the program and what does not. We have to make a judgment about success and failure if we want others to learn from the success or the failure. Last but not least, clients’ expectation and time restraint often make conducting evaluation for enlightenment impossible. It takes time to measure the program input, implementation, outcome, long-term impacts peoples’ attitude, etc. But most clients do not have so much time for an evaluation. Purpose: I prefer a more summative role for an evaluator for the following reasons. First of all, an evaluator is hired for his/her expertise in making rational judgments. If clients want only the vicarious experiences or a set of quantitative data, they can just hire some writers or statisticians to do the job. Second, even for the purpose to make a program better there is a difference between an evaluator and a program manager. At some point, an evaluator will give a summative opinion about what works and what does not in the program. An evaluator can make recommendation about how to make improvement; but ultimately it is up to the program manager to make that decision based on his/her knowledge about the program such as the available resources or people’s commitment. Lastly, being too involved in improving a program might hinder the objective and rational

International Education Studies www.ccsenet.org/ies

48

judgment from evaluators. A lot of personal bias is expected if the evaluator of a program becomes its advocate; since it is hard for an evaluator to make a statement that a program is not working and should be terminated if the evaluator believes in such program and has invested heavily in it. However, by reviewing other theorists’ position on the issue of evaluator’s roles, we can see each different role will have its unique strength and weakness and there is no specific role that fits all the evaluation. Instead of using another metaphor to describe my position, I will define the role of evaluator in the things an evaluator will do during an evaluation, as shown in the following table. (See table 2) Insert Table 2 Here Nevertheless, I understand my resolution for such fundamental issue is not perfect and is vulnerable to criticism in several aspects. First of all, the role I proposed for evaluators might not work in certain contexts; or it is very likely that evaluators who take different roles can still achieve great successes in conducting evaluations. Take Stake, Weiss, Wholey and Cronbach for example, their difference regarding the proper role for an evaluator doesn’t prevent them from presenting strong evaluation cases to support their positions. In other words, it is almost like I propose a role for an evaluator and then caution evaluators to take such role while in the same time indicate other roles might work even better in their contexts. Second, different cultural, racial and ethnic backgrounds of evaluators could also affect their judgments about issues such as clients’ expectations, dynamics between the program administrators and evaluators, the monetary relationship between evaluators, program administrators, federal government and stakeholder groups, as those issues might vary significantly in different background contexts. Since my resolution is based on my judgment of such issues, if my judgment is an incorrect reflection of the reality elsewhere, then my resolution might not fit well in that context. Last but not least, people can also attack the logic behind my approach to define the role for an evaluator. Maybe it is wrong to define the evaluator’s role in terms of what they should do during the evaluation; maybe an evaluator’s role should be associated with the purpose of an evaluation and such role actually defines what an evaluator should do in the evaluation. References Campbell, D. T. (1984). Can we be scientific in applied social science. Evaluation studies review annual, Volume 9. Beverly Hills, CA: Sage Publications. Scriven, M. (1969). An introduction to meta-evaluation. Education Product Report, 2, 36-38. Scriven, M. (1986). New frontiers of evaluation. Evaluation Practice, 7, 7-44 Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage. Shadish, W. R. Jr., Cook, T. D. & Leviton, L. C. (1991). Foundations of program evaluation. Newbury Park, CA: Sage. Smith, N. L. & Brandon, P. R. (Eds.). (2008). Fundamental issues in evaluation. New York, NY: Guilford. Stake, R.E. (1980). Program Evaluation, Particularly Responsive Evaluation. Rethinking Educational Research (pp.72-87). London: Hodder & Stoughton. Stake, R.E. & Trumbull, D. J. (1982). Naturalistic generalizations. Review Journal of Philosophy & Social Science, 7, 1-12. Rossi, P. H., and Freeman, H. E. (1985). Evaluation: A Systematic Approach. (3rd ed.) Beverly Hills, Calif.: Sage, 1985. Weiss, C. H. (1978). Improving the linkage between social research and public policy. In Lynn, L.E. (Ed.). Knowledge and policy: The uncertain connection (pp. 23-81). Washington, DC: National Academy of Sciences. Weiss, C.H. (1988). "If Program Decisions Hinged Only on Information: A Response to Patton." Evaluation Practice 9(3): 15-28. Notes Note 1. Readers gain new perspectives and ideas from reading the evaluation reports, which results in accumulated knowledge and evidence that can be used by them in many ways including defining a problem, suggesting a solution and making a policy. Differing from the instrumental use of evaluation results, enlightenment emphasizes the long term effect on policy-making.

International Education Studies Vol. 3, No. 2; May 2010

49

Table 1. A summary of theorists' positions on the issue of evaluator's role

Scriven Campbell Stake Weiss Rossi Program selection

“everything” (p.84)

Preference for pilot programs (p.136)

Preference for local programs

Programs whose evaluation results are likely to be used

Divide programs into 3 kinds: innovative, established and fine-tuning

Criteria selection

Prescribe value through needs- assessment

Describe values from political process (p.160)

Describe values from local stakeholders (p.307)

Describe values from different stakeholders (p.210)

Prefer criteria to come from stakeholder agreement but unclear how to avoid biases

Data collecting scope

The outcome effects of the program

Outcome consequence of the program (p.161)

13 kinds of data covering the program antecedents, transaction and the outcome (p.282)

Not only the immediate outcome of the program, but also its input, implementation and long-term outcome (p.205)

Collecting data regarding the program implementation, outcome as well as its efficiency

Data collecting method

Not specific Randomized-experiments or strong quasi-experiments

Qualitative methods, case studies

Both quantitative and qualitative methods (p.204-205)

Accept both quantitative and qualitative methods with a preference towards the former.

Evaluation findings

One final evaluative judgment regarding the program effects and “need”

Reports about the causal inference of program effect

Accurate portrayal of the program, diverse formats (p.282-283)

Separate summaries for different stakeholder groups, with knowledge and information that best interest them.

Include not only the program effects, but also its policy relevancy such as cost-efficiency, alternative options, etc. (p.409)

Use of evaluation results

Instrumental use of the results to choose the better evaluand or improve one. (p.109)

Instrumental use of the results to solve a social problem.

Provide vicarious experiences of the program

Enlightenment use of the results to accumulate knowledge and shape policy-making

Instrumental, conceptual and persuasive use of the results. (p.410-411)

Dissemination of the evaluation results

Not mentioned Dissemination is not the concern of an evaluator (p.162)

Not mentioned Actively facilitate the dissemination of results (p.207)

“ a definite responsibility of evaluation researchers” (p.410)

International Education Studies www.ccsenet.org/ies

50

Table 2. A summary of my position on the issue of evaluator's role

Phase of Evaluation Evaluator’s responsibility Program selection

• Determine the nature of the program

• Analyze the political influences that affect the program

Criteria selection

• Use values from different stakeholder groups

• Conduct needs-assessment to prioritize the values of different stakeholder groups

Data collecting scope

• Collect data that can answer stakeholders' inquiries and concerns

• Emphasis on the program outcome and look into program input and implementation

Data collecting methods

• Get to know clients' expectation and available resources

• Be familiar with both quantitative and qualitative methods

Evaluation findings

• Provide answers to the inquiry of different stakeholder groups

• Offer evaluative judgment regarding the program effect

Use of evaluation results

• Use the results as evidence to make decision regarding the program (promote, modify or terminate the program)

• Increase people’s knowledge of certain phenomenon

Dissemination of evaluation results

• Provide separate summaries of evaluation results to different stakeholder groups with information about their interest and concern