Notes on how to evaluate Educ RefNOTES on HOW TO EVALUATE EDUC REF? Hugo Zegarra This article contains my notes on Wiley’s article. SEE: Systems Research and Behavioral Science, May-June 1998 v15 n3 p209(7) Before making a summary of John Wiley article “Evaluating Educational System Design” (1998), I want to call the readers attention to the following arguments: NOTE 1 1. “Educational systems does not repond to the need of the learners”. This is one of the official stories about educational crisis. If you can change the word “learners” for society, parents or communities and you have the rest of similar hypocrit and coward arguments for reforming the school system. It is easy to talk in the name of “learners” because it refer to children and do not have voice. If they say that education do not repond to the “needs” of the society, parents or communities, there will be many voices that will jump to responde or ask questions such as Education for what development?. To export raw materials without adding value to them (mineral and agricultural products), for neoliberal economy we don’t need to make any school reform. All that we need is primary school, that is basic math, reading and communication skills to operate machines under the supervision and control of bilingual foreign managers. To pack saparragus or load trucks with gold, silver and other minerals we don’t need secondary education, this should be transfered to private institucions, they argue. If the State cannot afford to finance primary school, if it is in real bankcrupcy, then primary education should be transferred to local municipalities and since most of them do not have resources either, privatization is the solution. That is the way in which most central States are addressing the issue crisis of the education system. An honest argument about this crisis is to say that States have other obligations or prioritees to attend than providing enough resources to education, but the servants of the Central State are not pay to be honest. A suttle though direct way of addressing the issue would be not only talking about insufficient resources but the miss-management of the few resources given to education. That argument set the basis for the next one, the transfer of financial responsabilities to local communities or private institutions. We have to decentralize the system, that is the common feature and official way of thinking about education reforms nowadays. Does it mean that decentralization will solve the problem of insuficient resources in countries and local communities where more than 50% of their inhabitant suffer accute and chronic poverty?. If privatization of basic education is the hidden agenda for decentralization, how this policy is going to prevent a widen gap of intra-regional inequalities between poor and rich communities?, and what about urban-rual national gaps or differences?. How to explain the fact that a central State that tacitly admit having not power to set up national an regional “equalizing funds” to cop with current growing inequalities, will lather on have such a power?. The reader should keep in mind that -if decentralizacion implies a real transfer of economic and political power to region and municipales- the power gained by decentralized units of the State that will be at expenses of centralism. This means that later on the Central State will have less power to set up such “equalizing funds” to stop the gaps of intra-regional and national inequalities. These are some of the incositences of central state bureucrats regarding education reforms to cop with the current crisis of the educational system. It is paradogical that is the central State bureaucracy who is making the accusation that the school sistem does not respond to the “needs” of the learners, society, parents or local communities. That is that happens in developing countries, such as Andean countries in South America. There, the rulers and top servants of central States, are the ones who make such acusations, accusation that covertly or appenly goes against the teacher’s unions. What we have here is a twist in the “public opinion trial”, the delincuent is the accuser and the victim the accused. To understand this twist we should keep in mind that the State is the institution that control and manages the mega system called society or nation, to which educational systems belong to. So, the acusers are accusing themselves of not having the capacity to control and manage their own subsystems, the educational system. State managers should have been under trial for abandoning responsabilities, they are the delinquents. However, instead of being accused by teachers, communities and parents of the learners for not providing tochools with resources necessary for teachers to perfom their job, those State managers ?the ones who should have been in prison for no doing their job- are the ones who claim the right to accuse and not only that, the one who become re-designers of a new school system. This look like the war in Irak, the big business of destruction is being follow by the big business of reconstruction, different business under the same control, big corrupted corporations. That is the paradogical part of the issue educational reform in developing countries like Peru. And it is much more paradogical that Peru, the 2nd rich country in natural resources in the Andean zone, the one with the fastest rate of economic growth in several quinquenios during 1990-2005 in all Latin America cannot afford to finance its educational development. How to explain this fact?. We will see this later this topic. 2. Educational reforms are not designed to change the whole educational system, but to introduce piecemeal adjustments and mask-type changes to kepp the essence of the old system. That is the nature of most top-down education reforms. 3. If evaluations of educational reforms have to be performed, the evaluators should depart from those facts. The evaluators job is thus to demostrate at teoretical and methodological level the imcompetence of the designers and implementers of real and holist educational change in education. NOTE 2 “How to evaluate those incremental improvements in programs and a variety of “structural” changes called education reform?. Here we have some suggestions: “Evaluation theory development has been strongly influenced by an idealized problem-solving sequence for (a) problem identification, (b) generating and implementing alternatives to reduce symptoms, (c) evaluating the alternatives, and (d) adopting one or more of the most satisfactory (Shadish et al., 1991). Evaluation thinking differed substantially in terms of how evaluation theorists viewed such issues as purposes for evaluations, audiences for evaluation findings, roles of the evaluator, the stakeholders’ role in designing and conducting the evaluation, the extent to which goodness or value can be assigned by an external factor, the nature of knowledge produced by evaluations, and the data collection methods employed by evaluators”. J Wiley’s article [[Hugo’s note 2: PURPOSE OF EVALUATIONS]] BASIC QUESTIONS & ANSWERS REGARDING THIS RESEARCH PROJECT [[The first purpose of my evaluation will be to provide or suggest adjustments and corrections to the process of design and implementation of the designed reform from a systems view perspective. At the end of this study and according to findings I will provide some “efective” solutions to educational problems. The beneficiaries and audience of this study are ?at the beginning- the top policy makers who design the reform and those who check or control its implementation, since I will report to them some findings during the research process. The main beneficiary of this study are the communities and parents, teachers associations and stakeholders directly involved or close to the school system, to whom I will make available to final report. My role as evaluator will be strictly academic (limited to education topics of the proposed research) and neutral in political terms. I will avoid any political judgement that could hard interest groups at the top and the bottom of the spectrum. Regarding the stakeholders role in this research I will avoid their participation in the design and conducting of this evaluation, they have interest of their own that could bias the research process. Even if they are part of the donors financing this research I will make an agreement to limit our relation to information. Regarding the nature of knowledge that this study will produce; those will be the result of careful analysis and synthesis resulting from the methodology suggested by Bela Banathy. I will use his concepts, models or lenses to perform a Systems evaluation of education reforms. I will use also the original insights of theorists whose studies conform the background of Systems Theory. See in Annexes my article “Systems Theory Backgrounds”]] NOTE 3 “The critical omission of the various evaluation models produced is that they are largely silent concerning the evaluation of a system as a system. Instead, the concern is limited to the amelioration of problems possessed by existing systems or of selecting the best program or practice among competing alternatives. In short, they ignore system concepts. There is little acknowledgment that in open systems the fact that everything affects everything else, or that if one thing is altered, other conditions change in response.” Wiley’s article [[Hugo’s note 3: METHODOLOGY]] [[Does this means that “direct causal relationships seldom exist” so this type of analysis is/was irrelevant?. here we have 2 methodological points. 1st the idea that “everything affects everything else; if one thing is altered, other conditions change in response”: i guess there are key parts in a system that could produce such domino effect on other parts; however, not all of them have such impact. this means that when we analyze the structure of a school system we have to distinguish and select those parts for careful analysis if they are to be touched by a reform process. 2nd,“direct causal relationships seldom exist” . See “THE IMPACT OF SYSTEM DESIGN THINKING, 2nd papragraph, below Saying that all parts are interelated doesn’t mean that parts in a system are similar to a sack of potatos. parts in a system are structured either vertical, horizontal or a combination of both of them. so,what one part does could affect other or don’t. we can suppres some parts without affecting the continuity of the system at all, but we cannot supress some part without altering some other parts. in a school some parts of the subsystem teaching-learning ?for instance- are set in a way that a teacher of art does do not affect what the math teacher does, unless it affect time scheduled. in a factory it does, in the assembly-line units of machines, if one unit does not finish its job it affect the whole process. that is not the case of school systems. some parts are well connected, depends on one another, most of them don’t. loose coupling is the dominant feature of educational systems in the north,but direct causal relation do exist, even when multiple correlations are more relevant in some cases. at the end, the discussion of causality either lineal or multiple, is irrelevant in systems analysis. sytems approach doesn’t rely on quantitative methodology]]. “THE IMPACT OF SYSTEM DESIGN THINKING Recently, the thinking and particular formulations that have contributed to traditional evaluation theory and practice are being challenged. For example, and having substantial implications for evaluation of system designs, `naturalistic inquiry' has been proposed as a more appropriate way of thinking about and conducting research (Lincoln and Guba, 1985). The construction of a naturalist paradigm included the following axioms: (a) realities are multiple, constructed, and holistic; (b) the knower and the known are interactive and inseparable; (c) only time- and context-bound hypotheses are possible; (d) all entities are in a state of mutual simultaneous shaping, so that it is impossible to distinguish causes from effects; (e) inquiry is value-bound.” Wiley article Systems theory as naturalistic paradigm: this statement deserves some comments. Especially the 2nd paragraph with methodological insights. See hugo’s note 3, above NOTE 4 [[KEY QUESTION: WHAT IS THE PURPOSE OF THIS EVALUATION?]] SIGNIFICANCE, IMPORTANCE OR RELEVANCE? TO WHOM? “Banathy's review of approaches to `Design evaluation' reveals an increasing number of system thinkers and designers concerned with (a) evaluation as contributing to stakeholder decision-making about alternative design features (Nadler, 1981; Ackoff, 1981); (b) evaluation as helping establish `acceptability zones' for design solutions that must confront multiple perspectives (Jones, 1980); (c) evaluation as `argumentation' [[theory development?]](Rittel and Webber, 1984); (d) evaluation of design solutions through use of system criteria (Checkland and Scholes, 1990); (e) evaluation as a trade-off analysis at key decision points (Warfield, 1990); and (f) evaluation and design as complementary parts of the same process (Rowland, 1994)” [[hugo’ note 4 : more on purposes of this evaluation]] [[summarizing my purpose or objectives for this evaluation: the purpose of this evaluation study is to contribute with policy makers, stake-holders (defined as those that could or have veto power on school reforms), school actors (administrators, teachers and parents) and local communities related to the teaching-learning and developmental process of the learner in the following tasks: (a)provide corrections and adjustments to the ongoing project of education reform, or with alternative design features if the reform project has not yet approved; (b) design solutions to hot problems during the implementation process, previous evaluation of the process. this includes the suggestion to design of complementary (or subtitute) parts to the system’s aim to tackle with hot problems (adress the functional imperatives of a system, as parsons requested. (c) elaborate a report containing the analysis and synthesis of this study and diseminate it among policy makers, stakeholders, school actors and local communities. all of this with the help. (d) contribute with the theoretical development of reform evaluation and organizational theory in those areas that need more debate and reflexion, especially the issue of current community alternatives to finance and implement decentralization policies]]. [[GOOD QUOTE on WHY SYSTEMS THEORY. QUESTION WHICH PAGE IT BELONGS TO? FOLLOW BY WILEY TEXTS “Social system design seeks to understand a problem situation as a system of interconnected, interdependent, and interacting issues; and seeks to create a design as a system of interconnected, interdependent, interacting, and internally consistent solution ideas. System designers envision the entity to be designed as a whole. A systems view suggests that the essential quality of a part of a system resides in its relationships with, and contribution to, the whole” (Banathy, 1992) “Integration, then, is the sine qua non of a system design. “There are many other, equally important, characteristics of the `system design' technology; for example: (a) the importance of multiple perspectives and how these might be integrated; (b) the importance of creating an `idealized image' of the system to be that reflects the core values and needs of the system stakeholders, one that will serve as a target for design and from which, over time, a feasible and implementable operating design is extracted. “A FRAMEWORK FOR EVALUATING SYSTEM DESIGNS The main section of the paper is divided into three parts. The first part presents some general guidelines related to evaluating system designs. The second part describes a framework for thinking about evaluation as it applies to system designs. The final part discusses some limitations and complications that need to be recognized and taken into account by system design evaluators. [[First model]] “GUIDELINES FOR SYSTEM DESIGN EVALUATION “A set of guidelines are proposed below. [[OBJECTIVES]] “System design evaluations can be directed toward 1. (a) determining the appropriateness and worth of existing systems; (b) the appropriateness of new systems being designed but not yet implemented; (c) the implementation process for new systems; and (d) the `over time' appropriateness and worth of new systems. 2. The task of evaluating system designs begins with, is integrated with, and is concurrent with the systems design process itself. This means that the design process for most aspects of system design evaluations are accomplished with full involvement of system stakeholders. The design needs to be responsive to stakeholder interests. [[IN MY VIEW, THE EVALUATOR SHOULD BE INDEPENDENT, IF WANT TO PROVIDE RIGHT SOLUTIONS NOT MATTER WHAT INTERESTS ARE AT STAKE]] NOTE 5 [[hugo’s note 5: more on methodology]] “3. Social systems are in constant flux. Independent variables and, in some cases, dependent variables are vulnerable to change. Thus, capturing, understanding, and making sense of the process of change should have at least equal status to assessing effects”. (Wiley article) [[in systems theory the educ system is 1st a dependent variable and later on an independent one. the school system is 1st affected by the context and lates its outputs ?in 2nd place- affect the environment or society. so education is “in most cases” more vulnerable to changes than society is. the fact that societies can live without formal school systems but schools can not exist out of societal environments, explain the difference force and time of impacts between them. the impact of contextx on schools is stronger -especially the impact of the systemic context- than the impact of schools on society. thus, they do not have the same status, not in reality and not for the purpose of analysis and methodologycal research of reforms. on the other hand, making sense of changes in society and schools should consider that scieties are much more dynamic than schools. societies are in permanent change, schools no. the speed of changes in society cannot and will never be equated by schools. schools try to preserve, serve and reproduce yesterday social reality, are conservative in nature, while societies are revolutionary in nature, they are moving and creating today the reality for tomorrow. school systems will be always outdated, in permanent crisis because is imposible to keep up with the new demands of social change. this means that today school reform will be absolete tomorrow or in the nearest future. the school system that can survive the speed of changes of the society will not be the one without crisis ?that is a mirage- but the one that is able to respond inmediatelly to new possitive changes of that society. that is, the school that have contingency plans for tomorrows reality. to capture tomorrows reality with need to keep track and be in permanent contact with the whole society, with the external process of change in societies. this is why systems analisis became so relevant. there is not other theoretical and methodlogical framewrok that allow us to keep track of both whole systems, the society and the schools, so a quick changes in the school system or its parts can be intriduced without disrupting its cointinuitee. this is why we need to carry with us the three lenses suggested by banathy, the one to see relations between school systems and societal environments, the one to see functions & structures of the school system and the one to see both whole systems in process, the interelation and dynamic of both systems, so that we can say what need to be changed and fast inside the school system and inside the society. in sum, the process of studying education reforms start analyzing the environment, the nature and trends of their changes, that will serve the understanding of its effects on educ. then, we need to know the functional imperatives or problems to be solve in the whole structure of the school system. finally we need to study the dynamic relations of both systems to provide quick answers demanded at both levels, schools and society. we should keep in mind that impact of educ on the social context takes longer than the impact of social changes on schools. they do not have the same force nor does occur at the same time]] NOTE 6: ON THE WHOLE AND IST PARTS OR COMPONENTS 4. When system components are the objects of inquiry, evaluation activities should address the performance of the component itself, the contributions of the component [[GIVES AN anticipated IDEA OF THE performance of the larger system]], and the inevitable interactions that occur between the component and the rest of the system. Components that have no effect on a larger system [[THE WHOLE]] are not really a part of that system. [[TOTALLY AGREED WITH THAT]] 5. Performance indicators for the `system as system' are needed in addition to indicators for component performance. HUGO’s note 6: The need to discriminate between parts of the old system that can be kept & and those that no. I wonder if i can start here the analysis to please and not scary the donors. I mean, start in the relation between the whole-parts instead of in the analysis of the national economic and political inputs and the relation of the State regarding neoliberal policies [[Both performance analysis, parts and the whole, 4-5, the expected performance and the real one, can be captured by using the “function-structure” model suggested by Banathy]] NOTE 7 MORE ON WHERE TO START THE ANALYSIS: solving-problem orientation is the approach that the IDB likes] [[The evaluation of a designed school reform start in the study of the school problems. Usually this information is contained in the in the previous school system. this part is usually contained in the same proposal or designed project. Otherwise, we should design a small project to summarize first, a comparative analysis of the resources allocated by central states to schools in the last decade or quinquenio. The management of those resources by central, regional and local instances of school government should deserve special attention since some top officials said that the problem is not insufficient resources but its bad administration by school units. So special attention should be pay to the type of management at different levels of the system. Another key issue about resources is the portion of finances coming from the national sources, public and private, and those coming from foreign donors. in this last case we should check if they are conditioned as happens in Peru with the money coming from USAID-IDB Besides financing, and to complete the economic part we should see how neoliberal policies are afecting the whole and parts (curriculum and books design) of the educational system. finally we should pay attention to other key resources for schools and for teachers to make their job. Another key question is how the distribution of resources and the management of the system was related to the legitimacy and indicators of governability. In this regard, special attention should be given to the role of teacher’s unions, parents’ association and stake holders (donors among them) in the dynamic of the school system. The 2nd part of this overview is looking at the organizational structure, who control de school and how?. if the structure is decentralized, what were the open and hidden arguments for decentralization?. how was the processes of transferring competencies from central government to municipalities and small units of government?. if there evaluation of the decent process, check how was/is the re-distribution of power (administrative, political & economic). if there is function-duplicity, ask what has been done to correct or adjust the problem? and ask also if there are parts of the system that does not contribute much to the whole dynamic or solve any problem, why is still there?. The 3er part is the transformation of inputs into outputs; the focus will be on the teaching learning system and the participation of parents y the community in such process. finally, we should look at the results in the achievement tests, and other achievements]] NOTE 8: more on where to start or organize the research “FRAMEWORK FOR THINKING ABOUT EVALUATION OF SYSTEM DESIGNS “The most basic and most difficult issue for system evaluators is how to define the system so that it can be evaluated. The framework below describes a set of system areas for creating a definition and toward which the design of an evaluation can be directed, regardless of evaluation purpose. The framework is based primarily on the integration of three lenses for viewing social systems (Banathy, 1992) and a set of features defined as common to all social systems (Churchman, 1971; Checkland, 1981).” ‘The framework, following Banathy, is organized into three systemic areas. “The first guideline given above states that system design evaluations can be directed toward four different targets: the appropriateness and worth of existing systems, the appropriateness of new systems (being designed or not yet implemented), the implementation process, and the continued appropriateness and worth of new systems over time as experience, inquiry, and changing conditions provide new information” (Wiley article) I don’t know if this is Wiley approach or Banathy. Start analyzing the good and the bad of the existing system, is something that is already done in the diagnosis and the reasons alleged by the new proposal for reform. Then he said we should go to the wonders and appropriateness of the new proposal and finally go to the analysis of the implementation process. What I am sure is that Banathy said is the following: [[GUIDELINES: Banathy suggest that the investigation should be organized to cover 4 domains: (a) Analysis and description of an EAS(education activity system), the system that is on; (b) Design of a new system or re-design of the one that is implemented. (c) Analysis of the implementation and institutionalization of the new reform; and (d) Study of the system of management in charge of conducting changes. He add that in every domain we should focus on core ideas, values and organizing perspectives. (Banathy, 1992:18, figure 1.3]] NOTE 9 FROM BANATHY’S MODELS WE CAN GET GUIDELINES FOR ANALYSIS 1. MY VIEW ON SYSTEMS ENVIRONMENT MODEL [[Targets: Wiley missed the point in understanding the 1st model. In my opinion the 1st target is studying the relation between the school system and general societal environments (the international and national). Is in contact with such environments that school are defining their own purposes, goals and objectives]]. In my opinion with the 1st lens, the systems environment model, we can study the following issues: FIRST, Study the relation between the general environments and the school system. Banathy provides an explained figure with all the embedding system in which schools are nested in. This is a kind of Chinese set of boxes that suggest lines of research. We can either start analyzing the relation between the school system with the closest environment or from the largest one. Or get an interactive approach and start for instance in the description of the closest one and explain its problems by recurring to the next or the largest environment. SECOND, the system of boundaries set to protect the school system space from negative influences of the general environment. In this section, the topic what kind of influences were red-taped by schools and what other of the same nature were allow, and why, is an interesting way of addressing the ideology of school administrators. The time period for such openness and closedness of the school system in relation to its environments also deserve especial attention. Long lasting periods of school closedness could have ended up in the termination of the school system as happens with some school during the economic recession or during war time. THIRD, the relation between the systemic environment (that is, the national-local community) with the school system is the most important topic of analysis. The focus here is on what resources and how the school system gets what they have; FOURTH, Study the general transformation of inputs into outputs (though the 2nd model will provide a detailed analysis of this process). Here we will focus on the breaks to introduce resources into schools and the breaks to send back school outputs to society. This implies to study feedbacks and adjustments by administrators or managers of the school system and the phenomena of self-regulation (adaptation) and coevolution of schools with the society by transformation of their structure and functions. To address the FIRST topic above, starting from the largest environment, it is IMPORTANT TO ANSWER SOME general QUESTIONS: (a) How neoliberal policies have affected school system at economic, politic and ideological level?. More exactly How neoliberal adjustments and restructuring policies known as fiscal discipline (that includes the payments of the foreign debt in time), tax reform (that exclude big companies for paying taxes), the public expenditure priorities (that force cuts in social expending and education), privatization (via decentralization) have affected the school system?. Does deregulation policies has affected the labor rights of teachers?, if so, how they responded?. (b)How the government have respond to neoliberal policies described above?. Do schools are preparing students to cover the economic niches left by market globalization and trade liberalization?. Does the legitimacy of the central government and the indices of governability help to create good environment for implementing new changes in education?. Does THE CENTRAL STATE has enough resources to implement the PLANED educational reform?. (c) What ideological challenges are placed on education by current economic and political contexts. What about educational values like equity, efficiency and choice? Or education for democracy, peace, fredom and full respect of Human Rights? In sum, what kind of schools and demands on education derives from the such context?. In other words, what kind of education and for what development is being required and demanded by those social contexts and how national school systems have rsponded to them?. “Education are purpose-seeking systems”, they are guided by goals and ideal visions of the future. They are open and so able to react to and coevolve with their environment. They are “free to define their own policies/purposes and constantly seek new purposes and new niches in their environment” (Banathy, 1992,12-13)]] The study of feedbacks and adjustments by administrators or managers to the school system and the phenomenon of self-regulation (adaptation) set the idea that school reforms are condemned to introduce piecemeal changes to preserve the essence of the old school system. However the concept “coevolution” of schools with the society opens such a possibility and the idea of going beyond mere changes in the structure and functions of schools, but there is not much detail on this regard. In general there is not much room for radical change in Banathy’s view. How his conservative view explains radical school changes that start from the bottom up such as the desegregation movement in the North and radical community initiatives in the South? THE FUNCTION STRUCTURE MODEL THE MOTION PICTURE MODEL THE ACTIVATION OF THE SCHOOL SYSTEM OR IMPLEMENTATION OF SCHOOL REFORM NOTE 10: WILEY’s VIEW ON THE FIRST MODEL OR LENS OF BANATHY THE SYSTEMS ENVIRONMENT MODEL is called by Wiley : “THE SYSTEM'S PURPOSES, FUNCTIONS, AND STRUCTURE “This lens defines a system in terms of why it exists, what work is to be accomplished in carrying out its purposes, and how it will be organized for the work. These aspects of a system represent its most basic and core definition. The system features that relate to this area are as follows: 1. MISSION AND PURPOSES. Social systems have an ongoing mission and a set of purposes. General purposes, although not directly measurable, provide the starting point for the design of the entire system and for its evaluation. From these purposes, a set of system functions are selected that will ensure that purposes are being met. 2. Components. The system has components which themselves are systems having all the properties of the larger system. These components interact to some degree. The components are purposeful and are selected to perform the primary system functions and delivery key services to its clients. Organizational capacity building, resource development, monitoring, and evaluation are common functions of educational organizations for which components must be developed. The identification and organization of the key components help define the system. 3. Measures of Performance. Systems have measures of performance that are used to signify progress or regress in accomplishing its purposes. With schools, the primary measures of performance are focused on the quality and effects of its primary services to clients; e.g., instruction and learning. There are other kinds of system performances and related measures that are useful in sustaining system performance; e.g., cost-effectiveness of operations, staff development, internal and external communication processes and outcomes, and stakeholder support. But these tend to be enabling of the system performance and are not primary system properties themselves. 4. All systems have designers and decision-makers. The definition of this feature states who `owns the system', who has authority to change or redesign the system's major features, and what is the decision-making process for making design changes and taking action if the measures of performance are not matching expectations. THE ENVIRONMENT OF THE SYSTEM The environment of the system is the larger context in which the system exists, performs its functions, and delivers its services. Social systems are always embedded within larger environments and have boundaries (may be physical or communication) that set them apart from the rest of the environment. Generally, it is the more immediate environment with which the system must interact. Primary features related to the environment are: 1. Interactions with the environment. With regard to interactions with the environment, it is the intended nature, intensity, and frequency of the interactions that help define the system. Interactions might consist primarily of exchanges of information and resources. Alternatively, cooperative or collaborative efforts might be developed. At another level, some key functions of the desired educational system might be merged with other systems having similar purposes. The more intense and frequent the interactions, the more blurred become the boundaries between the system and its environment and the more likely fundamental systemic changes will occur. 2. System clients. Systems need to clearly define the clients for whom the system is designed to serve. When a traditional client base is expanded or contracted, the system needs to be altered, perhaps even redesigned, to accommodate the changes. Changes in the client base may create a need to rethink purposes, organizational structures, services, and the relationships of the system with its environment. THE PROCESS LENS The `purpose, functions, structure' lens was depicted as revealing a `still-picture' model whereas the `process' lens is depicted as revealing a `moving-picture' model. The functions-structure lens is concerned with what the system is, what it does, and how it is organized. The process lens is concerned with how the system works through time in order to accomplish its mission. Features include: 1. Communication. The system communicates both externally with the environment and internally so as to bring about expected system performance. The appropriateness, clarity, timeliness, and quality of communication is a critical measure of system health and potential survival. 2. Instruction. The procedures, practices, and arrangements utilized by the system to deliver services to its clients and bring about system performance is a key design feature of the system. 3. Monitoring and evaluation. Utilizing the defined measures of performance, the system observes itself, collects, analyzes, communicates, and uses information to evaluate performance and make needed adjustments. APPLYING THE FRAMEWORK The framework identifies a set of features which system designers need to define. These definitions, in turn, form the basis for designing an evaluation. Each feature represents a characteristic of the system that may or may not pass muster using criteria that will have been generated for them. In the process of designing the various system features and associated criteria, system stakeholders will have ideally generated and used large amounts of information relative to each for the purpose of choosing among possible designs. Thus, evaluation begins with the design process. The primary basis for judging the appropriateness of a system design, whether an existing one, or one that is aspired to, is `goodness of fit'. Goodness of fit is an overall measure that can be addressed by such evaluation questions as: 1. How closely do relevant design features, e.g., mission, purposes, system clients, respond to an aggregate of stakeholder needs and interests? Is the design authentic? 2. To what extent is the design consistent with the relevant research and practice knowledge base, e.g., instruction? What is the balance between personal values and preferences of stakeholders and what is believed to be known about learning? 3. Is the design ethical? Are important client needs addressed? Are some clients rewarded at the expense of others? Are/will stake-holders, including staff, be treated fairly? 4. Is the design systemic? Are all of the features defined? Have any features been defined in such a way that their contribution to the larger system functioning and sustainability is vague or problematic? 5. Is the design implemented (existing system) or implementable (new design)? What needs to be done in order to ensure that the design can be implemented? The choices tend to be (a) change the design to facilitate implementation or (b) spend time and resources developing system capability/readiness for implementation. 6. How well does the system perform? Are performance measures being met with reference to clients? How well is the system performing in terms of `system-level' indicators? These questions are appropriate for use when either an existing system is being evaluated or a new system design is being created. As was stated above, the framework may also be used to evaluate an implementation process by using the overall design and its features as developmental targets and assessing progress towards them. LIMITATIONS AND COMPLICATIONS The barriers to evaluating system designs are closely related to the system design process itself. The most problematic issues are described briefly below: 1. Evaluation must be valued by the organization so that use of data for the decision-making becomes routine. Organizations must provide the time and resources necessary and be committed to organizational learning. People within organizations need to become competent to engage with the evaluation process because experience suggests that the more traditional role of the evaluator, acting in isolation from the organization, does not necessarily lead to high information use. 2. The evaluation of a system design requires that an explicit design exists; i.e., its features are known to stakeholders. Given the complications of designing social systems, the future of system design evaluation literally depends on the willingness of schools to engage in relatively new behavior. The various difficulties perceived by educational administrators represent a major challenge. Clearly, a critical design task will be for stakeholders to design an `inquiring system' capable of coping with the traditional barriers to change (Jenks, 1995). 3. The principle of `requisite variety' (Ashby, 1958) states that a control mechanism's capacity for control cannot exceed its capacity as a channel of communication. In other words, a control mechanism must be at least as complex as, and matched to, the system it wishes to control. If the control system has less variety than the system, then it can control only part of the system, and such limited control may lead to unanticipated consequences in other parts of the system. The concept of `requisite variety' underlines the importance of developing evaluation designs that are conceptually appropriate with the system designs being evaluated. As system designs advance in complexity (as a result of progress from conceptual testing) evaluation designs keep pace. 4. Most of the challenges faced by system evaluators are the same ones with which all evaluators must contend. In general, the challenge is to design and conduct a form of disciplined inquiry that meets professional standards and provides useful and credible information. But system evaluators have additional challenges. There is the need to examine system variables that are not well understood or easily observed. Defining and measuring the effects of component interactions, the contributions of components to system-level performance, or the degree and quality of systemic integration, all acknowledged to be important characteristics of systems, are very complicated. Understanding and measuring them will require considerable effort and cumulative experience on the part of system thinkers and practitioners. CONCLUSIONS The problem-centered and program-oriented evaluation theory, models and methodologies created in the 1960s and 1970s to facilitate policymaking decision-making are not providing the kind of information and level of understanding needed to make changes in our educational systems. The recent development of a theory of system design and accompanying technology offers a foundation for the creation of more suitable models for evaluating systems and system designs. Moreover, a consideration of the nature of knowledge as suggested in the tenets of `naturalistic inquiry' and its associated methodology provides methods more suitable for understanding complex systems and sustaining systemic changes. REFERENCES Ackoff, R. L. (1981). Creating the Corporate Future, Wiley, New York. Ashby, R. (1958). Requisite variety, and its implications for the control of complex systems. Cybernetica 1(2), 1-17. Banathy, B. H. (1991). Systems Design of Education, Educational Technology Press, Englewood Cliffs, NJ, pp. 1-20. Banathy, B. H. (1992). A Systems View of Education: Concepts and Principles for Effective Practice, Educational Technology Publications, Englewood Cliffs, NJ. Banathy, B. H. (1996). Designing Social Systems in a Changing World, Plenum Press, New York. Checkland, P. (1981). Systems Thinking, Systems Practice, Wiley, New York. Checkland, P., and Scholes, J. (1990). Soft Systems Methodology, Wiley, New York. Churchman, C. W. (1971). The Design of Inquiring Systems, Basic Books, New York. Jenks, C. L. (1995). Educational systems design: making it more user friendly. Systems Practice June 1995. Jones, C. J. (1980). Design Methods: Seeds of Human Futures, Wiley-Interscience, New York. Lincoln, Y., and Guba, E. (1985). Naturalistic Inquiry, Sage, Beverly Hills, CA. McGlaughlin, M., and Phillips, D. C. (eds) (1991). Evaluation and Education: At Quarter Century, Ninetieth Yearbook of the National Society for the Study of Education, University of Chicago Press, Chicago, IL. Nadler, G. (1981). The Planning and Design Approach, Wiley, New York. Rittel, H., and Webber, M. (1984). Dilemmas in a general theory of planning. Policy Sciences, no. 4. Rowland, G. (1994). Designing and evaluating. Educational Technology, January. Shadish, W. R., Cook, T. D., and Leviton, L. C. (1991). Foundations of Program Evaluation: Theories of Practice, Sage, Beverly Hills, CA. Warfield, J. (1987). Features relevant to effective system design. A paper presented at the annual meeting of the International Society of Systems Sciences. Warfield, J. (1990). A Science of General Design, Intersystems, Salinas, CA. Worthen, B. R., and Sanders, J. R. (1973). Educational Evaluation: Theory and Practice, Charles A. Jones, Worthington, OH. C. Lynn Jenks, Correspondence to: C. L. Jenks, International Systems Institute, Buck Institute for Education, PO Box 734, Stinson Beach, CA 94970, USA Article A20860857 | 