Organizations frequently need to implement new programs, policies, or other initiatives in order to gain or maintain a competitive edge in today's rapidly changing marketplace. Evaluation research is the application of scientific methodology and research techniques to determine whether or not a program has met its stated goals. There are several steps in the program development life cycle: goal development, planning (including planning how the program will be evaluated), implementation, evaluation, and feedback based on the evaluation. In choosing how best to evaluate the effectiveness of a program, policy, or other initiative, the organization must develop appropriate criteria of success ranging from the reaction of affected personnel to the impact of the initiative on society or the environment.
Keywords: Business Model; Criterion; Evaluation Research; Policy; Psychometrics; Return on Investment (ROI); Scientific Method; Statistics; Survey
Many experts liken the organization to an organism that must constantly adapt and change to its environment if it is to survive. Sometimes, the organization must change its policies and procedures in order to meet the requirements of new government regulations. Other times, the organization must change its business model in order to adopt the new technology expected by its customer base. Still other times, the organization must change the way that it deals with its employees in order to increase their motivation, job satisfaction, or job knowledge in order to become a high performing organization. In short, if the organization is not constantly changing to meet the needs of its environment, it will fail.
For most large changes, the organization may need to implement new programs to launch and/or support the change. For example, the installation of a new inventory database may require a training program to teach employees how best to leverage its features into better customer service. The need to motivate the marketing staff to become more productive may involve the implementation of a new incentive or commission program to tie pay to performance. New government regulations for recycling may require programs to ensure that the new requirements are met by employees so that the organization does not run the risk of fines. However, as good as these programs may sound on paper, it cannot be assumed that they will be successful when implemented. Rather, it is important to perform evaluation research in order to determine whether or not the program has met its goals and how, if it has not, it can be made better.
Evaluation research is the application of scientific methodology and research techniques to determine whether or not a program has met its stated goals. Evaluation research is also often used to determine whether or not a program or policy has had negative impact or unexpected consequences that need to be addressed. In general, evaluation research can be thought of as the application of the scientific method to determining whether or not a program, policy, or other initiative has had the intended effect.
The Evaluation Research Process
As shown in Figure 1, the development of a new program or policy has several steps. First, one must specify what goals the new program, policy, or other initiative is to meet. Although this may sound simple (e.g., teach employees a new billing procedure), in actuality, this is often a complex task. The goal must be operationally defined so that when the program has been implemented, its benefits (or lack thereof) can be observed and measured. The choice of a criterion of success (a dependent or predicted measure that is used to judge the effectiveness of persons, organizations, treatments, or predictors) must be carefully made so that it has meaning in the real world and actually reflects the impact of the organizational program on its goals. For example, training programs are often evaluated using such measures of "success" as whether or not the trainees liked the training course and thought they had learned from it (neither of which opinion is necessarily grounded in reality) or whether or not they passed an end-of-training test. However, as any school child returning to school after summer vacation can attest, having passed a test at the end of a class does not necessarily mean that the information will be retained.
Figure 1: Program Development Life Cycle
An essential part of the process of evaluating the effectiveness of an organizational program, policy, or other initiative is to determine what goal the organization is trying to achieve. To say, for example, that one is instituting a new procedure for arranging inventory in the stockroom in order to increase the efficiency with which it can be retrieved is all well and good. However, if it costs more per piece to arrange the stock using the new procedure than is saved by the efficiency of retrieval, then the return on investment is not sufficient to justify implementation. It would be short-sighted to say that the program was successful because it was successfully implemented, when it actually costs the organization more than it saves. For this reason, it is important to carefully determine what the real goals of the program are both to articulate the vision so that the program can be designed to meet this goal and also to develop a priori criteria that can be used to objectively determine the success or failure of the program vis a vis this goal during the evaluation stage.
Collecting Support Data
Sometimes it is impossible to collect quantitative data to evaluate a program. For example, if a grocery store wanted to determine the change in customer satisfaction based on a change in floor plan, meaningful hard data may not be available, particularly in the short term. What is needed in such a situation is a way to determine customer satisfaction (or, ideally, the change in customer satisfaction) that results from the new store layout. In many cases, customers have their preferred grocery store and will continue to shop there despite all but the most egregious changes. Even if customers do leave to shop elsewhere, this change may be gradual, leaving the store without the feedback it requires in time to make adjustments. In cases such as this, it is necessary to use subjective data in order to acquire the feedback needed to evaluate the whether or not the program (in this case, the new floor plan) is working as expected. Although data such as customers' opinions of a new floor plan are subjective in nature (as opposed to the objective data of whether or not the customers continue to shop at the store after the implementation of the new floor plan), if handled appropriately, they can be used to give the organization meaningful feedback on the effectiveness of its programs. Psychometrics is the science and process of mental measurement. Psychometrics is used to adequately and accurately capture an individual's intangible attitude or opinion.
In order to develop a good data collect instrument, the concepts of interest first need to be operationally defined in terms that can be observed and measured. In the example of the grocery store layout, one needs to determine what "liking" or "disliking" the new floor plan means in practical terms that are relevant to the business of the grocery store. To develop a set of scales to measure this concept, the designers of the survey might first interview a representative sample of customers and/or draw on their own experience in trying to find things in grocery stores. Based on this information, they could then draw up a draft list of questions that could help elicit meaningful feedback about the shoppers' experience. Questions might include:
- How long does it usually take you to do your shopping?
- How long does it take you with the new floor plan?
- Did you find everything you needed today?
- What items couldn't you find?
- How many times did you have to ask for directions to find an item?
- Was the signage helpful?
- How likely are you to continue to shop at this store given the new floor plan?
The questions would then be refined and placed on a quantifiable scale so that they could be statistically analyzed and used to provide meaningful feedback to the grocery store concerning the new floor plan. Using the principles of good psychometric question design, the researchers would then develop a survey instrument which could be given to a representative sample of customers to determine the effectiveness of the new floor plan in meeting the goals of the organization.
Once the goal of the program has been articulated and operationally defined, the next step is to plan the intervention. This may be the design and development of a training course, a new policy or set of policies, reengineered business processes, or other business initiative. The planning needs to be done with the goal in mind and also with a view to how it will be evaluated. If a program, policy, or other initiative or intervention is implemented in such a way that its effectiveness cannot be evaluated, it will be impossible to determine whether or not it met the goals of the organization or if it yielded an acceptable return on investment. Part of the planning process should be the development of an evaluation plan for determining how the effectiveness of the program will be measured.
Once the program has been designed and developed, it is next implemented (e.g., the training course is run, the new policy goes into effect). After an adequate amount of time has passed for the program to take effect (e.g., employees to become proficient in a new method for doing their work, implement a...
(The entire section is 4343 words.)