Barba B. Affourtit, Vice President
Interaction Research Institute, Inc.
ABSTRACT
The song Ironic by Alanis Morrisette laments having "ten thousand spoons when all you need is a knife." The metaphor aptly describes the plethora of metrics that have been foisted upon an unwitting workforce. The late W. Edwards Deming often advised that we reexamine our management information systems for information utility, and discard measurements that are not instrumental in managing and improving a process.1 When measurement is used as a means to improvement, rather than as an end in itself, the result is fewer measures and more information. The right information leads to sound decisions that improve the quality of organizational processes.
THE PROBLEM
Bombardment with meaningless data is eroding productivity in the U.S. Information exists only in its assimilation. The output has far surpassed our ability to assimilate, and the sheer volume of output makes it difficult to extract the useful information. Michael Schrage uses the descriptor “cognitive malnutrition” to characterize information that is consumed but not digested.2 Like french fries, meaningless data leaves us full, but malnourished.
SPOONS
Mandate for measures. The Government Performance Results Act (GPRA) requires government entities to develop outcome measures related to program goals. An outcome is defined by the GPRA as "the intended result, effect, or consequence that will occur from carrying out a program or activity." The GPRA was initiated in response to the finding that most government agencies had numerous measures of activities, but scant measures of outcomes.
Although of intent of the GPRA is sound, in many cases it has propagated the very condition it was devised to correct. GPRA mandates have created a “search for measures,’ fostering the notion that measurement is an end in itself, rather than a means to generate improvements.
Technology: Enabler of What? Reengineering proponents espouse information technology as an essential enabler, reinforcing the acquisition mania that characterizes most improvement efforts. Just as copy machines fill shelves with unused documents, poorly designed information systems inundate overwhelmed employees with irrelevant facts. This reduces professionals into unwilling inspectors who have to sift through all the output to separate useful information from the scrap.
Last Resort Survey. When teams can’t figure out what to measure, the typical fallback is to develop a survey. Customers are becoming weary from filling out redundant surveys. Further, surveys often ask questions that don’t need to be asked. For example, a customer doesn’t need to be asked whether a program was on schedule. There are other means to gain that type of information. An important rule in developing customer surveys is “Never ask the customer what you can find out otherwise.”
Dysfunctional Goals. Indicators of level of attainment of arbitrary goals can be damaging as well as unproductive. For example, a goal to achieve a tenfold improvement in a major process sent managers in a software development lab scrambling to find something that could be improved by that order or magnitude. One program manager established an objective to find and fix ten times more errors during Verification Testing. Shortly thereafter, a research study found that an average of 1.7 new errors were inserted into the code for every error corrected during Verification Testing.
Apple orange comparisons. Data that serves a basis for benchmarking must be operatrionally defined. An example will demonstrate the point. In a comparison of employee absenteeism, what is the operational definition of an absence? Does it include scheduled doctor’s appointments for a portion of the day, or only unscheduled full days missed? The data is not comparable unless everyone is counting the same way.
KNIVES
The criteria for information utility is simple: useful information facilitates decision making. It provides knowledge to make more confident decisions with greater resolve. The type of information needed is a function of the decision at hand.
Provide Direction. Steven Covey popularized a key tenet of the scientific method: “Begin with the end in mind.”3 A study of the factors that contribute to team success conducted by IBM found that clarity of purpose to be the sole requisite for success. Teams with ambiguous charters flounder. A good metric translates an ambiguous concept into a concrete objective, thereby transforming indecision into action.
Pinpoint success. Peter Drucker observed that the most profitable source of innovation is to discover and exploit unexpected successes within the system you are seeking to improve.4 Innovations are employed everyday by employees who discover a better way to manage the processes that they govern. With today’s high turnover and transfer rates, many of these process improvements are lost when the innovator moves to another job.
Drucker discovered that boundless innovations are overlooked in most organizations. He attributes this myopia to the format of existing reporting systems. Data must be analyzed to disclose innovations that can be incorporated throughout the organization. These tried and tested interventions yield greater benefit than most "start from scratch" projects at much less cost.
A study of mortality rates for heart bypass operations in the late 1980's revealed significantly lower mortality rates for patients who were administered digitalis prior to the operation. The findings were verified through confirmatory experimentation, and preoperative digitalis has become standard practice. Mortality rates have been reduced dramatically by identifying and standardizing an existing procedure.
Crystal Ball. When we can predict what will happen under different circumstances, we can mold the future by creating the right circumstances. Rational prediction requires theory and builds knowledge by comparison of prediction with observation.
Schedules for software intensive programs are usually grossly overoptimistic; schedule and cost overruns are practically a foregone conclusion. Estimates are rarely based on past data due to the perception that "This program is different." Indeed, each program is different, but development is a repetitive process.
Dr. Edmund Conrow computed cost, performance, and schedule changes for approximately 50 programs, and found a growth index of 1.26 and 1.24 for cost and schedule respectively as compared to the Milestone II (or equivalent) estimate.5 Simply adding a 25% buffer is not the solution, but the research shows the extent of optimistic bias the current process is producing.
One program manager tackled the problem by using historical data of cycle time at the task level to derive an initial estimate of the time required to complete each task. He estimated human resource requirements by multiplying the number of tasks by the average task completion time. To monitor execution against plan, he developed a simple spread sheet itemizing each task, and recording a one (1) for each task completed and a zero (0) for each task remaining. The average task completion time was recomputed on a monthly basis to assess the reliability of the initial estimates and revise estimates when needed. The 22 month project was completed within days of the initial projection.
Create Beneficial Change. Change is not a synonym for improvement. Improvement emanates from change, but not all change produces improvement. Without understanding of cause, change is trial and error. Ascertaining cause is essential for improvement, since the type of cause dictates the type of intervention needed.
Clinical paths are an outgrowth of the observation that patients with similar treatments recovered at different rates. Investigation for cause revealed substantial differences in practices among health care providers. Clinical paths standardize, rather than change, health care delivery processes by specifying what interventions are needed (e.g., lab tests, diet, physical therapy, patient education) and when. Patient recovery is accelerated by providing the appropriate intervention at the appropriate time.
Converting Knowledge into Action. A team chartered to reduce the length of stay for cardiac bypass surgery patients at Medical Center Hospital of Vermont began with a literature reveiw.6 The team discovered a research study (conducted 10 years prior) that found that recovery was accelerated by reducing the amount of narcotics administered during and immediately following the operation. In response to this finding, the team created new protocols for narcotic administration and provided intensive training to anesthesiologists to develop the finesse to administer the amount of narcotics required, but not more.
Significant improvement was achieved within three weeks after implementation of the new protocols. The time required in Intensive Care was reduced from 3.4 to 1.3 days, and the average cost per case decreased by $1,500. The greatest beneficiaries are the patients who can now recover sooner and get on with their lives.
Useful information facilitates decision-making. It is generated when decision makers focus on what they want to accomplish and compile the information needed to get there.
Endnotes
1 Deming, W. Edwards. Remarks during seminar; Instituting Dr. Deming's Methods of Productivity and Quality. Arlington, VA, March 30, 1993.
2 Schrage, Micheal. No More Teams! New York: Doubleday, 1995.
3 Covey, Steven. Seven Habits of Highly Effective People. New York: Free Press, 1989.
4 Drucker, Peter F. Innovation and Entrepreneurship: Practice and Principles. New York: Harper & Row, 1985.
5 Conrow, Edmund H. Some Long-Term Issues and Impediments Affecting Military Systems in Acquisition Reform. Acquisition Review Quarterly, Summer 1995, 199-211.
6 Schiefer, Janice. Heart Surgery Process Improvements: Patients and Hospitals Win Together. 1993 Presentation Abstracts, The Fifth Annual National Forum on Quality Improvement in Health Care. Boston: Institute for Healthcare Improvement, 1993.