Information is not knowledge. Just as copy machines fill shelves with unused documents, information technology provides the capability to inundate overwhelmed employees with irrelevant facts. We need to examine our management information systems for utility, and discard measures not used for decision-making. When measurement is used as a means to acquire knowledge, rather than an end it itself, the result is sound decisions that improve the quality of organizational processes.
Improvement emanates from change, but not all change produces improvement. If a change is introduced into a process, what guarantees that the new way will be better than the old? Best efforts will not suffice, best efforts must be guided by knowledge. W. Edwards Deming often warned:
"An unstudied solution to a problem may yield immediate results in the right direction, yet in time bring disaster." 1
Solutions should address the underlying causes of performance. Without understanding of cause, change is trial and error. It is irresponsible to disrupt an organization with unstudied change. The soundness of our decisions is directly related to our knowledge of the dynamics of the process in question. Useful measures help develop the knowledge to create beneficial change.
Measurement develops knowledge when it provides the information we need. The right information is generated by posing a question, and collecting only the data needed to answer the question. The type of data required is a function of the question.
Tom Nolan provides a model for improvement comprised of three questions. 2
The first question establishes an objective to accomplish. An outcome measure is used to determine whether the change produced the desired result. A process measure enables change agents to identify the actions that are most likely to lead to improvement.
For example, the aim of the National Highway Traffic Safety Administration (NHTSA) is to save lives lost from vehicle-related accidents. Therefore, the outcome measure is number of fatalities due to vehicle related accidents. A reduction in the absolute number (or proportion of the driving population) indicates that the actions taken led to improvement.
Actions that lead to improvement are identified through causal analysis. There may be several hierarchical causal levels. For example:
This question sets the aim of an improvement effort. The boundaries of the core system are defined by the aim. Many health care systems have changed their aim from "Improve patient's health status" to "optimize the health status of the community." This new aim promotes more outreach preventative services designed to reduce the number of hospital inpatients.
Deming defined a system as "a network of interdependent components that work together to try to accomplish the aim of the system." Core processes serve external customers, support processes serve internal customers. The handful of core processes define an organizations reason for existence in terms of what it is trying to accomplish. The core processes for Defense Acquisition University are:
Support processes enhance organizational effectiveness. For example, the Lionheart system for on-demand printing enables DAU faculty to incorporate the latest policy and procedural changes into course materials, thereby improving the education process. DAU's support processes include:
Goals and objectives derived by answering the first question may be too ambiguous to serve as a basis for action. The second question impels us to translate the objective into an explicit indicator that everyone can understand.
For example, the core mission of the Federal Highway Administration is to enhance mobility. Mobility is defined as "getting to where one wants to go easily." It consists of two elements; (1) can travelers get there? and, if so (2) how convenient is the access to their destination? This example will focus on the second question.
Congestion is the major impediment to convenient access to our destination. Therefore, congestion mitigation is one of the three "Vital Few" focus areas for the Federal Highway Administration4.3 Measures of congestion include:
Annual Hours of Traveler Delay is an indicator of the total time an individual loses in a single year as a result of traveling under congested conditions. In 2002, the average driver lost 23.8 hours due to congestion.
Percent of Travel Under Congested Conditions is defined as the percentage of traffic on the freeways and principal arterial streets in urbanized areas moving at less than free-flow speeds. This measure increased from 29.4 percent in 2000 to 30.5 percent in 2002.
Percent of Additional Travel Time is an indicator of the additional time required to make a trip during the congested peak travel period, versus other times of the day. In 2003, a trip that would take 20 minutes during nonpeak, noncongested conditions would typically require 28 minutes if taken during the peak period of travel.
Any change introduced into a process is a prediction that the new way will be an improvement over the old. The prediction should be based on knowledge of the causes of process performance. Such knowledge is attained through the scientific method of investigation. Subject matter expertise guides the investigator to ask questions of the process and target data collection and analysis accordingly.
Deming categorized process causes into two types:
Special Causes: Variation caused by special circumstances that can be pinpointed to a specific time or location.
Common Causes: Net effect of numerous sources of variation inherent in the current system.
Ascertaining the cause is essential to beneficial change, because the type cause dictates the type of intervention needed. Special causes are correctable through local action by personnel who execute the process. Common causes can only be corrected by those with the authority the change the system.
In his study of successful interventions, Peter Drucker observed that the most profitable source of innovation is discovering and exploiting successes within an organization. Boundless innovations are overlooked in most organizations. Drucker attributes this blindness to the format of existing reporting systems. Data must be analyzed to pinpoint successes.
An educational institution exploited success by isolating exceptional instructors through comparative analysis of student evaluations. The exceptional instructors became mentors to the rest of the faculty, resulting in an improvement in quality of all the courses.
There are numerous strategies that could reduce highway congestion, e.g., adding more base capacity (increasing the number and size of highways), incident management (improving response time to clear accidents), and work zone management. The measures quantify their effect. For example, does adding more base capacity reduce congestion in the long run, or contribute to the vicious cycle of increased development that eventually increases congestion? Some urban areas have halted new road construction into the city to stop the vicious cycle.
If possible, a change is pilot tested on a small scale, and incorporated system wide once it has been confirmed that the change is indeed an improvement. Data collected prior to the change is charted to establish a baseline for process performance.
Unique congestion mitigation initiatives in urban areas across the United States enable pilot testing to quantify the impact of alternative mitigation strategies, and determine the most viable strategy to implement throughout the country.
Data that serves a basis for decision-making must be valid and reliable. Valid and reliable data is the product of a consistent methodology to collect and record the data. An example will demonstrate the point. If we wish to measure employee absenteeism, what do we count as an absence? Do we count scheduled doctor's appointments, or only unscheduled call ins? The data is not comparable unless everyone is counting the same way.
An Operational Definition transforms a measure into a metric. The metric defines an explicit procedure to observe and record the data. The following table depicts some examples of measures and their corresponding metrics.
| Measure | Metric |
|---|---|
| Productivity | FTE days per software program module |
| DAU throughput | # of DAU graduates per fiscal year |
| Readiness | % of inventory ready to use |
| Flight Test anonamalies | # unplanned events per flight test |
| Aircraft Maintainance | Maintenance man-hours per flight hou |
| Absenteeism | unscheduled missed workdays / total workdays |
| Mishaps | Lost time injuries per 200,000 hours worked |
| C-Section Rate | (# C-Sections / # of deliveries) per quarter |
| Hospital Length of Stay | Average # of days in hospital per patient |
| ER Patient Flow | # of hours between registration and disposition |
| Response time | # workdays between customer request and delivery of product/service requested |
Endnotes
1. Deming, W. Edwards. Remarks during seminar; Instituting Dr. Deming's Methods of Productivity and Quality. Arlington, VA, March 30, 1993.
2. Nolan, Thomas W. Making Changes that Result in Improvement. W. Edwards Deming Institute Conference Proceedings, October 2006.
3. US Department of Transportation, Federal Highway Administration. Status of the Nation's Highways, Bridges, and Transit: 2004 Conditions and Performance. Retrieved from http://www.fhwa.dot.gov/policy/2004cpr/index.htm.
References
Affourtit, Barba. "Statistical Process Control Applied to Software." Chapter 17 in Total Quality Management for Software , New York: Van Nostrand Reinhold, 1992.
Affourtit, T.D. "Leadership and Management Training & Development." Chapter 5 in Human Resources Management Handbook. New York: Marcel Decker, Inc., 1987.
Deming, W. Edwards, Out of the Crisis. Cambridge, MA: Massachusetts Institute of Technology, 1986.
Deming, W. Edwards, The New Economics. Cambridge, MA: Massachusetts Institute of Technology, 1993.
Drucker, Peter F. Innovation and Entrepreneurship: Practice and Principles . New York: Harper & Row, 1985.
Ishikawa, Kaoru & Lu, David J., What is Total Quality Control? The Japanese Way. Englewood Cliffs, NJ: Prentice-Hall, 1985.
Rodriguez, Antonio, Landau, Samuel & Konoske, Paula. Systems Approach to Process Improvement. Washington, DC: Total Quality Leadership Office, 1993.
Senge, Peter M. The Fifth Dimension: The Art & Practice of the Learning Organization. New York: Doubleday/Currency, 1990.
Author:
Barba B. Affourtit, Vice President
Interaction Research Institute, Inc.