Kiss Me
Please

Who's
Counting

Survey
Design
Strategy

Hire the
Handicap

In 1932, Albert Einstein was invited to address the problem of delivering mankind from the menace of war. Einstein’s treatise described a Utopia where mankind subordinated his instincts to the dictates of reason. In reply, Sigmund Freud pleaded for a more feasible and expeditious solution. Of Einstein’s idealistic venture, Freud wrote:

The Deming philosophy has not changed. His fundamental approach is the simple use of the scientific method, judiciously applied, to provide precise evidence of the causes of and the responsibility for discrepancies: the correction of which leads to creative, viable, and lasting changes in the system. Deming abhorred what he called the "misuse, overuse, and abuse" of fundamentally simple methods that are designed to narrow the focus of a scientific inquiry, and thereby efficiently lead to an effective solution.

The process flow diagram and process analysis techniques are prime examples of misguided emphasis by many practitioners. While these techniques can be used to spot quick fixes in a dysfunctional process, such as redundancies, unessential steps, and delay points, the real value lies in locating a condition on which to focus our more sophisticated analytical weapons.

Users of process flow techniques become preoccupied with the procedure because it is easy to understand, easy to apply, and easy to communicate. Some practitioners package and promote this technique as the essence of quality management. The technique has often become an end in itself, rather than a means to an end. As a result, use of the technique becomes an inhibitor to progress. A case in point is the team that covered a twelve foot conference table with a process flow schematic that involved 140 process steps, took six months to develop, and was only 90% complete! Another overstressed group hired a consultant to produce a detailed schematic so complex that it was never referred to during problem-solving sessions.

It is this kind of misapplication that turns a good concept like total quality or a good tool like the process flow into a fad; that eventually dissipates due to futile application and fruitless return for the time, effort, and expense invested.

It is not that the process flow charting technique is bad; if used to narrow the focus of an investigation, generate a reasonable scientific question, or to determine what is important to study. Deming would applaud any method that leads to the improvement of quality for any process. Deming would not; however, applaud the concentration on productivity increase through process flow charting that typifies the majority of process improvement applications. While emphasis on productivity can have a major impact on inefficient activities, true transformation to continuous process improvement occurs as a result of an emphasis on the quality of process performance. According to Deming, productivity increases as a consequence of quality improvement, not the reverse.

Only five or ten process steps out of 140 may be essential for an improvement effort, the rest may not be relevant, may not make a difference in the outcome, or may have no impact on improvement. The detail of many process flow applications is usually unnecessary. By the time it takes some groups to finish an overly detailed process flow, the "problem" can be completely out of hand.

There is no standard process flow format and no rule of thumb for application, since processes differ and the detail required may vary. Common sense, knowledge of the process, and a rational objective for improvement are determining factors. Above all, the K.I.S.S. (Keep It Simple Stupid) principle should be followed.

In some cases, it’s a matter of proper sequence. In planning a military amphibious operation for example, the are of operation (AO) is selected first, before a detailed study of each appropriate beach landing site is made. If the conditions of all beach sites were studied in detail prior to selecting an AO, it would take years before an operation could be conducted.

The application is also analogous to planning a trip from the East Coast to the West Coast. An interstate highway map is required, and street maps for only those areas where stopovers are planned. Street maps for each city along the way would be quite unwieldy and drastically reduce luggage space. In a similar vein, detailed study of the combustion engine schematic is not necessary when the oil warning light comes on. Only a few steps are necessary to determine the cause and get the vehicle moving again.

Over emphasis on the process flow is a symptom of micro-management, a disease that takes management away from the critical duties for which they are assigned. It’s not that it is wrong in principle to detail all aspects of the processes for which one is responsible. And, it’s not wrong to locate every delay point in the system (although delay points are usually caused by many other factors that should occupy attention). And, it’s not wrong to list every possible cause for a problem, or to chart several hundred characteristics of a part. It’s just not efficient! And, it leads to tampering, and a constant search for special causes.

After paying a large fee for a data base that allows for benchmarking, one health care institution discovered that the company supplying the database had not recorded more than 100 deaths that actually occurred. Mortality rate is a prime indicator in evaluation of health care organizations. Three months of analysis, management meetings, directives, and policy changes were wasted before an investigation discovered that employees who input data for data base comparisons were "dropping" (disregarding) diagnostic codes that were not understood.

Another health care organization could not reconcile their ancillary costs (other than direct) data against the data base of other benchmark companies. Investigation revealed that no operational definition was used to determine what ancillary costs actually were. Each institution had different criteria for this data. It was input as is, making benchmark comparisons totally invalid for years.

A recent GAO report sites that incomplete and poorly designed military health studies have left unanswered questions about the extent of exposure and range of problems concerning military personnel who may have been opposed to toxic substances during the Gulf War.

It seems like anywhere data is being collected on the same process…it will not be collected in the same way! Incomplete, off target, and contaminated data issues of Data Integrity, are main complications for any measure of performance according to Barba Affourtit, Chief Statistician for Interaction Research Institute, Inc.

At the most basic level, programs that use data to assess process change fail, because of the validity and quality of the input. "Data integrity is an issue in every endeavor…service, commercial, and industrial. Even the electronics industry has trouble defining solder defects consistently."

According to Affourtit, the reason for so much undercounting and overcounting is the lack of an operationally defined measurement system. Confusion leads to miscounting. The measurement system must be defined in such a way that people can understand. "The operational definition must be explicit, so that everyone collects and records the data in the same manner. Consistent interpretation of an operational definition can be confirmed by comparing independent measures of the same items, as recommended by W. Edwards Deming in his first book, Quality, Productivity, and Competitive Position."

Affourtit is currently assisting a high level Quality Management Board in developing an international measurement system to monitor the effectiveness of their occupational safety and health program. "The first step is to determine what measures provide the most useful feedback for reducing work related injuries, illnesses, and their associated costs. Once the measures are identified, an operational definition is derived for each key process indicator, specifying how the data is going to be collected."

If this sounds easy… it is not! There are many factors to consider. Defining the "WORK PLACE" for example requires consensus over parameters. The "ACTIVITY" often evokes differences in opinion. And, the sample size has to be considered, as well as the method of observation, how the data is received, and how the score is derived for each process indicator.

Most definitions are left open to interpretation. For example… the Bureau of Labor Statistics definition of lost workdays due to injury does not include the day the injury occurred. Misunderstanding leads to overcounting.

A normalization factor provides a level playing field for comparing unlike activities that have different risks of injury. For example, you can not compare raw scores for blue collar and office workers. Data recorders often look for patterns and aggregate data. This practice leads to faulty conclusions. According to Affourtit, proper implementation of the Government Performance Results Act requires sound emphasis on the issue of Data Integrity. This is the foundation that establishes the credibility of any measure of success.

Every since the Government’s Performance Review Act of 1993 was mandated there has been a resurgence in the use of surveys to assess judgments of performance, establish baselines, produce benchmarks, or otherwise serve as measures of positive momentum.

Agencies are designing, borrowing, and/or modifying item statements, or using existing survey instruments to assemble their version of performance assessment. Gains and loses due to year to year fluctuations in scores are reported either enthusiastically or explained as aberrations caused by external forces.

However, a study conducted by a Harvard University research group some years ago revealed that in one government department alone (DoD), only 37 of the nearly 2,000 survey instruments in use derived evidence of validity and reliability. Surveys were being used to make decisions about such issues as purchasing facilities, personnel assignments, equipment requirements, and combat readiness. According to the Harvard group, the accuracy of these instruments was no better than random guessing.

The study pointed to a number of survey construction and design issues that failed to meet the American Psychological Association Standards for Tests and Measurements.

Failure to establish validity, (criterion, concurrent, or predictive) and a lack of instrument reliability (internal consistency and test-retest) were prominent reasons for rejecting the surveys. Without evidence that changes in survey scores translate to a significant change in some performance objective, the survey may only be measuring common variation, sampling difference, or an artifact of the survey itself. Notwithstanding, performance validity and instrument reliability, the researchers identified a number of particular constructions errors:

A good survey instrument is a finely designed, and highly precise tool. Like any tool it must be carefully calibrated and tested before results can be offered as hard evidence of performance change or used for decision-making.

Hire the Handicap

"Hire competent statistical help!" is W. Edwards Deming’s answer to those seeking a solution to their production and service woes. "Get top management involved…Create a structure that pushes every day on the 14 principles of management" is more advice from the Bhagwan of statistical methods. Easy for him to say! But, enlisting competent help – the type who can do all that Deming espouses requires a master statistician with the character and will of John Wayne.

The modern day SPC expert must have integrity, tact, insight, wisdom, daring, and charisma; not to mention highly developed statistical skill with at least, according to Deming, ten years of experience as a practitioner. Not only should Deming’s disciple be both a man and woman’s man and/or woman, but he/she must also accept the salary of a technician. Deming’s ideal is an enigma; one who champions Jeffersonian principles, practices Machiavellian tactics, and accepts Buddha’s self-denial.

Expedient managers, hoping to tap the source before the talent is depleted, grasp for expertise in a limited field. As a result, a number of characters have emerged to assume their role as the corporate "critical mass".

The Professor is a self-involved egotist who normally displays elaborate, if unnecessary, graphs and charts as testimony of his or her self-importance. The Professor, speaking primarily in jargon suitable for valedictorians, is a formula freak who cannot complete a sentence without interjecting statistical or theoretical preposition as the object of communication. He maintains an aloof posture, especially when threatened, and he feels that dealing with the mechanical or practical aspects of a process is quite beneath him. Condescending in manner and tone, the Professor talks over you, at you, or past you, but never to you. Two Professors are less than one, since both will pass the time discussing or debating esoteric topics and grandiose designs of little concern to the common worker.

The Datamaniac is a human aberration who would much rather communicate with hardware than people, and who expresses himself to others in vague, cryptic half-sentences, interpretable only by cohorts of the same bent. Detached and often frustrated by attempts to deal with humans, the Datamaniac surrounds himself with state-of-the-art equipment, and usually responds to questions from others while looking directly at a monitor. It’s not that the Datamaniac blatantly ignores those seeking help. It'’ just that he is oblivious to all those who do not share his domain. When he does emerge from his cubical of calculations, he proudly displays reams of printouts and/or lists of numbers (often of dubious origin) that satisfies his sense of value and serves to reinforce his notion of man’s incapacity to function in the complex world of figures. In the end, we will all submit to the supercomputer and be replaced by robots anyway, so why be concerned about the timelines and practical application of current data?

The Enforcer alienates others by retaining and enhancing the tradition image of Quality personnel as "policemen" of the industry. Attempting to affirm his new role, the Enforcer often overreacts and projects an image of the inspector gone mad with power. Punitive in method, the enforcer usually evokes ill feelings and confrontation among and between various departments and, as a result, instigates submission of "messaged" data that eventually destroys the corporate quality movement and the Enforcer as well.

The Isolationist may have begun his or her assignment with the romantic, if naïve, notion of the contemporary Quality professional as the company "champion," destined to lead corporate America back to preeminence by showing all how to "do it right the first time." Disillusioned and confused by misoneism (resistance to change), the Isolationist regresses to a xeonphobic state and seeks solace within familiar surroundings, hidden from the source of discontent. Scheduling activities so as to avoid contact, the Isolationist leaves reports and other evidence of his work with secretaries or at unattended desks, communicates via memo, and is either unresponsive or noncommittal at management meetings he is forced to attend. Since he doesn’t really comprehend the working processes he is required to assess, or the nature of the data collected for his analytical benefit, his charts are often confounded by uncontrolled artifacts and unattended measurements that offer ridiculous representations of reality, and that no one can accurately interpret. As a result, the Isolationist continues to withdraw from decision-making activities and eventually is utilized for "special" projects and travel assignments that affirm his title. Eventually the Isolationist is forgotten or discounted as a vital contributor to corporate excellence.

The Fuhrer strikes an authoritarian posture, for affect. He seeks and needs power above all. Through SPC he will build an empire that befits his sense of self-esteem. He sees the Quality function as the controlling force in the organization, and desires, not to promote statistical understanding within each department, but to infiltrate each function with gestapo type agents (see The Informer) that report disloyalty to the Fuhrer’s cause. Through the Zeitgeist (spirit of the times) and personal distortion of Deming’s philosophy, the Fuhrer hopes to control the corporation today and the world tomorrow.

The Informer functions primarily as an agent for a manager who is attempting to "burn" those who fail to comply with, or properly respond to, the quality improvement mandate. Operating with the cunning of a KGB agent, the Informer uses statistical methods strictly as tools for evaluation, and he is responsible for most of the corporate energy being wasted on blame scaping, blame avoidance, or finger pointing. As a result, the Informer is more a source of annoyance than support.

Deriving great satisfaction from "catching" people, the Informer uses common problems, not as opportunity to foster solidarity, but as incidents to promote dissention and paranoia. Not interested in imparting his expertise to help co-workers do better, he tries to elevate himself by exposing weaknesses in others. Interrogation is substituted for advice and threat for encouragement, the Informer often mistakes interest in gossip or approval from above. He eventually evokes disdain rather than reward from superiors who, in turn, allow the informer to self-destruct.

The Evangelist, more of a preacher than a practitioner, believes he has a mandate from heaven to reform corporate America. With a penchant for proving the obvious, the Evangelist devotes most of his time to abstractions and rhetorical statement, expressed in pontifical overtones and self-righteous demands for repentance from those who do not follow the new quality religion. The ultimate true believer, the Evangelist has a way of saying what everyone already knows, but with more style and profundity. Global in scope, he has foresight without insight (can’t see the trees for the forest). Guidance from the Evangelist is strategical rather than tactical – often beyond the control of those who desire simple solutions to mundane problems. Those who can influence his pronouncements, tolerate his pomposity and use him as a protagonist for customer propaganda.

Selection of the individual to spearhead or oversee SPC projects is most critical to the success or failure of the entire corporate quality improvement movement. Notwithstanding technical expertise, the SPC program leader should be diplomatic, concise, and efficient. Essential data analysis and timely feedback directly to those who can benefit most from the signals is key. Ability and willingness to communicate findings in a cooperative and supportive, as opposed to an authoritarian or derogatory manner, will overcome most of the pitfalls and barriers that usually occur when projects cross department boundaries.

The SPC coordinator needs many highly developed talents, but contrary to Deming’s dictum, the least of which is statistical expertise. It is not profound knowledge of statistical theory and methodology that is most important. Generally, the statistical techniques necessary to achieve control and breakthrough are easy to master. Quality improvement usually requires simple techniques, systematically applied, and judiciously monitored. Knowledge of the process is crucial to proper utilization of basic statistical tools. Therefore, selecting in-house talent familiar with corporate processes and then developing his or her statistical capability has many advantages. Operators, supervisors, and middle managers usually have much more understanding of their processes than most statistical experts can hope to achieve. It is this intrinsic knowledge of the nature of the process that enables the professional statistician to select, recommend, and properly apply the right tools. Only by working together in a mutually supportive atmosphere can all members of an organization forge the best results of everyone’s benefit.