Quality Software ... But By Whose Definition. Is the End User King? Copyright CAUSE 1994. This paper was presented at the 1993 CAUSE Annual Conference held in San Diego, California, December 7-10, and is part of the conference proceedings published by CAUSE. Permission to copy or disseminate all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, that the CAUSE copyright notice and the title and authors of the publication and its date appear, and that notice is given that copying is by permission of CAUSE, the association for managing and using information technology in higher education. To copy or disseminate otherwise, or to republish in any form, requires written permission from CAUSE. For further information: CAUSE, 4840 Pearl East Circle, Suite 302E, Boulder, CO 80301; 303449-4430; e-mail info@cause.colorado.edu Quality Software ... But by Whose Definition. Is the End-user King? By Louise M. Schulden Cornell University Introduction Software is playing an ever-increasing role in critical business processes. Yet software quality has not received the attention needed for such an important company asset. Current software quality levels in the US result in software with approximately 4.5 defects per 1000 lines of executable code. This is an unacceptable level of quality. Japan is doing 3-fold better with 1.5 defects per 1000 lines. Motorola and IBM have launched quality programs striving for six sigma quality in software, or 4.3 defects per million lines of code. This may be excessive and addressing the wrong problem. How bad is the problem? A 1988 US Government Accounting Office surveyed the success, or otherwise, of software projects for their division and found that of a 6.8 million software budget the results were: Software Projects for US Governmental Accounting Office 1988 47% (3.2 million) software delivered but never used 29% (2.0 million) software paid for but not delivered 19% (1.3 million) software abandoned or reworked 3% (0.2 million) software used after changed 2% (0.1 million) software used as delivered Total quality management, quality improvement programs are common place in most industries, particularly manufacturing, and in most industries the payback has been incredible. The word quality is used in everyday speech to describe the degree of excellence of a product or service. But in the interum quality programs for software have been allusive. The first problem is a definition of software quality. There is confusion about what is meant by the term software quality. Part of this confusion may be caused by the different perceptions of software quality existing between people; software developers vs traditional quality assurance people vs end-users. There are different dimensions of quality which are important when considering the quality of a software product: performance and features, reliability, conformance, durability, serviceability, aesthetics, perceived quality, etc, It seems clear that quality is not easily defined, except arbitrarily, and that there are a number of dimensions to it. This paper would like to present the software quality challenge. It starts with the important definition of what is the meaning of quality software to your institution and more importantly those who ultimately stand in judgement of IT (Information Technology) products and services, the end-users. Then how does a company organize a Information Technology quality improvement effort? What is the process for addressing quality trade-offs? What role does the customer play in all this? What is their definition of quality? What software and system attributes are important and to whom and how do we measure them? What tools or processes or ideals to use and follow will improve the quality of our software? Finally, how do we evaluate if our efforts are successful... worthwhile? Misconceptions and ... The first misconception about software quality is that IT management and staff know what quality is. When problems occur or customers become dissatisfied, it becomes immediately obvious that the software is of poor quality. Yet the IT response to the quality question remains essentially reactive rather than focused on searching for ways to build quality into software and services. Second misconception, quality can be related to an "acceptable" level of failure. An old IBM advertising campaign asked: "if your failure rate is one in a million, what do you say to that one customer?" Unfortunately, all too often we are measured by our failures. Third misconception, quality is an expensive luxury. The cost of quality in software is the cost incurred by delivering faulty systems. These costs encompass not only the cost of correcting the fault, but the costs incurred by the business due to the fault such as, lost orders, uncollected tuition, dissatisfied customers. The cost of detecting and repairing software failures after they have occurred usually far outweighs the cost of preventing them. Fourth misconception, quality is free. Quality improvement efforts are by no means free. There are costs to efforts required to prevent mistakes, appraise work done, correct defects. However as long as these costs are less than the resulting benefits, they are worthwhile. The problem is that quality efforts require an investment up front, and it takes time before the benefits show themselves and can be assessed. Fifth misconception, lack of quality in software is caused by poor quality staff. In fact, most people prefer to do a good job, but will deliver the quality they think is expected of them. If people feel that no one cares whether they produce quality work, they won't. Sixth misconception, one can test quality into software...unit test, integration test, systems test, acceptance test, and finally quality is achieved. Testing does improve quality, but it is costly and still you can miss the mark. Truths First truth, users do not weigh equally everything that is right with software against what is found to be wrong. Unfortunately, we get judge by our mistakes. Software that works well is taken for granted. Software that is wrong for whatever reason, is remembered... and often talked about. Second truth, users do not distinguish between problems caused by the application software itself, and those which are caused by faults in the hardware, system or communication software. Third truth, whatever is wrong with the software, not meeting requirements, buggy programming, bad communications environment, slow response time, does not interface with vendor purchased or other software applications, etc.,etc. is the software developer's problem. It may not be his/her responsibility, but it will be their problem. It should be noted, this is getting better with more business partnering between the IT function and other business functions within the organization and team work across department, divisional, and institutional organizational boundaries. Still it has a way to go. Fourth truth, "the best you can do as a computer professional is defend yourself." (DeMarco, 1980) Why is software quality important? The crash of a Boeing 767 in May, 1991 was attributed to malfunction of software that caused the plane's engines to reverse thrust in midflight. I expect the people on the plane did not realize when they boarded the significance of that software, but without question the quality of that particular software was of paramount importance to their very well-being. Computers and the software they run from microcode to standard 4th generation programming languages touch every aspect of our life. For a university, the proper functioning and support of computer software makes sure we have an entering class, students get courses, students get tuition bills and financial aid, room assignments are made for classes, grades are recorded and tracked, and finally diplomas are issued. University software pays the bills (suppliers, employees), collects the revenue (tuition, outside support, alumni gifts), and even controls the heat in our buildings during our cold Ithaca winters. And yet software quality has not received the attention needed for such an important institutional asset. When facility building runs over budget, we cut moneys for parking to cover the loss. So software quality is the parking lots of many of our expensive high-rise system software development efforts. And yet because of hard financial times, institutions like Cornell must look to the strategic application of Information Technologies as a source of competitive advantage with other educational institutions in the future. A high quality purchasing system will allow Cornell to pay bills in a more timely fashion and consolidate orders, thereby saving millions by taking advantage of volume and early payment discounts offered by vendors. Quality administrative systems free up not only staff but faculty, allowing the institution to save dollars in staff reductions and better utilization existing employees. Freeing up faculty time, allows them more time to go after grants and perform better research and teaching so more moneys flow to the school. Poor quality software or information technologies solutions COST BIG TIME. When an administrator says that they'd rather fill out a form than use the system, IT has a problem. When a faculty member calls, and complains they just wasted a day trying to send a document because of faulty communications software, IT has a problem. When 70% of your IT staff is spending all their time fixing bugs and maintaining software so it runs in production instead preparing for the new and future needs of the institution, IT better start looking for work in another field. What is Quality? Quality Defined. One of the early works to define quality resulted in Garvin's 5 approaches to defining quality. Garvin recognized that one approach to evaluating quality would not fit all situations. Consequently, the result was five approaches with the advice to follow the one that will most likely give you the result you seek. What you can see in computing is an evolution of the quality definition. Garvin's 5 approaches to quality include: the transcendent approach, the product based approach, the manufacturing approach, the user based approach, and the value based approach. Transcendent approach is software is viewed as its innate excellence. In this case, the software would be viewed as a work of art: new, visionary, inventive. Quality is an unanalysable property. One can only evaluate on gut feel. Unfortunately, far too many computer professionals feel this way about their work. It is this path that has caused IT to find themselves in the predicament their in. For years, computer professionals were rewarded for reinventing the wheel. Now, there is just not enough time or money and there is far too much work, to encourage this behavior. Programming must stop being art, and be a business. If I have a print routine, writing another one that is unnecessary, is not excellence, it does not contribute to the quality of the IT function even if it is well written software. We very rarely have the resources to revisit the same problem or need twice. One step further, if I can purchase a print routine that meets the organization's needs for the optimal cost, then that is the quality thing to do. The transcendent approach may be how computer people judge each other, but is not an institutional approach to software quality. It was probably most applicable prior to the 1980's, when in fact computing was still in it's infancy and time of discovery. 1980's Quality Definition - Product and Manufacturing Approach Product based approach is software quality is related to the presence or absence of some attributes or characteristics and that these attributes can be objectively measured and consequently so can the software's quality. The manufacturing approach equates quality with conformance to stated requirements. The combination of the two, software that contained code possessing the professionally accepted quality software attributes/ characteristics and conformed with stated requirements was the goal of the 80's. It represented what 1980 programming shops consider acceptable and quality product. Those of us who got our computer training in the 80's, were brought up on attributes or software characteristics that were signs of quality programming. Quality Software Attributes and Characteristics What attributes or characteristics are relevant and traditionally have been considered when considering the quality of a software product? The software literature is full of the attributes such as: correctness, flexibility, efficiency, reliability, usability, extendability, portability, testability, understandability, re-usability, maintainability, interoperability, integrity, and survivability. Top of the list is performance and features. Performance relates to the primary operation characteristics of the software. Features refer to the secondary characteristics that supplements the software's basic functions. (NOTE: Both performance and features are measurable, but it does not follow that the user perceives differences between different software as significant in quality terms). Efficiency ,the amount of computing resources and code required by a program to perform a function. Usability ,the effort required to learn, operate, prepare input for, and interpret output of a program. Reliability ,the extent to which a program can be expected to perform its intended function with required precision or the probability of a software product failing with in a specified period of time. Unlike a manufactured product, software is more difficult to evaluate on this front due to the fact it doesn't "physically deteriorate". Extendability /flexibility ,the effort required to modify an operational program. Portability ,the effort required to transfer a program from one hardware configuration or software system environment to another. Testability ,the effort required to test a program to ensure it performs its intended function. Understandability , the effort required to understand the code and what it is doing. Re-usability ,the extent to which a program can be used in other applications, related to the packaging and scope of the functions that the programs perform. Maintainability ,the effort required to locate and fix an error in an operational program. Serviceability ,the ease with which the supplier of the software accepts responsibility and rectifies. Interoperability ,the effort required to couple one system with another. Integrity ,the extent to which access to software or data by unauthorized individuals can be controlled. Conformance, the extent to which the software meets the specification. (This must be measured before and after acceptance of the software by the customer. Deviations may become apparent only after the software has gone into service.) Correctness ,the extent to which a program satisfies its specifications and fulfills the user's mission objectives. Durability/survivability, the measure of the length of time that software can be used before replacement. Aesthetics, yes software can be beautiful. Perceived quality , the user opinion of the quality and usefulness of the software. This may in fact be the most important. Individuals may not have full information to judge by, but judge they will. Their judgement may also include price and reputation of the software supplier. As one can see there are many characteristics that contribute to the quality of software, and this list is certainly not exhaustive. These actual represent high-level attributes that can be shown to depend on other characteristics. For instance, if a piece of software is to be maintainable it must be understandable, testable, and modifiable. Given the state of the art in software engineering, growing the the tree in this way until the characteristics at its leaves are objectively measurable may not yet be possible but it is a necessary goal if software quality assurance is to develop. In 1987, Kaposi and Kitchenham proposed a quality profile model as a way of structuring the analysis of the quality of a piece of software. The quality profile of the software is specific to an individual and the application, but has the advantage of separating quantifiable and non-quantifiable factors. It provides a good basis for an explanation of why different people can simultaneously hold different views about the quality of the same piece of software. Thequality profile categorization follows: Quality Profile for a Person,Application Transcendental Properties (Non-quantifiable) Quality Factors (Objectively measurable) Quality Metrics (Quantifiable) Quality Attributes (Indicate presence or absence of a property) Merit Indices (Subjectively measurable) Quality Ratings (Quantification of value judgement) It should be noted that some of these characteristics are mutually exclusive. Quality is a trade-off. Which attributes should be emphasized? Quality is a Trade-Off In addition to identifying the "quality" characteristics there is a problem with conflicts between the quality attributes. After quality attributes of software for an application have been defined, the next major concern is determining which of the quality attributes to emphasize. It is impossible to optimize all quality attributes because of conflicts between the quality factors such as, maintainability being at odds with speed of execution or minimization of storage. A system that is easy to use requires easy access and system openness. By contrast, high integrity requires limited access and a closed system. In a trade-off environment, one must decide whether to emphasize the correctness characteristics (internal controls, data entry, and validation) or the maintainability characteristics (user documentation and simplicity of design). It is important to emphasize the qualities appropriate for the application. To add to the difficulty, is the issue of cost. People say quality is free. That's not exactly true. Total quality-related costs are often subdivided into four groups: 1)prevention costs (quality planning, employee training, supplier education, etc), 2) appraisal costs (reviews, walkthroughs and other forms of testing), 3) costs of correcting defects discovered before acceptance,and 4) costs of correcting defects discovered after acceptance which have to be borne by the developer. This complicates the cost of quality issue because the cost of quality assurance activities such as appraisal and prevention are more easily estimated than the expected savings. Over the years, depending on the software and its application some attributes have taken a back seat to others. For example, in the 80's portability was of little importance. Most administrative shops were running large mainframe applications. There was little thought to moving the applications to other platforms. Now with hardware cost plummaging, micro- and mini- computers competing with mainframes on raw computing power, and communications software and networks propagating and improving in reliability, portability is a very desirable software attribute. The type of application effects the ranking of relative priority of the characteristics. An application used by hundreds of decentralized users will place more importance on the quality of useability and nice GUIs, than a system used by a well-trained few. Motivation to undertake quality assurance activities may be to produce a good product, but usually not. More usual reasons include cost effectiveness or good customer relations or marketing. And despite the definition of quality characteristics and their prioritization, quality software and systems alludes us. What is missing from our definition? 1990's Definition of Quality In Garvin's user-based approach and value-based approach we may find a definition of quality that we can successful apply in the 90's and next century. User-based approach where quality is related to its fitness for use in a particular application. Quality is related to the software user's satisfaction. Value- based approach combines quality, which is a measure of excellence, with value, which is a measure of worth, by defining a quality product as one which provides performance at an acceptable price or conformance at an acceptable cost. Sample definitions reflecting this philosophy... "The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs (ISO 8402 standard)." "The totality of features and characteristics of a product or service that bear on its ability to satisfy a given need (BSI 1979)." "The degree to which the attributes of the software enable it to perform its specified end item use (DOD 1985)." What is not present in these widely accepted standards is the acceptability of cost. Now with finances being tight, acceptable cost must be added to the definition. Organizing for Quality Now we know what it is, how do we achieve it? First there must be management commitment. This can not be over-emphasized. Few things are more damaging to quality initiatives than a stated quality policy which is immediately contradicted by short-term imperatives and unrealistic deadlines. Without a highly visible commitment to software quality from management, no quality program will succeed. A separate functional group with responsibility for quality should be created within the IT function, being careful to make sure the achievement of quality remains the responsibility of every person involved in the delivery of software products and services. The quality group is to advise on procedures, techniques and tools, and provide external, objective quality assurance. The quality specialists must be viewed in a support role of assisting staff in the achievement of quality, rather than a policing role. Management must be the police, so the seriousness of this quality initiative is reenforced. The quality function should aim to prevent problems before they occur through education, and the introduction and support of appropriate procedures, standards, techniques, tools, and training. One of the key functions of this group is to take the customer's satisfaction pulse regularly. A second group will be needed to spearhead the quality improvement effort. This group would be comprised of members representing different roles in the IT function: business modelists, front-line consultants, analysts, designers, programmers, technical support staff, and operators. Their responsibility should be part-time, and a rotation through this group is advised. These people will define and plan the quality improvement effort, represent their concerns to the quality team, and the quality team to their function. They will be instrumental in the implementation of quality initiatives within their function. Metrics It will be difficult to register any improvements in quality unless some measures of quality are established. "You cannot control what you cannot measure" (DeMarco, 1982). The identification of suitable measure, and the assessment of the actual values of each of these measures, is an essential component of any effort to improve quality. One must be careful when selecting measurements. Selecting the wrong measurement could give undesired results. For example, measuring lines of code could result in the illusion of increase productivity, but more likely it will result in extraneous, inefficient code and reduce use of reuseable modules. Measurements might include: * number of problem reports, change requests received per period of time * problems or change requests outstanding at the end of each month * time taken to respond to problems * number of errors detected and type design, specification, misstated or misunderstood requirement There are numerous possibilities that should be limited only by imagination, need, and resources. It is important to implement the right measurements. Common sense and monitoring the results will tell you if you are measuring the right things to get the desired outcome. Secondly, it is important not to select too many measurement (5-6 is sufficient). Remember quality improvement is an iterative process and a long-term commitment. Too many measurements will distract and confuse the direction of the quality effort and be overly costly. Pick largest problem areas first. It may be difficult to obtain measurements. Inability to take needed measurements in itself is a quality problem, and should be attacked as such. Thesecond step in a quality effort may be developing the means to collect needed metrics after identifying what metrics are needed. Whatever means is used, keep it as simple and unobtrusive as possible! You'd be surprise how much development and maintenance records you are probably already keeping can tell you. For example, the costs of development efforts is usually readily accessible information. Currently, 79% of all development efforts are viewed as going significantly over dollar and/or time budgets. System usage records are also usually readily accessible due to IT's need to account for machine usage. Currently national usage statics show 45% of all systems never get used. During development: track costs, milestones, the success of unit testing, the amount of reusable code exercised, track record for user acceptance testing. The effectiveness of your systems/software development life cycle methodology can be seen in the number of changes made at each development stage to: the business model after its acceptance, the logical design after its acceptance, the physical design after its acceptance, file changes after physical design, and program changes after unit testing during user testing. Maintenance Metrics Maintenance tells you an incredible amount about the quality of existing software. Maintenance can fall under 4 categories: corrective, adaptive, perfective, and preventative. If your organization is doing a great deal of corrective maintenance, fixing bugs, etc. then it is a good indication your IT function needs better systems development cycle methodologies, or modeling tools, or programming standards, or testing procedures. Adaptive maintenance is due to a changing user or computing environment. Some of it is inevitable, but too much is again an indication that user requirements were not defined adequately during systems development. The user requirements required the ability to change and the specifications analysis lacked the quality to anticipate this need resulting in undesirable system inflexibility. Perfective maintenance is often referred to at Cornell as enhancements. Often, our enhancement list is longer than the original specifications. This is a combination of user not recognizing needs and analyst not discovering all user needs prior to software release. It is often a sign of an unrealistic implementation schedule, that was too rushed. Finally, preventative maintenance which is the periodic review of the system to uncover or anticipate problems. If your shop is doing mostly preventative maintenance you are probably running a quality environment and have control of your computing. Maintenance will tell you alot about the quality of the IT work. Track maintenance costs by system, by program, by programmer. Measure to number of failures per program. Calculate number of hours spent on maintenance and whether it was corrective, adaptive, perfective, or preventative in nature and emergency, urgent, or routine. Develop a profile of the most common maintenance requests and problems encountered. Look to correct these first in the development process. Quality improvement is incremental improvement. Tools The clear statement of quality requirements in the requirements specification is a major step towards the production of good quality software. Software developers must plan and implement software development projects with the objective of building in quality. A desire to produce a high-quality product must be supported with a willingness to commit resources to the three disciplines needed for the activity: development disciplines (such as analysis, design, and unit testing), product assurance disciplines (such as quality assurance, test and evaluation), and management disciplines (such as project and general management). It seems to be generally agreed that this involves activities in the following areas: 1. Establishment and maintenance of a requirements specification. This also serves for the basis for acceptance tests. 2. Establishment and implementation of a process for developing the software. This would include shop design and programming standards. A methodology. 3. Establishment and maintenance of an evaluation process. This involves the production of standards defining what must be done to complete a task successfully and also how the work should be done. Fifth truth, the biggest single problem encountered in the computing industry is the specification of requirements. Organizations seem to find it exceedly difficult to express what they want in clear and unambiguous terms. The fuzziness particularly is evident where a institution's own administrative function and information are concerned. If the business has problems in this area, computerization is often seen as the way forward. In such cases computerization only succeeds in producing more convincing chaos, not sense. The evidence for this lies in the hundreds of abandoned projects throughout the industry. If the goal or definition of quality is meeting the user's needs than IT will have to more closely align itself with the customer. Cornell has implemented Business Modeling to separate the software development process from the business analysis. Business modeling has help at Cornell better synchronize IT with the business and in many cases better synchronize the user's with their own business. Business modeling involves everyone who has a part to play in an elemental function that is being studied. It breaks down the function to its smallest parts. With everyone having a solid understanding of the business, it is a wonderful opportunity for reengineering and doing a critical study on where and what kind of support information technologies can best provide. The advantage of this, is time is taken to identify what IS the business. It is an opportunity to reengineer and optimize necessary activities and obliterate worthless activities. All this is done prior to thinking about computerization. Quality is in the eyes of the user. To understand what the user values, IT function has to move closer to the business philosophically to understand what's important. Computer people know what they value in a quality system, robustness, maintainability, etc. What they don't know is what the user values. To find this out the user should be asked. A simple software characteristic evaluation form filled out by the user will help communicate user defined quality. Early on communication is key, if for no other quality goal than a satisfactory user perception. Communication must be viewed as a key tool to insuring quality software. Business modeling assists that communication. Early survey sheets from users on what they are seeking and postmortem user survey sheets on satisfaction on prior software products move communication forward. Taking up residence with the user community is also appropriate. Communication is unfortunately often an under utilized tool. Aides to communication such as surveys, e-mail, structured modeling tools (DFD,ER diagrams) are invaluable to the quality effort. An important tool to system/software development is structured development approach. Often referred to as system development life cycle or SDLC or system development methodology. At Cornell, we have our Systems Development Methodology. This methodology describes: * the phases of the system life cycle, * the purpose and goals of each phase, * the items to be delivered and for each deliverable item, who is to prepare it, what it consists of, some idea of the methods and tools available to create it, and the review process by which the item is accepted, * the approval process for each phase, how we know it is completed. Tools to assist this process include modeling tools: data flow diagrams, entity relations diagrams, structure charts, and business function diagrams. These provide several benefits. First they act along with the methodology guidelines as a communication tool between IT and the customer. In many cases these days, automated modeling tools can serve to check for consistency andcompleteness of the model. And finally, the model serves as documentation. A glossary or data dictionary of terms for data elements and other items in the system is a must. Redundant and and inconsistent data definitions may exist throughout an organization's procedure manuals, source program documentation, data files, and in the minds of those in the organization. Ambiguity of what things mean is the makings of software disaster. One can not hope to build quality software, or purchase it, when there is ambiguity of the meaning of the data in the system and how it is used. Standards are of utmost importance to insuring quality. Well trained software specialists know the best practices for analysis, design, programming, implementation, documentation, and maintenance. Quality demands consistency. Consistency is insure by standards. Many systems development groups operate without standards or have standards they do not use. The most common reason for a lack of standards or not following them (though often not admitted to) is that standards inhibit creativity. I haven't met a systems analyst or programmer yet who did not believe they could determine a better way to do a task than the process proposed by the standards manual. When computer professionals are allowed to do this they have performed two jobs instead of one. They have develop the process that they follow AND follow that process to solve the user's needs. Waste. Programming standards come in all shapes and forms. Not everyone with a 4 year computer science degree knows how to program well. Standards can help teach good programming techniques. Modular design and reuseable code allows one to create the best code possibly and use in multiple places. Invest in new automation techniques. Case tools can be used not only to increase programmer productivity, but to institute shop programming standards. CASE generates code faster and reduces code variability. The biggest problem with CASE is sometimes getting IT staff to use it. Most of them got into this business because they enjoyed programming. The other problem though not as obvious, is customers wanting their own signature systems. With CASE tools you get a standard interface. The resulting purchasing system looks like the resulting budget system, etc. Many users through years of IT lacking standards, have gotten as use to creating their own world as the supporting IT staff have. Both sides of the house must be sensitized to the cost of individual interfaces built for functional area preference vs a common interface for the institution. One of the best tools, though often overlooked,for improving quality is staff training. Proper training in quality, programming, analysis and design, modeling, software tools, and communication skills is invaluable. Variability is the enemy. The obvious problem with variability is that the user and other IT staff never know what to expect. Each system operates differently. Maintenance may be easy in one system and difficult in another. It makes user training and new IT staff training a nightmare. And with variability, visible and invisible to the customer, IT credibility for knowing what they are doing suffers. Testing does improve quality, but it is a costly method of accomplishing the quality objective. Testing procedures should exist for all levels of testing: unit, module, integration, systems, and acceptance. It is important to have a good test environment, that mimics the production as closely as possible. All project plans must include adequate testing time. Post- Review and Evaluation are Important Tools The only tangible that matters is dollars. Does the system save more money than it cost to develop (or purchase), maintain, and run. The only intangible that matters is customer satisfaction. Though these are simply they encompass a world of sins. Money, where it is spent, how much is received, or how well it is utilized is just not that easy to track down. But scrutiny of the business, it's inputs and outputs, and an honest look will show you. Taking an honest look however is not easy. Pet projects, pet agendas, and business processes steep in tradition and sometimes mysticism stand in the way. It is IMPORTANT to quantify as many savings and costs as possible. Look for signs. * Staff working less/more hours or Staff reductions/increases * Reduction in costs or budgets * Reduction in identifiable waste * Reduction in paper * Fewer reports or sources of information without completeness or sufficiency of information suffering * Lower stress or anxiety levels among staff and service receivers * More business or higher quality business. In a university that might mean more and better applicants to admission, larger alumni gifts, better faculty. Customer satisfaction is also allusive. Customers sometimes do not know what they want until they see it. They almost always know what they do not want. And alot of the satisfaction depends on the expectation. It is important in addition to systems project management that the IT management manages customer expectations. Customers must be kept abreast of progress all along the way and be the center of system development. This is sometimes difficult. It is not always obvious who the customer is: is it the sponsor, who may never use it such as a VP of Finance, the heads of the function the system serves, such as the heads of budget for a budget system, or is it the clerk who actually USES the software. This is where business partnering becomes the key. Ultimately the clerk is the customer, but it should be kept in mind the VP and functional head has a broader picture of what is trying to be achieved. Both needs must be reconciled for success. This sometimes requires the administration to align their goals across the organization. But this is not a topic for this paper. To determine customer satisfaction...ASK THEM often, repeatedly. Quality of Outsourcing Service With many of our organizations seeking to cut costs by outsourcing all or part of the IT function, I feel a need to make a comment on outsourcing quality. For many of us the outsourced pieces of IT will become an integral thread of our institution's IT function. The same quality guidelines apply. The quality of products and services, cost-effectiveness, timeliness of deliverables, reliability of performance, flexibility, and responsiveness to the customer, and customer needs being met to their satisfaction is still the measures of quality. Project deadlines must be met or penalties imposed. Measures of performance reflecting customer priorities are part of the contract. In short, the quality guidelines of the IT function should apply to vendor supplied software. Software Quality is Only Part of Information Technologies Quality Software does not run in a vacuum nor do customers judge it solely on it's own merits. Unfortunately, this is truly a case of one bad apple can spoil the bushel. Software that runs in a poor quality communications environment is worthless. Bad response time or machine downtime reflects poorly and causes customer dissatisfaction no matter how good the software is. To insure Information Technologies quality one must look at the entire IT picture. Diane Wilson (MIT,1988) identified seven IT assessment methods to evaluate the IT function: Productivity. Efficiency of expenditure of IT resources. User utility. Customer satisfaction and perceived value of IT services. Value chain.. Impact of IT on functional goals. Competitive performance. Comparison against competition with respect to infrastructure components of business measures. Business alignment. Criticality of the organization's operating systems and portfolio of applications to business strategy. Investment targeting. Impact of IT investment on business cost structure, revenue structure, or investment base. Management vision. Senior management's understanding of the strategic value of IT and ability to provide direction for future action. In her research, it was discovered only a third of the organizations studied measured the business value or strategic impact of information technology on the business. The dominant measures were ones of cost reduction, increased productivity, and reduced head count. Although more than 70% of the organizations use surveys to determine user needs, about one-third used formal procedures to assess user satisfaction with IT services. Measurements are also needed to evaluate the value of the system/software in relation to it's performance. What is the strategic value of the software to the business? How does it contribute to the institution's competitiveness? Finally what may be the best expression of the two previous questions, is how satisfied is the end-user? These are the harder, but more important questions. How can these attributes be measured? IT Quality Assessment Attribute Instrument or Metric Organization satisfaction End-User Surveys Meeting Business needs/priorities End-User Surveys Contributions to Business Competitiveness Revenues Gross margin Reduced costs Improved productivity Improved cash flow Cost avoidance-Be careful of this one Improved transaction response time Improved receivables payment Improved departmental performance Impact on end customer (students,etc) Return-on-investment calculations Return-on-asset calculations Impact on products and services Strategic Value to Business Establish competitive barrier Create defendable market position Improve service level Introduce technology-based products Introduce technology-based services Summary I contend quality is simply meeting the user's requirements both expressed and implied for an acceptable cost. It is not an intangible or subjective factor of rightness or good design. IT IS directly measurable through the user's perceived satisfaction with the product or service, and its tangible costs and benefits can be calculated. The wider context of the service offered by the information technology function should be part of the formula, the majority of users do not distinguish between problems cased by the application software, and those which are caused by faults in the hardware or system software. What we need to aspire to is tight performance measurement linked directly to important business consequences. Quality is, above all, about people: a continuing commitment to produce quality software and provide a quality service is needed from people at all levels and in all parts of the IT organization. End-users perceive quality in software products and services which most meet their requirements and continue to do so. Techniques, tools, and procedures can improve software quality, but only if they are deployed in an environment which encourages every person to make a long-term commitment to the achievement of that quality as part of a team committed to quality performance. Of all tools, communication is the most valuable. If communication with users is poor, then the perceived quality is likely to be low. The bottom line. The user will judge the quality of the software, the system, the Information Technologies function.