By: Richard G. Russell
Updated April 2004
Minor Edits, Move to new blog, December 2009
As a software developer and a manager of developers, I often find myself in the position of communicating a project’s strategic goals to implementation teams, management, marketing and customers. This is rarely an easy or straightforward task. Various stake holders in a project generally have different views regarding a project’s overall goals and how to make important judgments and decisions – especially quality. I’ve often found it strange that issues centering on software quality very often take a back seat to requirements, schedule, cost, process, organization, and even look-and-feel.
To counter this, I take a somewhat unconventional approach by framing the entire discussion regarding goals in terms of software quality.
It would be tough to argue that the quality goals for a software project, no matter how small, and the decisions made during its development should always include an appropriate degree of thought and analysis regarding quality requirements. I find it reasonable, even preferable, that the entire framework for defining, managing, measuring, developing, delivering and deploying software should be discussed in the context of quality. This stems from the premise that the definition (or measure) of software quality is much more than just the number of bugs found during QA, or how many support calls are received during BETA. In my view, everything from the number of bugs found during code reviews, the development schedule, and even the end user’s price can be discussed in the context of meeting software quality goals.
Questions and problems quickly arise as soon as this concept is presented. This is especially true when a software development team is larger than a handful of people. Ask any two software engineers about software quality and I’m sure you’ll get at least three opinions. For example, a google search for software quality returns almost 90 million entries 3.31 million entries. This is an increase from 1.8M in early 2000, and 2.1M in early 2001, and 3.31 in early 2004.
The first issue I usually encounter is incoherency over the definition of quality and how it is measured. An executive once said to me “Our software must be of the highest quality”. He was thunderstruck when I told him his statement held no value. He was confident any product developed by ‘his’ company would be of the ‘highest quality’. However, he couldn’t tell me what that meant. I ask him what he was willing to do to achieve such a lofty goal. He was willing to talk (endlessly) about reaching the ‘highest quality’ and how the ‘customer demands quality’. However, he certainly wasn’t willing to let the development team spend an extra six weeks doing code reviews, extend the formal QA cycle, hire qualified technical writers to review and edit the documentation, reduce the projects scope, or buy more lab equipment. He certainly wasn’t willing to write fifty pages of requirements himself – which was our big problem. What he really wanted was as much quality as we could deliver and still hit a release date in time for the next big trade show. He really never cared one wit about quality – delivery date was his only requirement. All he cared about was the perception of quality.
Many discussions on software quality start off with a broad definition of the term “defect” such as the following definition from the Software Engineering Institute at Carnegie Mellon University:
We will define a software defect to be any flaw or imperfection in a software work product or software process. A software work product is any artifact created as part of the software process including computer programs, plans procedures, and associated documentation and data. A software process is a set of activities, methods, practices, and transformations that people use to develop and maintain software work products.
Many people also use the more informal term ‘bug’, a good definition of which is:
A bug is a condition – within a running program – that: is not expected to arise; which can not be recovered from if it does arise; and cannot be ruled out with certainty.
The Carnegie Mellon definition is a good starting point. The technical report from which it is derived establishes a practical defect discovery and tracking framework. However, this broad definition and a good tracking framework do not lay the groundwork with which a development team and management can effectively and practically discuss, define, measure, and ultimately deliver a quality software product. It also does nothing to help provide a framework for making key decisions.
This is because defining software quality and making key decisions can be extremely difficult and often complex – even for seemingly small projects. There is no single view or definition of software quality. This concept is difficult to quantify, and has many overlapping and sometimes conflicting definitions. The goal for a common understanding of software quality often becomes more elusive the bigger a group becomes.
In my experience, ideas on software quality are generally determined by a persons relationship to a software product. A developer will often view quality as the number of bugs and other issues he has to track down and fix during a QA cycle. Executive management’s view on the quality is often determined by the reviews received or the accolades and awards given (or lack thereof). The views on quality by an end user can be determined by many things such as easy of use, consistency of operation, or the number of blue screens of death.
The second problem I run into is that there is little or no consistency in the way decisions, judgments and tradeoffs are made during the definition, architecture, design, development, delivery and deployment phases of a software product. I’ve been on projects where most decisions were made by consensus. If everyone “felt good” about something then that was the way to go. Developing software by consensus is rarely a good thing. Consensus in and of it self isn’t bad, but using it as the sole or primary tool for decision making, direction setting and value setting is simply goofy. I’ve also worked on projects that were run by a dictator – sometimes known as an architect. The problem was that he didn’t effectively leverage experience of other smart people. All the key decisions were made by one guy – not a small team of experienced technical people and leaders. One person is never as smart – or effective – as a cohesive team of crack developers.
In my mind, the way a project is managed, how decisions are made, and the methods used to successfully accomplish its goals are a function of the people on the extended development team. For any software project, its extremely important that everyone involved understand the framework in which: work and communication will take place; decisions will be made; who makes value judgments; the values and measures used to make engineering and business tradeoffs; and the values by which results will be judged. I’ve picked my words very carefully here. I’m not trying to play semantic games or split hairs, but everyone on a project doesn’t have to agree or have buy in on the framework, but they must all clearly and unambiguously understand it. This is particularly true for portions of the framework by which success will be measured and judged. Communicating the framework is the job of the project’s leaders.
To bring coherence to the discussion of software quality, I use a framework that provides a common understanding of software quality among software project stake holders and a framework for making decisions. This framework consists of a single tenet and set of definitions. These are used to frame the discussion, definition, and measures of software quality.
Better, Sooner – the Tenet for Delivering Quality Software
Academic study can be very useful, but when it comes to the task of developing and delivering software, I tend to focus on practical matters. Starting off with a straightforward premise helps keep the discussion focused.
My simple tenet for developing quality software is that a team should focus on developing better software sooner. Sounds simple doesn’t it? It is.
I specifically chose the word “tenet” because its definition focuses on a key item – a group of people:
te·net 'te-n&t also 'tE-n&t: a principle, belief, or doctrine generally held to be true; especially : one held in common by members of an organization, movement, or profession
When discussing software quality, my first goal is for all the participants to buy into the above tenet as the fundamental rule used to build the development and decision making framework. This usually isn’t to hard: It is difficult to disagree with the idea of developing better software, sooner.
My next step – which is usually the hard part – is getting everyone to agree what it means to focus on this strategic tenet. Thinking strategically first and tactically second is vitally important. This helps the team to avoid the tendency to focus too much on what they are doing rather than why they are doing it.
Frequent over emphasis on using Object Oriented techniques is a classic example of focusing on tactics rather than strategy. OOP is simply a tool – it is not a way of life. I’ve personally seen three major programs go in the tank because the team decided that “We should use C++ and objects” or “Everything should be a COM component”. In every case it was because the team violated the above rule with reasoning such as “it would be fun” or “educational” or “we’d like to learn C++” or “it would be good for my career” or “everyone is using C++” or “objects lead to code reuse” or “smart pointers will eliminate the use of free and delete and save us from evil memory leaks” or “Microsoft says COM is the way to go”.
While developers are particularly susceptive to this phenomena, other groups are not immune: Marketing can cause creeping featurism, executives can easily induce schedule madness, and sales guys can focus a team way to much on that ever important customer or trade show demo. All of this generally happens because people loose sight of the above tenet.
My primary goal is to get everyone to agree that all actions, decisions, judgments, and tradeoffs material to delivering a software product must be judged in the context of the accepted tenet. This thinking leads to “RGR’s Iron Rule of Software Development“:
If it doesn’t help to develop better software sooner- don’t do it. Its that simple. The idea is to meet the quality goals on time. Anything that negatively effects quality or schedule is the wrong thing.
This doesn’t mean that a project’s scope, required delivery time frame, or other constraints can’t change – the can and they will. What this means is that the goal is to deliver better software sooner rather than later. Things that move a project closer to the goal are good, and things that move you further away from the goal are bad.
Once people agree to the iron rule, the rest is easy. The next step is defining a framework of definitions with which to establish software quality goals, requirements and measures.
Quality – Just what is Better Software?
The first step in defining a project’s quality goals is determining a project’s quality metrics. However, metrics by themselves are not enough: any measure, even subjective ones, must have a frame of reference. The level, measure, and value of quality must be compared against something. Quality goals are also specific to a project: Different software projects need different quality goals. Each project should have requirements for appropriate degrees of quality, not the “highest quality”. For example, the software for the space shuttle, the core controls of a nuclear reactor, or a remote surgical robot are held to a significantly higher level of quality than more mundane software such as an accounts payable software module, a word processor, the software in your VCR, or even a modern desktop operating system.
This kind of statement is often viewed as heresy – at best a lazy or laissez-faire attitude. However, it is simple reality that the constraints of time, people, scope, and money simply dictate that appropriate quality goals be set for any given project.
This does not mean that quality can be ignored on some projects! What it means is all projects should have a set of quality goals that meet the needs of the end user or customer while living inside its hard constraints. There are often business aspects to quality as well! It may not be enough just to meet customers expectations – it may be necessary to clearly exceed the quality bar set by the competition.
Quality also goes beyond just the technical. The quality discussion must go beyond just the development team to the extended team which includes developers, quality assurance and test folks, engineering management, senior or executive management, marketing, sales, business development, and sometimes even business folks (finance, accounting). All of these groups have a vested interest (whether they know it or not) in understanding – and having some material input regarding – the software quality definitions, goals, and measures for software projects.
Customers or end users of a software project also have a stake – after all, they will depend on the software to deliver some kind of value. The higher the quality of the software, the greater the value. The role of customers in developing a project’s goals typically focus more on making sure their requirements are understood and met rather than on the definitions used by developers. This is in contrast to focusing the extended development team on achieving a common understanding of what quality means for a specific project.
So, how does a team determine the appropriate level of quality for a project? By using the tenet of course! The first thing to do is define a frame of reference. The quality of a software product must be compared to something – it must be better than some alternative. I maintain that a quality goal that is not specified in relative terms isn’t a quality goal at all.
So, what is better software? That is an easy question: better software is software that is better than your competitor’s software, or better than old legacy software, or better than the alternative, or better than the previous version 1.0, or better than buying software outside (buy vs. build), or better than nothing at all. But its got to be better than something. If it is not, then why develop it?
The key idea is: develop some “better than” statements for your software project.
The ‘better than’ statements setup a frame of reference for defining quality goals. Once a frame of reference is establish the discussion can turn to quantifying which quality characteristics are important, how to define the extent to which they will be met, and how to objectively measure the degree to which the characteristics have been met. For this phase of the discussion, I use a set of definitions originally derived from the book “Code Complete” by Steve McConnel. Murray Cantor takes a similar approach in his book “Software Leadership”.
McConnel defined two basic categories of quality characteristics, internal and external. The end users, or customers, for the software are aware of and place value in external characteristics. The extended development team are also aware of, and place value in, a projects internal characteristics.
Things important to the end user or customer
Software’s external quality characteristics directly affect the value of the software (the product) to the end user or customer. The following list is by no means definitive and in no particular order, but it is a good set for discussion:
The money spent by the end user or customer in order to: develop, or have developed, or purchase; deploy; use; and maintain a software system. This is often called Total Cost of Ownership but is probably better measured or discussed using the notion of “economic utility”.
The degree to which the final system delivery schedule was met – or the degree to which the value of a software system is delivered inside the required time frame of the end user or customer.
The degree to which a system implements its planned scope with a particular focus on meeting requirements and delivering features.
The degree to which a system is free from faults in its requirements, scope, specification, architecture, design, implementation and deployment.
A system’s ability to perform its required functions under stated conditions whenever required. Also: having a long mean time between failures
The degree to which a system continues to function in the presence of invalid inputs or stressful environmental conditions.
The ease with which users can learn about and effectively use a system. The the quality of end user documentation and technical support can radically effect this characteristic. This includes traditional documentation, on-line help and web based information.
The measure of resources usage such as: memory, CPU utilization, disk space, network bandwidth, screen real estate and amount of user interaction to complete key tasks (see performance).
Speed or throughput: minimizing the time, or perceived time, between a system’s input events and output events – optimizing or maximizing the amount of useful work done in a given period of time. Note that software can be very fast, but still be a memory or CPU hog (see efficiency).
The ability or suitability for holding, storing, or accommodating data or information. The maximum amount or number of something that can be contained or accommodated. Capacity may be dictated by design, hard coded limits or requirements. It may also be dictated by the operating environment (see scalability).
The degree to which a software system’s capacity, efficiency, or performance is not limited by its design, implementation, the hardware platform on which it runs, other software systems with which it interoperates or communicates.
The extent to which a software system will function or communicate correctly, reliably and robustly with other similar systems that share the same data types, file formats, or user interfaces. Backward compatibility specifically applies to a software systems’ ability to work with previously versions from which it was derived or with versions ported to other systems.
The extent to which a software system will function or communicate correctly, reliably and robustly with other system using externally defined interfaces (hardware or software) or communications protocols.
The degree to which a system prevents unauthorized or improper access or modification to its code and data or other system resources and/or the degree to which it ensures that data or object state is maintained in a coherent and correct manner. The idea of integrity includes restricting unauthorized user access as well as ensuring that data is accessed properly by its intended users and other software.
The degree to which a system can be used, without modification, in applications or environments other than those for which it was specifically designed.
The degree to which a system’s operational parameters or physical makeup can be changed by the user.
The degree to which a system, as built, is free from error, especially with respect to quantitative outputs. Accuracy differs from correctness; it is a determination of how well a system does the job it is designed for rather than whether it was implemented correctly (see correctness).
The degree to which a system will repeatedly produce the same results given a consistent set of inputs and a consistent operating environment. Sometimes called Reproducibility.
The ease with which a software product can be reliably built from its individual components. Typically this focuses on people other than the original developer. However, it also applies to the scenario where the original developer has not built the system for a extended period of time. The use of SCM tools typically focuses on this characteristic. The term reliably is important: it implies that the build system is repeatable, dependable, timely and that when given the same inputs it will always build the same thing.
Some of these characteristics are similar, but all have different shades of meaning which help us define goals, distinguish between competing priorities, and make trade off decisions. For example, compatibility and interoperability are similar concepts that have fundamental differences. A software program can be compatible with another program, or a previous version but may not interoperate at all with other related software or components.
It is important to note that all of these external quality characteristics share an important trait: they can all be objectively measured, verified and tested. They are generally what the testing department looks at. Most bugs (or defects) are a result of a software system failing in one or more of these external categories.
All of these attributes can have quantitative measures. Performance is a good example. Metrics might included “bytes per second transferred” or “transactions per second”. It might simply be a requirement that a networking application handle a given amount of connections per second. For example, the statement “the sort component must be able to sort all the records in the database in 20 seconds or less” may be a requirement, but it is not a quality goal. Here is a way to restate this as a quality statement: “the sort component must be able to sort all the records in the database at least 15% faster than V1.0. The minimum bar is 20 seconds on the reference hardware”.
While they are all quantifiable, some of these characteristics are only measurable and quantifiable in a relative context. This is where the “better than” statements come into play. Take configurability for example. There is no absolute measure for configurability, so this characteristic must be measured comparatively. A better than statement might be made like this “V2.0 of IDE must be significantly more configurable that our competition”. The development team would translate this to some objective requirements such as:
- The user must be able to select any third party programmer’s editor
- Tab sizes must be settable by file type
- The user must be able to switch between multiple pre-defined include paths
- The user must be able to select 64-bit or 32-bit targets for each project
All of these things add up to better configurability and can be directly measured and tested.
Internal Qualty Characteristics
The quality with which a software system is built has a direct bearing on the the development team’s ability to meet a projects external quality requirements, development cost targets and schedule. As with the list of external quality characteristics, some of these internal characteristics have similar degrees of meaning, but their differences are significant enough to be valuable:
The ease with which a developer can modify a software system to change or add capabilities, improve performance or efficiency, or correct defects without adversely affecting other internal or external quality characteristics.
The extent to which a developer can modify a software system for uses or environments other than those for which it was specifically designed without adversely affecting other internal or external quality characteristics.
The degree to which a system, or a system’s components, can be used in an operating environment different from that for which it was originally designed or developed without adversely affecting other quality characteristics. There are two types of portability – run time and compile time.
The extent to which, and the ease with which, a developer can use parts of a system in other systems without adversely affecting the internal or external quality characteristics of the target system or the reused elements.
The ease with which a developer can read and understand the source code and technical documentation of a system, especially at the detailed source code statement level.
The degree to which someone can unit-test, system test and functionally test a software system. This idea also extends to the ease with which a test plan can be developed from the projects requirements.
The ease with which someone can comprehend a software system at both the system-organizational and detailed-statement levels. Understandability has to do with the coherence and cohesiveness of the system at a more general level than readability. Understanding includes not only understanding what the system does, but why it does it. Good detailed design documents can greatly enhance a systems understandability.
Many people have made statements like “if you can’t measure it, then it doesn’t exist”. Well, that’s a mighty fine academic sentiment, but its not very helpful from a practical standpoint. Unlike external quality characteristics, internal characteristics are much harder to measure. However, just because they are difficult (or impossible) to quantify doesn’t mean they should be ignored or marginalized. Ignoring them is foolish – all of these ‘ilities’ are real and tangible characteristics of components (especially source code) that make up a software product.
In contrast to external characteristics, internal characteristics are subjective attributes; Defining truly objective measures for these items is difficult or impossible. Take “understandability” – we all know it when we see it, but how do you put a number on it? Portability is another characteristic that is difficult to quantify; Someone once pointed out to me that portability is indeed very measurable – “Its either portable or it isn’t” was declared. This notion is simply wrong: Software portability means a lot of things. For example, a piece of software could be portable among many Microsoft Operating systems including Windows 95, Windows 98, Windows Millennium Edition, Windows NT 4.0, Windows 2000, Windows XP. I would agree such software is quite portable… in a 32-bit x86 Microsoft environment. But how portable would it be to 16-bit versions of Windows, such as Windows 3.11 or to MS-DOS? What about portability to 64-bit versions of Windows? What about portability to alien operating systems such as Linux, Unix, Solaris, Symbian, Plam-OS, Windows CE, VxWorks, BE-OS, VMS, QNX, OS-X, or Multix? Or, how portable would it be to a non x86 processor environment such as ARM, PowerPC, EPIC, MIPS, SuperH, TriCore, or Sparc?
I’m sure somebody could come up with metrics for some of these internal characteristics, but how much value would they have? Would they be truly reproducible, objective, quantifiable or absolute? For example, is it possible to create a tool that measures source code understandability? While it might be a good doctoral project for someone, I don’t believe that putting metrics on these characteristics would be time well spent.
So, how do does a team evaluate these characteristics? How does a team know when the software is portable enough, easy to build, that the code is testable or that the comments and code are appropriate?
There is a good answer: How these attributes are valued, and the degree to which a component meets these values, must determined by:
peer review and the application of experienced judgment
Entire libraries of books have been written regarding how teams should develop software. Reams of material have been developed on peer reviews – how to achieve “good code” is beyond the scope of this essay. Once again, a simple concept gives us an effective framework for handling a complicated problem: judging the “goodness” or the “level” of internal quality. This task is the responsibility of senior development team members and leaders. They must work as a team to balance the need for good internal quality and delivering the product. These judgments and peer reviews don’t happen at a specific point in a project: Judgment must be applied on a daily basis from the beginning of the project to the end and peer reviews must be handled in a constructive manner that is an inherent part of the definition, analysis, architectural development, design, implementation, testing and deployment of a software project.
Setting Quality Goals
The development team is responsible for meeting a project’s quality goals. But, who defines these goals? In my mind, there are three areas of responsibility that must be accounted for: technological, business, and the customer (or end user). In a typical organization, this responsibilities might be represented respectively by: a technical lead, architect, or senior engineer; a marketing or product manager; and a senior sales person or even the customers themselves.
However represented, all three of these areas must be appropriately considered when defining a project’s quality goals. This is true even for small projects such as small internal development utilities. Do you need to consult marketing when writing a small development utility? Usually not. But – even for seemingly small things – you should take technological, business and customer related perspectives into account. Even if you decide that a program really is a throw away utility, by taking the time to think about it, you have made sure you are doing the right thing. This doesn’t take time consuming analysis. Often just a little thinking or consulting with a colleague can help shape your decision. For example, you may discover that the QA department would love such a utility. This might lead you to take a couple of extra hours to write a decent how-to, add a some extra reporting capability, or provide an online help file.
Meeting some level of quality is a requirement for any software project – no matter how small. The first step in any project must be to define the projects quality goals. Under this framework, the quality goals include scope, requirements, and schedule in addition to the common approach of minimizing or eliminating defects. An essential part of this process is instilling a common understanding of software quality among software project stake holders and establishing a quality based framework for making decisions This framework provides those definitions and a framework used to guide the decision making process. Having a common set of quality definitions and a framework for making decisions is essential for setting the goals, meeting them and judging the results.
After the quality framework is in place, the team can focus on the processes, methodologies and techniques used to scope, architect, design, implement, test, and deploy the software. The team must always keep in mind that all goals are defined in terms of quality and that the tenant described here is used to guide all decisions – no matter how small.
I hope you have found my essay helpful. I would certainly appreciate your feedback. Just email me!
- Joel on Software
- Painless Software Management: this is a great site and is full of insightful articles on software engineering.
- A Software Engineering Glossary
- From CERN: a useful glossary with many hyperlinks.
- Three questions you should ask about each bug you find
- By Tom Van Vleck: a very practical essay on bugs. Be sure to read his other excellent essays.
(I apologize for the missing and out of order footnotes. HTML editors don’t handle footnotes well and I haven’t had time to renumber them)
 (go back) To give credit where credit is due, I must refer to the book “Code Complete” by Steve McConnell’s. The basis for these definitions came from Chapter 23 of this book which is a most excellent book and I recommend it highly.
Code Complete, A Practical Handbook of Software Construction; Steve McConnel, 1993 Microsoft Press. ISBN 1-55615-484-4 (amazon)
(go back) On Price, Cost and Value:
The idea here is that the cost of ownership and use, combined into the total cost of ownership (a very overused phrase), should be appropriate to the utility of the software to the end user or customer. The main levers that the extended implementation team has to control this is time and resources. The less time and resources spent on a piece of software, the lower the cost to the end user can be. Note I say “can be” not “will be”. For example, the price of a commercial software program includes some amount of margin (profit) to the seller. The margin is generally set by the executive management and/or sales. This the price of the software equals the companies cost to develop and produce it added to the amount of total margin required by management.
 (go back) Software Quality Measurement: A Framework for Counting Problems and Defects, William A. Florac, Software Engineering Institute Carnegie Mellon University, Technical Report CMU/SEI-92-TR-22 ESC-TR-92-022, September 1992
 (go back) The IEEE Standard Glossary of Software Engineering Terminology (IEEE Std 610.12-1990) defines the following terms:
- “Robustness. The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.”
- “Error tolerance. The ability of a system or component to continue normal operation despite the presence of erroneous inputs.”
 (go back) I hate the phrase “work product” too. I just don’t have a better phrase for it. Management science sometimes tries way to hard to quantify people – this is inherently difficult, maybe impossible, where tasks are dependent on someone’s intellectual capacity and ability. Don’t get me wrong, there is definitely a strong scientific and engineering aspect to software development – there is also an inescapable human element. Management often forgets this.
I wrote this document primarily to clarify my thinking. I put this on the web with the desire that others would read it and provide feedback – publishing this on the web seems very appropriate given today’s development culture. I hope you find this essay valuable.
I completed the first version in January of 2001 but I wasn’t satisfied with it. I’ve been working on it off and on since then.
It is important to note that I specifically call this document an ‘essay’ instead of a paper, treatise, or article as it consists primarily of my thoughts and musings on this topic. An essay is a literary composition of moderate length, dealing in an easy, cursory way with a single subject, usually representing the writer’s personal experience and outlook.
I also intentionally developed this document using very simply HTML… this allows it to be printed, or easily saved as a single document. Things such as pop-ups, frames or other fancy navigation mechanisms, while nifty, make it more difficult to do these things.
Copyright © by Richard G. Russell, All Rights Reserved