INTRODUCTION
There are millions of software projects in the world, and thousands of software technologies available.  This means that research into topics that affect software project outcomes is of necessity a complicated issue.  By concentrating on the extreme ends of possible results, it is easier to see the root causes of success and failure.  Projects that set records for productivity and quality are at one end of the scale.  Projects that are cancelled or have problems severe enough for litigation are at the other end of the scale. This article concentrates on “worst practices” or the factors that most often lead to failure and litigation.

For the purposes of this article, software “failures” are defined as software projects which met any of these attributes:

1. Termination of the project due to cost or schedule overruns.
2. Schedule or cost overruns in excess of 50% of initial estimates
3. Applications which, upon deployment, fail to operate safely.
4. Law suits brought by clients for contractual non-compliance.

Although there are many factors associated with schedule delays and project cancellations, the failures that end up in court always seem to have five major deficiencies:

1. Accurate estimates were either not prepared or were rejected.
2. Accurate estimates were not supported by objective benchmarks
3. Change control was not handled effectively.
4. Quality control was inadequate.
5. Progress tracking did not reveal the true status of the project.

Let us consider each of these topics in turn.

PROBLEM 1: ESTIMATING ERRORS AND ESTIMATE REJECTION
Although cost estimating is difficult there are a number of commercial software cost estimating tools that do a capable job:  COCOMO II, KnowledgePlan, True Price, SEER, SLIM, CostXpert, and Software Risk Master ™ (SRM) are examples available in the United States.

However just because an accurate estimate can be produced using a commercial estimating tool that does not mean that clients or executives will accept it.  In fact from information presented during litigation, about half of the cases did not produce accurate estimates at all and did not use estimating tools.  Manual estimates tend towards optimism or predicting shorter schedules and lower costs than actually occur.

Based on 50 samples of each, manual estimates and parametric estimates produced similar results below 250 function points.  However as sizes increased manual estimates  became progressively optimistic and understated both costs and schedules by more than 25% above 5,000 function points.

PROBLEM 2: MISSING DEFENSIBLE OBJECTIVE BENCHMARKS
Somewhat surprisingly, the other half of the cases in litigation had accurate estimates, but they were rejected and replaced by arbitrary forced “estimates” based on business needs rather than team abilities.  These pseudo-estimates were not produced using parametric estimation tools but were arbitrary schedule demands by clients or top executives based on business needs.

The main reason that the original accurate parametric estimates were rejected and replaced was the absence of supporting historical benchmark data.  Without history, even accurate estimates may not be convincing. A lack of solid historical data makes project managers, executives, and clients blind to the realities of software development.

Suppose you are a project manager responsible for a kind of software project which no company in the world has ever been able to build in less than 36 calendar months.  As a responsible manager, you develop a careful estimate and critical path analysis, and tell the client and your own executives that you think the project will require 36 to 38 months for completion.

What will often occur is an arbitrary rejection of your plan, and a directive by either the client or by your own executives to “finish this project in 18 months.”  The project in question will usually be a disaster, it will certainly run late, and from the day you receive the directive the project is essentially doomed.

A situation such as this was one of the contributing factors to the long delay in opening the Denver Airport.  Estimates for the length of time to complete and debug the very complex baggage handling software were not believed, according to the article on “Software’s Chronic Crisis” in the September 1994 issue of Scientific American Magazine by T. Wayt Gibbs.

For more than 60 years the software industry lacked a solid empirical foundation of measured results that was available to the public.  Thus almost every major software project is subject to arbitrary and sometimes irrational schedule and cost constraints.  However the International Software Benchmarking Standards Group (ISBSG), a non-profit organization, has started to improve this situation by offering schedule, effort, and cost benchmark reports to the general public.  This data is available in both CD and paper form.  Currently more than 5,000 projects are available, and new projects are added at a rate of perhaps 500 per year.   Other companies such as Namcook Analytics LLC, Reifer Consulting, and the Quality and Productivity Management Group also provide quantitative benchmarks.  (Samples of Namcook Analytics quantitative data can be found on our blog and web site.)

There are other collections of software benchmark data, such as those gathered by the Galorath Associates, by Gartner Group, by the David’s Consulting Group, by Price Systems, by Software Productivity Research (SPR), by Process Fusion, and by other companies too.

However, much of this benchmark data is usually made available only on a subscription basis to specific clients of the organizations.  The ISBSG data, by contrast, is available to the general public.  The Reifer data is also available commercially.

Note: in September of 2013 a joint benchmark report was published by Peter Hill of ISBSG, Capers Jones of Namcook Analytics, and by Don Reifer of Reifer Consulting.  The title of the report is “The Impact of Software Size on Productivity.”   This new report is available from the ISBSG web site, www.ISBSG.org. 

The report deals with the overall impact of applications size from below 100 function points up to about 100,000 function points.  IT applications, systems software, web applications, and other types of projects are discussed.

PROBLEM 3: RAPIDLY CHANGING REQUIREMENTS
The average rate at which software requirements change is has been measured to range between about 1% per calendar month and as high as 4% per calendar month.  Thus for a project with a 12 month schedule, more than 10% of the features in the final delivery will not have been defined during the requirements phase.  For a 36 month project, almost a third of the features and functions may have come in as afterthoughts.

These are only average results.  The author has observed a three-year project where the delivered product exceeded the functions in the initial requirements by about 289%.  A Canadian lawsuit dealt with a project that doubled its size in function points due to requirements creep.  A recent arbitration in 2011 in Hong Kong dealt with a project that went from 15,000 to more than 20,000 function points at a rate of change that approached 5% per calendar month.

It is of some importance to the software industry that the rate at which requirements creep or grow can now be measured directly by means of the function point metric.  This explains why function point metrics are now starting to become the basis of software contracts and outsource agreements.  Indeed the governments of Brazil and South Korea now require function point metrics for all government software contracts.

The current state of the art for dealing with changing requirements includes the following:

• Effective mapping of business needs to the proposed applications
• Estimating the number and rate of development changes before starting
• Using function point metrics to quantify changes
• Using high-speed function point sizing on all changes
• A joint client/development change control board or designated domain experts
• Model-based requirements methodologies
• Running text-based static analysis tools against text requirements
• Calculating the FOG and Flesch readability indices of requirements
• Full time involvement by user representatives for Agile projects
• Use of joint application design (JAD) to minimize downstream changes
• Training in requirements engineering for business analysts and designers
• Use of formal requirements inspections to minimize downstream changes
• Use of formal prototypes to minimize downstream changes
• Planned usage of iterative development to accommodate changes
• Formal review of all change requests
• Revised cost and schedule estimates for all changes > 10 function points
• Prioritization of change requests in terms of business impact
• Formal assignment of change requests to specific releases
• Use of automated change control tools with cross-reference capabilities

Unfortunately in projects where litigation occurred, requirements changes were numerous but their effects were not properly integrated into cost, schedule, and quality estimates.  As a result, unplanned slippages and overruns occurred.

In several cases, the requirements changes had not been formally included in the contracts for development, and the clients refused to pay for changes that substantially affected the scope of the projects.  One case involved 82 changes that totaled to more than 2,000 function points or about 20% of the original size of the initial requirements.  Although the contract did include clauses for funding “out of scope” changes, the defendant asserted that the 82 changes were merely refinements rather than changes.  It is obvious that contracts need to be very specific about what constitutes “change.”

Since the defect potentials for changing requirements are larger than for the original requirements by about 10%, and since defect removal efficiency for changing requirements is lower by about 5%, projects with large volumes of changing requirements also have severe quality problems, which are usually invisible until testing begins.  When testing begins, the project is in serious trouble because it is too late to bring the schedule and cost overruns under control.

One of the observed byproducts of the usage of formal joint application design JAD sessions is a reduction in downstream requirements changes.  Rather than having unplanned requirements surface at a rate of 1% to 4% per month, studies of JAD by IBM and other companies have indicated that unplanned requirements changes often drop below 0.5% per month due to the effectiveness of the JAD technique.

Prototypes are also helpful in reducing the rates of downstream requirements changes.  Normally key screens, inputs, and outputs are prototyped so users have some “hands on” experience with what the completed application will look like.

The Agile method of having a full-time user representative attached to the project is also valuable, if this is possible.  However for very large projects with perhaps millions of potential users (such as Microsoft Windows 8) one user cannot speak for every user.  Therefore truly representative customer involvement is feasible only for projects with a fairly small number of users who are likely to utilize the application in similar ways.

Several of the new model-based requirements methods are very successful in eliminating requirements defects.  Some are so fast that they outpace requirements creep.

A new kind of static analysis tool that finds errors in text requirements is also a useful approach to minimizing requirements issues. This kind of tool can be paired with older tools that calculate the readability indexes of text documents; i.e. the FOG and Flesch readability indices.

New kinds of rapid-sizing tools such as Software Risk Master™ (SRM) can now predict both function points and logical code statements in about 90 seconds which is fast enough to allow requirements changes to be sized immediately without lengthy delays.

Requirements changes will always occur for large systems.  It is not possible to freeze the requirements of any real-world application and it is naïve to think this can occur.  Therefore leading companies are ready and able to deal with changes, and do not let them become impediments to progress.  For projects developed under contract, the contract itself must include unambiguous language for dealing with changes.

PROBLEM 4: POOR QUALITY CONTROL
It is dismaying to observe the fact that one of the most effective technologies in all of software is never used on projects that turn out to be disasters and end up in court.  Formal design and code inspections have a 50 year history of successful deployment on large and complex software systems.  All “best in class” software producers utilize software inspections.  The measured defect removal efficiency of inspections is more than twice that of most forms of software testing (i.e. about 65% for inspections versus 30% for most kinds of testing).

(The term “best in class” is subjective.  In this article and other studies by the author, it refers to projects that are in the top 15% of all projects measured in terms of quality, schedules, and productivity rates at the same time.)

Effective software quality control is the most important single factor that separates successful projects from delays and disasters.  The reason for this is because finding and fixing bugs is the most expensive cost element for large systems, and takes more time than any other activity.

Successful quality control involves defect prevention, defect removal, and defect measurement activities.  The phrase “defect prevention” includes all activities that minimize the probability of creating an error or defect in the first place.  Examples of defect prevention activities include the Six-Sigma approach, joint application design (JAD) for gathering requirements, usage formal design methods, usage of structured coding techniques, and usage of libraries of proven reusable material.

The phrase “defect removal” includes all activities that can find errors or defects in any kind of deliverable.  Examples of defect removal activities include requirements inspections, design inspections, document inspections, code inspections, automated static analysis of code, complexity analysis, and all kinds of testing.

Some methods can operate in both defect prevention and defect removals domains simultaneously.  The most notable example of a method that is effective in both defect prevention and defect removal roles is that of formal design and code inspections.   Inspections are the top-ranked defect removal method in terms of efficiency.  Also, participation in formal inspections is one of the top methods for defect prevention.  After participation in several design and code inspections, participants spontaneously avoid the kinds of problems that were encountered.  The net effect of inspections in terms of defect prevention is a reduction of about 50% of potential defects.

Both “defect potentials” and “defect removal efficiency” should be measured for every project.  The “defect potentials” are the sum of all classes of defects; i.e. defects found in requirements, design, source code, user documents, and “bad fixes” or secondary defects.  It would be desirable to include defects in test cases too, since there may be more defects in test libraries than in the applications being tested.

The phrase “defect removal efficiency” refers to the percentage of defects found before delivery of the software to its actual clients or users.  If the development team finds 900 defects and the users find 100 defects in a standard time period after release (normally 90 days) then it is obvious that the defect removal efficiency is 90%.

The author strongly recommends that defect removal efficiency levels be included in software outsource and development contracts, with 95% being a proposed acceptable level of defect removal efficiency. For medical devices and weapons systems a higher rate of about 98% defect removal efficiency should be written in to the contracts.

(The U.S. average is only about 85% so a rate of 95% is a significant improvement.  For some mission-critical applications, a higher level such as 98% might be used, but it is technically challenging to achieve such high levels of defect removal efficiency.  Formal inspections and pre-test static analysis plus at least 8 forms of testing are needed to top 98% in defect removal efficiency.

Following are some of the methods for defect prevention, pre-test defect removal, and testing that are associated with high quality levels and high defect removal efficiency levels:

Defect Prevention

• Joint application design (JAD) for gathering requirements
• Thorough analysis of business and technical needs
• Formal architectural analysis before starting design
• Formal design methods
• Structured coding methods
• Formal defect and quality estimation
• Formal test plans
• Formal test case construction
• Participation in formal inspections
• Formal change management methods
• Security analysis for the application
• Six-Sigma approaches (customized for software)
• Utilization or the Software Engineering Institute’s capability maturity model (CMM) or (CMMI).
• Utilization of the new team and personal software processes (TSP, PSP)
• Utilization of Quality Function Deployment (QFD)

Pre-Test Defect Removal

• Requirements inspections
• Requirements static analysis
• Design inspections
• Document inspections (user’s guides, tutorials, etc.)
• Text static analysis
• Code static analysis
• Code inspections
• Test plan and test case inspection
• Defect repair inspection
• Software quality assurance reviews

Test Defect Removal

• Unit testing (manual and automated)
• Component testing
• New function testing
• Regression testing (manual and automated)
• Performance testing
• Refactoring
• Usability testing
• Security testing
• System testing
• External Beta testing
• Acceptance testing

The combination of defect prevention and defect removal activities leads to some very significant differences in the overall numbers of software defects compared between successful and unsuccessful projects.

For projects in the 10,000 function point range, the successful ones accumulate development totals of around 4.0 defects per function point and remove about 98% of them before delivery to customers.  In other words, the number of delivered defects is about 0.2 defects per function point or 800 total latent defects.  Of these about 10% or 80 would be fairly serious defects.  The rest would be minor or cosmetic defects.  Stabilization or the number of calendar months to achieve safe operation of the application would be about 2.5 months.

By contrast, the unsuccessful projects accumulate development totals of around 6.0 defects per function point and remove only about 85% of them before delivery.  The number of delivered defects is about 0.9 defects per function point or 9,000 total latent defects.  Of these about 15% or 1,350 would be fairly serious defects.  This large number of latent defects after delivery is very troubling for users.  The large number of delivered defects is also a frequent cause of litigation.  Stabilization or the number of calendar month to achieve safe operation of the application might stretch out to 18 months or more.

If these low-quality applications contain “error prone modules” with very high defect densities, stabilization may be impossible.  Error-prone modules are often so complex and difficult to fix safely that they may need surgical removal and complete replacement before stable operation is possible.

Unsuccessful projects typically omit design and code inspections and static analysis, and depend purely on testing.  The omission of up-front inspections causes four serious problems:  1) The large number of defects still present when testing begins slows down the project to a standstill;  2) The “bad fix” injection rate for projects without inspections is alarmingly high; 3) The overall defect removal efficiency associated with only testing is not sufficient to achieve defect removal rates higher than about 85%; 4) Applications that bypass both inspections and static analysis have a strong tendency to include error-prone modules.

(Studies by IBM and other leading companies noted that bugs are not randomly distributed in large systems.  They tend to clump in a small number of very buggy modules.  In one case, there were 425 modules in a major data base application.  Of these 300 were zero-defect modules with no customer-reported bugs.  About 57% of the entire volume of reported defects were against only 31 modules out of the total of 425.)

PROBLEM 5: POOR SOFTWARE MILESTONE TRACKING
Readers of this article who work for the Department of Defense or for a defense contractor will note that the “earned value” approach is only cited in passing.  There are several reasons for this.  First, none of the lawsuits where the author was an expert witness involved defense projects so the earned-value method was not utilized.  Second, although the earned-value method is common in the defense community, its usage among civilian projects including outsourced projects is very rare.  Third, empirical data on the effectiveness of the earned-value approach is sparse.  A number of defense projects that used earned-value methods have run late and been over budget.  There are features of the earned-value method that would seem to improve both project estimating and project tracking, but empirical results are sparse.

Once a software project is underway, there are no fixed and reliable guidelines for judging its rate of progress.  The civilian software industry has long utilized ad hoc milestones such as completion of design or completion of coding.  However, these milestones are notoriously unreliable.

Tracking software projects requires dealing with two separate issues:  1) Achieving specific and tangible milestones; 2) Expending resources and funds within specific budgeted amounts.

Because software milestones and costs are affected by requirements changes and “scope creep” it is important to measure the increase in size of requirements changes, when they affect function point totals.  However there are also requirements changes that do not affect function point totals, which are termed “requirements churn.”  Both creep and churn occur at random intervals.  Churn is harder to measure than creep and is often measured via “backfiring” or mathematical conversion between source code statements and function point metrics.

As of 2013 there are automated tools available that can assist project managers in recording the kinds of vital information needed for milestone reports.  These tools can record schedules, resources, size changes, and also issues or problems.

For an industry now more than 60 years of age, it is somewhat surprising that there is no general or universal set of project milestones for indicating tangible progress.  From the author’s assessment and baseline studies, following are some representative milestones that have shown practical value.

Note that these milestones assume an explicit and formal review or inspection connected with the construction of every major software deliverable.  Formal reviews and inspections have the highest defect removal efficiency levels of any known kind of quality control activity, and are characteristics of “best in class” organizations.

Table 1:  Representative Tracking Milestones for Large Software Projects

1. Application sizing completed using both function points and code statements
2. Application risk predictions completed
3. Application size and risk predictions reviewed
4. Requirements document completed
5. Requirements document inspection completed
6. Initial cost estimate completed
7. Initial cost estimate review completed
8. Development plan completed
9. Development plan review completed
10. Cost tracking system initialized
11. Defect tracking system initialized
12. Prototype completed
13. Prototype review completed
14. Complexity analysis of base system (for enhancement projects)
15. Code restructuring of base system (for enhancement projects)
16. Functional specification completed
17. Functional specification review completed
18. Data specification completed
19. Data specification review completed
20. Logic specification completed
21. Logic specification review completed
22. Quality control plan completed
23. Quality control plan review completed
24. Change control plan completed
25. Change control plan review completed
26. Security plan completed
27. Security plan review completed
28. User information plan completed
29. User information plan review completed
30. Code for specific modules completed
31. Code inspection for specific modules completed
32. Code for specific modules unit tested
33. Test plan completed
34. Test plan review completed
35. Test cases for specific test stage completed
36. Test case inspection for specific test stage completed
37. Test stage completed
38. Test stage review completed
39. Integration for specific build completed
40. Integration review for specific build completed
41. User information completed
42. User information review completed
43. Quality assurance sign off completed
44. Delivery to beta test clients completed
45. Delivery to clients completed

The most important aspect of table 1 is that every milestone is based on completing a review, inspection, or test.  Just finishing up a document or writing code should not be considered a milestone unless the deliverables have been reviewed, inspected, or tested.

In the litigation where the author worked as an expert witness, these criteria were not met.  Milestones were very informal and consisted primarily of calendar dates, without any validation of the materials themselves.

Also, the format and structure of the milestone reports were inadequate.  At the top of every milestone report problems and issues or “red flag” items should be highlighted and discussed first.

During depositions and review of court documents, it was noted that software engineering personnel and many managers were aware of the problems that later triggered the delays, cost overruns, quality problems, and litigation.  At the lowest levels, these problems were often included in weekly status reports or discussed at team meetings.  But for the higher-level milestone and tracking reports that reached clients and executives, the hazardous issues were either omitted or glossed over.

A suggested format for monthly progress tracking reports delivered to clients and higher management would include these sections:

Suggested Format for Monthly Status Reports for Software Projects

1.  Status of last months “red flag” problems
2. New “red flag” problems noted this month

1. Change requests processed this month versus change requests predicted
2. Change requests predicted for next month
3. Size in function points for this months change requests
4. Size in function points predicted for next month’s change requests

1. Schedule impacts of this month’s change requests
2. Cost impacts of this month’s change requests
3. Quality impacts of this month’s change requests

1. Defects found this month versus defects predicted
2. Defects predicted for next month

 

1. Costs expended this month versus costs predicted
2. Costs predicted for next month

1. Deliverables completed this month versus deliverables predicted
2. Deliverables predicted for next month

Although the suggested format somewhat resembles the items calculated using the earned value method, this format deals explicitly with the impact of change requests and also uses function point metrics for expressing costs and quality data.

An interesting question is the frequency with which milestone progress should be reported.  The most common reporting frequency is monthly, although exception reports can be filed at any time that it is suspected that something has occurred that can cause perturbations.  For example, serious illness of key project personnel or resignation of key personnel might very well affect project milestone completions and this kind of situation cannot be anticipated.

It might be thought that monthly reports are too far apart for small projects that only last six months or less in total.  For small projects weekly reports might be preferred.  However, small projects usually do not get into serious trouble with cost and schedule overruns, whereas large projects almost always get in trouble with cost and schedule overruns.  This article concentrates on the issues associated with large projects.  In the litigation where the author has been an expert witness, every project under litigation except one was larger than 10,000 function points in size.

The simultaneous deployment of software sizing tools, estimating tools, planning tools, and methodology management tools can provide fairly unambiguous points in the development cycle that allow progress to be judged more or less effectively.  For example, software sizing technology can now predict the sizes of both specifications and the volume of source code needed.  Defect estimating tools can predict the numbers of bugs or errors that might be encountered and discovered.  Although such milestones are not perfect, they are better than the former approaches.

Project management is responsible for establishing milestones, monitoring their completion, and reporting truthfully on whether the milestones were successfully completed or encountered problems.  When serious problems are encountered, it is necessary to correct the problems before reporting that the milestone has been completed.

Failing or delayed projects usually lack of serious milestone tracking.  Activities are often reported as finished while work was still on going.  Milestones on failing projects are usually dates on a calendar rather than completion and review of actual deliverables.

Delivering documents or code segments that are incomplete, contain errors, and cannot support downstream development work is not the way milestones are used by industry leaders.

Another aspect of milestone tracking among industry leaders is what happens when problems are reported or delays occur.  The reaction is strong and immediate:  corrective actions are planned, task forces assigned, and correction begins to occur.  Among laggards, on the other hand, problem reports may be ignored and very seldom do corrective actions occur.

In more than a dozen legal cases involving projects that failed or were never able to operate successfully, project tracking was inadequate in every case.  Problems were either ignored or brushed aside, rather than being addressed and solved.

Because milestone tracking occurs throughout software development, it is the last line of defense against project failures and delays.  Milestones should be established formally, and should be based on reviews, inspections, and tests of deliverables.  Milestones should not be the dates that deliverables more or less were finished.  Milestones should reflect the dates that finished deliverables were validated by means of inspections, testing, and quality assurance review.

An interesting form of project tracking has been developed by the Shoulders Corporation for keeping track of object-oriented projects.  This method uses a 3-Dimensional model of software objects and classes using Styrofoam balls of various sizes that are connected by dowels to create a kind of mobile.   The overall structure is kept in a visible location viewable by as many team members as possible. The mobile makes the status instantly visible to all viewers.  Color coded ribbons indicate status of each component, with different colors indicated design complete, code complete, documentation complete, and testing complete (gold).  There are also ribbons for possible problems or delays.  This method provides almost instantaneous visibility of overall project status.  The same method has been automated using a 3-D modeling package, but the physical structures are easier to see and have proven more useful on actual projects.  The Shoulders Corporation method condenses a great deal of important information into a single visual representation that non-technical staff can readily understand.

SUMMARY AND RESULTS
Successful software projects can result from nothing more than avoiding the more serious mistakes that lead to disaster:  1) Look at the actual benchmark results of similar projects; 2) Make planning and estimating formal activities; 3) Plan for and control creeping requirements;  4) Use formal inspections as milestones for tracking project progress;  5) Include pre-test static analysis and inspections in quality control; 6) Collect accurate measurement data during your current project, to use with future projects.

Overcoming the risks shown here is largely a matter of opposites, or doing the reverse of what the risk indicates.  Thus a well-formed software project will create accurate estimates derived from empirical data and supported by automated tools for handling the critical path issues.  Such estimates will be based on the actual capabilities of the development team, and will not be arbitrary creations derived without any rigor.  The plans will specifically address the critical issues of change requests and quality control.  In addition, monthly progress reports will also deal with these critical issues.  Accurate progress reports are the last line of defense against failures.

SUGGESTED READINGS

Charette, Bob; Software Engineering Risk Analysis and Management; McGraw Hill, New York, NY; 1989.

Charette, Bob; Application Strategies for Risk Management; McGraw Hill, New York, NY; 1990.

DeMarco, Tom; Controlling Software Projects; Yourdon Press, New York; 1982; ISBN 0-917072-32-4; 284 pages.

Everett, Gerald D. and McLeod, Raymond; Software Testing – Testing Across the Entire Software Development Life Cycle; IEEE Press; 2007.

Ewusi-Mensah, Kweku;  Software Development Failures; MIT Press, Cambridge, MA; 2003; ISBN 0-26205072-2276 pages.

Fernandini, Patricia L.; A Requirements Pattern; Addison Wesley, Boston, MA; 2002; ISBN 0-201-73826-0.

Flowers, Stephen; Software Failures: Management Failures; Amazing Stories and Cautionary Tales; John Wiley & Sons; 1996.

Galorath, Dan and Evans, Michael; Software Sizing, Estimation, and Risk Management: When Performance is Measured Performance Improves; Auerbach; Philadelphia, PA; 2006.

Garmus, David and Herron, David; Function Point Analysis – Measurement Practices for Successful Software Projects; Addison Wesley Longman, Boston, MA; 2001; ISBN 0-201-69944-3;363 pages.

Gibbs, T. Wayt; “Trends in Computing: Software’s Chronic Crisis”; Scientific American Magazine, 271(3), International edition; pp 72-81; September 1994.

Gilb, Tom and Graham, Dorothy; Software Inspection; Addison Wesley, Harlow UK; 1993; ISBN 10: 0-201-63181-4.

Glass,  R.L.; Software Runaways:  Lessons Learned from Massive Software Project Failures;  Prentice Hall, Englewood Cliffs; 1998.

Hill, Peter; Jones Capers; and Reifer, Don; The Impact of Software Size on Productivity; International Software Standards Benchmark Group (ISBSG), Melbourne, Australia, September 2013.

International Function Point Users Group (IFPUG); IT Measurement – Practical Advice from the Experts; Addison Wesley Longman, Boston, MA; 2002; ISBN 0-201-74158-X; 759 pages.

Johnson, James et al; The Chaos Report; The Standish Group, West Yarmouth, MA; 2000.

Jones, Capers and Bonsignour, Olivier; The Economics of Software Quality; Addison Wesley, Boston, MA; ISBN 10 0-13-258220-1; 2011; 587 pages.

Jones; Capers; Software Engineering Best Practices; McGraw Hill, New York, NY; ISBN 978-0-07-162161-8; 2010; 660 pages.

Jones, Capers; Applied Software Measurement; McGraw Hill, 3rd edition 2008; ISBN 978-0-07-150244-3; 662 pages.

Jones, Capers; Assessment and Control of Software Risks; Prentice Hall, 1994;  ISBN 0-13-741406-4; 711 pages.

Jones, Capers; Patterns of Software System Failure and Success;  International Thomson Computer Press, Boston, MA;  December 1995; 250 pages; ISBN 1-850-32804-8; 292 pages.

Jones, Capers;  Software Quality – Analysis and Guidelines for Success; International Thomson Computer Press, Boston, MA; ISBN 1-85032-876-6; 1997; 492 pages.

Jones, Capers; Estimating Software Costs; McGraw Hill, New York; 2007; ISBN 13-978-0-07-148300-1.

Jones, Capers; Software Assessments, Benchmarks, and Best Practices;  Addison Wesley Longman, Boston, MA; ISBN 0-201-48542-7; 2000; 657 pages.

Jones, Capers: “Sizing Up Software;” Scientific American Magazine, Volume 279, No. 6, December 1998; pages 104-111.

Jones, Capers; Conflict and Litigation Between Software Clients and Developers; Software Productivity Research technical report; Narragansett, RI; 2007; 65 pages.

Kan, Stephen H.; Metrics and Models in Software Quality Engineering, 2^nd edition;  Addison Wesley Longman, Boston, MA; ISBN 0-201-72915-6; 2003; 528 pages.

Pressman, Roger; Software Engineering – A Practitioner’s Approach; McGraw Hill, NY; 6^th edition, 2005; ISBN 0-07-285318-2.

Radice, Ronald A.; High Qualitiy Low Cost Software Inspections;  Paradoxicon Publishingl Andover, MA; ISBN 0-9645913-1-6; 2002; 479 pages.

Robertson, Suzanne and Robertson, James; Requirements-Led Project Management; Addison Wesley, Boston, MA; 2005; ISBN 0-321-18062-3.

Wiegers, Karl E.; Peer Reviews in Software – A Practical Guide;  Addison Wesley Longman, Boston, MA; ISBN 0-201-73485-0; 2002; 232 pages.

Yourdon, Ed; Death March – The Complete Software Developer’s Guide to Surviving “Mission Impossible” Projects; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-748310-4; 1997; 218 pages.

Yourdon, Ed; Outsource:  Competing in the Global Productivity Race; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-147571-1; 2005; 251 pages.

WEB SITES

Information Technology Metrics and Productivity Institute (ITMPI): www.ITMPI.org

International Software Benchmarking Standards Group (ISBSG): www.ISBSG.org

International Function Point Users Group (IFPUG): www.IFPUG.org

Namcook Analytics LLC: www.Namcook.com

Namcook Analytics Blog:  http://NamcookAnalytics.com

Software Engineering Institute (SEI): www.SEI.cmu.edu

Software Productivity Research (SPR): www.SPR.com

SUGGESTED WEB SITES

http://www.IASAhome.org This is the web site for the non-profit International Association of Software Architects (IASA).  Software architecture is the backbone of all large applications.  Good architecture can lead to applications whose useful life expectancy is 20 years or more.  Questionable architecture can lead to applications whose useful life expectancy is less than 10 years, coupled with increasing complex maintenance tasks and high defect levels.  The IASA is working hard to improve both the concepts of architecture and the training of software architects via a modern and extensive curriculum.

http://www.IIBA.org This is the web site for the non-profit International Institute of Business Analysis.  This institute deals with the important linkage between business knowledge and software that supports business operations.  Among the topics of concern are the Business Analysis Body of Knowledge (BABOK), training of business analysts, and certification to achieve professional skills.

http://www.IFPUG.org This is the web site for the non-profit International Function Point Users Group.  IFPUG is the largest software metrics association in the world, and the oldest association of function point users.  This web site contains information about IFPUG function points themselves, and also citations to the literature dealing with function points.  IFPUG also offers training in function point analysis and administers.  IFPUG also administers a certification program for analysts who wish to become function point counters.

http://www.ISBSG.org This is the web site for the non-profit International Software Benchmark Standards Group.  ISBSG, located in Australia, collects benchmark data on software projects throughout the world.  The data is self-reported by companies using a standard questionnaire.  About 4,000 projects comprise the ISBSG collection as of 2007, and the collection has been growing at a rate of about 500 projects per year.  Most of the data is expressed in terms of IFPUG function point metrics, but some of the data is also expressed in terms of COSMIC function points, NESMA function points, Mark II function points, and several other function point variants.  Fortunately the data in variant metrics is identified.  It would be statistically invalid to include attempt to average IFPUG and COSMIC data, or to mix up any of the function point variations.

http://www.iso.org This is the web site for the International Organization for Standardization (ISO).  The ISO is a non-profit organization that sponsors and publishes a variety of international standards.  As of 2007 the ISO published about a thousand standards a year, and the total published to date is approximately 17,000.  Many of the published standards affect software.  These include the ISO 9000-9004 quality standards and the ISO standards for functional size measurement.

http://www.namcook.com This web site contains a variety of quantitative reports on software quality and risk factors.  It also contains a patented high-speed sizing tool that can size applications of any size in 90 seconds or less.  It also contains a catalog of software benchmark providers which currently lists 20 organizations that provide quantitative data about software schedules, costs, quality, and risks.

http://www.PMI.org This is the web site for the Project Management Institute (PMI).  PMI is the largest association of managers in the world.  PMI performs research and collects data on topics of interest to managers in every discipline: software, engineering, construction, and so forth.  This data is assembled into the well known Project Management Body of Knowledge or PMBOK.

http://www.ITMPI.org This is the web site for the Information Technology Metrics and Productivity Institute.  ITMPI is a wholly-owned subsidiary of Computer Aid Inc.  The ITMPI web site is a useful portal into a broad range of measurement, management, and software engineering information.  The ITMPI web site also provides useful links to many other web sites that contain topics of interest on software issues.

http://www.sei.cmu.edu This is the web site for the Software Engineering Institute (SEI).  The SEI is a federally-sponsored non-profit organization located on the campus of Carnegie Mellon University in Pittsburgh, PA.  The SEI carries out a number of research programs dealing with software maturity and capability levels, with quality, risks, measurement and metrics, and other topics of interest to the software community.

http://www.stsc.hill.af.mil/CrossTalk This is the web site of both the Air Force Software Technology Support Center (STSC) and also the CrossTalk journal, which is published by the STSC.  The STSC gathers data and performs research into a wide variety of software engineering and software management issues.  The CrossTalk journal is one of few technical journals that publish full-length technical articles of 4,000 words or more.  Although the Air Force is the sponsor of STSC and CrossTalk, many topics are also relevant to the civilian community.  Issues such as quality control, estimating, maintenance, measurement, and metrics have universal relevance.

Capers Jones, Vice President and CTO, Namcook Analytics LLC