The purpose of this article is to provide some background as to why software can be so complicated and expensive to develop. The target audience is Business Management. This article is not comprehensive. It is also general. However, it should explain the problem space in a way that is understandable, conferrable, and defendable.
The problems listed here are ordered according to anticipated priority of concerns. These are not theoretical problems, as the author has witnessed each of them. Note that these problems are more specific to internal Development Groups.
The information source is often the Development Group. Verification is generally infeasible and may generate resentment. It is not in the interest of Development Groups to provide information that will alarm or invoke scrutiny from Business Management. Such scrutiny inevitably causes further inefficiency for the Development Group.
The source of expert council is generally the Development Group itself. Such council is constrained to the expertise of the Development Group. It is also influenced by the particular interests of the Development Group. The Business Manager must then trust that the council will be defendable under scrutiny. Thus, the Business Manager becomes tied to the technical success of the Development Group. This subtle leverage is quite significant.
Software Development is divergent from other engineering disciplines in regard to the personality types of the people involved. It is a creative discipline that requires individuals who are adept at assimilating abstract and complicated problem spaces. Consequently, the internal dynamics are complicated. External interference of the internal dynamics is rarely productive. That is, Business Management will have limited leverage to address process issues within the Development Group.
Development Groups like working on interesting and visible projects. Implementation decisions are often influenced by the technical motivations of the individual members. Tedious and low visibility features receive less attention, despite their business merits. Career interests may direct the choice of inferior new technologies over older proven technologies. Experienced developers often push to implement the more visible features, instead of the more crucial.
While Development Groups are cohesive during a project, individual members often move to other groups after completion. The stigma of being on a failed project has little residual effect the individual members. The reasons for failure are cooperatively obscured by the Development Group during the project. It is usually too difficult to determine individual responsibility, let alone hold anyone accountable.
Unfortunately, Business Management is usually responsible for this. They have a general budget and a sizable list of business goals. The allure of finding an optimal set of features given the various budget constraints is intoxicating. New information and constraints may often change the proposed goals and the resulting Feature Set. Business Management is motivated to postpone feature set decisions, or risk critical peer review for inadequate deliberation. Typically, a determination of a Feature Set is postponed too long, and even then it is not very stable.
The problem is that the schedule and technical assessment of a given Feature Set takes much longer to determine than the Business Manager realizes. This is important: Feature Set changes typically generate the most significant cost, schedule, and quality impacts on a project. It becomes even more costly during Implementation and Testing. Coincidently, it is one of the few variables that the Business Manager has the most control over.
Development Groups are over-optimistic when estimating the time and efforts needed to complete features and projects. They necessarily qualify their estimates with phrases like "if nothing unexpected occurs", "it isn’t unreasonable", "it should", or "theoretically".
The number of events that can negatively affect a project is staggering. Historically, doubling a reasonable estimate has been about as accurate a method as any for predicting the actual cost. However, this method rarely gets used, since it is hard to defend. More rigorous methods exist, but they are costly, and generally produce marginally better results.
Unable to clearly explain large buffers in estimates, Development Groups and individual members are essentially forced to provide qualified estimates for best case scenarios they know will rarely occur.
During the course of a project, business needs will often change. Such changes may result in feature set changes or modifications. The resulting feature changes are expensive and difficult to avoid.
Complications are often discovered during Implementation and Verification. The worst of them are related to incidental instability and performance issues. Other types are that external components do not function according to their interface or functional descriptions.
Incidental problems are time consuming to reproduce and resolve. They are often left unresolved until late in the project. Their underlying causes may never be determined. Workarounds requiring some amount of redesign may be needed.
Issues with external components may require significant redesign. The resolution of such issues can be quite time consuming, depending on support and licensing issues. Sometimes, such issues can invalidate the entire feasibility of the project.
While some buffer is generally provided to account for such complications, it is rare that an external component or technology is more stable or performs better than expected.
Changes to features, general constraints, unexpected events, and unanticipated complication all generate additional tasks. If the completion date is already extended as much as possible, the only way to meet the deadline is to cut features or bring in more resources. Using these resources incurs an additional unbudgeted cost to a project.
Typical end-user usage and utilization is difficult to predict. Seemingly small aspects in performance may reduce the utility of a feature. For example, if the search mechanism of an inventory control program is too cumbersome, the end-user may avoid using it, preferring other less efficient methods.
New features may actually generate additional problems. For instance, a search feature may be used in unanticipated ways causing severe performance issues for other applications as it overloads the database server.
The end-user may be able avoid a required, but cumbersome, business workflow using new functionality in intended ways. For example, if creating a product description requires a complex approval process, but new functionality allows an author to directly edit an existing article, an author may just rewrite an existing article, avoiding the review process workflow.
An external condition which was a basis for the project may change. For example, a new inexpensive external application addresses the very problems that the custom project was designed to address.
A portion of this problem is due to the difficulty in prioritizing actual problems and determining an optimal feature set. It is often due to some dynamic unassociated to the core requirements. For instance, upper management may require the inclusion of a technology for strategic reasons, incurring additional costs and possibly restricting the ability to address a core requirement.
Organizationally, Development Groups are generally uniform. However, given the same project goals and constraints, even similar Development Groups will often produce very different results. Subtle differences in the views of individual members can have quite an effect on component choices and feature implementations. The total amount of divergence is related to the specificity of the project goals and constraints.
The idea of Software Engineering has been around for a very long time. However, the technical instability of the industry combined with the creative nature of the participants generally repels the ability to enforce standardization. Even minor issues like variable name notation remain indefinitely unresolved.
The result is that quality, reliability, supportability, and efficient development of a project will all differ significantly based on the particular Development Group implementing it.
Development Groups are full of creative people who are often disproportionately affected by their work environment. Morale and motivation are primary factors of a projects success and efficiency. Organizational, policy, benefit, management, and equipment changes will affect morale and motivation. The affect of morale and motivation on Deliverables is significant.
Either through direct updates, or updates of dependant components, the stability of an external component can change. For instance, your component may relying on a web browser that altered the way it utilizes frames during a minor update, causing your released product to malfunction to a such a degree that patches and new releases must be made.
For example, the project results were a feature rich reporting tool which used a log file generated from a shared web server. However, it was determined that the generation of such log files could be a performance or security issue. So company policy no longer allows the generation of such log files.
For example, the project design may rely heavily on what was thought to be a secure component. However, after the project is completed, it is discovered that the component exposes security vulnerabilities.
For example, the proprietary voice recognition component utilized by your application has a licensing agreement that is incompatible with the way the product needs to be competitively marketed.
Like most problems of some scale, the causes of Software Development problems are interrelated. It can best be viewed in terms of chaos theory. That is, determine those variables which have the strongest influence on the system. Granted, other variables may emerge. It is possible that some such variables may create even worse conditions. However, it is the opinion of the author, that such risk is worth taking.
Software components, technologies, and operating systems are too instable. They are instable in terms of performance, security, consistency, reliability, and longevity. This creates nearly intractable complexity for Business Manager to address when analyzing the cost-benefit analysis of Software Development.
An understated analogy would be attempting to estimate the cost of a building when it is uncertain what materials will be used, whether those materials will perform as expected, how they will be utilized, and even how long they will last. Fortunately for home owners, the building industry is reasonably constrained to use proven materials and construction methods. Businesses wanting custom software solutions are not so lucky.
Stability is counter to the financial interests of the industry. Stability reduces the frequency at which applications, systems, and system component need upgrades. It is the simply the unwillingness of the industry to act directly against their interests without significant pressure to do so.
An application is dependent on the questionable stability of many external components. The complexity associated with this instability enables the industry to avoid legal and financial liability. Software Licensing Agreements generally state that they may not work correctly, and will assume no liability. Certainly, they cannot assume liability for the instability of external components they must utilize.
A key problem stemming from general instability is the difficulty of costing and planning a Software Project with a reasonable amount of confidence. The planning and costing of a project end up being significant portions of the total cost of a project. Whereas, the actual cost of writing the source code for the software is relatively small.
General system instability is not the only complicating factor. Software projects often require solutions which may not have well understood, or existing patterns to follow. They may utilize interfaces and techniques that add additional schedule and cost risk. However, such complexity is less artificial and less avoidable than that associated with general instability.
Business Managers typically make decisions based on cost and benefit analysis. A confident determination of such is typically not prohibitive. For software development, a reasonably confident cost and benefit determination may be too costly to make, or too qualified to be useful.
Software Development is complicated. Even if major strides were made in terms of overall stability of operating systems and components, new complexities would emerge. Hopefully, the new complexities would add greater value to the end results. Software Development follows other industries where the improvements to process and design allow for the addition of features and their corresponding complications.
Therefore, the Business Manager must rely to a great extent on the advice of experts. Too often that advice comes from the Development Groups that are planning and implementing the Software Projects. Such advice is difficult to refute without an independent expertise. This situation is easily leveraged to the continued benefit of the Development Groups.
Consumers and businesses are paying much more than they should for software that assumes little, if any, liability or responsibility for its quality.
The current state of the Software Industry primarily benefits the market leaders both financially and strategically. Free market dynamics have been largely suspended. The inability of the governing authority to provide meaningful remedies is likely to continue. Therefore, the problems will continue.
This isn’t to say that Software Development problems cannot be effectively solved in some context. Addressing such problems is the basis for Cybron's business model. However, such solutions are not likely to engineered, implemented, or sustained broadly until the industry problems are remedied.