15 March 2010

Why Rewrites Fail

Last week on Twitter Kay Johansen asked, "Why do "rewrite" projects so often fail?" After all, rewriting an existing application should be easy, right? We are simply copying the business functionality, and not adding anything new. How could that possibly go wrong?

Actually, there are a number of reasons. The first one that I thought of came from the Standish Group's CHAOS reports, in which they say that some 45% of features in software are never used, and only 7% are always used. I can understand that some features of a commercial product may fall into the never used category - you may need to add a feature in order to stay even with a competitor - but I seriously doubt that those types of features account for 45%!

What that statistic does tell me is that there was precious little business involvement in the process of defining what features the software should have. Alternatively, the delivery cycles were so long that the business was tempted to cram in as many features as possible. Those two issues result in a lack of timely feedback that can be incorporated back into the software, and lead eventually to features that weren't really needed in the first place.

How does that lead to failure on a rewrite project? Well, if you blindly march into the process without re-engaging the business, you're potentially wasting 45% of time & money on features the business didn't really need in the first place!

Also, consider why a rewrite would be necessary. Was the software buggy? Was it not performing well? Was it so old that the hardware and platform software vendors didn't exist anymore? In the first two cases, the business people for whom the software was written likely weren't terribly happy with their system in the first place. As a result, they probably wouldn't want to devote a lot of time to the rewrite... "Just make it work this time!"

In the end, business involvement is exactly what is required when rewriting a system. I'm obviously biased towards Agile processes, but they force that involvement and are much more likely to deliver the system that the business needs at that time, rather than just what it did before. By focusing on the current business needs and providing constant feedback, there is much less risk of failure and a much higher chance that the delivered functionality will meet the business need.

Another factor that can feed into the causes of failure is the belief that the new system must be 100% complete before it can be released and replace the existing system. There may indeed be cases where this is a requirement, but there are likely as many or more cases where the business people could live with the duplication of effort for a certain period of time. At one client I had, a Lean Transformation in the business completely obviated the need for one rather large system - whiteboards and face-to-face communication had replaced the complex workflow that the system possessed and was no longer used. That system, however, was a "system of record" and data still needed to be entered for legal purposes. The business made a conscious decision to continue to use the system in as minimal a way as possible in order to satisfy the legal requirements. There was duplication of effort, but it was deemed worth it until a new system was in place (which will likely occur some 2-3 years after the manual methods were first used). Meanwhile, some simple automated tools were created to assist the teams working within their new process. All of this falls into Lean's principle of optimizing the whole rather than just certain aspects of a process. Completely replacing the existing system first would not have done so, and likely would have eliminated any benefits to replacing the older system. The replacement of that system is now taking place when the business understands their new process much better, and thus is much better positioned for success.

The real moral of this story is that a simple rewrite should be anything but. The business must be as engaged as they would during the development of a new system. The level of support from management needs to be the same. In the end, what is developed should be a new system supporting the current business processes, not what was in place when the system to be replaced was released!

7 March 2010

The Wrong Metaphor

This past week I read a post on the Cutter Consortium blog by Bob Benson entitled, "A Shift in the Notion of Portfolio Management Opens Opportunity for IT". Bob begins the post with,
For many years,Tom Bugnitz and I have been recommending that IT organizations think of themselves as a service business, with five fundamental service portfolios.

1. The Application Portfolio (e.g., providing the applications and support to business units)
2. The Project Portfolio
3. The Infrastructure Services Portfolio (e.g., e-mail, network attach, remote access)
4. The User Services Portfolio (e.g., help desk, PC repair)
5. The Management Services Portfolio (e.g., providing procurement, planning support to business units)
He goes on to describe how he's now seeing the boundaries between these service portfolios become blurred, with which I agree. What I don't agree with, though, is the suggestion that IT organizations much act like a service business, at least not for the part of IT that build applications for use by the business.

The problem is that the level of collaboration required for effective software development precludes the use of the Service-Consumer model. As I wrote in Embedded Collaboration in 2008,
There is no functional separation of business and IT on a project - they are living and working together, breathing the same air, meeting at the same water coolers.
The most obvious way to accomplish this is to have the software development people "go native" and physically sit with the business people for whom they are building software (I also wrote about this in Agile Circa 1988). Create product teams within each business unit whose sole reason for existence is to serve the software development needs for that unit. The business unit is responsible for establishing it's own software development budget, and for feeding the product team with a constantly groomed list of prioritized features that they require. There are some potential problems with that model, though.

First, it can create a local optimzation at the business unit level, i.e. the immediate software needs of that unit are being satisfied, but perhaps what is created could have been shared with the organization as a whole. To address this you would require some form of communication and collaboration between the software development groups of each business unit.

Second, once an application or business process crosses business unit boundaries, you're not in Kansas anymore - you can't simply "go native" across the whole organization. In that case, the discipline of product management become even more important, with the need for a more strategic view of the organization to be used to guide the team. That said, though, there is still no reason for the people who are building the applications to belong to an IT service organization - they should be part of a product team that includes close, frequent (ideally daily) participation from the business.

The third problem is that not all business units are created equal. It can be much more difficult for Human Resources to justify its own software development budget compared to a business unit that is directly creating revenue for the company (ironically, the first time I encountered Embedded Collaboration, it was in a Human Resources Systems group). To address this, you may need to have a product delivery group that spans multiple smaller business units. The key, though, is not to look at what a business unit either contributes to or takes away from the organization's bottom line, but rather at the impact of that unit on the business. Human Resources may be considered "overhead", but its impact on the business of any medium to large organization far outscales its size.

I do agree with Bob Benson that there is a blurring of the lines between the activities performed in IT groups now. However, I don't agree that the Service-Consumer model is appropriate for software development activities. I've met Bob, and have a lot of respect for his experience. However, I've seen that model when applied to software development create many more problems than it has solved. It has created artificial divisions that impeded true collaboration, and thus negatively impacted the delivery of software.

If you believe that the Service-Consumer model is appropriate, allow me to quote Dr. Phil,
How's that workin' for ya?