21 February 2006

Agile, circa 1988

When I first heard about Extreme Programming back about 5 years ago and poked around its practices, I recognized the On-site Customer as one that I had done before back in 1992-1993 and found it very beneficial. So, I decided to get together with my manager from those days to discuss the motivation for co-location, when conventional wisdom at the time was to keep the bloody Customer as far away from the developers as possible in order not to disturb them!

I started off the "interview" with a request for a history lesson on the creation of the Whole Team. What ensued was a description of Agile Development that was only missing automated tests.

This project started in late 1987, and was for a public sector organization's HR system. It was intended to handle functions such as classification, staffing, staff relations, and training, and would be written in PowerHouse on a Data General mainframe. My old manager decided at the time that he was tired of having 2 roles in charge of the project - the Customer in charge of the requirements and providing the funding, and the IT group in charge of building the system. The Customer would throw their requirements together and hand them to the IT group to be implemented. When, 18-24 months later, the system didn't do what the Customer wanted, IT became the scapegoat.

Additionally this manager felt, and continues to feel, very strongly that a good overall multidisciplinary team was essential to success. So, he pushed hard to be placed in charge of all people who would work on the project. This wouldn't mean that people would change the actual reporting relationships, but rather that while working on the project they were part of his team. These people included the development team and the business or domain experts, which was the departure from the past. He was given complete control over how the team worked, and how the system would be developed and deployed. He had already realized that the absentee Customer was a non-started, so he insisted that the development team and Customer sit together - the On-site Customer.

The next step was to figure out what to build. The development team sat down with the assembled domain experts for the different parts of the HR process. They started by talking about the first event in the process of hiring someone, and then stepped through that process to retirement. This wasn't done at a ridiculously detailed level, but rather in terms of the titles of the different steps in the process. With these basic steps in place, this group then determined what part or module of the system needed to be built first. Bear in mind that most of these modules were quite dependent on the others - you couldn't build one and leave the others as manual processes. Regardless, the development was broken down into these modules so that development would be focused, and in order to minimize the risk that the system as a whole would never ship.

For that first module, the development team really didn't have a concrete idea of how big it was, or how long it would take to develop. They knew it wasn't too big, and that there wasn't much risk that they wouldn't be able to develop it at all. They communicated all of this to the Customer, which was another departure from tradition at the time. The development team, mainly the manager also explained that this first module would very much be a learning experience, and would likely be completely rewritten over time. They essentially performed the Planning Game, and Refactoring.

So, the team built that first module, and it took a bit longer to build than they anticipated. However, the Customer was kept in the loop at all times and could see the progress of that first module. When it was completed, the team and the Customer sat down to review how things went (a retrospective) and to plan for the next module (Release Planning). The team had learned quite a bit, and shook out a lot of issues along the way. Even though they hadn't actually shipped anything to production, they had already built up trust with the Customer.

When they started planning the work on the next module, they had the domain experts of the each of these modules in the room. The expert for the first module was asked, "On a scale of 1 to 10, how big would you say that module is in the overall HR process?" The person answered, "Three." They then asked the expert for the second module, "How big do you think this next module is?" They responded, "Nine." With that answer, the team then estimated that developing the new module would take 3 times as long as the first. Forget COCOMO, forget Function Points, they just tripled their time based on the Customer's evaluation of how big in a relative sense the next module would be. The time estimate and budget for the new module were based on Yesterday's Weather. And, they were right. Remember, this was 1988!

This process was repeated for each of the mandatory modules, and the system was placed into production in late 1989 or 1990. The full production cycle had taken 2 years, but the Customer (in the form of domain experts for each module) had been involved from start to finish.

I didn't actually join the team until early 1992, when I was hired on contract to evaluate the progress on a stand-alone interim version of the training module being developed in the same general group as the main HR system, but as a separate product. After a day or two I recommended a rewrite, and found myself the lucky winner of the contract to perform that work! I was immediately placed in a desk in close proximity to the Customer for the system, and worked very closely with her over the next few months to build it.

The interesting thing about that training system is that it had been intended from the start to be included in the main HR system. However, it wasn't a core part of the HR process, and it never reached a priority level required for it to be incorporated. Again, the organization as a whole identified the modules, and prioritized them according to the overall business value. For the training module, it could provide equal value in an interim, desktop system while the main team focused on the higher value modules. (As an aside, that training system was called ITS - Interim Training System - and was first released in 1990. In 1995 or 1996, I finally convinced people that the 'I' should be changed to mean something other than Interim so it became the Integrated Training System!)

Meanwhile, the main system was having releases to production every 8-12 months. When the government department for which the system was built merged with a larger department in 1993, it was decided that this system would be the HR system for the new department. In 1995, PeopleSoft arrived on the scene and was to replace what was now perceived as an old, character-based mainframe system. The head of HR (the Gold Owner) made it abundantly clear that PeopleSoft wouldn't replace the system until it provided the same level of functionality. PeopleSoft, the last I heard, is slated to roll into production in 2007.

So, I came away from my meeting this morning with a renewed sense that I had been part of something quite special, although I didn't consciously know it at the time. Many of the development practices from XP/Agile such as automated testing and true iterative development weren't done, but the overall team practices were. We had the On-site Customer, Planning Game, Frequent Small Releases, Collective Ownership, Refactoring (although not in the modern sense), and Sustainable Pace (there was overtime, but not all that often).

We were successful. We were Agile!

16 February 2006

The Disengaged Customer - For want of a Product Manager

In The Disengaged Customer - Balkanization, I discussed how the Customer's organization began to work against itself as the system grew to incorporate different business lines (and thus different stakeholders).

A picture emerged from all of this where the team had originally built the system for a single Customer handling a single line of business. Using XP worked wonderfully in that situation. Then we added another line of business, with its user community. That process went OK, but not as well as the original line of business since we had a shuffling of our Customer, and no real feedback on the new work. Then we added another line of business, with its community including the old Customer mentioned above. As new lines of business were added and encompassed new user communities, the complexity of dealing with those people increased dramatically (I’d argue exponentially).

The problems came to a head shortly before I left. Again, the limited communication between the Customer’s organization and the development team caused a misunderstanding to occur, and the Gold Owner took it out on our Kevlar-underwear-wearing manager. I was pretty ticked at that point about the whole issue. Our manager met with the Gold Owner, and it finally became clear to her what was happening. There was talk of “a lot of crying on the floor” when the Gold Owner was done dealing with some of the people. The development team had been absolved of most of the blame, and we were going to get our Customer back and engaged.

I spoke a few days ago with one of my now former colleagues on that project. The Customer had been diverted to focus on a “problem child” project. Plus que ├ža change…

In all of this, the Gold Owner had her own vision of how all of these lines of business should be integrated to support the organization. This was a good thing, in fact an excellent thing, except that she wasn’t the Customer. I dearly wish she had been, since that vision was an excellent way to categorize and prioritize the work. It also would have meant that the Customer was in a sufficiently high position of authority to be able to deal with the Balkanization issue. That person would have effectively been what Jim Shore refers to as the Product Manager.

As a Product Manager, she would have been in a position to either directly enforce some discipline on the organization, or to escalate the problems very quickly to someone who could. With her vision, she would have been very easily able to establish priorities and to weed through issues that may have seemed important to individual stakeholders, but weren't all that important to the organization as a whole.

A healthy dose of financial oversight would likely have helped as well, but that's another issue.

The Disengaged Customer - Balkanization

Wikipedia states that, "Balkanization is a geopolitical term originally used to describe the process of fragmentation or division of a region into smaller regions that are often hostile or non-cooperative with each other." This was a very bad thing in the Balkans. It's also a bad thing for the Customer.

Sometimes, agile teams can be victims of their own success. The Gold Owner sees that all is going quite well with the project, and has the Customer disengage somewhat to deal with other issues. In The Disengaged Customer – A Funny Thing Happened… I spoke of the effect that this disengagement had on the team’s development efforts. Without the Customer being fully engaged on the project, communication became more difficult and much lower bandwidth and the team drifted. Eventually, the Gold Owner chewed out the development team’s management over her perception that the team wasn’t doing the work that they were supposed to.

At that point, the team took on new development to replace an old system by integrating its functionality into the system. This meant that they were dealing with new users, although with the same Customer who was still rather disengaged. However, by having an existing system from which to work we were able to move ahead reasonably well. This was when we began to realize that there were, um, some issues in the Customer’s organization.

The person who had been the Customer for the old system was retiring. She therefore wasn’t the Customer for the new system, although she was always sought out for input and information due to her experience in that business line. She did have some quirks, though. The biggest was that she insisted that the statistical reports generated from the system be clean. That is, the totals at the bottom of the report should equal the sum of all the data. That makes sense, of course, except that the data received in the field and entered into the system were rarely like that. Often data were from third-party providers who didn’t really have much interest in capturing the details of an event, other than it happened. As a result, the system contained a considerable number of fields with the value of Unknown.

The Customer for the older system didn’t like Unknown. It didn’t look good. To her, it meant that people weren’t doing their job. So, she had directed the developers of the old system to create default values, for example if a report came from a particular country but didn’t have any city information, the system would set a default city. She was also given the ability to edit the data after input, which allowed her to clean things up even more. This was fine for true data entry errors, but she used it to massage the data to better fit what she saw as proper. So, the statistics generated by the system were very clean indeed. They were also pretty close to fitting the saying, “Lies, Damned Lies, and Statistics”.

So, when we as the development team embarked on developing the new replacement system, we were dealing with a new Customer. This Customer didn’t know about the data massaging, but when told immediately directed that it stop. The new system wouldn’t default values like the old, and the ability to make edits after the initial data entry would be limited and audited. This didn’t sit well with the old Customer, and things became, um, tense.

Meanwhile, we were directed to look at revamping a small chunk of our system. No big deal, really, we discussed it with that particular user and in conjunction with our Customer created the stories to make the changes. Then that user left on stress leave. When it was deemed that we should start work on the changes, we ran them by the new replacement user, again in conjunction with the Customer. The new user indicated that the changes were all wrong, even though they had already been approved by her management. Those changes were on hold as of the end of January when I left the project.

The Customer was quite frustrated at this point, having to deal with her peers and not being able to get any consensus. She didn’t know the lines of business as well as these other people, so she didn’t feel that she could make the decisions on her own. Sometimes she did, and people went behind her back to the Gold Owner and complained. Other times, these other stakeholders would talk directly to the development team members, giving the impression that what they wanted had been cleared. The Gold Owner didn’t really understand that the development team needed the Customer’s single voice in order to be able to proceed. In essence, the Customer organization had become Balkanized.

Continued in The Disengaged Customer - For want of a Product Manager.

14 February 2006

The Customer Role in Agile Development

Lack of user involvement traditionally has been the No. 1 reason for project failure. Conversely, it has been the leading contributor to project success. Even when delivered on time and on budget, a project can fail if it doesn't meet user needs or expectations.
Software Magazine, Feb. 2001, Collaborating on Project Success
Based on the Extreme CHAOS 2001 Report from The Standish Group.

Agile development teams deal with the issue of customer involvement by having a person who is empowered to specify the User Stories, set their priority and to answer questions about them in order to clarify the real requirements. Ideally, this Customer is co-located with the development team, although an even better approach is to locate the team with the Customer.

The effect of having the Customer working so closely with the developers is twofold:
  • Communication improves dramatically. Verbal discussions provide much more valuable information than e-mail or hard-copy documents, and thus costs and the time taken to make decisions are reduced. This isn't to say that no documentation is produced, but rather that the documentation should be the result of a conversation first, and not vice versa.

  • Decisions are made more quickly. Because the Customer is readily accessible, questions can be answered and decisions made in very short order. This allows the team to keep moving forward, rather than waiting for answers or performing speculative work.

"The Vision Thing"

Because of its importance in ensuring the success of a project, the Customer role requires certain traits in the person who fills it. They must be able to make informed decisions quickly and not waffle on their answers. If they don't know the answer to a question, they need to be able to say so and then follow through with getting the answer as quickly as possible. They need to know the business that will be supported by the system quite well. They also need vision.

For a small system with a focused group of users, having an end-user as the Customer is fine. When a system is larger, for example it consolidates several lines of business, the Customer needs to have an overall strategic vision of how the system is going to be used to support all of its users. This requires someone at perhaps a management level, or they have a reporting relationship with C-level management. This is required because the issues faced by this Customer increase considerably with the number of lines of business in the system. Often, these issues require resolution from a higher level of management, and having a Customer either at that level or with a direct reporting relationship to that level will speed the resolution process.

More than One Customer

What happens when a system has multiple Customers or Stakeholders? Getting multiple Customers to agree on the time of day let alone User Story priorities is very difficult task. In situations such as that, there needs to be a higher level of leadership from whole organization, not just the Customer or IT groups. One strategy is to have one person appointed as the single Customer who can make the vast majority of decisions about the system on the spot. Another is to have a comptroller or similar person from the financial part of the organization to assist in determining the true cost and benefit of requested features.

A strategy that simply does not work is to have a Customer Committee (read Change Control Board) to determine the requirements and set priorities. In almost all cases, such committees are too slow to respond, and impede the team's progress. What can work, however, is to have a steering committee comprised of executive-level members (CxO, VP's etc.) to provide oversight for the project and a consistent means of escalation of issues from the Customer level. There still needs to be a Customer who will be the single voice for the project in order to provide consistency for the developers, but that Customer can rely on being able to obtain quick resolution of issues by having the ear of senior management.

Customer "Balkanization"

Different people have different priorities, they work differently and see the world differently. Some people may want reports to show the data with all its warts, while others may want to massage it so that it appears more consistent. Some people want a very spartan interface so that they aren't distracted by reams of data, while others want as much as can possibly be crammed onto a page in order to avoid scrolling. Also, although we're all supposed to be adults, personality conflicts can enter into the equation. I personally have witnessed situations where if person A said the sky was blue, person B would disagree as a matter of principle.

Resolving issues such as these is critical to the success of any project, but again it's even more important on agile projects.

Even when there a single Customer has been appointed, sometimes other people work behind the scenes (e.g. talking to developers directly) in order to have the features that they see as important moved up in priority. The best solution that I can see is to ensure that the leadership of the Customer's organization is aware of the issues, and that the development team needs the Customer to speak with a single voice.

To provide resolution, the Customer needs to either be or report to a sufficiently high level of management to be able to impose a certain level of order. There are times when a decision simply has to be made, and the Customer must be in a position to make it themselves or to escalate to their management to have the decision made.


The Customer role is the single most important and difficult on any type of project. It is especially important for an agile project due to the hands-on approach required. They need to be located with the development team (or the team with them) in order to facilitate communication.

The Customer must have an overall vision for the project, and how it fits into the business that it will be supporting. They must have the authority to make decisions required to allow the team to keep moving forward.

By having a dedicated person located with the development team and able to make decisions quickly, the chances for success increase tremendously.

The Disengaged Customer - A Funny Thing Happened

In The Disengaged Customer - Introduction, I spoke of a nice, rosy Agile Development microcosm in which dog and cats... er, Developers and the Customer were living together in harmony. The Customer was fully engaged with the project, and all were reaping the benefits of that involvement. Then a funny thing happened...

Well, it was funny in the peculiar sense. This client has a very large, dysfunctional project in the works. It has sucked up about a quarter of a billion dollars so far, with at least 2 years remaining. They are trying desperately to do a good job of using Waterfall to build this system. It isn't working. At any rate, our new Customer was directed by the Gold Owner to start working with the dysfunctional project team, while also working with us. After all, our project was humming right along, and didn't require as much attention as the problem child project.

Suddenly, the Customer wasn't available for meetings. Decisions that once took a couple of hours or a couple of days to be made were taking a couple of weeks. Where once the Customer attended our meetings, the Business Analyst on our team now had to go meet with the Customer when time permitted. All of these issues resulted in a critical loss of feedback.

We continued developing, assuming that the path we were on was the correct one. Of course, we had no way of knowing otherwise, since our Customer wasn't available to provide the feedback. We were able to get some feedback from a previous, interim Customer, but that person had a tendency to flip-flop on decisions (which is likely why she was a previous, interim Customer).

So, as a development team, we started to drift.

We weren't sure what our priorities were. We weren't exactly sure what to work on next. We pulled stories from the overall list, but there was precious little from the Customer in terms of what we should be working on. This went on for a few months.

Then, we found out that the Gold Owner was pissed - really pissed. We hadn't been working on what this person thought we should. Not being at the meeting in which my boss required Kevlar underwear, I'm not sure how much of a defence he put up with regards to the lack of involvement from the Customer. However, I'm pretty sure that it was a "read-only" conversation.

So, we were getting our knuckles rapped for not doing our jobs. However, from my perspective it was the Customer who wasn't there to steer the project. Similarly, the Gold Owner directed the Customer to focus on another project. The result was that our team didn't have the focus required nor was our work visible to the absent Customer. In the end, I don't believe that the Customer's organization fully understood the importance of the Customer's role.

Continued in The Disengaged Customer - Balkanization.