14 November 2005

Bloodletting

A Google Alert came my way about how Microsoft was having success using the Scrum methodology.

The responses on slashdot reminded me of a talk given here in May by Craig Larman about Iterative and Incremental software development. In it, Craig had a couple of great slides about Bloodletting. This, of course, was a medical practice that was supposed to cleanse the body of all things evil and cure the patient by draining some of their blood. Quite often, though, rather than saving the patient it killed them! For example the first President of the US, George Washington, died less than 24 hours after a bloodletting session to cure laryngitis.

Regardless, bloodletting was used for centuries until about 150 years ago when more rigorous scientific study began to show that the practice was harmful rather than helpful. However, even into the 1920's bloodletting was used to treat pneumonia. There was a significant backlash to the notion that bloodletting was harmful and its opponents were vilified for stating such heresy.

Eventually, the practice of bloodletting was abandoned in favour of modern approaches to medicine. However, it didn't take the application of reason and large body of evidence to convince those that still believed in the practice. What it took was them eventually dying.

Well, based on what I saw on slashdot and what I see at times when I speak to people here, it's going to take a generational change or extinction to rid this industry of the bloodletters... er, BDUF, Waterfall, Code & Fix types who don't see that what they're doing is harmful.

6 September 2005

Not Good Enough for Agile?

Ron Jeffries recently wrote in the XP Yahoo Group that,
Agile methods cannot work if the design quality is not kept 'good enough'. A team needs to be very sensitive to this notion, and, frankly, I feel that the majority of programmers are not sufficiently skilled even to recognize what's good enough.
So, does this mean that the majority of teams aren't 'good enough' for Agile Development? Finally, an agile development thought leader has acknowledged an issue that I have been talking about and dealing with for a few years now... Assumed Competence.

Regardless of what they feel their skill levels are, the people behind the Agile Software Development movement are stars in their field. People such as Ron Jeffries, Bob Martin, Kent Beck, Martin Fowler and Ward Cunningham, to name just a few, are people who through intelligence or experience can simply "think OO". Those people are at least one and perhaps 2 orders of magnitude more productive than I am, and I'm pretty good at this software thing. In turn, I'm at least one and perhaps 2 orders of magnitude more productive than some other people with whom I work.

So, if the team I'm on has people with my level of expertise as its thought leaders, and is staffed with others less productive, are we doomed to failure if using agile methods? Must the thought leaders be in the top 25 in the industry in order for the team to succeed?

Traditional methods look to commoditize the people on a project. People of very high calibre rail against such a notion, since those people are so much more productive and are anything but a commodity. Perhaps, though, commoditization isn't such a bad thing for the lower-tier people. I'm beginning to wonder if there is a need to make the practices used in agile methods so bloody easy that anyone from the unemployed Fine Arts major to the most experienced OO guru can use them without feeling pain or embarrassment.

I don't have any answers at the moment, but I'd at least like to start the discussion. My main motivation is so that I can learn something.

17 August 2005

"Hired goons!"

There was an episode of the Simpsons in which Homer was the union negotiator dealing with Mr. Burns. Homer, sitting at home, receives a knock on the front door. When he asked who is there, the answer was "Hired goons!" I always found that really funny for some reason... maybe it took me back to the Saturday Night Live glory days, and the Land Shark.

What isn't so funny, is that those Hired Goons are knocking at the door on a project at one of my current clients right now. A recent thread in the XP Yahoo group about Technical Debt had me thinking about how as a team we've maxed our high-limit, high-interest credit card. We've tried to pay it down by refactoring, but the interest payments (just keeping the existing code running) are too high. In fact we aren't paying down the debt - it's increasing. The Hired Goons are at the door, and they've got baseball bats.

So, to avoid being beaten savagely by the Goons, what can be done? Well, it's time for bankruptcy and debt restructuring. I don't mean the wonderful Chapter 11 kind where life goes on and debts are magically written off, but the personal kind where you can conceivably lose your home, must sell off possessions, and have a note on your credit report about the bankruptcy for 6 years (in Canada). Let's apply that analogy to software development: the system is in ruins, the team breaks up, and you all bear the stigma of a failed project.

In our case, the system isn't quite in ruins. There are places where there is reasonably good test coverage. Not surprisingly, very few defects are found in that code. The problem is that a good 2/3 of the code has little or no tests. This includes the GUI itself, and the controller logic behind the GUI. As I mentioned in People Are Funny, the system started as a single application, and other lines of business have been folded in over time. The first application was a success, the second worked quite well (though it hasn't been used all that much in production), and the third line of business was a great success. In all of these cases, the overall system had remained small enough that the manual testing efforts were manageable. However, by the time the third line of business was completed, it was becoming more and more clear that we were incurring a significant amount of technical debt that would have to be paid off.

What has happened since, not unlike the credit card analogy, is that a number of factors have conspired to prevent us from performing refactoring work that we know needs to be done. First, a certain level of detachment from the customer has occurred because the main person was needed on a "problem child" project. Second, a rather aggressive schedule for implementing new lines of business was established. Third, the developers charged madly ahead with the new lines of business, without applying new techniques or changes to the older code. Finally, our beleaguered testers were still way behind the developers, so they weren't testing what we were working on.

When the first new line of business was just about completed, some of the team started on the second. They did perform some refactoring in order to include the next LoB. The problem is, that refactoring wasn't really supported by any tests and broke a whole whack (and this is Canada, so it's a metric whack!) of the functionality we had just completed in the first new line of business. The decision made was to fix the code, and not to roll it back to before the changes. Hindsight being a clear as it is... well, suffice to say that wasn't the best decision.

So, 8 months later we have just endured a lengthy testing process that uncovered numerous defects that simple unit tests should have caught. The deployment was challenged, and our credibility as a team took a pretty severe hit. Ain't life grand.

"Enough whining!! Give me something concrete!!"

I hear you. When I joined this particular project about 7 months after it started, I noticed (and voiced my concern about) 2 things right away - the use or hard classes rather than interfaces in most cases, and the lack of tests in the GUI and controller layers. In the 22 months since I started, some baby steps have been made towards using interfaces, though there are only a few more GUI and controller tests.

One of the developers feels very strongly that we should start using the Spring framework. I think that's a great idea, but I also feel that it wouldn't really make life much easier considering our current code. Our model layer isn't bad, but does rely too heavily on inheritance, IMO. That's OK, though, we can work with it.

One big problem is that we have classes to represent our Code Tables, and these classes are hard-coupled to the entire code table structure all the way to the EJB and database. It's a huge pain in the ass, and I'd love to rip as much of it out as possible. This is a place where even a marker interface could provde the type safety that necessitates the use of different types for each code table.

For example, why not have:

public interface Gender {}

public class TestGender implements Gender
{
  :
  :
}

public class CodeTableBasedGender implements Gender
{
  :
  :
}


Our code could still refer to a Gender type, but the class that actually implements Gender is completely hidden. It could come from the database, XML, flat file, Mars, I don't care!! It would also mean that testing would be considerably easier. (I'm sure that the Smalltalk crowd is all shaking their heads about now. Please bear with me...) Anyway, this is a simple step that we could easily perform with the refactoring tools built into our IDE of choice.

This isn't the only place where we're in need of interfaces. We don't have any interfaces in our model layer, although that isn't hurting too much at the moment. Where it is a problem is in the GUI and controller logic in both our Swing and web applications. The Swing application has some nasty hard-coded references in many places, in addition to embedded business logic. Testing these classes is difficult because of the dependencies on other classes (which have dependencies on other classes, etc.). We can use jMock with classes to a certain extent, but even that can't be done in all cases. So, the introduction of interfaces for some of the classes would simplify the testing. So, this is definitely a refactoring target, but we're leaving it alone until we have stories that require changes to the Swing app.

The web application, however, is where virtually all of the development is occurring at the moment. It's using the Struts framework, which is another place in desperate need of interfaces. Testing the Struts actions isn't easy, although using jMock has helped somewhat. However, these tests were written for precious few Actions (which, not surprisingly, have had few if any defects reported). Older code was not retrofitted with tests as we learned how to mock out dependencies (sounds like debt to me).

Then, just to make things really interesting, a couple of developers decided that we needed to add a new abstraction on top of the HTTP Session. This isn't a bad thing in an of itself, but I didn't believe that introducing it late in the release supported by limited tests was such a great idea. I lost that battle, their code was checked in, and all sorts of things that once worked began to break. Unfortunately, the debt we had accrued by not having tests in the controller layer (Actions) was coming back to bite us at this point. The developers making the change should have been able to push a button and see if anything had broken. So, the problems were found in a very slow, sequential process of finding one bug, fixing it, finding the next, fixing it, etc. Even more debt was accrued when the developers decided that it was too time-consuming and/or too difficult to always write tests to expose the bugs. Sigh.

Since we have to start adding more functionality to the web application, we essentially do have stories that support the refactorings that are necessary. Put simply, if we don't add tests and refactor properly, it's going to take even longer to develop the next lines of business.

One issue that has arisen several times is that we are trying to perform too many large refactorings. During the worst of these, the application was dead for 6 weeks. That means that we couldn't run the application, let alone test it, for 6 weeks. That's 30 working days. This was during a large database refactoring that wouldn't have been possible without the test we do have in our EJB and data layer. This was an extreme case, but our team history is rife with examples of multi-day refactorings. Put simply, we need to make all (and I mean ALL) of our refactorings in tiny, incremental steps. I have been shouting that message from the hilltops for months, and I believe that some people are now beginning to listen. Big, long refactorings bad... small, incremental refactorings good. It isn't rocket science (unless you're Bil Kleb).

So, to get to the Promised Land of defect-free code, what do we need to do right now? Here are my ideas:
  • Get at the very least some interaction level tests in place in the web application using a tool such as Selenium, etc. This isn't perfect, but provides a baseline from which we can measure

  • We do need to implement the abstraction of the HTTP Session, but we need tests surrounding everywhere that refactoring is performed. The tests also need to be written and tested first in order to prove that the change hasn't broken the code.

  • We're going to need to apply the new testing and coding techniques to older code that isn't using them. That's a wad of debt that's going to bite us if we don't address it.

  • Finally, we need to communicate all this to our customer. It might not be as specific, but more along the lines of, "In order to add new functionality, we need to fix some plumbing as we go. Therefore, it's going to take us longer for this release.
The last point is something that has really come to the forefront recently with all our tribulations - accountability. I better understand now why Kent Beck was on about it so much in XP Explained 2nd Edition and in the newsgroups. Our team hasn't been held accountable for its actions. If we're going to keep the Hired Goons out of the house, we have to get back to being accountable.

15 August 2005

People are funny

People are funny. We know that smoking, drinking and eating too much are all bad for us, and that we need to get regular exercise. Here in Canada, the top New Year's Resolutions are always to lose weight, start exercising, and to stop smoking. I suspect that it's the same anywhere that such resolutions are made. Of course, those resolutions are usually broken.

So, I suppose that it shouldn't be a surprise that agile teams break resolution as well. Here are some practices and techniques that we know provide tangible benefits to the development process:
  • No production code is written without corresponding programmer tests

  • No code is checked into the source repository until all of its programmer tests are passing

  • Programmers work in pairs

  • Each story has at least one acceptance test

  • The acceptance tests are automated

  • Continuous integration, including the automatic running of the programmer and acceptance tests

  • When a build fails, the whole team stops to correct the problem

  • The customer is involved on a continuous basis

  • If there are multiple stakeholders in the customer organization, they speak with one voice (consensus)

  • The whole team is involved in the planning process at the story level – the customer provides the priority, and the programmers provide the estimates

  • When planning an iteration, the team only signs up for as much work as they were able to complete in the last iteration
At a current client, I'm part of a team of 17 good to extremely good people. A little over two years ago, we started using XP to build the first part of an integrated system. A very conscious decision was made to avoid stovepiping, and to integrate the data and functionality of several lines of business into a single system. The users would see different front-ends depending on their role, with a rich GUI client locally, and web interfaces for users outside of the local area and overseas.

When I joined the team in October 2003, the team was smaller, having only 10 people. They were actively pairing, tests were being written religiously, the customer was very involved, and the planning game was being used. We didn't have many tests for the GUI, but the system was still simple enough for both the programmers and testers to test manually. We also didn't have continunous integration, but it didn't seem to be a problem since the number of programmers was relatively low. The acceptance tests were manual, mainly due to internal political issues with obtaining a testing tool that didn't require the programmers to do the bulk of the work.

The first production release of the system took 6 months, and a second release was finishing up when I arrived, after 2 months of work. Further releases were already in the pipe, at about 2-3 month intervals. Even though we weren't doing the XP practices at 100%, we were certainly an agile team, and the customer was very pleased with the results.

Fast-forward to the end of the summer, 2005. The system has now absorbed 4 new lines of business, and has both the rich GUI and the web interface. This past week, we performed an emergency release to fix a couple of defects that only manifested themselves in the production environment. During the deployment, a quick test of the web application showed that one of the bugs wasn't fixed. However, it was working just fine the testing environment. After a few tense moments, we realized that the wrong WAR file was installed. This came about 10 days after the initial release had to be rolled back because of problems. That resulted in a few days of panic, leading up to a release that was deployed successfully, but had the above mentioned defects. All of this happened after a protracted development cycle of almost 9 months. The system had been unstable, and we spent about 6 weeks just fixing defects prior to the release. Our boss, who is usually very calm in the face of “issues" lost it, and rightfully so.

So what happened over the course of almost 2 years that took the team from agile to code & fix? During that time, we didn't just sit on our collective behinds, but actually did try to keep improving. Among the initiatives we undertook were:
  • Expanding our tests into the GUI code using tools such at HttpUnit/jWebUnit and HtmlUnit

  • Using Mock Objects to better focus the tests and reduce the dependencies on external classes

  • Using CruiseControl for automated builds, and to run our unit test suites

  • Investigate the use of Fitnesse for higher level testing at the integration and system level

  • Bringing in outside experts to coach us on testing and the use of frameworks like Spring in order to improve our code
Each of these initiatives was met with at least some enthusiasm, and people were initially using the tools and techniques. However, it seemed that invariably after about a month the enthusiam would wear off and the use wouldn't continue.

A prime example is the use of CruiseControl. Our team has a tradition of going out for lunch every Friday. We also had an unwritten rule that if anyone broke the build and had left for the day, they had to buy a round of drinks on Friday. With CruiseControl running, the incidence of broken build suddenly increased, and several people (myself included after I left too hastily one day without adding a new class to the repository!) had to shell out on Friday. The entire team suddenly became very careful about what they were doing, and we were the better for it! However, one day after a few weeks, one of the developers was hit with a second “round" in two weeks and protested his penalty. The rule of buying a round was repealed. In very short order, the build was breaking again, and as time went on people cared less and less. I was able to have CruiseControl execute our unit test suites, and to flag the build as broken when they didn't pass. Of course, the tests weren't passing, the build was failing, and the team decision was to disable the running of the tests (which I protested to no avail).

Another problem we encountered was that we had a tendency to focus on tasks rather than the completion of stories. We made a decision for one particular release to define the tasks, but only track the stories. We also decided that we wouldn't sign up for any more stories in an iteration than we completed in the previous iteration (Yesterday's Weather). The entire team made an admirable effort to do this, and we completed the release a few days early, and with very few defects during acceptance testing. Everyone – the team, our management, and the customer – was happy.

So with this success under our belt when we began the next release, what did we do? We lapsed back into a focus on tasks. The justification was that some people felt that they needed the tasks to be enumerated so that they knew what had to be done. I had no problem with this, but it was done at the expense of estimating that the story level, i.e. only task-level estimates were provided. The result was that we did indeed complete the tasks, but there was never any real determination as to whether the stories were completed. Thus, at the end of the iteration, we would find that we hadn't finished everything. With that happening, we also started overloading iterations with work, despite the fact that everyone knew we couldn't possibly complete it all in the time available. That “next release" was the one that I spoke of above, that was a near disaster.

It would seem that each time we found a practice that improved our results but took either some extra work or was simply different from what we had done before, we abandoned it. We broke our resolutions. We started smoking again, eating too much, and gave up exercising. We were feeling better and people were telling us how great we looked, but it took effort and time to keep it up.

We became lazy.

Back in the early 1970's, the Canadian federal government started a program called Participaction, which was intended to prevent us from turning into a nation of 30 million couch potatoes. One television commercial in particular received a lot of attention, in that it told us of how the average 60-year-old Swede was more fit than a 30-year-old Canadian. We were lazy, and it was going to cost us in the long run. It made perfect sense, and they were absolutely right. We needed to eat right and exercise more in order to improve our lives. The people behind Participaction also recognized that our health care system couldn't cope with the future costs of dealing with the various ailments that would be encountered as the result of our unhealthy lifestyles.

In the 30-odd years since Participaction was established, we have improved somewhat. Smoking nationally is at an all-time low. People are exercising more, although we still aren't as fit as that 60-year-old Swede. Essentially, we have moved from the equivalent of Waterfall or Code & Fix to a more agile approach. However, once we realized that it takes time and effort, we couldn't maintain an XP level of fitness.

So, are current agile methods such as XP doomed by the inherently lazy people that use them? I know that not everyone is lazy, and I'd certainly like to think that I'm not. So, perhaps there will always be a group of people who will have the necessary discipline to follow the practices, and those who either don't care or don't want to stay with them. Similarly, there will be a group of people who have the discipline to stick with the practices. I suspect that group is the same as those that would make the heroic efforts to save a project on a Death March.

I believe that a key question, then, is will we be able to sustain agile development practices when they move to the mainstream and are used by people who don't have the same motivation as we early adopters? More importantly, how can we make the practices so easy to use that anyone, regardless of their level of motivation, will follow them willingly?