26 November 2008

Embedded Collaboration

You keep using that word. I do not think it means what you think it means. - Inigo Montoya, The Princess Bride
In a meeting today at a client, people were talking continuously about how the IT organization and business have to improve their collaboration. The business feels a considerable amount of pain about the lack of collaboration with IT, and IT in turn is feeling some pain collaborating with the business during a transition to Agile.

The problem is, though, that what these people are saying is collaboration isn't really what they think it is. Get together for a meeting, come to some decisions, write up some action items and go away to do some more work until the next meeting. All good stuff, right? Sure, and there's communication going on and people were working together. But is it the type of collaboration that's required to create a true partnership between business and IT? No.

Someone at this same meeting talked about how business was the Customer and the IT group served them. That doesn't sound like collaboration to me. It's a model for doing business, but it's quite deficient with respect to the delivery of systems.

What's required is Embedded Collaboration. There is no functional separation of business and IT on a project - they are living and working together, breathing the same air, meeting at the same water coolers. I worked in this model from 1992 to 1994 at a government client (see Agile Circa 1988), and at a smaller scale in earlier work. Having the development team located in the same physical space as the business it supports is an immensely powerful way to promote and support the level of collaboration required.

I'm interested to hear about other organizations that have used this model. Is it universally good? Are there hidden challenges?

20 November 2008

Agile Ottawa Meetup - Nov. 24th

There is going to be a meeting of the Agile Ottawa group (or groups!) on Monday, November 24th at 7:00 PM. This is an initial session, intended to get some discussion going. Here are the details:
Agile Meetup 7-9PM, November 24, 2008
At the The Code Factory, 2nd Floor - 246 Queen Street, Ottawa, ON
Mayford Technologies is proud to be sponsoring this event, so there is no charge.

Hope to see you there!

19 November 2008

The Practices Do Matter... At First

With all the recent talk about the Decline and Fall of Agile, I've seen a number of comments about how we should focus on the Values and Principles of the Agile Manifesto and not on the practices of any one approach. I agree with this to a point. Once a team has matured within a particular process then absolutely they can really begin to tailor it to suit their local circumstances using the Values and Principles as their guide.

However, a team needs to learn to walk before it can run, and the Japanese martial arts learning concept of Shu Ha Ri is an excellent model to use in this case.

In the first stage, Shu, a team is given fundamental practices that they watch and perform. Everyone performs these practices in the same way, regardless of their own personal traits.

When the team reaches the Ha stage, they have integrated the practices into their own way of thinking and are good enough at them to begin to expand beyond simply mimicking what their master (or coach) has shown them. The literal translation of Ha is "to break free" or "to frustrate", a dual meaning that explains very well what happens at this stage. The team is learning how to tailor the initial practices to their own environment and personalities, and the coach does indeed have some frustration with respect to the team departing from using the practices as taught.

The final stage, Ri, is when the team has fully integrated not only the "what" of the practices, but the "why". The translation of Ri is "to break free", which reflects how the team no longer needs the coach to guide them with respect to the practices. The team has improved to the point that they know what works for them and what doesn't, and can work within themselves to self-improve.

Going back to the Agile Manifesto for a moment, you can equate the Shu Ha Ri model to integrating Practices, Principles, and the Values. The problem is, the Manifesto only speaks to the latter two, with no mention at all of practices. So, how can a team reach the Ri stage with an Agile process without having a Shu stage to start?

Well, there have been a number of discussions recently about the use of XP practices with Scrum. In a software development context, Scrum avoids prescribing any technical practices, preferring instead to let the team self-organize to complete their work. This lack of guidance contributes to both Scrum's higher adoption rate, and to the challenges encoutered by teams using Scrum that continued to use their existing practices that had led them to seek out a better development process in the first place!

So, teams are using Test-Driven Development. They Refactor their code, and this is something that Ken Schwaber has recognized is required. They're using Continuous Integration. Pair Programming is being used, or some other form of near real-time code oversight.

In short, teams are realizing that the Practices do indeed matter. Without them, a team will improve somewhat, but it will likely be a case of We Suck Less.

To be able to achieve all of the potential improvement, a team needs to take the Practices and do them by the book until they are good enough to decide whether they work for that team. At that point, and only at that point will a team be capable of making the decision to drop or modify Practices.

Shu = Learn the Practices as is until you're good at them.
Ra = Ask "why" about the Practices, and experiment with changes.
Ri = Forget the damned Practices and just build some software!

The last point is only partially tongue in cheek - when a team reaches that stage, they are able to use the Values from the Manifesto to frame what they do on a daily basis. They know what works for them to enable the sustained delivery of high quality software and what doesn't. Furthermore, they know how to (and do) seek continuous improvement.

A team cannot reach this last stage without starting at the first - the Practices matter.

18 November 2008

Waterfall vs. RAD vs. RUP vs. Agile (Scrum, XP, etc.)

When I broke into this industry in the mid to late 1980's, it was pretty well a Waterfall world. By that I mean the minimal definition of waterfall from the top of page 2 of Winston Royce's seminal 1970 paper Managing the Development of Large Computer Systems that showed the flowing steps of Requirements, Analysis, Design, Coding, Testing, etc., and not the more comprehensive definition that followed in the rest of that paper. I was a good soldier, defining and designing everything up front... well, OK, sometimes I cheated and did some coding work because I was impatient or bored and needed to get moving forward, but generally I followed those steps.

In the early 1990's, along came this cool concept called Rapid Application Development (RAD). The consumers of the systems we were building were also impatient - they wanted value NOW, and were willing to have corners cut with the design and documentation. So, RAD espoused getting something - anything - into production RIGHT NOW!! That worked fine, except that you didn't spend much time on any of the activities that promoted sustainable development.

So, we went from too much process to too little.

At the same time, Object Oriented languages were coming into prominence and the development models created by Grady Booch, Ivar Jacobsen and James Rumbaugh (with contributions from Philippe Kruchten) were combined into a single process know as the Rational Unified Process. The RUP dealt with many of the shortcomings of both Waterfall and RAD - it was iterative allowing the constant feedback missing in Waterfall, and was heavily weighted towards architecture and risk management which was missing in RAD. Cool, now the industry could move forward with a standard process.

But there was a problem. It wasn't so much the process contained in the RUP but how it was packaged. The RUP was put together as a process framework, which was not unlike a large pantry containing all the ingredients you would need to make any number of meals. Unfortunately, people didn't seem (or didn't want?) to realize that you didn't need all of the ingredients for every project. The older Waterfall mentality was used when attempting to tailor the process to local needs, and when faced with paring down a large process people seem to err on the side of not paring enough.

What resulted was many projects using way too much process, which in turn resulted in more documentation, more design and less software. This was not what the RUP's creators had intended, not unlike what happened to Royce's definition of Waterfall.

So, in the late 90's, people tired of the heavyweight processes started rebelling against them. Along came what eventually became known as the Agile processes such as Extreme Programming, Scrum, Crystal, Feature Driven Development, etc. They focused on delivering value, and value to them was running software. Many of the Agile processes also added technical practices intended to provide the sustainability missing from RAD. They also overtly identified the people as being as important as the process itself.

Success stories abounded, not unlike what happened with each of the prior waves of processes. There were also, though, some failures. Those have, in my observation, increased over time as more teams started to use Agile processes. When looking at the causes of these failures, they often show that teams weren't using all of the practices in a given process, or they didn't use good technical practices at all and just adopted the management practices.

So now, is the pendulum swinging back towards more process?

There was an interesting post from David Anderson on the KanbanDev Yahoo Group this morning. In that post he states,
At this point, it is impossible to take the 'David' or 'Jeff" or 'Israel' factor out of the achievement of the high maturity.
This is in reference to increasing the maturity level of organizations beyond just "Agile", and to do so he believes that we must remove the human element not from the process itself, but from the stewardship of the process.

I think this is the correct direction - acknowledge, retain and leverage the human factors of executing the process, while at the same time removing the human element from managing it. Any team could be successful using XP if Kent Beck, Ron Jeffries, Jim Shore, etc. were running it. What we want to achieve, though, is a point where any team could be successful with (insert process name here) regardless of who is running it.

To answer the question, then, yes I believe that some more process is required. However, I think the amplitude if the pendulum swings is decreasing and eventually we're going to find a happy medium that's sustainable both within an enterprise setting, and over a long term.

16 November 2008

Agile Circling the Drain?

Yesterday, James Shore posted an interesting blog entry entitled The Decline and Fall of Agile. In that entry, he opined as to how the lack of specific direction with regards to engineering practices in Scrum has secured its place as the leading agile method in use today. That lack of direction is also a leading reason why organizations have had challenges sustaining efforts to move to more agile methods of delivering software. In a nutshell, I couldn't agree more. I've said roughly the same things in the past in my own blog entries, Scrum is not Enough and Call it What It Is!

"Uncle" Bob Martin responded in the Object Mentor blog as well as on Twitter, indicating that he believes the notion that the process is the problem is hogwash. He's correct, but only to an extent in my opinion. Individuals do indeed need to be responsible - they need to strive to constantly improve themselves and their techniques. However, the problem isn't only with the software developers! Testers need to do this, business analysts, interaction designers, business people - they all have to learn to improve how they do their jobs in order to make any process successful over more than just the short term. However, a process that leaves people to their own devices to figure out how to attain these improvements is courting disaster. Common sense, as they say, isn't very common.

That's why, to me, the more prescriptive approach of XP (1st edition) makes more sense in that it follows the Shu Ha Ri method of learning. Initially, you must show everyone practices that work, rather than expecting the people to just find them. I once heard Scrum described as "the VB of Agile Methods", for this very reason. It's easy to pick up and start, but unless you already have excellent practices in place you're going to struggle after a while. Think of the fancy 3D fonts and drag & drop controls that got rave reviews in demos, but the overall system was shipped full of bugs, if it even shipped at all. Don't even get me started on DLL Hell!

My point behind the Call It What It Is blog entry was that saying you're doing Scrum when in fact a number of practices were added out of necessity does a disservice to Scrum and the other processes from which those practices were taken. If you're doing XP but using Scrum terminology, just say so!! If you say you're just doing Scrum, the next group in your company that sees your success is going to say, "Wow, let's adopt Scrum!" If they do so without the extra practices, and aren't disciplined enough to be using them already, they will struggle after some initial success.

So, assuming that James Shore is right and Agile is on its way out, what's next? With one client, I'm seeing tremendous synergies with the application of Lean in the business and Agile (or whatever you want to call it) in the software development group supporting that business. The business people understand what they have to do in order to support the development people, and vice versa. Even the approach of introducing Lean to the business was the same as my approach to bringing in Agile for the development team - a short initial training workshop, then right in the deep end with a a coach there to guide the team through the first few iterations.

I can see that two-pronged strategy as being much more effective than simply trying to convert the development teams. Certainly it's more difficult than just changing the software development method, but isn't that the case with most activities in order to make them sustainable over the long term?

2 November 2008

Learning from Dogs

Our 6-year-old Golden Retriever Hannah is pretty typical of that breed. She's big, happy, playful, lovable and thinks she's a lap dog. She and our cat wrestle as if they're littermates, and she's best buddies with our rabbit. A born predator if there ever was one! :)

Hannah has also had to have surgery on both her hips because of hip dysplasia, a genetic defect in the formation of the ball and socket joint that leads to severe, painful arthritis. Her first was when she was 11 months old, and the second when she was 3-1/2 years. Both of those were successful, and she recovered to being a normal dog.

Late last February we were walking in a dog-friendly park here in Ottawa, and had taken Hannah's brother (who lived around the corner) with us. The two dogs were running around and as best as we can tell Hannah was bumped from the side while both her back legs were planted in the deep snow. She ended up having a sprained MCL ligament in one leg and ACL in the other. The MCL sprain healed fine, but in early April she completely tore the ACL.

The ACL tear, unfortunately went undiagnosed for quite some time. We had been told it was just a sprain, and to keep her resting. She would improve somewhat, but then would start limping quite a bit. Finally, after taking her to another vet, the complete ACL tear was properly diagnosed. The only way to correct it was surgery.

I do need to point out, though, that the only indication that something was wrong with Hannah was the limping. She never whined, wasn't aggressive at all, and still got excited about the "W" word (that's 'walk' for those of you who don't have dogs!). In my 25+ years of playing basketball, I saw 2 people injure their knees with torn ACL's, and both screamed in pain. One, a man in his late 20's, was literally in tears from the pain. Hannah, however, would still climb stairs and jump up on the bed if we'd let her (and sometimes even when we wouldn't!).

This past Thursday, Hannah went in for corrective surgery for her bad knee. The procedure is known as Tibial Plateau Leveling Osteotomy (TPLO), and the surgeon literally cuts off the top of the tibia bone and repositions it to support the motion of the knee. This is done in place of a graft or synthetic replacement for the ACL ligament, and is generally better tolerated in dogs. Hannah came through the surgery just fine, and we picked her up from the vet about 6 hours after she was finished. She was still pretty groggy, and was understandably not putting any weight on the repaired leg at all. It's going to take 8 weeks for the bone to recover sufficiently for her to begin rehab, which I guess will be her Christmas present.

So now on to the point of this entry. This is a dog who has just had her third major surgery in 6 years. On the first night, she moaned a couple of times, but generally just slept. On Friday she was much more alert, but still not quite herself. On Saturday, she was almost back to being normal in terms of her behaviour, and she was actually putting some weight on the repaired leg. This morning (Sunday), you would think that she hadn't even had the surgery. She putting weight on the leg (which we're trying to dissuade), she's as alert and perky as ever, she was playing with the cat, and was begging for food at the table during breakfast.

I do realize that the medication she's receiving is helping to control the pain quite well, but this is still quite remarkable. When it gets right down to it:

Dogs don't know how to be wimps.

A human in the same situation would be lying in bed or sitting on a chair. They would be moaning and groaning about how sore they are, and how long they're going to have to convalesce. Hannah, meanwhile, seems to be saying, "Hmmm, my leg hurts. Anyway, LET'S PLAY!!".

Perhaps when your lifespan is only 12-15 years, you realize that you don't have any time to waste wallowing in your fate. I think there's a lesson to be learned or some inspiration to be gained from this. While we can't always live like today's our last day alive, maybe we can live like we're only around for 15 years.

Suddenly, the ache in my lower back isn't quite as bad.