17 December 2009

Predictions for the New Decade

"Prediction is very difficult, especially about the future."
Niels Bohr
I was asked yesterday to provide any predictions that I might have for the computing industry for the next year. Initially, I wasn't sure I had anything significant to predict, but then I remembered a thought that has been rattling around in my head for a while now.

First, it's hard to believe that this decade is coming to an end already. Ten years ago in December 1999 I was working on that project, which was the absolute antithesis of anything resembling Agile. A year later, I had found out about Extreme Programming and the rest is history. So, onto my predictions...

This prediction isn't so much for 2010, but for the upcoming decade. Looking back at two previous game-changing trends in computing - Relational Database Management Systems (RDBMS) and Object-Oriented Programming - you see a very distinct pattern. There was some initial early work in both areas, but once marketable products became available, both of those trends took about 10 years to become entrenched enough in the market to be considered by mainstream companies. At about that same 10-year mark, there was also significant breadth in terms of the number of products and companies involved. Indeed, it could be considered that at the 10-year mark those respective industries were quite fragmented - different products and companies pushed different approaches and tools.

At the end of 2009, I see the same thing happening in the Agile Software Development world. We are at a point where the mainstream is beginning to consider Agile as the way they should be approaching their software delivery process. We are also at a point where there is a good level of fragmentation in the market - Scrum, Kanban, Lean, Extreme Programming/Industrial XP, Feature Driven Development, RUP and DSDM are all in use to various degrees. There is a level of competition among these approaches not unlike that which occurred between various vendors of RDBMS systems and OO languages. We're also seeing a burgeoning market for tools in the Agile space, especially for those for project management and those that facilitate communication among distributed teams.

For 2010 and on into the coming decade, I see a similar pattern to these other trends. Market leaders will emerge, even if their approach isn't necessarily considered the 'best' by purists (which is arguably already being seen with Scrum). There will be consolidation among approaches, companies and tools, and even a standardization of Agile approaches. For the latter, early work is already occurring for the technical practices in the form of the Software Craftsmanship and Agile Developer Skills movements.

By December 2019, I expect that the number of approaches to Agile Software Development will be down to just two or three. These may not look exactly like the approaches we see today, but their roots will be quite recognizable because they will follow the Values and Principles of 2001's Agile Manifesto. Agile as an approach will be firmly entrenched in the Late Majority, with even Laggards considering it.

Finally, the success rate of software projects will become so high that The Standish Group will cease to publish the CHAOS reports and shut down CHAOS University by around 2016 or 2017. OK, that's a stretch but, with no disrespect intended towards Jim Johnson and the people at Standish, it's a laudable goal nonetheless!

All the best to everyone in the new year and new decade!

[New addition based on feedback from Bob Marshall and Lisa Crispin]

Something I wasn't clear about above - the few processes or methods left in 2019 will likely incorporate facets from all of the ones that we currently see. There will be some variation in order to accommodate different types or sizes of projects or teams, but in the end all processes will be high-level enough to allow individual teams and organizations to tailor the process to their own unique circumstances.

Something Lisa mentioned that I hadn't was that the term Agile itself will slowly diminish in its use over the next decade. This will occur to the point that it will simply be the 'default' way that software is produced, much the same as how the Waterfall became the de facto standard during the 80's.

14 December 2009

On Estimation

My family moved over the past weekend to the lovely little town of Richmond, ON (actually part of Ottawa, but it's still a community of its own). We decided that for this move we would hire a company to both pack and move for us.

A couple of months ago I had a few moving companies come to our old home and give us estimates of the cost. Each company sent out a sales representative who walked around the house, attempting to determine how many people and how much time it would take to pack and perform the move. One company came out with a significantly lower estimate, and they had moved other people that I knew. So, we decided to go with them.

On 'packing day' the two packers showed up just when they said they would. Within 15 minutes, though, they said that the sales rep had underestimated how much was there. In the end, what was estimated to be an 8 hour job took 12.5 hours.

The next morning the movers arrived, also right on time. It only took them about 5 minutes to start saying that there was more than estimated. The lead mover then spent the rest of the day griping about how the move had been underestimated. I spent the rest of the day griping about how the sales rep said that the move wouldn't cost more than the estimate.

By the time the movers had the truck loaded, they had determined that several substantial objects wouldn't fit without a second trip (and the associated extra costs). So, I had to cut scope.

OOOPS... I think I just gave away the point of this blog entry!!

Here I was, the Customer, being told that I had to cut scope or face additional charges. I didn't get to choose what was going to be left behind for me - the movers told me what wouldn't fit.

It sucked. A lot.

After some reflection, I wondered if sending at least one other person to handle the estimation would be better. The sales rep assured me that he had 25+ years in the moving business, and I have no reason not to believe him. But this is the same failing as having a business or systems analyst go off and gather requirements and determine the size of a project:
No matter how much experience you have, estimating in the absence of the people who will actually do the work will lead to errors that affect the ability to deliver all that is required, a quality product, or both.
When estimating, include as much of the core Project Community as possible. If that means juggling some schedules to fit everyone in, then do it. Having that input is critical since it provides sanity checks on the amount and type of work to be performed. Involving more people will increase the likelihood that someone may have an idea for a simpler implementation of a particular feature. Involving more people increases the level of understanding about the problem to be solved for everyone. Involving everyone increases the cohesiveness of the community that will work together.

I knew intuitively that, from the Customer's or Product Owner's perspective, the traditional estimation process where people talked to you then went away and dreamed up a schedule just didn't work very well. Now, having been that Customer, I know for certain that it's the case.

8 December 2009

WAQB Plagiarism

For the second time, the World Agile Qualifications Board (WAQB) has plagiarized the work of Lisa Crispin and Janet Gregory. The WAQB's Agile Testing Foundations course outline is almost an exact copy of the Agile Testing book's table of contents.

The Agile community wants nothing to do with these people.

20 November 2009

Agile in the Public Sector

A good 80-85% of my work since 1991 has been on contract for the federal government here in Ottawa. For those not familiar, Ottawa is Canada's capital and seat of government. There are some 200,000 people in the region that are federal government employees, and nearly as many who are secondarily employed as terms, contractors and consultants or support staff for companies that provide those people. In short, the government is a huge part of business in this town, and I've been part of it for a good long time. In terms of the groups with whom I've worked, I've had ups and downs, seeing the good, bad and ugly of organizations and their management.

Since I first learned about Agile in late 2000 (actually it was Extreme Programming then - the term Agile Software Development wasn't coined until a couple of months later), I've been asked more than a few times if it applies to the public sector. After all, I was told, there really isn't that much change in the business of running the federal bureaucracy!

Really?

Since I first started doing work for the government in 1991, there have been 6 general elections with 2 changes in the ruling party, and one with a new leader (effectively a change in party!). Each of those changes brought major changes in policy and thus legislation. Even after elections in which the ruling party did not change, new policies that were part of the election platform were enacted. There have also been numerous re-organizations of departments and agencies in that time, each of which changes the focus and direction of the business.

In other words, the business changed, sometimes dramatically.

Additionally, there have been external events that have affected the business of government. A group I was with from 2003-2006 works on an immigration system whose raison d'ĂȘtre was the 9/11 attacks. That system was also affected after the 2004 Indian Ocean tsunami when the Immigration Minister quickly created a program to fast-track requests from that area. The development group was able to quickly respond to the required changes and put them into production so that the program could be supported.

All of this illustrates that even in the public sector business change does occur. It can be rapid and disruptive or move at a glacial or tectonic pace. Regardless, it happens and organizations need to be in a position to deal with it. Of course, being a huge proponent of Agile Software Development, this is where I say, "And Agile can help you deal with that change!"

That's true, of course, but I would suggest that where Agile provides the most value for public sector organizations (and arguably all organizations) is not with these macro-level changes but rather with the much smaller changes that occur as a team, group or organization learns what problem they are really trying to solve with an automated system. This occurs on a daily basis - during visioning and chartering exercises, story creation, release planning, iteration planning, daily standup meetings, ad hoc discussions, iteration demos, retrospectives and even after release when more real-world feedback is obtained.

All of what's learned in those opportunities for collaboration is what sets Agile apart from other processes. The ability to quickly apply the new knowledge and get it into the hands of the people who need it is by far the most important factor. This is what pushes the features most required to the top of the priority list. This is what eliminates features that aren't required, and reduces the priority of those that are less important. This is how knowing the Done State of work allows the whole team and project community to understand how long it will take for features to be released. This is how transparent progress on projects allows the organization to better understand and manage their entire portfolio of IT work. This is how an organization can position itself to be proactively ready for the next macro-level change.

And that is agility in the public sector.

16 June 2009

A New Generation, A New Perspective

At the beginning of May I attended the Cutter IT Summit in Cambridge, MA. I quite enjoyed it, since it was not just about Agile but many other aspects of the IT industry. The summit gave me the opportunity to soak in knowledge from many thought leaders from various sectors of IT.

A common theme that I encountered is that we are in the midst of a sea change in IT. The current economic circumstances are certainly a driver of this, with organizations abandoning old business relationships to save incredible amounts of money moving their operational IT work to the cloud, and its natural extension of Software as a Service (SaaS). Indeed, I'm an advisor for (and did a pile of development work with) local social media startup Favequest that is entirely run in that manner.

The other change vector we are seeing is a new generation of people both entering into the IT industry and consuming its services. I'm talking about those kids (which is now anyone more than 5 years younger than me!) who can send text messages from their smartphone while it's still in their pocket. To them, technology is a given - it's always on, always available, and these "kids" have very little brand loyalty. If your product or service no longer appeals to them you're done! Many of the traditional rules of business have been turned upside down.

This reminded me of a presentation I gave at the CUSEC conference in 2005 on Extreme Programming. After giving my talk, I fielded the usual questions. One question, though, absolutely stunned me. A student asserted more than asked that "XP seemed like a very heavyweight process". I hadn't realized that my perspective, and that of those who have been in the IT industry for any more than a few years, had skewed my view of what constituted a lightweight process. In the absence of any other perspective, XP looked very prescriptive and heavyweight to this student. Indeed, most if not all of the development processes that form Agile were reactions to what were perceived by my generation as heavyweight and prescriptive.

The lesson in this, I believe, is that our perception of something is precisely that - our own! We may share that perspective with others who have similar experiences, but we must be judicious in the use of that assumption.

So what, then, will the software development world look like when the Agile methods are indeed heavyweight and prescriptive? I think the App Store is a hint of what's to come. There was also a session accepted for Agile 2009 entitled, Agile's Too Slow: Developing a Facebook App for the Obama Campaign. Finally, at a recent Agile Ottawa gathering, Luc Levesque of TravelPod described the development process his company uses, which includes a deployment to production as soon as a developer commits to source control.

These micro-processes are all predicated on gathering usage statistics and user feedback in near real-time and executing immediately on the results. The business environment can literally change hour to hour, so everything in the process must be geared towards handling that flux. As a business, you simply can't afford to complain that things change - you have to adapt or you'll go out of business.

Essentially, in the always on, always available world even Agile circa 2009 isn't sufficently lightweight. I'm sure there will be those that say micro-processes don't apply to them.
What about life-critical software such as that in medical devices and avionics? What about control systems for nuclear power stations? There's no way we could use a micro-process for embedded software development!
Funny, I heard the same questions almost 10 years ago about XP and other Agile processes.

The sea change discussed at the Cutter IT Summit is happening whether we want it to or not. We need, therefore, to decide if we want to ride the wave, get washed away, or just get the hell out of the water altogether. For the latter two cases, there are plenty of young people ready to cast off the burden of our overly prescriptive Agile processes and get down to really delivering some value!

14 June 2009

Rounding the Corners

I was cutting the grass this afternoon and remembered back to when I was a kid. My parents had a relatively large lot, and it used to take me a good two hours to cut the lawn. The problem was, I hated cutting the lawn. Even when I was a teenager, I couldn't understand why we want to "maintain" our lawns, making them look as artificial as possible without resorting to Astroturf.

I would do whatever was possible to get out of cutting the lawn. I still do, for that matter! Raining in Sudbury? Well, it could make it to Ottawa at any time... that would be unsafe! Inevitably, though, I had to cut it.

Even then, I wasn't going to go out of my way to extend my agony any more than was necessary. The back lawn in particular was quite large, and by itself it took a full hour to cut. I guess I have the process improvement gene turned on, because I spent a good amount of that one hour trying to figure out how to make it something less than, well, one hour. Eventually I figured out what slowed me down - square corners. Each 90 degree turn meant stopping, turning the mower and starting again. If I could remove as many of those corners as possible, I would reduce the amount of time it took to cut that lawn.

This particular yard wasn't perfectly rectangular, and had several trees. I determined that there were a few places along the edge I could cut first to remove some of the variance in the shape of the yard, and that I would have to do at least one full pass to cleanly cut the outside perimeter (OK, so I did care a bit about the quality as well!). After that, I kept to cutting in one continuous strip that was curved around the corners rather than square. This didn't bother my Dad at all, so I then set about optimizing the new process.

It took a couple of tries, but I eventually reduced the cutting time from a full hour to 45 minutes. Being a rather impatient teenager at the time, having an extra 15 minutes to do what I wanted was much better than nothing!!

This relates to practices in Agile that allow a team to move faster, or at the very least sustain their current velocity. Automating anything that is repetetive, tedious and chews up time goes a long way towards reducing and even eliminating those 90 degree turns. Testing, builds, deployments, etc. are all practices that can be automated and have a tremendous amount of tool support now. These "rounded corners" will keep the team moving ahead at a steady pace.

Oh yeah, after I moved out of my parents' house, my Dad bought a riding mower. Cutting the grass is now considered fun, I'm told. I have never had the opportunity to use the riding mower myself. I'm sure there's another lesson there somewhere! :)

11 June 2009

Revert to Training

I was having a "conversation" on Twitter this morning with Esther Derby and Scott Duncan about how good engineering practices are required in addition to Scrum's project management practices.

This isn't anything new - I've blogged about it before.

We discussed how it often seems that it's easier to introduce the management practices of Scrum than the engineering practices of XP. Esther asked me why I thought this was the case, and a thought struck me - it's how the developers were trained.

(Warning - here comes yet another aviation analogy!)

During my flight training, my first instructor made a point of saying early and often that it was very important to listen closely, do what she said the way she said to do it, and to practice that way afterwards. She emphasized the importance of this, saying that "when faced with an emergency, you'll revert to your original training". That was in 2005, and even now I hear her voice in my head when I'm flying and practicing things such as forced landings.

How does this apply to software development? Well, consider the poor overworked software developer. They've had this bloody Scrum process thrust upon them meaning they have to deliver more, deliver it faster and have those annoying business people around all the time to pester them. On top of it, the damned "Agile Coach" is harassing all the developers to write more code that tests the code they've already written. Near the end of the sprint, this poor developer is running out of time and still has work to complete - they are facing an emergency! So what do they do? They revert to their original training.

What was that training? If they went to university or college, likely long hours working alone or in very small groups trying to get assignments done. They hacked something together that met the requirements for the assignment, and could afford to make it throwaway. Automated tests? BAH!! Simplest thing that could possibly work? How's that going to impress the prof? In the end, the attitude is to get something - anything - done in order to get the marks.

It's this training environment that leads to the problems when the developers are "faced with an emergency" later in their careers.

26 May 2009

(Almost) Instant Karma

Recently, a team at a client that had been having success with the managerial practices of Agile was beginning to feel the dull pain of gradually mounting technical debt. The lack of automated tests and a slew of code smells were pointed out, and the developers were quite keen on learning how to apply their new skills for identifying and eliminating debt taught to them by technical coach Mark Levison.

It was pointed out to the team's management and the Customer that there would be some cost to paying down the debt, not unlike the Principal and Interest ratio in a loan payment. While the payment is the same (the length of the iteration), the amount paid as interest (technical debt) gradually becomes less and more is paid towards the principal (stories). This was accepted, although the Customer correctly wanted to be able to at least see what work was being performed to pay down the technical debt.

After this meeting, the development team's manager talked to me. He was concerned that the developers would be spending time trying to make the system "perfect". While I agreed that "good enough" would suffice, I didn't agree with this manager's next assertion: "The system isn't going to be changed after this next bit of work is completed." I pointed out to him that the vast majority of work on a system is performed after it has been released into production, but he still had a lingering concern.

Fast-forward a few days, and I'm notified by this manager that the team is losing one of its developers and half of the time of one of its business analysts. I was not happy, to say the least, and asked why this was happening. The manager said that their last project needed them back to do some new work. I asked, "What new work?". The manager replied, "Some new requirements came up after the users got the system in production."

He didn't seem to understand the irony of the situation.

20 May 2009

Agile Project Management Public Course - July 28-29, 2009

Mayford Technologies will be holding a 2-day public Agile Project Management course in Ottawa, ON on Tuesday, July 28 and Wednesday, July 29, 2009. For this session we're offering summer special pricing of $995.

The course provides an overview of Agile Development, and allows the participants to understand its background, motivation, and benefits. They will learn how to manage and track Agile Development projects, and strategies for introducing Agile Development to their organization.

Registration is limited, so sign up today!

Agile Ottawa Meeting - May 26, 2009

Topic: Season Retrospective

Date: Tuesday, May 26 from 6:00-8:00 PM

Location: The Code Factory, 2nd Floor - 246 Queen Street, Ottawa, ON

For the Agile Ottawa May meeting we will be doing some self-reflection on the season 2009-2010 season. The retrospective, like many other Agile retrospectives, will be to look back on the season ending with this meeting and look at what we did well, what we need to improve, and in this case what we want to become.

More details at the Agile Ottawa Site.

16 April 2009

Agile Ottawa Meeting - April 28, 2009

The Agile Ottawa group will have its April meetup next Tuesday, April 28th from 6-8 PM at The Code Factory.

This month's focus is on the Product Owner role, with Paul Relf from Nortel as the guest speaker. Paul helped to implement Scrum at Nortel, and will discuss his experiences learning about Agile and rolling out Agile on a large distributed project. Paul has also helped train other product owners.

More details available at: http://agileottawa.wordpress.com/2009/04/16/april-meeting/ .

Hope to see you all there!

11 March 2009

Software Professionals, or Joe the Plumber?

Over the past few weeks I've seen considerable discussion about how we in the software development industry need to advance our profession. Bob Martin et al have penned the Manifesto for Software Craftsmanship, there has been an active discussion on the Software Craftsmanship Google Group, and the Agile 2009 talk I proposeed on Professionalism has seemed to strike a nerve.

Something that has struck me during these discussions is that when we speak of being professionals, we always equate ourselves with Doctors, Lawyers and other very visible occupations with professional designations.

This has been gnawing away at me. I absolutely agree that we need to move away from the ad hoc nature of how people become software developers now, i.e. anyone can just say that they're a software developer, and who am I to argue otherwise?! What I don't necessarily agree with is attempting to gravitate towards what are considered the higher professions in medicine and law.

Maybe what we need are masses of plumbers, mechanics or electricians.

I don't mind the analogies between medicine and software development - both areas have significant, rapid change as well as role specialization. Contrast that with the work my Father-in-law did as an Aviation Maintenance Engineer (AME), starting in 1943... not a whole lot changed for him in terms of the type of work or even the tools or materials with which he worked between WWII and the last Cessna 150 he checked out in the summer of 2008.

So does that mean we have to be doctors rather than mechanics? Well, I can say that the materials and tools with which I work today have indeed changed since I first started programming in 1981 - or have they? Certainly how I use the tools of the trade hasn't really changed all that much - the technology has simply allowed me to do it much faster. But I still use the same programming structures I learned in the 80's. I still have to compile code, though now I have a runtime or VM that handles the linking for me. The languages have changed somewhat, but not really all that much.

The point is that perhaps we aren't all that far removed from a trade like that of an AME. The commitment to quality that they have is not unlike Bob Martin's Clean Code Wristband concept. Perhaps viewing ourselves as AME's - aircraft mechanics - could move us past software development as a craft an towards a trade.

Maybe being a trade is good enough, and attempting to be a profession is gold plating. I'm interested in your thoughts.

1 February 2009

Scrum is not Enough - Redux

Last year I opined that Scrum without technical practices is not enough to provide teams with sustainable agile development. I've also commented on the topic more recently in False Advertising and Call it What it is!

This past week Martin Fowler, Ron Jeffries and Joshua Kerievsky have said the same thing, as did James Shore back in November. I would say that all of these gentlemen were more eloquent than me, but the message is the same. Scrum for software development without technical practices, such as those from XP, is not sustainable.

Code must be refactored. Continuous integration (or at least regular, i.e. daily, builds from source control) must be used to verify the state of the system. Testing must be applied at the unit, system and acceptance levels. Automating that testing will provide feedback on a time scale that will enable teams to be agile. If the tests aren't automated, manual testing will become a bottleneck over time, slowing the entire team down.

We have solutions for those problems but, humans being humans, we want the easy answer. Scrum is an easy answer in the sense that it doesn't tell development teams that they must use technical practices in order to sustain their initial gains. Well, interest-only, 50-year amortization mortages seemed like a great idea that allowed people to own homes much more expensive than they could otherwise afford.

To get all Dr. Phil on you, "How's that workin' for ya now?"

9 January 2009

First Agile Ottawa Meetup of 2009

The first Agile Ottawa meeting of 2009 has been announced on the Agile Ottawa blog.

It's Tuesday, January 27th from 6-8 PM at the Code Factory.

Hope to see you all there!