6 February 2012

Pragmatic Agile - Be Principled!

In my previous post on Pragmatic Agile, I mentioned the 6th and 8th principles of the Agile Manifesto.  Just to be clear, I'm not a religious zealot who has the principles memorized chapter and verse and can recite them in order... naming them by their 'number' simply provides an easy reference without having to restate the principle every time!

In that previous post I related how I have encountered numerous people and teams who avoided changing their existing ways of working in the name of being pragmatic.  I can understand to an extent why they felt they were being pragmatic, especially given the first definition above - to them Agile is a theory, and they are afraid to make changes based on theory rather than their own experience.  However, that is NOT Pragmatic Agile, that's simply... well, doing what you have always done and saying that it's Agile.  If that is working for you, then great!  Just don't say that you're Agile.

So what does it mean to be Agile, then?

Around a decade ago I would have told you that you had to be doing Extreme Programming.  Others would have said Scrum.  Others, Feature-Driven Development.  Some, DSDM.  Yes, those are Agile methods that each have their own set of practices, which often overlap from one method to another.  When Agile was defined at Snowbird in 2001, the people behind those various methods boiled down their practices into what was common among them.  That led to the Agile Manifesto's Values and Principles.

The values are a statement of where the software industry was in the late 1990's and where the people at Snowbird felt it should be going.  In North American culture, we would refer to the values as "Mom and Apple Pie" statements - no right-minded person would disagree with them!  By being values, though, they were deliberately vague on how one would implement them, and they also represented a spectrum that could vary from organization to organization, team to team, even person to person.  That's good, but not enough to really be able to make concrete changes.  "No more documentation?!  Hell yes!!  We'll just bash out code until the cows come home!"  Well, it doesn't really work like that.

The real meat of what came out of Snowbird was the 12 principles of the Agile Manifesto.  They are what truly define Agile and what have endured over the ensuing 11 years.  Interestingly, if you were around the Agile world back then, you can also see the compromises between the various methods reflected in the principles. ;)  What isn't a compromise, though, is that those principles are the benchmark for saying, "We are Agile!"

The first principle states,
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Since first reading that, I've heard almost every excuse in the book against implementing that principle.  We can't possibly ship something incomplete!  Our users can't cope with changes every few weeks!  Our system is so large that it takes our customers weeks to install and validate - just when we'd be releasing again!  We have a certification process to follow for each production delivery that's very expensive so we can't afford to delivery early and often!

Yes, those are valid points... when you are working from older assumptions.  There are plenty of ways to address those concerns and still deliver early.  Scrum has spoken for years about completing a Potentially Shippable Product Increment each Sprint, although you may wait until you have a Minimum Viable Product (MVP) before you release to production.  Defining the MVP is a key part of the Lean Startup movement.

As a concrete example, consider credit card billing integration.  In one case I encountered, a system looked like it was taking a credit card and having it authorized automatically, but in reality it was storing the card information in a secure area and generating a quick e-mail to a mailing list.  Someone in the company would then take the information and manually call the card company for authorization.  It worked just fine, the system's users weren't aware there was a manual process in place, and it meant that the credit card billing portion of the system could be put into production much sooner than if it had been integrated with an automatic authorization service.  Bear in mind this was a number of years ago and today there are numerous SaaS solutions for handling credit card processing, but it illustrates the point that you can think outside of the "it all has to be automated" box in order to ship your Minimum Viable Product sooner.

Perhaps you don't have to deliver to production at all, but rather an environment that people can test the system as if it were live.  Another approach is that people could also still use the old system, but be redirected to the new functionality when they needed it.  As for users being overwhelmed by changes, the shorter the release cycle, the fewer changes that will be introduced and the more whelmed your users will be!  If you don't believe this, ask Google and Amazon how overwhelmed their users are when they release new functionality every 2 weeks (I suspect that, like me, you are one of those users!).  Sometimes that new functionality is only released to a small population of users in order to get quick feedback and minimize the risk that all users will be bothered by the change.  There is a well-known concept known as A/B Testing in which a system's users will be presented with two different presentations of a system in order to determine which works better.  The ultimate form of this kind of fast feedback mechanism is the Feature Fake, in which you don't actually implement anything, but just a button or link that pops up a form telling the user that the feature doesn't exist yet, and possibly asks some questions about whether they would actually want that functionality.  In other words, a simple, cheap concrete experiment that obtains real results to help determine if further investment is required.  That's very pragmatic, is it not?

What I find interesting is that the Lean Startup movement has taken this principle, embraced it, and moved to a new level.  Eric Ries' assertion is that following this principle is almost the only way you can be successful as a startup.  Yes, we hear about the really successful companies that started as a couple of people putting some code together, but we don't hear about all the failures that take untold amounts of money down with them.  Eric argues that you must release as early as possible in order to receive and act on feedback in order to ensure that you're meeting the market need you thought existed.

That sounds like it makes perfect sense, right?  Steering your work and your investments of time and money using what Eric calls Validated Learning... actual practice instead of theory.  Pragmatism.

The second principle follows this pattern as well,
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. 
If you're following the first principle, then you have to be ready to accept that the requirements from which you've been working will change.  Not might, not could, but will.  Indeed, even if you have been following a traditional serial development process when have the requirements not changed?

I wrote last year in Whither Requirements that the real goal of any requirements process is to achieve a shared understanding of what is to be accomplished.  This is a really, really tough thing to do and requires much more time than people care to think it does.  The biggest issue with serial processes is that they work from the assumption that once the requirements phase is done, well, it's done.  There is no room for clarification.  There is no room for learning while implementing the requirements.  There is no room for "oh, well I actually meant...".  Change Control Boards are created explicitly to prevent change, because any change is considered scope creep which represents a threat to the budget and schedule.  Budget and schedule are indeed important, but not at the expense of innovation and the inevitable learning about the real requirements of a system while it is being built.

If you stop for a moment and look at the second sentence of this principle, you'll notice that the Manifesto authors went out of their way to explain why changing requirements are important.  If you are working on a project with a 12-month release cycle that locks down the requirements after 3 months, even if that system ships on time the requirements will be 9 months out of date.  That might be acceptable in some domains, but in my experience it just isn't good enough even in public sector domains!

On many teams with whom I've worked, an early release plan contained everything but the kitchen sink in terms of features that just had to be delivered.  Once those teams started working in a mode where they delivered early and often and accepted changing requirements and priorities, an interesting thing happened - many of those "kitchen sink" requirements simply fell off the list.  Once the system was being touched by its users, the true priorities and truly important requirements became clear, and the pet requirements or wish-list items simply were no longer considered necessary.  One group dropped a full 20% of their initial items after 5 iterations... think about the cost savings that represented for the business!

As before, the Lean Startup folks have taken this approach to the next level.  They start from the position that you may have an idea for something that people might want, but until you turn that idea into an experiment, run the experiment and receive feedback, your idea is simply a hunch with as much chance of success as randomly picking a feature an implementing it.  In the Lean Startup world, each experiment will likely create changes in your requirements and thus embracing that change is required if you want to be successful.

So again, success is tied to actual validated practice vs. some unproven theory.

It's pragmatic.

2 February 2012

"Pragmatic Agile"

I've never been one you could call academic.  I'd like to think that I've always had my feet firmly planted in the real world, however you define it.  My focus has always been on delivering value rather than adhering to some theoretical principles.

For example, when I first learned about Extreme Programming in 2000, I didn't read Kent Beck's Extreme Programming Explained first, but rather Extreme Programming Installed by Ron Jeffries, Chet Hendrickson and Ann Anderson.  While both books described XP, the former wandered more into the theory while the latter was more along the lines of, "This is XP and here's how we implemented it."  That approach of applied theory appealed to me, and I was able to apply it myself shortly thereafter.

I'm a huge fan of the Pragmatic Programmers, Dave Thomas and Andy Hunt.  Their first book, The Pragmatic Programmer, is a must-read for anyone entering into the software development profession (and a should-read for most who are already developing software professionally!).  Again, their real world approach appealed to me, and they have turned the whole Pragmatic thing into a franchise of its own.  The books that they publish for other authors all have that feeling of providing concrete advice that you can use immediately.

However, I do have to admit that I cringe every time I hear the word pragmatic.  While Mssrs. Thomas and Hunt have what I believe the right approach to pragmatism, time and again I see that word abused.

Cherry Picking
I can't begin to count the number of times that, as a coach, I've heard programmers, testers, business analysts, line managers, product managers, directors, executives, and probably even the cleaning staff say that they couldn't possibly use some Agile practice.  Instead, they would insist on continuing to do what they had always done.  After all, we have to be pragmatic in how we apply Agile, right?  Well, yes... and no.

This notion of taking what you like from Agile methods such as Scrum, XP and Kanban is known as cherry picking, and in my experience it leads to serious problems almost 100% of the time.

Agile Values and Principles
I'm OK with people substituting practices.  I honestly believe that there are many ways to accomplish a task.  However, if you don't link the practices back to the Principles and Values of Agile, then you are going to experience difficulty.

For example, consider the 7th Agile principle:
Working software is the primary measure of progress.
What that means is... well, I think it's pretty clear what it means.  It does not mean a colourful Gantt chart generated by MS Project by someone who attends the standup meetings and tracks each task by each person down to the hour.  Even if that is done in a spreadsheet, completion of tasks and even the completion of Stories or Backlog Items simply represents activity.  True progress is shown when software is working and in the hands of those intended to consume it such that you can receive concrete feedback on that software's suitability to the business task.

The use of Velocity - Story Points or some other measure of the size of items completed over time - as a means of measuring progress can be (and is often) easily abused.  Despite that, I've always loved that the original XP team chose that term.  Speed is a simple scalar value, denoting distance over time, whereas Velocity is a vector, which has a directional component.  If you aren't receiving feedback on whether what you're building is actually what is supposed to be built, you could be going in completely the wrong direction.  In other words, you aren't measuring Velocity, you're measuring Speed!

The use of the term working also means that the teams don't simply hack together some spaghetti code in order to get the current stories completed.  That leads to the 9th principle:
Continuous attention to technical excellence and good design enhances agility.
I love asking groups of developers to whom I'm speaking if anyone there is a crappy developer.  Of course, there's always one or two who put their hands up and we all have a chuckle.  I then put up some of their code that I had asked for prior to the talk, and start to talk about all the Code Smells in it.  The excuses then absolutely pour in:
  • That was written by the people in (any site but theirs)!
  • We had to get that done before the release!
  • I wanted to clean that up, but my manager said I didn't have time!
Sorry, but a 3,000-line function with a Cyclomatic Complexity of 500 that grew to that size over 10 years is inexcusable.  Apparently, though, it isn't uncommon whether you're working in the procedural or the object-oriented world.

I see developers constantly taking shortcuts in the name of expediency, being pragmatic, about meeting a deadline vs. creating solid, maintainable code.  The ironic result is that rush to meet the deadline creates defects and opaque code that causes them to miss the deadlines even in the short term!

The same applies to automated testing of code.  I agree fully that it's neither practical nor necessary to ensure that you are hitting 100% code coverage in your low-level tests.  That's especially the case when you are applying tests to existing legacy code - you need to evaluate the risks associated with the work you're doing, and test appropriately.  However, for greenfield work it's relatively simple to start testing immediately and maintain those tests over time.  That applies to languages from C to Ruby, Java to Python, and almost everything in between.

Regardless of whether you're dealing with a legacy Big Ball of Mud or working from a clean slate, making the investment in automated tests will make you go faster.  That is the pragmatic approach!

So, if you are choosing to change or omit some practices from Scrum or XP, you need to ask yourself if you can still satisfy the Agile Principles that are part of the foundations for those practices.  If you can, then you are probably being truly pragmatic in your approach.  If not, then you need to look deeper at what needs to be done in order to improve how you work.

Stay tuned - more on pragmatism within the context of the Agile Principles in a later post!