How do you convince engineers that it’s okay to build quick experiments, and deploy code that you will probably throw away? How do you convince rigorous engineers that it’s actually more efficient to work in a lean startup style?
That is a topic that Sam McAfee from Change.org addressed during a panel at the LeanStartupConference last December in San Francisco.
A little bit about Change.org
Change.org is “the world’s petition platform”, and a very cool social activism site where you can start petitions about issues you care about. These aren’t just petitions about your neighbor’s annoying cats, many of these petitions are about real life changing events on a global scale.
For instance, right now there are petitions highlighted on change.org to fight corrective rape in South Africa, to end driving license corruption in India, to allow women to drive in Saudi Arabia, and many more issues. Many of these petitions actually effect the change they are asking for, and Change.org marks those as “won.”
Sam McAfee’s journey to Lean
Sam is director of Web Engineering at Change.org, and I appreciate how generous he was with his time. Sam and I talked more about the issues that he discussed on the engineering panel at the Lean Startup Conference, and we started with talking about his journey to the world or agile and lean startups.
Sam got into software development at the tail end of the dot com boom. He was first exposed to agile around 2002 or 2003. At the time he was building a consulting company and stumbled on Kent Beck and Martin Fowler, and was inspired to incorporate their eXtreme Programming (XP) techniques into his consulting company.
The methods of XP are great, and things like Test Driven Development and Pair Programming are a great way to become more agile. But in Sam’s experience, it’s easy to get lost in the mechanics of those practices but still lose sight of your core goals. The practices by themselves don’t automatically make you “agile”, and it took Sam a while to really grasp that. So far Sam’s story is very similar to my own, and so this was really ringing true to me.
The lean movement got it’s start more in overall business process design and some higher level concepts, and Sam started reading Mary and Tom Poppendieck and getting more exposed to the lean movement. The intersection of agile and lean was originally presented to him as “agile for the project and lean for the enterprise”. So Sam read all he could, going all the way back to reading about Deming’s process improvement work in the post WWII era. Reading Deming really helped him solidify his understanding of the goals that agile has.
As you can imagine by now, Sam is really jazzed about the increasing overlap of the lean and agile worlds.
Combining Lean Startup and Agile Engineering at Change.org
Prior to joining Change.org, Sam had already read Eric Ries’ book on Lean Startups. The team at Change.org was already embracing the “Build Measure Learn” loop, but getting a little lost in the practices. They were also using scrum, but it was only marginally successful. Estimation was all over the place, and stories were not always finished. They were starting to deploy regularly manually, but it wasn’t automated yet. Ultimately they decided that Scrum was holding them back.
Sam reached out to Eric Ries, and asked him to get on the phone and chit chat about how Change.org was applying lean startup ideas. They decided to start separating the work at Change.org into two broad areas: 1) work on well known domains, and 2) more unknown areas where Minimum Viable Products (MVPs) would be helpful.
Coding to learn
How do you get rigorous agile TDD engineers to think like Lean Startupers? For Sam, it required convincing engineers that sometimes code has the sole purpose of learning a thing rather than doing a thing. Crossing this mental block means it’s okay to throw that code away and not be too strict about how you code it.
Here are a few challenges that Sam and Change.org ran into initially:
- The team had code quality very high in mind, which is great for a known problem. But when building a new business, it gets in the way of testing ideas.
- You might not keep experiments or A/B tests – there’s at least a 50% chance that you won’t keep the code.
- Engineers were aghast at the waste of a lean startup process initially.
- They were afraid that quick & dirty code would get integrated with the rest of the app and left there as technical debt.
- The team initially struggled to modularize the experiments, and send user segments to the experiments
- Business stakeholders had to understand that even successful tests still need to be re-engineered
It took patience and some architectural changes, but now the team is pretty accomplished at running frequent lean experiments before they invest a lot of time in new ideas for Change.org.
Most of the experiments are a couple days to a week in length. They are integrated into the scrum sprints but not in a real formal way, and the experiments are not bound by the time box of the sprint. Product Owners have to indicate if a user story is related to an experiment or a permanent change on the site.
From the customers’ perspective, the features on the site just gradually change for users. There are no “big bang” releases or big announcements.
On the tech side, Change.org runs a Rails stack. Their Continuous Integration (CI) system is custom designed for their needs. The team can spin up AWS server instances to run tests on production type environments.
Code deploys happen multiple times a day with small clusters of features in them. These deployments are not fully automated yet, but the team continues to iterate towards that.
Automated testing is important for an environment that changes this frequently, and so the team writes RSpec and Cucumber tests at various levels of the system. Tests have to run parallelized now because the test suite is pretty large.
Closing thoughts
Lean Startups help tie together business and engineering concerns. This is a key point to Sam, and one of the big benefits. It makes it easier to show that every decision you make is ultimately an economic decision. Will this change help us achieve some goal with our customers or not? No tool should be purchased or software developed until you have proven it’s benefit in advance with lean experiments.
Engineers understandably want to wait to delay all code deployments until everything is perfect. And while it may feel dirty to deploy code that you know is not perfect, and doesn’t have fully automated tests around it, you need to make a distinction between code that is intended to be permanent, and code that is a temporary experiment.
As Sam pointed out, even Kent Beck says it’s ok to write some code without TDD. The agile engineering practices are good, but you have to understand the intent.
The lesson I am drawing from Change.org is that if you can draw a clear line between experiments and permanent code, build an architecture that allows you to run experiments easily, and attend to the cultural concerns of engineers and the business, then you too can balance rigorous agile engineering with highly efficient and dynamic lean startup software development.
