Who’s ever heard of a telecom lean startup? That’s sort of what Twilio is, and it’s a pretty cool company and interesting case study.
When I attended the LeanStartup.co conference in San Francisco recently, one of the parts I was most excited about was the tour of area companies. The first tour that I did was at Twilio, and it was very interesting from a lean startup and an agile engineering perspective.
What exactly is Twilio?
Twilio.com tells you that they “bring voice and messaging to your mobile applications.” They have built a great API that developers can use to quickly and easily provision new phone numbers, and build in voice and text messaging automation to your applications. That’s even cooler than it may sound, and if you setup a trial account with them then Twilio will quickly give you a new phone number and guide you through a short tutorial. In less than 15 minutes, I had setup a customer voice mail message on my new phone number, send myself text messages from that number, and setup auto return text messages from that phone number. All of this was done using extremely simple code that they gave me.
Twilio’s guided tour of their features is probably the best I’ve ever seen. I got a quick sense of their capabilities in a very short tour, and I was impressed with how simple they made it look. They have taken an intimidating topic, broken through the learning curve barrier, and got me hooked. Within ten minutes later I was asking their sales team if they have any phone numbers available in Central America since that is where I do a lot of business. I’ve got ideas brewing for how to use Twilio!
Cool office? Check.
Talk about Twilio has been trending in the development community for a while now, so I was excited to see their office and meet some folks. Their office feels like a startup. They have the obligatory indoor bike racks, big monitors, and open room feel.
The desks are all handmade by the team members out of doors and cheap lumber. They started out with a lean budget for office furniture and see no reason to change it. It actually looks good, and beats the used-door desk I had in college. The conference rooms are all named after famous technical people who have inspired them, like Steve Wozniak or Hedy Lamarr (an Austrian actress and scientist who invented frequency hopping).
We sat down at large tables (hand made of course) in the kitchen area. Joel Franusic (@jf) is a developer evangelist and gave us a great demo. Then Kyle Conroy (@kyle_conroy) talked about engineering at Twilio. Kyle is a former intern at Twilio who is now a developer there.
They gave us a great tour of the API, similar to what you’ll see on the website. Joel was getting our phone numbers from people on the tour, and sending us text messages from his console in seconds. It was an impressive taunting of the demo-gods, but it went flawlessly and Twilio has a lot of confidence in their API. It’s the keystone of their business.
Agile Engineering? Check.
How do you develop that much confidence in your software? I asked about agile engineering techniques like test automation, and how they use QA teams. Kyle’s answers were very interesting. Twilio has no QA department, and no testers. They preach code ownership and so the developers are responsible for all testing before it goes to production.
They do lots of automated testing, and those tests run regularly under Continuous Integration servers so that they know how stable the main branch of the code is. They have tried Cucumber, which is one of my favorite tools for automated acceptance testing. For them though, Cucumber was too verbose. They are mainly testing an API, and they felt that the “Given/When/Then” syntax was simply too cumbersome.
I can see where that would be the case. One of the big benefits I think of Cucumber is how it gives you “plain English” executable tests that you can show to your customers. However, in Twilio’s case, everyone understands the technical talk, because serving developers is everything to them. So the extra verbosity of Cucumber is not worth it.
The development process is automated at Twilio so that they can do frequent deployments whenever the developers have small changes ready.
Applying Lean Startup stuff? Yeah, pretty much.
Those of us on the tour also asked Kyle and Joel about their application of lean startup methodologies to their business model. Twilio didn’t start out applying those concepts, but they have done more of it over time.
They pointed out that they suffered some in the past because they had not applied lean startup methodologies or customer development. They spent a long time building a new product called Twilio Connect because they were sure they knew what developers wanted. After investing a lot of time, it was not received well when they launched it.
How long should your betas be? Although I’m not fond of the term beta, Twilio has found that they have to be careful about making too frequent changes to an API. Developers expect you to maintain reverse compatibility, and you can’t change the API all the time or you will break companies’ production code and alienate customers. It’s not as simple as a UI driven company where you have much more freedom to change the look and feel of your website, or make constant small tweaks to the user interface.
So Twilio has found that long beta cycles work well for them. This gives developers a long lead time to start testing with the new features. Twilio tries to communicate clearly that the code is not ready for prime time yet, and so they have the freedom to change the new API calls significantly during the beta period based on feedback from early adopters. This allows them to get very valuable feedback early on from developers, without jeopardizing the stability of the official API.
Once they have a solid understanding of what the development community wants in the new features, then they can publish them into the official production version of the API.
Metrics? Check and Double-check!
One final conversation that came up is very important to those of us interested in the application of metrics to lean startups. Twilio originally did a lot of funnel analysis to get more people signed up for the service. But they realized that they were over-optimizing the sign up process, and the use of their API and customer retention was bad. In other words, they were getting more people to sign up, but they were the wrong kind of people. Some people thought they were signing up for a consumer focused VOIP solution, not a development API.
Twilio had to tweak their metrics to focus on what really matters to them: use of their API. Lower sign up conversion rates are acceptable if it means that the people who do sign up are actually using the API more. Metrics matter a lot in lean startups, but make sure you are focused on the metrics that actually matter to your core business model.
Go check ’em out…
By the time I finished this blog post, their sales team had already got back in touch with me. That’s pretty great for the day after Christmas as I was writing this. While it turns out they don’t have phone coverage in Costa Rica yet like I hoped, there is still so much cool stuff to play with that I will have to make some more time for fun holiday hacking with Twilio. You should too!
