Using Safe-to-Fail Experiments in Continuous Planning
How adopting experimentation helped a software engineering team successfully shift from annual to quarterly planning.
I was fortunate enough to be working at an enterprise SaaS company during a multi-year, large scale devops and modernization initiative, as we shifted from onprem hosting to public cloud, from manual testing and deployments to automated, from a monolith architecture to microservices, and from big bang quarterly releases to continuous deployments using feature flags.
Of all the changes we made, one of the most difficult was shifting from annual to quarterly planning. Like most global enterprises, we had an annual planning cycle, with budgets created & finalized in Q4, after which teams would be sent off in Q1 for goal alignment exercises and roadmap creation.
Annual Planning Nightmares
It feels strange to admit this today, but I thrived in this environment. Each year during annual planning, as the leader of our Internal Tools, I would check in with the company’s Senior Leadership Team and give them a choice of 4 or 5 big strategic projects that my team could tackle the following year. It was always important to me that my team was working on things that mattered to the business. Once I knew what was most important, I would work with my team to set ambitious goals that might take 6 or 8 or 10 months to deliver. We put our heads down and we delivered on them. Year after year.
We had a reputation for “doing what we said we would” and “keeping our promises” and that was something I was quite proud of.
But after a few years something changed. Senior execs stopped being able to guide me to the most important priority of the upcoming year. I would approach them in October or November with a choice of 4 or 5 business problems and asked them which was most important. And they couldn’t answer. They couldn’t tell me the number one thing I should work on anymore.
It was during one of these frustrating annual planning cycles when I made the decision with my team to change from annual to quarterly planning. If the execs couldn’t tell me what they needed in a year, we were at least going to figure out what the best thing we should do in the next 3 months.
From Annual to Quarterly
Moving from an annual planning cycle to quarterly was much harder than I expected it to be and requires a fundamental psychological shift, otherwise you’ll find yourself doing all the work of the annual cycle, but repeating it every 3 months.
To successfully switch from annual to quarterly planning, I worked with my team to accomplish 3 things:
- Accept the unknowable. (Mindset)
- Know what’s important, aka your guiding North Stars (Strategic)
- Embrace Safe to fail experiments (Tactical)
While you do need all 3 to achieve high performance, I found it was easiest to start with safe-to-fail experiments because it creates psychological safety, gets a flywheel spinning, and will free up the team leader to focus more on 1 and 2 which are required for sustainable experimentation.
What is a safe to fail experiment?
A safe to fail experiment is exactly as it sounds. It’s just an experiment you can run where no matter the results or what happens as a result of the experiment, nothing too bad can happen.
What does experimentation have to do with business?
We know experimentation is important to science. The scientific method is based on hypothesis and experimentation for the purpose of discovery. Scientists run experiments to learn what they don’t or can’t otherwise know.
So what does this have to do with business? Remember the senior leaders who couldn’t tell me what they needed my team to build in the coming year? It wasn’t because they were inept or incapable. No. It was because they could no longer predict 12 months into the future.
Of course the future was always unknowable and impossible to predict accurately. But as technology advances, the future is getting more and more volatile and unpredictable, not just in the long term but in the short term as well.
So maybe there was a time when annual planning was effective. But no more.
Which brings us back to experiments. In the business world today, we must conduct experiments because the future — even in the short term — is unknowable. There are too many variables and any assumptions we make today will have changed within just a few months. Just like scientists, we need experiments to discover and learn. (If you’re looking to better understand the necessity for experimentation in modern work, the book Sooner Safer Happier by Jon Smart, Zsolt Berend, Myles Ogilvie, and Simon Rohrer makes the best case I’ve read.)
Tips for conducting safe to fail experiments
The key when designing experiments is to make sure they are safe to fail. The goal of running an experiment is to learn something. So, the cheaper, faster and safer, the better. If something is not going to work, the sooner you know the better.
Some good questions to ask when designing experiments:
- Can it be reversed?
- What is the worst thing that could happen?
- How much will it cost?
- What might we learn?
- Will we actually change based on results?
- Is there a smaller, cheaper, safer way?
- What objective does this advance?
And remember, do not try to design experiments that will always be successful. If you don’t have failed experiments, you’re not getting the most out of this experience.
A way of life
Designing and conducting safe to fail experiments was a critical piece to our successful shift away from an annual cycle with its emphasis on being “correct” towards a quarterly cycle with its emphasis on “what can we learn” and “what value can we create today”.
We started running experiments instead of delivering a roadmap, and we churned out small bits of value out on a regular basis. We avoided the common and tragic fate of working on a huge project, only to later abandon it at 60% because it was no longer a priority. We didn’t spend as much time in meetings trying to figure out what to build and feeling pressure to be “right” because the goal of all the experiments is to learn.
Experimentation puts the science back into computer science.
Have you adopted Safe to Fail Experimentation? What was your experience? How did you first hear about this approach? What was the hardest or easiest part? Did you have good results?
