Designing Great Tests
“A problem well-stated is a problem half-solved”
Tests are your friend.
They catch issues while they are fixable, they give you confidence in your ideas, and they reveal new opportunities that can make/save you a lot of money.
They are also wildly unpopular with a lot of new entrepreneurs, as they feel high-risk or confronting.
Neither is true; tests reduce your risk exposure, and they are the least confronting way of receiving surprising or unpleasant news.
A poorly designed test can feel expensive, unnecessary, time consuming or inconclusive – all things that a founder can’t afford.
Luckily, this is a learnable skill, one that you’ll use across every business project you work on in the future.
First things first, you need to decide that you do want to run a test…
Declaring Your Intent
If you resent the testing process, you’re unlikely to get the benefits and will wrap it up too quickly.
We want to be good scientists, forming ideas and checking them in a fair and unbiased way.
Our intuition and gut feelings are helpful for forming hypotheses, but then we need to boot them out of the room while we run the experiment.
Finding out that our ideas are wrong is actually a win for our projects, even if it means taking a few “backwards steps”.
It’s helpful to change how you frame the situation, try this template for yourself:
“I am thankful for my gut feeling and experience because ___________________.
However, now we need more than that, because we’re about to commit a lot of time, money and talent towards this next stage of growth.
We need validation, we want quickly but also can’t afford to rush it.
We are excited to run a test because we’ll validate the
o Desirability
o Feasibility
o Viability
o Impact
of the business.
We’re going to use fair, objective measurements, not wishy-washy descriptors.
For us, a failed test is as good as a win.
We would rather a failed fair test than a rigged and flawed false positive.
Naming The Most Crucial Assumption
Next, we articulate what assumptions should be tested.
These are usually the ones that are catastrophic if they are wrong, or lucrative if they’re right:
“We believe that:
o Customers want…
o Partners can provide…
o We can serve…
o People are willing to pay…
o We need to own…
o People will sign up for…
o It will cost us…
o This problem can be solved by…
o
And this must be verified.
We don’t want vague compliments or to score highly in vanity metrics that don’t matter.
We want the truth.
More specifically, we want commitment and advancement, our customers, partners and team putting their money where their mouth is.
And we want them to be enthusiastic, not twisting their arms.”
Choosing What To Do
Now we describe what will verify those assumptions.
This should be something we can reasonably do it a short timeframe, and involves an action rather that passive research.
We need a sample size that is large enough to be statistically valid, but not an administrative burden (two customers is likely too small, 500 is overkill, it depends on your industry).
The output from our actions should be measurable, not a “vibe” or verbal feedback.
We also need to decide what constitutes a pass or fail ahead of time, so that we’re not “shooting the arrow and painting the target around it”.
It helps if you have an accountability partner, who can act as a sounding board and stop you from skewing the data.
“In order to verify this quickly, cheaply and accurately, we will…
We’ll have a large enough sample size if…
And measure...
The test is successful if...
The test is failed if...
Our accountability partner will be…”
Signing Your Work
It’s important not to limp into this.
You want to be resolute and excited – no matter what the outcome is.
If you’re hating the idea of this test, now is the time to talk about it.
It’s helpful to remind yourself why you’re doing this, and to have a reward or incentive for completing the test, even if it’s unflattering.
That way we’re likely to continue testing our ideas in the future, not resenting the process.
“These tests require emotional labour, creative energy and vulnerability.
We want more of these in our work.
Therefore, after these tests we will reward ourselves by...
I promise to conduct a test that is brutally honest and relentless fair, because it helps us to...
Signed ______________________”
All together it reads like this:
“I am thankful for my gut feeling and experience because...
However, now we need more than that, because we’re about to commit a lot of time, money and talent towards this next stage of growth.
We need validation, and urgently.
We are excited to run a test because we’ll validate the
o Desirability
o Feasibility
o Viability
o Impact
of the business.
We’re going to use fair, objective measurements, not wishy-washy descriptors.
For us, a failed test is as good as a win.
We would rather a failed fair test than a rigged and flawed false positive.
We believe that:
o Customers want…
o Partners can provide…
o We can serve…
o People are willing to pay…
o We need to own…
o People will sign up for…
o It will cost us…
o This problem can be solved by…
o
And this must be verified.
We don’t want vague compliments or to score highly in vanity metrics that don’t matter.
We want the truth.
More specifically, we want commitment and advancement, our customers, partners and team putting their money where their mouth is.
And we want them to be enthusiastic, not twisting their arms.
In order to verify this quickly, cheaply and accurately, we will…
We’ll have a large enough sample size if…
And measure...
The test is successful if...
The test is failed if...
Our accountability partner will be…
These tests require emotional labour, creative energy and vulnerability.
We want more of these in our work.
Therefore, after these tests we will reward ourselves by...
I promise to conduct a test that is brutally honest and relentless fair, because it helps us to...
Signed ______________________”
What Sort Of Tests Might We Use?
You have a lot of options for different tests, and you don’t need to follow a strict recipe (so long as it gives you a meaningful and measurable result).
Books like Testing Business Ideas are great, full of examples to suit all stages of development.
Here are some real examples from our clients:
Launching a crowdfunding campaign to raise funds and build momentum for a new product, service or venue.
The founders create a video and landing page, sponsorship/purchase tiers, a financial goal and a deadline.
They do not need to actually build the product or run the service as yet.
With most crowdfunding platforms, the money only goes through if the financial target is reached, or else it gets refunded to supporters.
We’ve seen this raise several hundred thousand dollars, and we’ve seen campaigns reach only 3% of their funding target, which served as a helpful wake up call.Hosting a pop-up stall or space for several weeks, e.g. in a vacant retail store or mall, or as a co-tenant in an existing shopfront.
The founders might sell their wares, talk to customers, measure foot traffic, measure peak times and evaluate whether their income will outpace their expenses.
One entrepreneur set up a table at a popular market and started offering free personal finance advice – her conversations at that market became the basis for a full course and gave her some prototypical customers.
In several cases, this showed the founders that they did not want to proceed in a retail space, or highlighted some specific needs for their future physical spaces.Creating a landing page and driving traffic to it, measuring clicks and email signups.
Sometimes this is done organically (e.g. with existing fans or an audience you know), sometimes this is done with paid marketing (e.g. search engine marketing or social media ads).
It’s easy to test several versions of the page and compare conversion rates, and you can tweak different elements and measure the differences.
This also has hides your popularity (or lack thereof) – while crowdfunding tells all visitors how much money is committed, your website doesn’t have to tell people how many others have indicated their interest.Customer interviews in the style of The Mom Test.
These are low stakes conversations where nothing is being “sold”, they are purely questions for prospective customers about their preferences and past behaviour.
Customers tell stories about what they’ve bought in the past and why they chose what they did, which is far more honest than fishing for compliments or asking them if they’d choose your brand in the future.
Once they’ve followed the process outlined in the book, founders consistently gather brilliant insights from these interviews, both from the reassuring and surprising responses.
These can send founders back a few steps (for good reasons), or spark new product ideas, validate price points, and gives a realistic picture on who your customers might be and how they behave.Creating a mock-up.
These might be physical prototypes, a digital rendering of a product, app or venue, a wireframe of a digital experience, etc.
Founders are able to show these to partners and customers, gathering feedback on design elements and learning from their questions.
The aim is to look as real and professional as possible, or else the new idea isn’t taken seriously.
These work well on landing pages, social media posts and in investor presentations.Customer interactions with white-labelled products/services.
A founder can take an existing product from a competitor, change the label and work with customers to see how they treat it.
This allows them to learn about what delights customers, what might be missing or what customers are willing/able to pay.
It might reveal that the current options are woefully inadequate or perfectly adequate, which will shape what you choose to design next.
Similarly, you might coach a customer through the purchase process, to learn about what they need, see how they make a choice as well as how happy they are in the long run.Asking interested customers for a commitment.
This is the moment that separates polite conversation from genuine buyer interest – two things that might be hard to determine depending on your local business culture.
You get to ask for the sale.
The entrepreneur segues the conversation towards a nice sounding offer, perhaps with an incentive or discount, for the other person to consider.
You don’t have to make it awkward, the aim is to give customers a chance to make a commitment or advance the conversation, like a second meeting or a pre-registration for a launch.
One client was confident that they had built a “reference group” of peers and end users who would become the first batch of major customers.
When this person presented each of the reference group with an offer (pay $20k for three years’ worth of access to the platform), they were stunned to see zero out of the fifteen people take them up on the offer, leading to a pivot in the idea.
Another entrepreneur received almost a thousand responses to a survey, so they sent an event link for interested customers to sign up – and they did!
One Test Per Major Assumption
It would be more efficient to test every assumption at once, but in practice this ends up obscuring the truth.
If the test works, you won’t know why, and if it doesn’t work you also won’t know why.
You’ll likely find that different assumptions require totally different types of test.
Some might be quantitative, others qualitative.
Some might need deep insight, some might be looking at broader surface trends.
Some might be highly personal, others might be focused on Google Analytics.
A neat way of thinking about this is to use one test card per assumption or focus area.
e.g. one for desirability or feasibility or viability, one for your impact model, etc.
You might create a test card around “Which social channels should we focus on?”, and another test card on “What are these specific customers willing and able to pay?”.
Each test card has four prompts:
1. We believe that…
2. To verify that, we will…
3. And measure…
4. We are right if…
These prompts can feel annoying, but it’s because they bring a helpful rigor and thoughtfulness to your work.
They are totally safe for you to use, but can feel risky and daunting.
If you’re stuck, try this:
Write out four bad test cards and show them to a friend or coach.
Your friend might have some suggestions, but more importantly the process of you talking about them to someone else causes the proverbial light bulb to switch on (usually just as you finish talking them through the bad cards).
Then the two of you can create a better version of each test card, and you’ll have an accountability partner who will keep you on track.
If nothing else, remember this:
Tests are a little bit daunting, but they shouldn’t be terrifying.
If you feel ill at the idea of running a test, it’s almost certainly because of a story you’re telling yourself, not because there’s actually a lot at stake.
Try out the design process above, and see where you feel the most “resistance”.
Talk to someone you trust, and fill in the template together.