What is the product market fit survey?
The product market fit survey is built on one question from startup marketer Sean Ellis. In a blog post dated 27 July 2009, The Startup Pyramid, Ellis described asking existing users how they would feel if they could no longer use the product, and wrote that product/market fit requires at least 40% of them to answer very disappointed.
In the copy the Wayback Machine saved on 31 July 2009, Ellis says the line came from comparing results across nearly 50 startups; by the December 2009 copy, the same sentence said nearly 100, the figure the live page and Vohra's article give. Ellis also wrote: "Admittedly this threshold is a bit arbitrary." Startups struggling for traction were always under 40%, Ellis wrote, while most that gained strong traction were above it. The same post adds a condition: the users also need to represent a target market large enough to build an interesting business.
Rahul Vohra, founder and CEO of the email app Superhuman, described the four-step process Superhuman built around the question in a First Round Review article published on 12 November 2018. Vohra describes Ellis as having run early growth at Dropbox, LogMeIn and Eventbrite before coining the term growth hacker. The article also cites Hiten Shah, who asked 731 Slack users the question in a 2015 open research project; 51% said very disappointed.
| Answer | What Superhuman did with this group |
|---|---|
| Very disappointed | Counted as the score; used to define the target persona and the main benefit |
| Somewhat disappointed | Listened only to those whose main benefit matched the fans' main benefit |
| Not disappointed | Politely disregarded their feedback |
What four questions did Superhuman ask?
Superhuman emailed a link to a Typeform survey with four questions. It went to users who had recently experienced the core of the product, following Ellis's recommendation to focus on people who had used it at least twice in the last two weeks. Superhuman had between 100 and 200 users to poll at the time, but Vohra wrote that smaller startups should not hold back, because "you start to get directionally correct results around 40 respondents".
Here are the four questions as Vohra published them, with [product] in place of the name:
| # | Question | Answer type | What it tells you |
|---|---|---|---|
| 1 | How would you feel if you could no longer use [product]? | Very disappointed, somewhat disappointed, not disappointed | Your score |
| 2 | What type of people do you think would most benefit from [product]? | Free text | Who it works for, in their own words |
| 3 | What is the main benefit you receive from [product]? | Free text | Why your fans stay |
| 4 | How can we improve [product] for you? | Free text | What holds the fence-sitters back |
Vohra notes that question 2 is powerful because happy users almost always describe themselves, in the words that matter to them. Those words are useful for marketing copy too.
How did Superhuman go from 22% to 58%?
Superhuman's score rose from 22% to 58% within three quarters of product work, by Vohra's account. All figures below are self-reported by the company in the 2018 article.
| When | Very disappointed | What changed |
|---|---|---|
| 2015 | No survey yet | The team starts coding Superhuman, desktop first |
| Summer 2017 | 22% | First survey, before launch, with a team of 14 |
| After segmenting | 33% | Counting only the personas found among very disappointed users |
| Within three quarters | 58% | Roadmap split between what fans loved and what held others back |
| November 2018 | Not given | Article published; team of 22 |
Vohra says the score became the company's most visible metric, tracked weekly, monthly and quarterly, with custom tooling that kept surveying new users. The product team ran an OKR whose only key result was the very disappointed percentage. Superhuman never surveyed the same user twice, to avoid skewing the 40% benchmark, and rebuilt its roadmap every quarter with the same process.
One warning from the article: as you sell beyond early adopters, users expect more and the score may well drop. That is a reason to keep measuring by segment.
How did Superhuman segment the answers?
Superhuman segmented by persona first, then by benefit. These are the steps in Vohra's article:
- Group answers by question 1. Very, somewhat and not disappointed.
- Tag each respondent with a persona. In Vohra's simplified example, the personas in the very disappointed group were founders, managers, executives and business development. Superhuman temporarily ignored everyone else, which took the score to 33%.
- Profile the fans. Using Julie Supan's high-expectation customer framework and the answers to question 2, the team wrote a detailed profile of its most demanding target user: a busy professional, such as an executive, founder, manager or business development lead, who reads 100 to 200 emails and sends 15 to 40 on a typical day.
- Find the main benefit. Question 3 answers from very disappointed users came down to speed, focus and keyboard shortcuts.
- Split the somewhat disappointed. Those who also named speed as the main benefit got close attention; the rest were set aside. Their answers to question 4 pointed to one big gap, the lack of a mobile app, then integrations, attachment handling, calendaring, a unified inbox, better search and read receipts.
- Build a half-and-half roadmap. Half the work went to doubling down on what fans loved, half to what held others back. Each project was rated low, medium or high for cost and for impact, starting with low cost, high impact.
The logic Vohra gives: only doubling down on what users love won't raise the score, and only fixing what holds people back means competitors will likely overtake you.
How to run the survey on your last 40 customers
You can run this in a week with your most recent customers. The steps below are our suggestions for a small B2B team, built on Superhuman's method.
- Pick the people, not the accounts. Survey the people who use the product at each customer, filtered the way Superhuman filtered: people who used it at least twice in the last two weeks. Keep sending to recent customers until you have about 40 responses. Vohra says results only start to be directionally correct around 40, so treat a smaller sample as a rough signal.
- Don't ask what you already know. Pull each respondent's role, company size and industry from your CRM so the form stays at four questions.
- Survey each person once. Superhuman never asked the same user twice; keep a surveyed-on date for each contact in your CRM.
Score everything in one sheet, one row per person:
| Column | What goes in it |
|---|---|
| Persona | Role plus company type and size, for example founder, agency, 5 to 20 people |
| Q1 | Very, somewhat or not disappointed |
| Q2 to Q4 | The answers, pasted word for word |
| Main benefit | One or two words from Q3, such as speed or fewer manual steps |
| Surveyed on | The date, so nobody is asked twice |
Send it from the founder's own inbox, in plain text, on a weekday morning. Send one reminder after three days to people who haven't answered, then stop.
Point outbound at your highest-scoring persona
This part is our suggestion, not something Superhuman described: once the answers are in, aim outbound at the segment with the highest very disappointed share. Group responses by persona (role plus company type and size) and calculate the share for each. The table below is an illustrative example with made-up numbers, not real data:
| Persona (illustrative) | Responses | Very disappointed | Share |
|---|---|---|---|
| Founders at 5 to 20 person agencies | 14 | 9 | 64% |
| Heads of sales at 50 to 200 person SaaS companies | 12 | 5 | 42% |
| Marketing managers at companies over 1,000 people | 9 | 1 | 11% |
| Everyone else | 5 | 1 | 20% |
| All respondents | 40 | 16 | 40% |
Trust only segments with enough answers
We suggest at least 8 to 10 responses before acting on a segment's share. Below that, one person changes the result by 10 points or more.
Write the ideal customer from question 2
Copy how very disappointed users describe the people who would benefit, word for word. Siela's AI SDR accepts a short ideal-customer summary in place of your website, so those words can go straight in.
Open with the question 3 benefit
Use the main benefit, in your fans' words, as the first line of every message to the top segment. Our lead scoring framework shows how to rank the people you find.
Pause the lowest-scoring persona
Stop prospecting the segment that scored lowest, as Superhuman temporarily ignored personas outside its fan group. Review in a quarter.
Re-survey the new customers
After 60 to 90 days, survey the customers outbound brought in. If their share is lower than the segment's, your targeting may have drifted.
Frequently asked questions
What is the Sean Ellis test?
The Sean Ellis test asks existing users how they would feel if they could no longer use a product and counts the share who answer very disappointed. Ellis wrote in July 2009 that product/market fit needs at least 40% answering very disappointed. The first archived copy of that post bases the bar on nearly 50 startups; by December 2009 it said nearly 100. Ellis called the threshold a bit arbitrary.
How many responses does a product market fit survey need?
Rahul Vohra wrote in 2018 that you start to get directionally correct results at around 40 respondents. Superhuman itself had between 100 and 200 users to poll when it first ran the survey in summer 2017. Below 40 answers, treat the score as a rough signal at best and pair it with customer conversations.
Who should get a product market fit survey?
Send it to people who have recently experienced the core of the product. Superhuman followed Sean Ellis's recommendation to survey users who had used it at least twice in the last two weeks. It also never surveyed the same person twice, so repeat answers would not throw off the 40% benchmark.
What was Superhuman's product market fit score?
Superhuman scored 22% very disappointed when Rahul Vohra first ran the survey in summer 2017. Counting only the personas in the very disappointed group lifted it to 33%. Within three quarters of product work, it reached 58%. These are the company's own figures, published in First Round Review in November 2018.
What should you do with somewhat disappointed users?
Split them by the main benefit they name. Superhuman paid close attention to somewhat disappointed users who also named speed, the benefit its fans loved, and set aside the rest. The first group's improvement requests, led by a mobile app, filled half of the roadmap; the other half doubled down on what fans already loved.
Sources
- First Round Review: How Superhuman Built an Engine to Find Product Market Fit (Rahul Vohra) (12 Nov 2018)
- Sean Ellis: The Startup Pyramid (startup-marketing.com July 2009 page, current version) (27 Jul 2009)
- Sean Ellis: The Startup Pyramid (Wayback Machine copy, 31 Jul 2009) (27 Jul 2009)
- Sean Ellis: The Startup Pyramid (Wayback Machine copy, 13 Dec 2009) (27 Jul 2009)