When it comes to market research, there are many tools at our disposal. And one of the most effective mediums for gathering data and building a full picture of customer behaviour is the survey. Sam McNerny is a survey expert who goes far beyond the surface level of yes or no questions and his research philosophy is worth hearing.
Read on to learn more about his story and how the French philosopher Michel de Montaigne inspired his work.
What’s your background and how did you get to where you are now?
I do polling and customer surveys for small and medium-sized e-commerce brands. Typically they’re trying to solve the problem of driving organic growth and spending less on performance marketing and budgeting efficiently.
As for how I got here, it’s hard to delineate a clear starting point. I’ve always been interested in the behaviour of people and that manifested itself in doing a philosophy major in college. This led me into learning more about behavioural science and popular psychology.
From there, my first real job was at a big ad agency in New York and I was there for 5 years doing strategy and insight work. While I was there I learned how to do online surveying and run polling experiments.
Around the pandemic, I went solo and started working with e-commerce brands that have a variety of customer questions that need to be answered by surveys and polls. That’s how my career manifested into market research.
You mentioned you did a philosophy major and when we first met we had a great conversation about the practical benefits of philosophy.
How has the subject impacted your life?
I’ve thought about this a lot. I could tell you a romantic story about how I fell in love with the topic and use it to inform my decisions every day. But really it’s just about me appreciating underdogs. And I think that’s what a lot of memorable philosophers were – underdogs who ended up changing culture in a meaningful way. Socrates is a really good example of this. So is Bob Dylan.
So, my understanding of philosophy came from it being taught to me chronologically from the ancients up to the modern philosophers. It was seeing it as a story of people one-upping each other.
A philosopher that really stands out to me is Michel de Montaigne because he did something that no one had done before. Back in the 16th century, he examined himself by starting the genre of the essay and there were a couple of things that were innovative about his approach.
One is he commented on everyday living. There are essays where he talked about taboo subjects like farting and burping, which went unaddressed by other philosophers up to that point. When I read an essay, I felt like Montaigne is right there with me.
The second thing he does is have a very interesting relationship with the truth. Montaigne never bullshits in the Harry Frankfurt sense of the word, who made a great distinction between bullshitters and liars.
This line of thought is that liars respect the truth. They have to because they’re shielding the truth, whereas bullshitters are indifferent towards the truth. They’re engaging in a different enterprise.
Montaigne valued the truth, but he was content to make an observation and explore a few different perspectives. Then let it go without having to prove the truth in the Socratic tradition.
In his writing, there’s also a sense of Montaigne being along for the ride as much as the reader with philosophy. That’s how I feel when I’m learning philosophy and that’s down to Montaigne’s meandering and easy-to-follow tone.
My intuition from Montaigne’s work is that you can draw a line from him to standup comedians. I don’t think you could necessarily make that comparison with any other philosopher in terms of observing daily life. Now, Montaigne wasn’t trying to make you laugh. But he was trying to make you think.
I do agree with you about Montaigne’s ability to make it feel as if he’s talking directly to the reader.
Moving into your role as a market researcher I imagine there must be lots of details you have to sift through day to day.
So, I’d love to hear more about your research philosophy and how it impacts your approach to surveys.
When people think of surveys, prosaic or ham-handed questions might come to mind e.g. would you recommend this product to a friend?
But I’d like to offer a defence and context for survey. The concept of the survey emerged in the late 19th century with the British social reformer Charles Booth. Between 1890 and 1900 he and a team of researchers walked the streets of London collecting personal income data, which you can imagine to be gruelling in that time period.
After ten years of asking people how much money they made, Booth collected the data and published it in some really beautiful maps. We could compare these maps to modern data visualisations.
Booth’s survey led to the Old Age Pensions Act passing, which was the first welfare law ever passed in the West. So, this case study is a great lesson in how progress can only really come from measuring the thing that you’re trying to change.
Another personal survey hero of mine is George Gallup. In the 1930s, he upended the go-to-approach for polling voters. This was magazines with large readerships polling their readers. Gallup scaled the stats of representative sampling. He’d go out and make sure his sample reflected the census on the grounds of age, gender, ethnicity, income and geography.
His big moment came in the 1936 election where he correctly forecasted Roosevelt winning when popular newspapers were saying Roosevelt wouldn’t win.
Fast forward and the survey today is maligned as a market research relic. But you really can’t write the history of the 20th and 21st centuries without a survey.
I find when I’m using surveys, clients have questions about their total addressable market. So, that would be things like age, gender and income distribution. There’s a lot of science behind making this data work for you in understanding a market.
I see surveys as an interface. Think of a theatre stage or a TV screen. What’s on that stage or screen is deliberately trying to influence an audience. Surveys can be used in the same way with the right sequence of questions to draw out certain opinions and preferences from an audience.
Specifically, I’m talking about asking people if they find a brand relevant or not to them and if they intend to purchase from the company or not. Then using their responses to challenge them.
There are always people who’ll say ‘this brand is relevant to me. But I wouldn’t consider making a purchase.’ And I love asking those people why not and using their previous answers to set it up.
Their next response could be ‘I don’t intend to buy right now.’ And my survey answer might be ‘Well, what’s stopping you? You just said this product is relevant to your needs.” It’s getting the customer to explain themselves and create some tension.
There’s an argument to be made that I’m leading the witness or customer in that context. But in that situation, I’m not using a survey like a scientific tool. I’m using a survey as a creative tool to get people to react in a certain way.
That’s some fascinating history around surveys. Digging deeper, what kind of tactics would you recommend to make a survey engaging to people?
When I think engaging I don’t mean in the way that platforms like Reddit engage readers. I mean making it as easy as possible for the customer to answer. That will make the results more insightful.
I was recently reviewing a client survey and they asked the question ‘When choosing products within our categories, which of these benefits is most important to you?’ If you think about it, that’s a leading question because it assumes that assessing the category and finding a product with the right benefits is something you actually do.
When reading the question I struggled to imagine a situation where people would choose a product within this particular category. The question only provided a vague mental state. You don’t want to put a customer in a vague mental state with any question you ask.
My suggested change was ‘Since consuming this product, which of these benefits has been most important to you?’ That’s a lot easier to answer.
Another thing that makes surveys engaging in this sense is good writing. It’s about using simple words over complex language. And when the writing is done, it’s important not to get too close to the words. It’s always worth having a second set of eyes and asking for the questions to be read aloud. If any questions sound weird or are hard to answer, then you’ll want to edit them.
I do think in my research philosophy there’s some Montaigne lurking in the background. He’s easy to read so it’s about making everything I write easy to read too.
Lots of great practices for survey creation. I’m also curious to know what your process is like for survey research.
It starts with understanding the client’s problems and their objectives e.g. driving organic growth or building certain segments. I might begin by chatting with the founder or VP of marketing and then learning as much as I can about the product. This will take different forms.
It may be speaking with current users, asking more in-depth questions with the marketing team, looking at online reviews etc. The closer you can get to the customer, the better because you come to appreciate what they like and what they don’t like.
When designing a survey, I normally field it with a group of 10 – 15 respondents which takes an hour or less. I get the results, see what works and adjust as necessary. I typically do that between 10 and 20 times before pressing go on the main data collection. I do this to mimic the fail fast, fail early approach to products that come out of Silicon Valley.
This approach differs from big research agencies that will labour over a questionnaire for weeks or months. This also requires a lot of stakeholders and debating, whereas my strength as a solo consultant is to iterate rapidly.
What’s your opinion on AI in regard to market research or surveys?
I’m really excited about it and I’ve embraced it and have ChatGPT 4 open most days. That said, there’s a lot of talk about this kind of technology being overrated in the short term and underrated in the long term. I’ll give you a few short-term things that have helped me personally.
Firstly, ChatGPT is really good for analysing text data. I’ve included more open-text responses in my surveys than I normally would because I can throw the results into it. I’ve done this with old surveys that I’ve analysed and it’s better. There’s a decent-sized market for text analytics that I think AI really disrupts, if not renders completely useless.
I’ve also found it useful when it comes to pointing me in the right direction for sampling resources. So if I have questions about census data or if I can use zip codes to infer income, I find ChatGPT better than Google. This is because you’re having a dialogue with the tool and asking it to try different things.
In the long term, I think people are underestimating how dramatically ChatGPT will change things. For me, surveys are the most enduring kind of market research tool simply on the grounds that they’ve been around for the longest. Other than conducting interviews, which has to be the original form of market research.
So, a lot of contemporary tools that seem sophisticated like social listening or analysing transactional data via credit cards will likely die a lot faster than surveys. But it wouldn’t surprise me in 10 years time if technologies like ChatGPT have transformed how people are doing surveys.