Quantitative Market Research: 9 Criteria for Selecting Quantitative Market Research Companies
Recently a fellow market researcher posed set of questions in a market research forum about what criteria might be used to select market research suppliers. Having responded to these questions there, I have revised and refined nine criteria that could be used to choose quantitative market research firms.
First, full disclosure: My business is as a market research supplier. I conduct both qualitative and quantitative market research studies for my clients. No doubt, the criteria that follow are influenced significantly by having been a supplier for some fifteen years.
I also have almost fifteen years’ experience as a purchaser of market research services, in positions of both market research management and product management for my employers, mostly notably one of largest telecommunications firms in the country. So I have much experience dealing with market research vendors from a client perspective.
In these positions, I developed several criteria and areas of inquiry for selecting market researchers for quantitative projects. (As a manager for a very large company, you might imagine how many solicitations for business I received.) Several of these criteria I learned to employ based on years in the school of hard experience.
Here is how I evaluate quantitative market research suppliers:
#1: Evaluate Their Approach to Market Research
First, what is their market research philosophy? How do they approach projects? Are they focused on actionable, practical results?
What I am really seeking with this series of inquiry is the supplier’s willingness to understand my needs and assist me in reaching my goals.
Of course, my first requirement with any supplier is their adherence to market research standards, to ensure that the study is representative of the population(s) I am looking to assess, to collect the data in a professional manner, and to keep me apprised of any issues.
Understanding how they approach projects is my first clue to their level of competence in these areas.
#2: Are they listening or are they selling?
Second, are they trying to sell me a solution in search of a problem or do they really attempt to listen to my needs (and limitations, particularly in the budget area) and offer me a solution to my problems?
This is absolutely a deal-breaker. If I get a sense that a potential supplier is flogging one single tool as a solution to all problems in market research, the supplier is off my list. This happens with surprising regularity. The “solution” they propose is often proprietary (see criteria three below in the next post) and usually carries a high price tag.
I expect a supplier to listen carefully to my needs and the project specifications and act and bid accordingly. I am especially concerned that they demonstrate a clear understanding of the project objectives.
There are only so many solutions in a market researcher’s bag of tools. I do not expect a supplier to invent a new tool. I simply expect them to give thought and consideration and a solid rationale for their recommended approach, methodology and analysis plan for my project.
#3: Black Box Solution—Bye, Bye, I am out of here
Third, are they offering me a black-box solution? Is their solution something I (and colleagues) can understand or am I going on faith that they know what they are doing? Can what they are offering be validated and replicated?
Here is another criterion that if failed will send me running. Some hot-shot statistical guru has invented a proprietary analytic process that answers more questions, solves more problems, and is more accurate than any test, process, or model known to market researchers, statisticians or even God. And they will deploy it just for you. But, wait a minute, they cannot tell you how it works, they cannot give you enough information so you can understand it, and, of course, no one else has it so it cannot be validated or replicated. But, boy, does this snake oil get results!
Right, and I have a CDO for sale.
Run, and run fast.
The fact of the matter is that there is no black box solution, just as the Wizard of Oz ultimately proved to be a humble man from Kansas.
Don’t buy it. You are MUCH better off with tried-and-true market research methods and analytic techniques and tests. Hey, they might not be sexy, but they are proven to work and have been endlessly validated and replicated.
I am reasonably well-educated, have market research experience and a good head on my shoulders. My inability to understand a black box solution does not lie with me. Instead, it is that the black box solution is a Rube Goldberg machine. The guys who invented the black box talk and act so much smarter than I am, so smart that a mere mortal could not possibly understand their technique. They are selling smoke and mirrors. Let them peddle their brilliance on Wall Street—the quants there will buy anything (and did).
There is a saying in Las Vegas that if I sit down at a poker table and I cannot identify the patsy, the patsy is me.
Just don’t buy into solutions you don’t understand. It’s not that you’re not smart enough to get it; it is that the sellers of black boxes either aren’t smart enough to explain their solution effectively, or that they don’t understand it either. Regardless of which is the case, you don’t want to be a buyer.
(Pardon me if I sound passionate about this topic—I am. I have seen this happen too many times. A black box seller persuades someone in the company to buy it and inevitably the project blows up. Lots of money wasted and lots of time expended for naught.)
Show them the door and find a market research supplier who meets the criteria here. You will get solid research and the pride of a job well done.
#4: Demand a Single Point of Contact
Fourth, will I have a single point of contact at the firm that will take full responsibility for the development and execution of my project? If that person is not the person selling me their services, can I meet and talk with the person that will be responsible?
This criterion is crucial to me. Before I hire a market research supplier, I MUST meet, talk with and get to know the people who will actually be handling my project. I MUST have access to these folks. And one of them MUST take overall responsibility for the projects, for running interference, for problem-solving and for communicating regularly with me about the project. If the sales person does not have operational responsibility, I have to talk with the person who does.
Fundamentally, do we get along? Can we work together? If you really like and relate to the sales person, but he or she drops out of the project picture once you are on board, you could be left working with a person to whom you cannot relate. Don’t simply take the sales person word for it that the project manager is a great guy or gal and you will love him or her. Meet that person and ask that person some of the same questions you have asked the sales person.
Too many times I tried suppliers based on the likeability and assurances of the sales person, only to discover that the project manager or other key team member on the vendor side had the business and social skills of Frankenstein. Or worse, there was no one person who could or would take responsibility for the project, leaving me to call all over the firm to find someone to answer my questions or solve problems with the project.
Know who is responsible before you ink the deal.
#5: Are They Willing to let the Relationship Grow?
Fifth, are they willing to take a small project as a test or do they only want the bigger pricier ones?
Market research suppliers often specialize in the types of projects they conduct. Before you contact suppliers, it is wise to determine if they have an industry specialty and then to get a sense of the types of customers they serve. Select those that fit your market research needs.
I may be headed into a sensitive area, but so be it.
Some firms position themselves as all things to all potential clients, at least in my experience as a client. In actuality, some firms specialize in what the industry calls “field and tab,” that is, relatively simple studies of attitudes and perceptions, basic customer satisfaction, direct mail tests, concept evaluation and the like. Such a study basically involves developing a questionnaire, locating an appropriate sample, fielding data collection, analyzing the data, generally with frequencies and/or cross-tabulations and report-writing.
Other firms specialize in (or are definitely prepared to handle) more sophisticated studies using conjoint analysis (which requires specific types of questions) or discrete modeling. Such projects require significantly more statistical analysis and therefore expertise and the requisite staff to handle this level of analysis.
Narrow your field of potential market research suppliers beforehand, based on the types of projects they tend to do.
Having done so, you will probably want to test your chosen supplier with a relatively small project to get a sense of how well you work together.
Is the supplier open to, and appreciative of, such an arrangement? Or do they push to either up-sell your project or want to hold off for a larger opportunity? If so, this supplier might not be a good fit for you.
#6: Evaluate Their Responsiveness
Sixth, how responsive are they? How soon are my messages answered? Do I have to pull teeth to get answers? Are they proactive in keeping me informed about the status of the project or do I have to pester them for this information?
This aspect of a client-supplier relationship is difficult to ascertain short of conducting a project with them. In my experience, almost all suppliers are very responsive during the initial sales cycle.
One way to get a peek at potential communication issues is your conversations with both the person who will be your primary contact at the firm and the other significant team members (the data collection manager and the analyst to name two).
This is also a main reason to test new suppliers with a small project. If they don’t meet this test, I would decline to continue to do business with them.
#7: Get Clear Pricing
Seventh, how clear is their pricing? What is included in their services and what is extra?
Pricing clarity is critical. You may or may not want to always seek the lowest price. In evaluating market research services, pricing is driven by the confluence of complexity of the project, the sampling costs, the timeframes, and most critically, length and quantities of the survey or interview, analytic requirements, and quality control.
That said, you do want pricing clarity.
I have worked with suppliers who would low-ball the initial project costs and then nickel-and-dime me to death on the back end. Oh, you want a frequency? Well, that’s extra. You want SPSS output? That’s extra. You want a daily status report? That’s extra.
Insist on knowing exactly what the deliverables are and how much they will cost.
#8: Check Their References
Eighth, do their references check out?
It continually surprises me that otherwise rational, hard-headed business people, let alone market researchers, do not ask for or check references. I surmise that the rationale for not doing so is that if a potential supplier can give references, they must check out okay. What company would provide references that would not give the supplier a good review?
Checking out references is a matter of tone. When you check references you listen for what is said, how it is said and what is not said. Unless a supplier was truly abysmal, I would not give a negative review to any supplier if you called me for a reference on the ones I use, for example. But I would certainly choose my words very carefully and use intonation to convey my lack of enthusiasm. If a supplier is truly superior, I would let you know and I would let you know what they do that makes them superior.
#9: Check Their Experience
Ninth, ask about the last three projects the supplier did.
A reputable supplier will NOT reveal client names or project specifics, but will give you an overview of the projects.
The answers to these questions are important; how they are answered, with what speed, with what depth and with what openness are critical. You are also listening for what is not said.
Along with probing the areas above, I would also verify their credentials and qualifications. If anyone is insulted by your request for credentials and qualifications, run fast and far. If you ask them for their credentials, they should be happy to supply them and to give you the information to verify them. It is not insulting…if they have the credentials. If they are insulted by this request, either they don’t have the credentials they say they have, or they are neurotic. In either case you would not want to work with them.
