Part 2, Part 3, Part 4, Part 5 and Part 6 of this series discussed the first eight criteria for selecting a quantitative market research firm. Part 7, the last in the series, introduces the ninth.

Ninth, ask about the last three projects the supplier did.

A reputable supplier will NOT reveal client names or project specifics, but will give you an overview of the projects.

The answers to these questions are important; how they are answered, with what speed, with what depth and with what openness are critical. You are also listening for what is not said.

Along with probing the areas above, I would also verify their credentials and qualifications. If anyone is insulted by your request for credentials and qualifications, run fast and far. If you ask them for their credentials, they should be happy to supply them and to give you the information to verify them. It is not insulting…if they have the credentials. If they are insulted by this request, either they don’t have the credentials they say they have, or they are neurotic. In either case you would not want to work with them.

In the next posts, I will explore the criteria I use to evaluate qualitative market research suppliers.

If you have additional criteria you use to evaluate quantitative market research suppliers and would like to share, please feel free to comment below.

Comments welcome.

Dr. Bob

Part 2 of this series discussed the first two of nine criteria for selecting quantitative market research firms. Part 3 discusses the third criteria.

Third, are they offering me a black-box solution? Is their solution something I (and colleagues) can understand or am I going on faith that they know what they are doing? Can what they are offering be validated and replicated?

Here is another criterion that if failed will send me running. Some hot-shot statistical guru has invented some proprietary analytic process that answers more questions, solves more problems, and is more accurate than any test, process, or model known to market researchers, statisticians or even God. And they will deploy it just for you. But, wait a minute, they cannot tell you how it works, they cannot give you enough information so you can understand it, and, of course, no one else has it so it cannot be validated or replicated. But, boy, does this snake oil get results!

Right, and I have a CDO for sale.

Run, and run fast.

The fact of the matter is that there is no black box solution, just as the Wizard of Oz ultimately proved to be a humble man from Kansas.

Don’t buy it. You are MUCH better off with tried-and-true market research methods and analytic techniques and tests. Hey, they might not be sexy, but they are proven to work and have been endlessly validated and replicated.

I am reasonably well educated, have market research experience and a good head on my shoulders. My inability to understand a black box solution does not lie with me. Instead, it is that the black box solution is a Rube Goldberg machine. The guys who invented the black box are so much smarter than I am, so smart that a mere mortal could not possibly understand their technique. It is that they are selling smoke and mirrors. Let them peddle their brilliance on Wall Street—the quants there will buy anything (and did).

There is a saying in Las Vegas that if I sit down at a poker table and I cannot identify the patsy, the patsy is me.

Just don’t buy into solutions you don’t understand. It’s not that you’re not smart enough to get it; it is that the sellers of black boxes either aren’t smart enough to explain their solution effectively, or that they don’t understand it either. Regardless of which is the case, you don’t want to be a buyer.

(Pardon me if I sound passionate about this topic—I am. I have seen this happen too many times. A black box seller persuades someone in the company to buy it and inevitably the project blows up. Lots of money wasted and lots of time expended for naught.

Show them the door and find a market research supplier who meets the criteria here. You will get solid research and the pride of a job well done.

More to follow.

Comments welcome.

Dr. Bob

One of my best experiences as a market researcher was as part of a small business case team at a Fortune 500 company.

The business case was going into its third attempt. Twice, other teams had worked the case and then slowly made the rounds up the managerial ranks, only to have the case rejected before reaching the ultimate decision-makers, the board of directors of the firm.

As you might imagine, to require approval at that level, the business case was asking the firm to make a multi-billion dollar investment in hardware and software capacity.

My boss had been a member of the second team that tried and did not succeed. His boss, in turn, had been on the first team. So, I was somewhat taken back when my boss stopped by my office one day to discuss my participation in a third effort.

I was the new kid on the block–a fresh recruit from the marketing research organization, I was now working my first (and only) stint in product management. It was a chance to get my ticket punched and, possibly, to receive a promotion, if all went well. But to have my boss entrust me with the somewhat overwhelming responsibility of representing the group, as the primary interface with market research, was surprising. I was still learning the lore of my new organization, grappling with the specialized business lingo there, and still working to make my mark in the group, in essence, to prove that I was capable of contributing solidly to the organization’s goals and responsibilities.

I had worked for my boss prior to us working together in product management. He had been my manager in market research. We worked well together and he had brought me over to add a fresh perspective to the somewhat stolid product team.

He was taking a pretty big chance. Clearly, both his success and his boss’s success were linked to the business case and both of them had worked long and hard on it without reaching the final goal of approval.

Now they were giving me an opportunity.

Parts 2 and 3 to follow.

Comments welcome.

Dr. Bob