Part 2 of this series discussed the first two of nine criteria for selecting quantitative market research firms. Part 3 discusses the third criteria.
Third, are they offering me a black-box solution? Is their solution something I (and colleagues) can understand or am I going on faith that they know what they are doing? Can what they are offering be validated and replicated?
Here is another criterion that if failed will send me running. Some hot-shot statistical guru has invented some proprietary analytic process that answers more questions, solves more problems, and is more accurate than any test, process, or model known to market researchers, statisticians or even God. And they will deploy it just for you. But, wait a minute, they cannot tell you how it works, they cannot give you enough information so you can understand it, and, of course, no one else has it so it cannot be validated or replicated. But, boy, does this snake oil get results!
Right, and I have a CDO for sale.
Run, and run fast.
The fact of the matter is that there is no black box solution, just as the Wizard of Oz ultimately proved to be a humble man from Kansas.
Don’t buy it. You are MUCH better off with tried-and-true market research methods and analytic techniques and tests. Hey, they might not be sexy, but they are proven to work and have been endlessly validated and replicated.
I am reasonably well educated, have market research experience and a good head on my shoulders. My inability to understand a black box solution does not lie with me. Instead, it is that the black box solution is a Rube Goldberg machine. The guys who invented the black box are so much smarter than I am, so smart that a mere mortal could not possibly understand their technique. It is that they are selling smoke and mirrors. Let them peddle their brilliance on Wall Street—the quants there will buy anything (and did).
There is a saying in Las Vegas that if I sit down at a poker table and I cannot identify the patsy, the patsy is me.
Just don’t buy into solutions you don’t understand. It’s not that you’re not smart enough to get it; it is that the sellers of black boxes either aren’t smart enough to explain their solution effectively, or that they don’t understand it either. Regardless of which is the case, you don’t want to be a buyer.
(Pardon me if I sound passionate about this topic—I am. I have seen this happen too many times. A black box seller persuades someone in the company to buy it and inevitably the project blows up. Lots of money wasted and lots of time expended for naught.
Show them the door and find a market research supplier who meets the criteria here. You will get solid research and the pride of a job well done.
More to follow.
Comments welcome.
Dr. Bob
