Part 2, Part 3, Part 4, Part 5 and Part 6 of this series discussed the first eight criteria for selecting a quantitative market research firm. Part 7, the last in the series, introduces the ninth.

Ninth, ask about the last three projects the supplier did.

A reputable supplier will NOT reveal client names or project specifics, but will give you an overview of the projects.

The answers to these questions are important; how they are answered, with what speed, with what depth and with what openness are critical. You are also listening for what is not said.

Along with probing the areas above, I would also verify their credentials and qualifications. If anyone is insulted by your request for credentials and qualifications, run fast and far. If you ask them for their credentials, they should be happy to supply them and to give you the information to verify them. It is not insulting…if they have the credentials. If they are insulted by this request, either they don’t have the credentials they say they have, or they are neurotic. In either case you would not want to work with them.

In the next posts, I will explore the criteria I use to evaluate qualitative market research suppliers.

If you have additional criteria you use to evaluate quantitative market research suppliers and would like to share, please feel free to comment below.

Comments welcome.

Dr. Bob

Part 2, Part 3, Part 4 and Part 5 of this series discussed the first six criteria for selecting a quantitative market research firm. This part introduces the seventh and eighth.

Seventh, how clear is their pricing? What is included in their services and what is extra?

Pricing clarity is critical. You may or may not want to always seek the lowest price. In evaluating market research services, pricing is driven by the confluence of complexity of the project, the sampling costs, the timeframes, and most critically, length and quantities of the survey or interview, analytic requirements, and quality control.

That said, you do want pricing clarity.

I have worked with suppliers who would low-ball the initial project costs and then nickel-and-dime me to death on the back end. Oh, you want a frequency? Well, that’s extra. You want SPSS output? That’s extra. You want a daily status report? That’s extra.

Insist on knowing exactly what the deliverables are and how much they will cost.

Eighth, do their references check out?

It continually surprises me that otherwise rational, hard-headed business people, let alone market researchers, do not ask for or check references. I surmise that the rationale for not doing so is that if a potential supplier can give references, they must check out okay. What company would provide references that would not give the supplier a good review?

Checking out references is a matter of tone. When you check references you listen for what is said, how it is said and what is not said. Unless a supplier was truly abysmal, I would not give a negative review to any supplier if you called me for a reference on the ones I use, for example. But I would certainly choose my words very carefully and use intonation to convey my lack of enthusiasm. If a supplier is truly superior, I would let you know and I would let you know what they do that makes them superior.

More to follow.

Comments welcome.

Dr. Bob

Part 2, Part 3 and Part 4 of this series discussed the first four criteria for selecting a quantitative market research firm. This part discusses the fifth and sixth criteria.

Fifth, are they willing to take a small project as a test or do they only want the bigger pricier ones?

Market research suppliers often specialize in the types of projects they conduct. Before you contact suppliers, it is wise to determine if they have an industry specialty and then to get a sense of the types of customers they serve. Select those that fit your market research needs.

I may be headed into a sensitive area, but so be it.

Some firms position themselves as all things to all potential clients, at least in my experience as a client. In actuality, some firms specialize in what the industry calls “field and tab,” that is, relatively simple studies of attitudes and perceptions, basic customer satisfaction, direct mail tests, concept evaluation and the like. Such a study basically involves developing a questionnaire, locating an appropriate sample, fielding data collection, analyzing the data, generally with frequencies and/or cross-tabulations and report-writing.

Other firms specialize in (or are definitely prepared to handle) more sophisticated studies using conjoint analysis (which requires specific types of questions) or discrete modeling. Such projects require significantly more statistical analysis and therefore expertise and the requisite staff to handle this level of analysis.

Narrow your field of potential market research suppliers beforehand, based on the types of projects they tend to do.

Having done so, you will probably want to test your chosen supplier with a relatively small project to get a sense of how well you work together.

Is the supplier open to, and appreciative of, such an arrangement? Or do they push to either up-sell your project or want to hold off for a larger opportunity? If so, this supplier might not be a good fit for you.

Sixth, how responsive are they? How soon are my messages answered? Do I have to pull teeth to get answers? Are they proactive in keeping me informed about the status of the project or do I have to pester them for this information?

This aspect of a client-supplier relationship is difficult to ascertain short of conducting a project with them. In my experience, almost all suppliers are very responsive during the initial sales cycle.

One way to get a peek at potential communication issues is your conversations with both the person who will be your primary contact at the firm and the other significant team members (the data collection manager, the analyst to name two).

This is also a main reason to test new suppliers with a small project. If they don’t meet this test, I would decline to do business with them.

More to follow.

Comments welcome.

Dr. Bob

Part 2 and Part 3 of this series discussed the first three of nine criteria for selecting a quantitative market research firm. This post explore the fourth criterion.

Fourth, will I have a single point of contact at the firm that will take full responsibility for the development and execution of my project? If that person is not the person selling me their services, can I meet and talk with that person?

This criterion is crucial to me. Before I hire a market research supplier, I MUST meet, talk with and get to know the people who will actually be handling my project. I MUST have access to these folks. And one of them MUST take overall responsibility for the projects, for running interference, for problem-solving and for communicating regularly with me about the project. If the sales person does not have operational responsibility, I have to talk with the person who does.

Fundamentally, do we get along? Can we work together? If you really like and relate to the sales person, but he or she drops out of the project picture once you are on board, you could be left working with a person to whom you cannot relate. Don’t simply take the sales person word for it that the project manager is a great guy or gal and you will love him or her. Meet that person and ask that person some of the same questions you have asked the sales person.

Too many times I tried suppliers based on the likeability and assurances of the sales person, only to discover that the project manager or other key team member on the vendor side had the business and social skills of Frankenstein. Or worse, there was no one person who could or would take responsibility for the project, leaving me to call all over the firm to find someone to answer my questions or solve problems with the project.

Know who is responsible before you ink the deal.

More to follow.

Comments welcome.

Dr. Bob

Part 2 of this series discussed the first two of nine criteria for selecting quantitative market research firms. Part 3 discusses the third criteria.

Third, are they offering me a black-box solution? Is their solution something I (and colleagues) can understand or am I going on faith that they know what they are doing? Can what they are offering be validated and replicated?

Here is another criterion that if failed will send me running. Some hot-shot statistical guru has invented some proprietary analytic process that answers more questions, solves more problems, and is more accurate than any test, process, or model known to market researchers, statisticians or even God. And they will deploy it just for you. But, wait a minute, they cannot tell you how it works, they cannot give you enough information so you can understand it, and, of course, no one else has it so it cannot be validated or replicated. But, boy, does this snake oil get results!

Right, and I have a CDO for sale.

Run, and run fast.

The fact of the matter is that there is no black box solution, just as the Wizard of Oz ultimately proved to be a humble man from Kansas.

Don’t buy it. You are MUCH better off with tried-and-true market research methods and analytic techniques and tests. Hey, they might not be sexy, but they are proven to work and have been endlessly validated and replicated.

I am reasonably well educated, have market research experience and a good head on my shoulders. My inability to understand a black box solution does not lie with me. Instead, it is that the black box solution is a Rube Goldberg machine. The guys who invented the black box are so much smarter than I am, so smart that a mere mortal could not possibly understand their technique. It is that they are selling smoke and mirrors. Let them peddle their brilliance on Wall Street—the quants there will buy anything (and did).

There is a saying in Las Vegas that if I sit down at a poker table and I cannot identify the patsy, the patsy is me.

Just don’t buy into solutions you don’t understand. It’s not that you’re not smart enough to get it; it is that the sellers of black boxes either aren’t smart enough to explain their solution effectively, or that they don’t understand it either. Regardless of which is the case, you don’t want to be a buyer.

(Pardon me if I sound passionate about this topic—I am. I have seen this happen too many times. A black box seller persuades someone in the company to buy it and inevitably the project blows up. Lots of money wasted and lots of time expended for naught.

Show them the door and find a market research supplier who meets the criteria here. You will get solid research and the pride of a job well done.

More to follow.

Comments welcome.

Dr. Bob

Here is how I evaluate quantitative market research suppliers:

First, what is their market research philosophy? How do they approach projects? Are they focused on actionable, practical results?

What I am really seeking with this series of inquiry is suppliers’ willingness to understand my needs and assist me in reaching my goals.

Of course, my first requirement with any supplier is their adherence to market research standards, to ensure that the study is representative of the population(s) I am looking to assess, to collect the data in a professional manner, and will keep me apprised of any issues.

Understanding how they approach projects is my first clue to their level of competence in these areas.

Second, are they trying to sell me a solution in search of a problem or do they really attempt to listen to my needs (and limitations, particularly in the budget area) and offer me a solution to my problems?

This is absolutely a deal-breaker. If I get a sense that a potential supplier is flogging one single tool or solution as a solution to all problems in market research, the supplier is off my list. This happens with surprising regularity. The “solution” they propose is often proprietary (see criteria three below in the next post) and usually carries a high price tag.

I expect a supplier to listen carefully to my needs and the project specifications and act and bid accordingly. I am especially concerned that they demonstrate a clear understanding of the project objectives.

There are only so many solutions in a market researcher’s bag of tools. I do not expect a supplier to invent a new tool. I simply expect them to give thought and consideration and a solid rationale for their recommended approach, methodology and analysis plan for my project.

Part 3 to follow.

Comments welcome.

Dr. Bob

Recently a fellow LinkedIn member posed set of questions in a market research forum about what criteria might be used to select market research suppliers. Having responded to these questions there, I have revised and refined nine criteria that could be used to choose quantitative market research firms.

First, full disclosure: My business is as a market research supplier. I conduct both qualitative and quantitative market research studies for my clients. No doubt, the criteria that follow are influenced significantly by having been a supplier for some fifteen years.

I also have almost fifteen years’ experience as a purchaser of market research services, in positions of both market research management and product management for my employers, mostly notably one of largest telecommunications firms in the country. So I have much experience dealing with market research vendors from a client perspective.

In these positions, I developed several criteria and areas of inquiry for selecting market researchers for quantitative projects. (As a manager for a very large company, you might imagine how many solicitations for business I received.) Several of these criteria I learned to employ based on years in the school of hard experience.

All the criteria will be discussed in the next few posts.

Comments welcome.

Dr. Bob

At first glance, the numbers are both dismal and daunting.

Telephone surveying is undergoing a sea change.

According to the Pew Research Center (based on figures from the National Center for Health Statistics), 25% of U.S. households are cell phone only.

Households are jettisoning landline service at an accelerating pace.

Cell only households are more prevalent in various subgroups of the population: half (49% of adults aged 25 to 29) and 30% of Hispanics are cell only.

As Pew states, “survey researchers…face a difficult decision as to whether to include cell phones in their samples. Doing so adds significantly to the cost and complexity of surveys at a time when respondent cooperation is becoming increasingly difficult to obtain.”

Some in the market research community have opted for Web-only surveying as a replacement for telephone. But this presents an even larger issue. In spite of the shift to cell service, phone service overall is near ubiquity. Only 2% of U.S. households have neither landline or cell service. Some 38% of U.S. households do not have internet service at home.

The market research industry is arguing furiously about how to solve the problems with telephone surveying. Thus far, there is little consensus. But the effort must continue.

Meanwhile, market researchers in the corporate world can rest relatively easily. Surveying out of customer databases by phone continues to work and continues to be statistically robust. The challenge on the corporate side is in sampling and data collection outside their own customers. Hybrid phone surveys, combining both cell and landline, are still relatively untested and their reliability is still largely unknown with unanswered questions about the representativeness of surveying via cell in dual landline-cell households.

Market research will clearly become only more challenging in the near future, at least.

But, I guess that’s why market researchers get paid the big bucks, right?

With a smile,

Dr. Bob

In the prior post on this site, I discussed the challenges in market research surveying of measuring consumer behavior.

For years, consumer market research has found that consumers support Green, while their actions in the marketplace belie this support.

A recent study published in the Journal of Personality and Social Psychology highlights shifts in Green thinking and behavior among consumers, showing that the Green movement might benefit from encouraging consumers to be conspicuously Green.

Researchers developed a series of experiments in which college students were offered equally priced products in both green and conventional versions. Indeed, the green models were less feature-rich than the conventional products. The test group was then primed to increase their awareness of status while the control group remained neutral.

Both groups then made their product selections. A significantly higher percentage of the test group chose the green products.

The study suggests the motivation for this difference: “costly signaling theory says that although costly, altruistic acts may benefit the altruist indirectly, by establishing a ‘reputation’.”

In a telling follow-up study, researchers found that preference for Green lessened among the test group as the price of the Green products were lowered.

For companies marketing Green, market researchers may want to measure these possibilities.

For more details about the study, click here.

Dr. Bob

Market researchers tend to explore the realms of consumer and customer perceptions and attitudes. Depending on the topics being investigated, a long history of both qualitative and quantitative research methodologies have established norms for obtaining valid and relatively precise data from respondents.

However, caution is required in market research when dealing with consumer and customer behavior.

A recent report by the Economist newspaper on the state of television illustrates this continuing issue. The Economist reports on the findings of Sarah Pearson, a research from England, who has almost 100,000 hours of recording of consumers watching television. As the Economist says, “There turns out to be an enormous gap between how people say they watch television and how they actually do….In surveys they almost always underestimate how much television they watch, and greatly overstate the extent to which they watch video in any other form.”

The accompanying data from Nielsen shows that consumers understate their television consumption by some 40% and overstate their consumption of online video by more than 100%.

Market researchers do well to proceed with caution in assessing the validity of self-reported behavior, such as product and service usage.

Comments welcome.

Dr. Bob

[Note: A subscription to the Economist is required to access this report online. It is available for purchase on their website.]