Questions to Ask Before You Start Market Research, With John Marling

John Marling has been conducting market research for almost four decades. In this episode, he shares insights on what marketers and communicators need to think about in order to get meaningful, actionable data from their research. Learn more at pioneer.coop/surveys.

Creators & Guests

Host

Andy Johns

Vice President of Marketing

What is The StoryConnect Podcast?

StoryConnect features interviews with marketers, communicators, CEOs and other leaders at cooperative and independent broadband companies, electric cooperatives and municipal power providers. The goal of the podcast is to help listeners discover ideas to shape their stories and connect with their customers. It is produced by Pioneer Utility Resources.

Intro:
A production of Pioneer Utility Resources.

StoryConnect, helping communicators discover ideas to shape their
stories and connect with their customers.

Andy Johns:
What are some of the questions you need to ask yourself before
you start doing any market research?

That's what we'll be talking about on this episode of
StoryConnect: The Podcast.

My name is Andy Johns, your host with Pioneer, and I'm joined on
this episode by John Marling, who is the founder and president of

Pulse Research. John, thank you so much for joining me.

John Marling:
I appreciate the opportunity to get together this afternoon,
Andy.

Thank you.

Andy Johns:
Absolutely. Now, John, like I said, is founder of Pulse Research.

We'll talk a little bit more about that in a moment.

They're Pioneers market research partner, and we're excited about
everything happening there.

But John, let's start right where that question kind of opened
up, because you've been doing this long enough to have seen the

good and bad of it.

What are some of the things folks need to consider before they
jump in and start doing market research?

John Marling:
Great question, Andy.

And we always, and we pride ourselves, in getting actionable
information that doesn't sit on the shelf.

Now, the way we accomplish that is the very first step before we
start anything with the research is getting together with you

guys and really putting your feet to the fire.

What do you want to find out?

What you want to learn?

What action steps?

What decisions do you need to make, you know from the research?

You know, so that there's a very clear objective and mission so
that then when the research is completed, you will have

information that, as I said a moment ago, is actionable and
usable and meaningful for you.

And so, we make sure that in that first step, we ask you the
question, what do you want to learn?

What's most important to you?

Do you have any decisions?

You know that your board is going to be making?

And are there some standard elements relating to, you know,
client satisfaction?

But in quick summary, we make sure that we clearly identify
together what your

priorities are, what your needs are, what action steps you want
to get out of the research.

Andy Johns:
You touched on a lot there, and particularly asking what the
board might want to know, I think is a tip worth coming back to.

But I'll preface this interview, and John and I were talking
beforehand, in my undergrad and graduate

program, I think I wound up with three C's overall, and two of
them were in the two statistics classes that I had to take.

So, John, I'm relying on you for quite a bit of this, but that's
good because when it comes to experience doing

surveys, you've been doing it for a while.

John Marling:
Yeah, 39 years, I'm proud to say, doing thousands and thousands
of surveys.

And speaking of sample size, you know, in this type of research
where you want to

get some insights to make decisions, to get some clarity in
terms of where you stand, over all the

years, you know what we've advised, what we found out is
important, is that either ends of the spectrum.

In other words, decision making research is what's important.

You know, what's the takeaway?

And what do I mean by that?

What that means is – and I'll just give you a for instance.

If nine out of ten respondents to the survey say you're doing a
heck of a job, you'd agree with that.

That's meaningful, right?

You know, if one out of 10 or 10% say you're doing a good job,
meaning 90% say you're doing a bad job, that is also quite

meaningful, isn't it?

And so –.

Andy Johns:
The truth hurts.

John Marling:
Exactly. But either end of the spectrum is where you make the
decisions.

Okay? So if your board is making a decision on a new facility or
something – and we've had those questions come up in the research

we've done – you know, nine out of ten people say stay with and
just remodel the current facility.

That is obviously insightful, you know, in terms of direction.

You know, and but coming back to sample size, you don't need for
this type of research, you know, a sample of

1,000, because, once again, you're making a decision based on
either end of the spectrum.

And so many of the cooperatives have 1,000 members.

Well, the gold standard in research is 400 completed interviews.

Well, you're not going to get 400 out of 1,000, a 40% response
rate.

You know, so we suggest 10%, you know, 10% of your member base,
you know, which a thousand, you know, members would be

100 sample. But if 90 out of those 100 say you're doing a great
job, once again, you're getting the results you want.

You know, the insights you want, not the results, but the
insights you want from the research.

Andy Johns:
And I think that's a really important piece because like you
said, some of the co-ops out there, especially, you get some, I

think in Iowa, there's some with 700 members, there are a lot of
them in that, you know, 1,000 or 10,000.

And there are some that are huge.

There are some with over 100,000.

But the research, the techniques, all of that, it can be applied
whether you're talking about a real small organization or a much

bigger one.

John Marling:
Absolutely. Speaking of the research, one of the key elements
that you know, we advise on and assist with is making sure

that each of your members have an equal opportunity of
participating in the survey.

And what that means is you don't just, you know, put out one
invitation in your newsletter.

You use multiple channels available to you.

So you're getting a diversity of outreach, and therefore getting
a diversity of your members, both young, old.

You know, members who have been a member for a year.

Members have been, you know, for 40 years.

So using all the channels available to you to get a valid
representative sample response from your

membership.

Andy Johns:
Perfect. And I think that's important to bring up.

I want to kind of shift gears a little bit.

So if we're talking about, you know, the ways to promote
surveys, the audience, the sample size.

Once somebody determines, okay, there's some things that we need
to know.

And this is something that we have struggled with because every
other year we do a Pioneer member survey that we ask our members

and our clients to tell us about.

Just like anything else, one department may have these 3 or 4
questions to ask, and then another department has 3 or 4 more.

And then, you know, the CEO wants a few more.

The board wants a few more. All of a sudden you've got a 50
question survey.

How have you tried to help folks zero in on the key questions?

Is there a magic link to, you know, "no, you can't go above X
number of questions." And then how do you fight that survey

creep? Because sometimes when you're asking good questions and
other folks may have more, like you're saying actionable things

they want to do, how do you help kind of keep folks focused on
what they want to know for the survey?

John Marling:
Once again, a very good question.

But in that process of the initial meeting, we establish
priorities.

You know, what is most important?

Because obviously, like you just stated, there's a lot of
stakeholders, you know, who want to participate in the survey,

obviously, with questions that are germane to them.

So one of the things we ask is, have you done a survey before?

You know, if they haven't, then we want to touch all the bases.

What do we mean by that? You know, you've got questions relating
to satisfaction, questions relating to perception, questions

relating to needs, demographic questions, etc.

Communication, which is an extremely important set of questions
relating to the takeaways, the action, you

know, from the research.

So in answering your question, Andy, if it's a first time
survey, we'd want to make sure and our advice would be, you know,

to prioritize and to touch each one of the primary elements, you
know, of

a survey. If they've done a survey before, what questions do you
ask before, so we can do some benchmarking, you know, to your

responses from before.

You know, the good news is though, Andy, this is not a phone
survey.

You know, in a phone survey, you're severely limited by time,
number of questions.

Okay. Particularly in today's world, you know, you don't do so
in phone surveys because nobody wants to answer them.

And Pulse was one of the pioneers of doing online research many
years ago.

And so the good news is when you doing an online survey.

And here's the other positive, in this case, you're doing a
survey of those who have a germane interest in the survey.

They're members, you know, of this utility, or they're members of
this broadband provider.

And so they've got a vested interest in doing the survey.

So you're going to get a much higher response rate.

So we at Pulse are less worried about the number of questions
because the people are interested.

And they will take the time because it's their survey, their
utility, etc.

And also, you know, Pulse is not going to limit per se the
number of questions.

We're going to do the questions that are needed to get the
results that you've prioritized when we have that initial

meeting. You know, so it could be 40 questions.

It could be 60 questions.

It could be 50 questions.

You know, the importance here is getting, you know, asking the
questions to get the information that are a priority for

you and knowing where you stand and what decisions you're going
to be making, etc.

Andy Johns:
Got it. That opens up a few other questions that we haven't
talked about before.

But is there a frequency?

Can you over survey? And I know this may be a loaded question for
a guy who makes surveys for a living.

But, you know, are they better off if there are those kind of
disparate requests from different departments, would they be

better off doing a number of short surveys throughout the year
instead of one longer one?

Or do you hit a point where if you're surveying too frequently
then the response rate starts to drop?

Have you seen anything to point you to a direction there?

John Marling:
Well, once I get a good question, it goes back to what have you
done before?

What are your objectives?

So, you know, in a pretty first phase survey, you do it maybe a
year to

two years afterwards.

Maybe a year to see how you've improved in areas that maybe you
were soft.

But let's say, you know, a particular client, you know, came to
Pulse, this utility had a specific decision or a specific

area. Maybe they were thinking about offering a new service.

You know, we found out that in that initial survey, you get some
pearls of insight that beg a follow up survey to get

more detail.

And so it really depends on the client and the focus objective
of the research in terms of the follow up

and, you know, kind of the frequency thereof.

Andy Johns:
Got it. You talked about it earlier that a lot of the folks
taking these surveys, particularly with the co-op, with the

PUD, with, you know, a telephone cooperative where they are
members, they have a stake in it.

Maybe that's incentive enough.

But with some of the other surveys you do, and you see things
like contests to, you know, win certain things.

Do you guys normally recommend doing incentives or does that
help at all?

John Marling:
Even with all the differing types of research Pulse does, this is
the one that is, how should we say, easier.

Because they have a stake hold.

They are members.

And I want to put it out. I've been a member of a utility co-op
for about 49 years.

Andy Johns:
That's right. You know it firsthand.

John Marling:
I would love it if they would do a survey.

I would immediately fill it out.

But you know, either the good, bad, and indifferent.

You know, I'm a member, and bottom line is it's my utility.

Okay. But to answer your question, we normally suggest you do a
thank you, an appreciation.

You know, a drawing for, you know, maybe a Visa card, $100 Visa
card or something that's universal.

Maybe $100 in groceries or something, just to say thank you.

We appreciate you being a member.

We appreciate you being a client customer and for taking the
time.

It would be less important, in this case, to enhance response.

But I think it's important to say thank you.

We appreciate you being a member, a customer you know, of us.

Andy Johns:
Perfect. So as we're kind of doing an overview of the surveying
process here, are there things

that you've seen and maybe it was early in your career, and
you've learned those lessons now.

Are there some common things that kind of lead to those follow-up
surveys, or the people do the survey, and then they wind up

thinking, oh, I wish we would have asked about this?

Or wish we would have asked about that?

What are some of the common things that come up that you're able
to help folks realize, now, before we launch this, don't you want

to consider A, B, and C?

John Marling:
I think in response to them.

If I'm understanding your question –

Andy Johns:
It was not my best question, to be clear.

It's a little intimidating asking questions to who a person who
is a professional at asking questions.

John Marling:
In every survey we've done, in thousands of surveys we've done,
there's always every single time an, oh my gosh.

And I call those pearls; things that you don't expect.

Andy Johns:
Okay.

John Marling:
And also in the surveys we do, there's a lot of reinforcements.

Well, we thought that, and now it's nice to have that
reinforced.

Okay? So now in response to your question, in those aha's that
come up, those pearls,

that's where they normally, clients want to follow-up and get
more detail, get more insight, get more depth,

you know, to take an applicable response.

Because in the first time you do a survey, it's the broad brush
.

You're getting feedback on satisfaction, perception, needs.

You know, communication.

You know, type relating questions, effectiveness of
communication.

And then when you go in maybe a phase two, you dive deeper
because of what you learned from the first level.

You know, so that's, and so yeah, every single time we find out
the aha's we point what we recommend you do about it.

And it ends up being a very, very positive because it helps them
overcome a weakness and turn

it into an opportunity.

Andy Johns:
You mentioned another example was about a facility being built.

And again, we don't want to identify anybody, but are there any
other examples for the kinds of things that the utility

organizations have wanted to know about?

John Marling:
Well, you know, at a board level.

You know, I've made presentations of the results of the utility
survey to the board.

And I think that has been very positive for all the stakeholders
involved, because, number one, it

reinforced, in this case that the satisfaction of the services
was overwhelmingly positive.

But number two, it pointed out some areas, of softness that the
board was very, very positive to find out.

You know, they could respond.

And then of course, in all the surveys we do, we ask, what
decisions do you need to make?

What decisions do you have on the horizon in terms of services
you're thinking about or offering, or facilities that we've asked

about here?

You know, there's a range of – how should we say – as the
utility industry looks forward in time, a

balance of renewable versus legacy power sourcing.

You know, so questions relating, querying their membership as
you look into the future, what are they willing to pay for, not

pay for?

You know what are they willing to commit to, etc.

So at a board level, those are phenomenally beneficial insights
.

And what I'm thinking about it before I forget, one of the things
that we do in the survey is ask a couple of open ended questions.

Andy Johns:
Oh, okay.

John Marling:
And oh my gosh.

Oh my gosh, Andy.

Do they open up.

And that's one of the benefits of doing an online survey.

When you're doing a phone survey, you're limited time.

But an online survey, you ask an open question about, you know,
what do you think about x, y, z and let them just

respond, just go on their little keyboards.

It is awesomely insightful to really digest and really analyze,
you know, the

gist of the open ended responses.

Andy Johns:
Well, and that's interesting. And I'm glad that you took us
there, because having done a few very basic surveys, not anything

to the level that you guys do, but.

You know, you can see a shape, on the the curve, on the responses

to be positive or, you know, whatever it's showing.

It's easy with the quantitative numbers are on there.

But it can be tough to me, when you, the open ended responses,
you know, it can

show that, like you said, nine out of ten are really happy with
this.

But one comment down there, in the open ended part that says,
you know, is there anything else you'd like to add?

That's negative, sometimes that rings louder in our ears than
what the statistics would show up above.

And maybe that's just the questions not being crafted properly
when I've done it.

But how do you weigh that?

How do you weigh those open ended requests versus the bigger
picture statistical data coming in from the survey?

John Marling:
It's through experience, recognizing that people are people and
some people are overly positive, some people are

overly negative.

And so you've got to be able to sort out, you know, as you're
analyzing the open-ended responses

to really identify the true meaning.

And then – and this is important also – then you qualify that or
corresponded correlate it

to, you know, the specific data set relating to satisfaction
scoring, like one out of five.

And so there's no clear cut, Andy.

You know how do you – it's based on the experience and knowing
that human

beings will be there.

There's a group who are over positive and a group who are over
negative.

And you've got to sort through that and then compare it to, you
know, the statistical, you know, results and come up with a

recommendation. But I'll tell you, the clients love the
open-ended responses.

Andy Johns:
I'm sure that kind of puts flesh on it, for sure.

Well, John, just a couple of other final questions for you here.

What are some of the trends?

Obviously, you've been doing this almost 40 years.

What are some of the things that are different about surveying
and doing member research, market research now?

You know, is everybody's attention span really short as we think
?

Or what are some of the trends and changes and maybe where even
do you see it going, just as you've seen the changes in the way

that we do market research or people's attitudes towards it?

John Marling:
I think in response to that, Andy, is there's a lot of – and I'll
be quite blunt – a lot of entities who think they know it.

They know their audience, and they know their members, etc.

And the reality is, in today's world, it's a very complex world.

We unfortunately don't.

And so I mentioned a moment ago and in terms of, in response to
your question, the important reason to doing research today is

you're going to get reinforcement, you know, to those areas you
think you know.

Well, I thought we knew that, and it's been reinforced.

So you know then that management team or the board is much more
confident to move forward.

But then there's always the ahas, the unknowns.

You know that you learn from that research, you know moving
forward.

And then taking it to the next level, and I've thought a lot
about this because our commitment is to actionable information to

be a real catalyst, you know partner not with just Pioneer, but
with the individual utilities to be able to

provide information, to be able to assist in the communication
to those members.

You know, being a member of a co-op, it's important, you know,
for me, you know, and I was doing a presentation a while ago

to a board of one of our utilities.

We had just completed the research.

And I've been a member of this co-op for, you know, almost 50
years.

And what I was shocked to find was in the presentation to the
board, that there is a high

probability of electrical outages.

That there isn't a reliability of power sourcing, and there could
potentially be rolling

brownouts. And I go, oh my gosh, I mean, what does that mean to
me in my, you know, co-op?

Point being in response to your question, in today's world, you
can't take things for granted, and you've got to

have an eye to the future because the world is changing and, you
know, power sourcing, the balance of green

versus legacy, you know, power source is very important.

But what the real element here is what the real takeaway is,
Andy, how do you communicate that?

How does that public utility, the co-op, you know, communicate
to its members?

You know, not only the power reliability, but, you know, a fee
increases, price increases.

You know, and that's why I think it's really important in answer
to your question is you find out so you can be able

to – and this is important – tell your story.

That's what you know Pioneer is all about.

You know, tell your story.

Andy Johns:
I've heard that a time or two before.

Yeah.

John Marling:
You know, the research is discover your story.

Discover your story so you can communicate that story.

I mean, that's critically important because members of these,
it's their utility.

Okay? And they feel that the utility board, that utility
leadership, you know, has a

responsibility to communicate to them.

And so in response to your question, I think it's more important
than ever for, you know, those reasons to reach

out and not just assume you know, but to find out from your
members where you stand.

And also dip your toe into the future, so you're able to
effectively communicate.

Andy Johns:
Perfect. I like that a lot.

Last question I have for you.

Obviously I would love folks to go to pioneer.coop/surveys.

You'll get a link to case studies and the other information to
get in touch with with us and with John and everybody,

pioneer.coop/surveys.

We would love for them to do the market research, with us, with
you, through this partnership.

But whether they do or whether they don't, what's the advice
that you would have for somebody who's looking to do market

research? Who's looking to know a little bit more about their
story, like you were saying.

What are some of the things that they need to keep in mind for
that?

John Marling:
Like we, and I think it's worth restating.

What do you want to learn?

Know what the actionable areas of information before you start.

Because the worst thing that happens is you just, you know, do
the research, the time, effort, expense, and it just sits on the

shelf. You know, in this world today, it's too important.

You have to know.

And so if and when you do it, make sure you have clear cut
objectives that your whole team is on board and

what you want to get out of the information so that you do, in
fact take action from it.

And that's the mission of Pulse is, is to provide information
that gives you that insight, that gives you, you know, that

reinforcement, if you will, in that areas of aha, that you
didn't know those pearls to take action.

Andy Johns:
Perfect. That's good advice from a long career of doing this sort
of thing.

So it would be good for us to take heed.

John, thank you for joining me.

John Marling:
Thank you, Andy. I appreciate the partnership.

Andy Johns:
He is John Marling.

He is the founder of Pulse Research, founder and president.

I'm your host, Andy Johns with Pioneer.

And until we talk again, keep telling your story.

Outro:
StoryConnect is produced by Pioneer Utility Resources, a
communications cooperative that is built to share your story.

StoryConnect is engineered by Lucas Smith of Lucky Sound Studio.