Lessons on Internet Surveys

It's really not matter of whether your respondents are randomly selected or self-selected.

The question is whether your respondents are representative of the population, specifically in the areas you are interested.

The reason why random sampling is so important is that it is hard to know whether or not a sample is representative without having surveyed the population. How can you know?

And so, if your self-selected respondents tend to be younger than the entire population of TidBITS readers, that might introduce a bias in the survey results. For example, it might be biased in favor of game players. However, it is also possible that the respondents ARE representative of the the TidBITS reading population.

This is a key difference, randomness and representativeness. The latter us what matters, and the former is just a method to get there. Much of the statistical technical mathematical stuff is about how likely a random sample is to be representative of the population.

An alternative method to random sampling is purposeful sample, which is what I believe Knowledge Networks does. They promise a representative sample, not a random sample, and they do that by selecting who will get the survey, among their partners/customers/employees/whatever you want to call them.

The problem with low response comes into play if that causes the sample to be unrepresentative. For example, if men are less like to participate, and you have reason to believe that men and women might answer questions differently, that's a problem (though it can be somewhat corrected for). However, if you think that everyone is equally likely to refuse to participate, then it doesn't matter how low the response rate is (except for how much it will cost to get X number of responses).

-- =Alex Hoffman Leadership Policy & Politics Teachers College, Columbia University

on 3/6/07 1:33 PM, Bill Sveinson at bilimac.com wrote:

The real problem with survey accuracy isn't response rate - it's the
randomness of the selected sample of respondents within an identified
population. To the extent that the sample isn't random (eg is self
selected or otherwise biased) the conclusions you can legitimately
draw about the entire populations based on responses to your survey
are inaccurate.


[No argument, but the numbers on the slides Jon Krosnick displayed didn't show huge differences in accuracy for the Internet survey firms he researched. -Adam]


Unfortunately, Web surveys tend towards self-selection bias. You are
more likely to respond to the survey if you feel a strong interest in
the issues raised by the survey questions than if you don't.

Most formal surveys are now biased by ‘opt-in’ selection - they don’t include that subgroup of people that refuse to respond. And this is more prevalent than 20+ years ago when people tended to have more of a “for the general good” bias toward occasional polling (usually about matters of more perceived weight). This is much more significant than Roper, Gallup, et al. acknowledge.

Regards, Jim

On 3/6/07 11:33 AM, "Alexander Hoffman" <ahoffmanaledev.com> wrote:

> This is a key difference, randomness and representativeness. The latter us
> what matters, and the former is just a method to get there.

The other method to achieve this representative sampling is to maximize
response rate, and thus ensuring that most outliers in the random sample are
respresented.

Dr. Krosnick didn't put the lie to this approach through his research, but
he did point out that even in a low responding population, the low response
tends to still be fairly representative of the population. (With level of
education and economic status being the demographics most skewed by low
response.)

In the specific case of TidBITS, it appears you have ideal circumstances for
an excellent online survey:

Because everyone subscribed to TidBITS has internet access, there is no
inherent sampling error due to posting an online survey.

Also because of TidBITS inherent demographics (predominantly, if not
entirely, tech-savvy individuals with Macintoshes, which would put them
toward the upper end of education and socioeconomic status), those
demographics greatest affected by low response aren't likely to shift much.
There are very few (if any) TidBITS readers who ARE of low education or
economic status.

So Adam should feel pretty confident in the results of his survey, provided
he gets a moderate response rate. The population he's sampling is ideal for
an online survey.

--
Nik :: gerberiNik.net
Make a developer cry! Vote for the top Mac software ever!
<http://www.squidoo.com/topmacsoftware/>

On Mar 6, 2007, at 8:50 PM, Muckerheide wrote:

> Most formal surveys are now biased by 'opt-in' selection - they
> don't include that subgroup of people that refuse to respond. And
> this is more prevalent than 20+ years ago when people tended to
> have more of a "for the general good" bias toward occasional
> polling (usually about matters of more perceived weight). This is
> much more significant than Roper, Gallup, et al. acknowledge.

My response to telephone polls is "I don't do surveys." <click>.
(Unfortunately, Congress in its wisdom (what a wonderful phrase!) saw
fit to exempt the polling companies from the do not call registry.)
My response to paper surveys is, almost always, the recycle bag which
sits at my right hand as I type this.

And I actually did fill out the TidBITS poll, but skip most Internet-
based ones. So Adam got at least one response from the geezer and
curmudgeon cohorts. ;-)

   --John

>Most formal surveys are now biased by 'opt-in' selection - they
>don't include that subgroup of people that refuse to respond. And
>this is more prevalent than 20+ years ago when people tended to have
>more of a "for the general good" bias toward occasional polling
>(usually about matters of more perceived weight). This is much more
>significant than Roper, Gallup, et al. acknowledge.

I think that Adam (and the speaker, Jon Krosnick) were specifically
addressing that directly.

That is, it is true that the response is MUCH lower than used to by.
But they are questioning how much that matters. More technically,
does that lower response rate affect the confidence intervals and
when might it be a source of systemic bias.

And so, I return to my point: the question isn't randomness but
rather representativeness. If Jim's explanation of the lower response
rate is accurate, I can see that impacting polls about civic
engagement, citizenship and perhaps even voting preferences. But
would it have a significant impact on polls concerning brand loyalty,
spending patterns, banking priorities and vacation interests (just to
name a few of the polls I've participated in in the last couple of
months)? The question is whether the skew in that particular aspect
of the sample affects its representativeness of the general
population ON THE ISSUES ABOUT WHICH THE POLL ASKING?

For real scientific/academic work, there are numerous things that
researchers do to combat the issue of responsiveness, including
following up/nagging people into participating and purposeful
sampling, weighting different subgroups of the sample, and others.

The whole talks, it appears, was premised on the you raise and others
that are threats to the effectiveness of polls.

I believe that what Adam was relating was that when all of these
things are taken into account, with cost and accuracy being the
important outcomes, Internet polling is not as bad/inaccurate as most
think and door-to-door polling is not as expensive as most think.

(Furthermore, Roper, Gallup, et al. have an enormous interest in
finding valid and cost effective techniques for continuing to supply
useful data to their customers. While they don't publicize these
issues, doubtless they are working on them, and trying to take
advantage of others work in the field, as well.)

--

=Alex Hoffman
Leadership Policy & Politics
Teachers College, Columbia University

In the specific case of TidBITS, it appears you have ideal circumstances for
an excellent online survey:

Because everyone subscribed to TidBITS has internet access, there is no
inherent sampling error due to posting an online survey.

Also because of TidBITS inherent demographics (predominantly, if not
entirely, tech-savvy individuals with Macintoshes, which would put them
toward the upper end of education and socioeconomic status), those
demographics greatest affected by low response aren't likely to shift much.
There are very few (if any) TidBITS readers who ARE of low education or
economic status.

So Adam should feel pretty confident in the results of his survey, provided
he gets a moderate response rate. The population he's sampling is ideal for
an online survey.

This depends on the question. It is certainly true for a “TidBITS user” survey among this group who have opted-in (but even then you would have a bias for the self-selected respondents to the specific topic/questions).

But this group doesn’t fully represent “all OS X users” (most of whom have not opted-in - the 22 million? :-), or computer users in general.

Regards, Jim

> Dr. Krosnick didn't put the lie to this approach through his research, but
> he did point out that even in a low responding population, the low response
> tends to still be fairly representative of the population. (With level of
> education and economic status being the demographics most skewed by low
> response.).....

Not everyone thinks in text, primarily. Mac users think in GUI's, don't
they? A very well documented phenomenon. How can we answer a survey
that is in text unless it is designed to conjour up our personal
preferences in sensory systems?

Then, believe it or not, there are some people who think in systems. If
a question demands a cut and dried 'Yes or No' answer where none exists
to a deep thinker...

Well, let's ignore all the Einsteins and Newtons who read TidBits; they
are insignificant anyway.

George

At 9:46 PM -0800 3/7/07, Alexander Hoffman wrote:
>The whole talk, it appears, was premised on the things you raise and others
>that are threats to the effectiveness of polls.

Yes, that's pretty much it. Although we were able to follow it, it
was an academic talking to other academics in the same field, and
there were undoubtedly many other layers of meaning that were
accessible to people who had the background.

>I believe that what Adam was relating was that when all of these
>things are taken into account, with cost and accuracy being the
>important outcomes, Internet polling is not as bad/inaccurate as most
>think and door-to-door polling is not as expensive as most think.

Or rather, Internet polling is not as bad as you may think, but it's
also not as a cheap as you may think, and therefore face-to-face
polls are not quite as expensive as they may seem. That said, phone
and mail polling are less accurate than both Internet and
face-to-face polling, and not any cheaper than Internet polling.

cheers... -Adam

>And I actually did fill out the TidBITS poll, but skip most Internet-
>based ones. So Adam got at least one response from the geezer and
>curmudgeon cohorts. ;-)

Oh, there was no lack of responses from the geezer and curmudgeon
cohorts. ;) We have more readers who are over 81 than under 21, for
instance.

cheers... -Adam

On 7-Mar-2007, at 22:46, Alexander Hoffman wrote:
> That is, it is true that the response is MUCH lower than used to by.
> But they are questioning how much that matters. More technically,
> does that lower response rate affect the confidence intervals and
> when might it be a source of systemic bias.

Well, I've been very suspicious of Nielsen ratings for the last
twenty years. I don't know if the problem with them is the sample
size or the selection, but there is certainly something wrong with
them, and there has been for a long time.

Where I notice it is not with the popular shows where everyone is
watching but with the very low rated, high quality shows where either
Nielsen is way off or I run into a hugely disproportionate number of
people seeing these shows. And I'm not talking about my friends and
acquaintances, but just people I hear talking about "last night's "
show.

The most famous examples of these for me are South Park, which it
sure seems like everyone watches but gets ratings lower than the vast
majority of the network shows and Buffy The Vampire Slayer which even
at its height of popularity never pulled the ratings it seemed to
deserve based on the number of people I heard talking about it.

It seems that Nielsen is still predominantly families on broadcast
TV, or maybe basic cable, living in the midwest and south. And I've
been a 'diarist' twice for Nielsen. The first time was a nightmare
because they had no way of accounting for my TiVo, and I watch
_everything_ off the TiVo.

I generally don't do any surveys, but do occasional ones online. I
don' do phone surveys because the ones I get are political and
obviously not 'real' surveys, but rather someone with a specific
political slant trying to generate a poll in support of their cause.
You can always tell these by the phrasing, "If you oppose this unfair
practice press 1, if you are a godless commie pinko scum that hates
your country, press 2".

On 8-Mar-2007, at 11:35, George Wade wrote:
> Mac users think in GUI's, don't they?

Er... no?

I am much more of a word person than a picture person, and I think
you will find a higher proportion of word people in the Mac world
than in the world in general. The reason I use the Mac is not
because it looks good (that's just a bonus) but because it doesn't
get in my way as much when I am trying to write something.

My primary application is mail, then terminal, then BBEdit. Oh sure,
I spend more TIME in Firefox or Safari, but a lot of that is wasted
time as I am either searching for what I want, or off on some wild
tangent while I forget what I was doing. And still, the vast
majority of my time on the web is spent reading, not looking at
videos or such.

On Mar 8, 2007, at 11:35 AM, Adam C. Engst wrote:

> That said, phone and mail polling are less accurate than both
> Internet and face-to-face polling, and not any cheaper than
> Internet polling.

Careful there!

Krosnick indicated that internet polling is more internally valid
than telephone or mail. However, external validity can be compromised
by the self-selected sample, as might reliability.

But I'm getting into some jargon, so let me try to break this down:

In a scientific/statistical test, you're trying to achieve Reliable
and Valid results.

Reliability relates to how repeatable your results are. If you survey
one sample and then survey another, do you get similar results?
That's your margin of sampling error in a random representative sample.

Validity is split into two measures: Internal Validity, or how well
your test/survey measures what you think it measures; and External
Validity, which indicates how accurate your test is when generalized
out to the larger population.

Obviously, an unreliable survey that generates different responses
from different solicitations of supposedly representative samples can
NEVER be considered valid. The basic test or sampling method is
incorrect if you can't get some measure of consistent results.

What Dr. Krosnick seemed to indicate, is that people are less honest
and/or thoughtful when responding to mail or telephone surveys. This
affects both axis of validity, since you're getting inaccurate
results, due to issues of trust, empathy, etc. However, these surveys
remain highly reliable, because people tend to respond inaccurately
in predictable ways (they try to please the interviewer, rush through
answers, etc.).

Web surveys, on the other hand, are more internally valid, because
people respond honestly and therefore in a well designed survey,
you'll get an accurate measure of what those people think. However,
web surveys generally have a non-representative sample (people who
can afford computers and typically volunteers who have reason to lie
about their demographic information), so you lose external validity
as you can't reasonably generalize your results to the population at
large. (Although you CAN generalize the results to the population of
people who take online surveys -- hence your TidBITS reader survey
should be pretty accurate.)

As for sample size, basically Krosnick was saying that neither
reliability nor external validity are seriously impacted by low
response. So really, getting a good sample and having a good survey
design are the key components to getting accurate results. Then you
want to serve that sample in the best possible medium, time and money
constraints notwithstanding.

Very interesting stuff. Thanks so much for bringing it to our attention!

--Nik

Lots of interesting discussion already, and I'll try not to repeat.

Krosnick's talk is very interesting. He covers a lot of topics in an hour,
and I found most of it to be very sound.

One thing that does not come across clearly in the talk (and this is one of
the very few places I fault him for imprecise language use) is that his
data did NOT imply that a low response rate does not affect the external
validity. What he said (supported by his data) is that a low response rate
does not affect the *demographics* -- age, sex, race, etc -- of the
respondents. One of the papers on his web page summarizes a paper "Response
Rates in Surveys by the News Media" etc and links to the slides. The slides
are very clear in limiting the conclusion to the effect on demographics --
which is certainly an interesting conclusion, but not the same as a
conclusion on the survey results. But in the talk he slips too quickly from
mentioning the demographics to implying that the conclusion applies to the
survey results also, and the jump is too quick even from the expert
audience which was present.

In the talk, he cites a couple of other studies, but doesn't give enough
detail to find them without some Real Work. In one, he mentions a study
which tested by using only the first 30% of the responses and found no
difference -- but in this case does not make it clear whether the result
applied to the demographics or to the actual result. And I would question
the assumption that taking only early responses is equivalent to getting a
low response rate.

At any rate, Krosnick's main focus was on the response rate and attacking
what he says is a common practice of requiring extremely high response
rates (like way over 50%) for some kinds of surveys. It's true that a high
response rate can set an absolute limit on the non-response bias, and that
there's no mathematical way to set a lower limit. This differs from
sampling bias, where you can set probabilistic (though not absolute) limits
on the bias by proper technique, and by doing so can reach high accuracy
with small samples. Studying non-response bias requires examining human
factors, psychology, etc. Krosnick is addressing this, and I think he's
addressing important questions which have not been addressed until
recently, if at all.

As Alex Hoffman has been saying, "the question is whether your respondents
are representative of the population". With a completely open poll, your
"sample" is the entire population, giving a very low response rate -- and
more importantly, obvious biases in who actually responds. Traditional
surveys limit the sample from the outset, but are still subject to
non-response bias. So where does a TidBITS reader survey lie? Somewhere
between the two I think. I gather the survey links were individualized, so
you have a clearly defined sample group smaller than the population. But
what is your response rate? If it's like 1%, then it's really pushing it to
claim there's no self-selection bias. And what are the possible selectors
for response? Not caring enough to be bothered to click the link
immediately and then forgetting it? Just not wanting to take a survey?
Being a curmudgeon like me and John (though I took the TidBITS survey too)?
Being busy? It's a lot easier to let any of these factors affect your
decision to respond or not when it's just a link in a newsletter, compared
with getting a phone call or a knock on the door.

So there's still a LOT of potential bias if the results are used to direct
the content of TidBITS. Does it matter? Maybe not. Maybe the people who
respond are representative. Or maybe they are the subset which TidBITS
needs to address anyway. Maybe they are the people who will recommend
TidBITS to others and thus affect its readership. Maybe they are the people
who TidBITS advertisers want to see their ads. Maybe they are the people
who donate to TidBITS. Maybe addressing this subgroup results in articles
of interest to the larger group anyway, even if their survey responses differ.

In the end, the question is who TidBITS should be directed at, as much as
how to direct it to those people. That's a question which a survey is
unlikely to help answer.

Edward

(Oh, and Krosnick cites $1000 as the cost of a one-hour door-to-door
survey. That's expensive in any comparison.)

(And Internet polls may exclude curmudgeons even more than the other kinds.
Krosnick says that one of the reasons online polls seem to be more
effective is the one-question-at-a-time technique. He mainly notes that
this allows branching so that the poll taker doesn't see a lot of
irrelevant questions. But some polls also use this to force answers. I've
tried to take polls where I came to a question with yes and no for the
answers, and my answer was I don't know or some other non-yes-no answer,
and the poll wouldn't take nothing for an answer. When I reach that point,
I always cancel out -- creating another bias in favor of satisficers, the
name Krosnick uses for those who want to satisfy the poller. Who pretty
much by definition are the opposite of curmudgeons.)

At 12:15 PM -0700 3/12/07, Edward Reid wrote:
>As Alex Hoffman has been saying, "the question is whether your respondents
>are representative of the population". With a completely open poll, your
>"sample" is the entire population, giving a very low response rate -- and
>more importantly, obvious biases in who actually responds. Traditional
>surveys limit the sample from the outset, but are still subject to
>non-response bias. So where does a TidBITS reader survey lie? Somewhere
>between the two I think. I gather the survey links were individualized, so
>you have a clearly defined sample group smaller than the population.

Actually, they weren't individual - the Web Crossing URL format makes
it look that way, though. The population is the readership of TidBITS
as a whole.

>But what is your response rate? If it's like 1%, then it's really
>pushing it to
>claim there's no self-selection bias.

We're closing in on 3,400 respondents, which is about 10% of our
email subscribers. Maybe 5-6% overall, though those numbers are
harder to pin down.

>So there's still a LOT of potential bias if the results are used to direct
>the content of TidBITS. Does it matter? Maybe not. Maybe the people who
>respond are representative. Or maybe they are the subset which TidBITS
>needs to address anyway. Maybe they are the people who will recommend
>TidBITS to others and thus affect its readership. Maybe they are the people
>who TidBITS advertisers want to see their ads. Maybe they are the people
>who donate to TidBITS. Maybe addressing this subgroup results in articles
>of interest to the larger group anyway, even if their survey responses differ.

It's all very fuzzy, and we knew it would be going in. However, it
gives us a baseline about what our more-committed readers think and
how they work. For instance, I can tell quickly that our readers
don't use aggregator Web sites like MacSurfer much as a way of
getting technical information, nor do they read personal blogs all
that often. And the two most popular "more like this" categories were
Productivity Software and Utility/Customization Software.

>In the end, the question is who TidBITS should be directed at, as much as
>how to direct it to those people. That's a question which a survey is
>unlikely to help answer.

Well, yes and no. Obviously, TidBITS should be directed at the
current readers, since, for instance, switching to writing entirely
about games, educational software, and scripting languages would
cause a huge exodus.

But the other half of that question is, if we wish to expand our
audience, in what direction would we like to expand it? Knowing more
about our current audience will enable us to tweak things in ways
that might draw in new readers without old readers even noticing a
difference.

cheers... -Adam

At 09:49 03/13/07 -0700, Adam C. Engst wrote:
>Actually, they weren't individual - the Web Crossing URL format makes
>it look that way, though. The population is the readership of TidBITS
>as a whole.

Ah, but then the population is the entire Internet population, since it's
not restricted to TidBITS readers. I'll agree that very likely there were
few non-reader entries, but without a check, neither of us actually knows.
Someone could have gathered friends to stuff the ballot box. Again, not
very likely on this kind of survey, but that's the kind of thing you need
to think about if you want valid results from surveys.

Edward
--
Art works by Melynda Reid: http://paleo.org

--- begin forwarded text

From: George Allen
Subject: Survey theory & practice

Hi Adam --

It's always impressed me how easily people are misled about survey
research, the matter of sample size that you recently referred to
being perhaps the most commonly abused idea.

The most important step, and often the most difficult, in survey
research is identifying the so-called sampling frame, which is a
(nearly) complete listing of the entities (people, businesses,
whatever) that you intend to survey. Without a valid sampling frame,
you have no idea of what your results apply to. Professional and
academic surveys maintain carefully created databases that validly
represent the universe to which their survey applies.

Once you have the sampling frame, then the next step, drawing a
random sample, is pretty straightforward. You determine the N by the
outcome precision you desire: for example, the usual political poll
uses a few hundred respondents, yielding precision of +/- a few
percent, depending on the mean percent (close to 50% has higher
variability, since p*(1-p) is maximized when p=0.5). Large N's get
you more precision, but you seldom need more precision, given the
other things that can go wrong with your survey.

Then comes the other really hard part, namely getting a response from
every entity in your random sample. If you don't then there usually
has been some sort of non-random bias in the selection process,
working against your original random draw. Rensis Lickert (who
created the first Lickert scales and established the Institute fro
Social Research at the University of Michigan) was one of the first
to demand extended follow-up visits in order to get interviews with
the specific person identified in the specific house identified in
the specific city block... you get the idea. If you want a clever
example of how this worked in a real-life situation, consider how how
relatively valid estimates of civilian casualties in Baghdad were
obtained.

This work is described in 2 Lancet articles (Vol 364, Nov 20, 2004,
pp 1857 ff; and Vol 368, Oct 21, 2006, pp 1421 ff). First of all, I
should clarify that they sampled the population of all of Iraq, not
just Baghdad. In the first survey, they divided up the area of Iraq
by Governorates, and then, using a cluster-sampling design,
identified 33 sub-areas that each represented 3% of the population of
Iraq. Then, "Once a town, village, or urban neighborhood was
selected, the team drove to the edges of the area and stored the site
coordinates in a ...GPS unit....A pair of random numbers was selected
between zero and the number of 100 m increments on each axis,
corresponding to some point in the village....Once at that point, the
nearest 30 households were visited" (2004, p 1858). This method thus
guaranteed that every household in Iraq had an equal chance of being
in the sample, making it a truly random sample. It's of some interest
to note that, in the 2006 paper, the authors write that they sampled
"...streets or blocks rather than with the...GPS, since surveyors
felt that being seen with a GPS unit could put their lives at risk"
(2006, p 1422).

Anyway, I thought this was a really cool way to get as close to a
random sample as possible in an area where no "list" of the
households or people existed. The fact that they put their lives at
risk -- probably in both studies -- just enhances the impressiveness
of the work.

So you can see what a minor place sample size plays in the validity
of a survey. Choosing respondents at one's convenience or not
following up to get everyone in the sample can bias the results so
badly that the survey can become useless. That's why nearly every
survey result published in the popular press is not worth even
looking at -- unless it's for a classroom deconstruction as to what
could and probably did go wrong.

Best regards

--
George Allen, PhD, Professor
Michigan State University, East Lansing, MI 48824

--- end forwarded text

Adam,

johnbaxterlistsmac.com wrote:
> On Mar 6, 2007, at 8:50 PM, Muckerheide wrote:
>> Most formal surveys are now biased by 'opt-in' selection - they
>> don't include that subgroup of people that refuse to respond. And
>> this is more prevalent than 20+ years ago when people tended to
>> have more of a "for the general good" bias toward occasional
>> polling (usually about matters of more perceived weight). This is
>> much more significant than Roper, Gallup, et al. acknowledge.
>
> My response to telephone polls is "I don't do surveys." <click>.
> (Unfortunately, Congress in its wisdom (what a wonderful phrase!) saw
> fit to exempt the polling companies from the do not call registry.)
> My response to paper surveys is, almost always, the recycle bag which
> sits at my right hand as I type this.

You know what. I'd answer most of those calls if I knew it was from "Pew
Research" or similar. But calls with a caller id of "Out of Area" or
"New Brunswick" get ignored. And I think most polls fall into those
kinds of caller ids.

On Mar 18, 2007, Dan Hardt wrote:

> Maybe you can start a section titled "Only For People Born Before
> 1935".........

Make that before 1942, that cuts out all the boomers and includes
anyone eligible for social security. <g>

Michael Logue