Bill Horne, W1AC wrote:

I'd guess that a "valid" survey would have provisions to account for all
those surveyed, including a method to weed out silent keys, and
provision for guarding against "false positives", i.e., knee-jerk "Yes,
I'm active" responses.



What little I remember from college statistics tells me that the design
of the questions is all-important. The survey mustn't cue the respondent
as to "right" or "wrong" answers, and must provide "discriminator"
questions to confirm and/or deny the accuracy of previous answers.

It's a job for an expert: if we called someone up and asked "Are you
active?", the results would be skewed, as you point out. However, if the
question is, e.g. "Will you help with disaster preparedness as a ham?",
you risk getting a "novelty" response, i.e., a respondent who says "Yes"
just because he/she hasn't done it before.

Questions about purchasing are less likely to show bias, but there's
always the problem of "what do the answers mean?": if a ham says he's
going to buy a new rig this year, is he just trying to please the
questioner, is he window shopping, or is he just wishing out loud?

This is all theoretical, of course. The first issue is to define what
"active" means, and then we'd need a survey that accurately measures the
ham population for that metric. Short of putting remote RF sensors at a
statistically-valid percentage of ham operator's homes, I'm out of ideas.


For an accurate survey, instead of defining active, we would need to
have several questions related to activity. We'd want to first ask the
respondent if they considered themselves active, then questions would
follow asking about how many times per month they are involved in any of
several Amateur related activities.

Just a definition is almost impossible to arrive at. Even if a group
came to a consensus, the next person might not accept that at all.

Just here we see where I was looking at activity relating to things on
a weekly basis, another poster on more of a monthly/yearly basis, and
yet another looked at active as one who takes the trouble to renew their
license. All of those opinions are valid, even though that spans an
extreme range from someone like me who spends several hours each day
involved in one activity or another related to the ARS, to someone who
never gets on the air, but renews their license.

In the end, the survey folks tend to express results in terms of
percentages, such as "20 percent of those who responded use their radios
on a daily basis." 30 percent of respondents participate at least once a
year in a public service event.

The nasty little line in all that is "those who responded". And just
like college football rankings, no matter how sophisticated the computer
program, somewhere, someone is going to make the first decisions which
will be based pretty much on opinion. GIGO, so to speak.

It truly isn't simple, eh?

- 73 de Mike KB3EIA -