Usage vs ratings...

At 07:43 07/14/06 -0700, Roberta L. Millstein wrote:
>Are the current ratings more a reflection of who has gone to the
>still-very-new website

That's exactly what it is, and what it will always be. It's a self-selected
sample. Self-selected samples are useless -- fun sometimes, but always useless.

This argument does not negate the other claims of the site, though I have
my doubts. "Organize your applications"? Not obvious. "Discover new ones"?
You already pointed out some of the problems, plus it would be a great way
to spread malware -- just what the Mac community needs, a web site which
enables malware distribution. "Store your apps"? If you don't back up your
data, then your apps won't have a lot of value.

Edward
Art works by Melynda Reid: http://paleo.org

On Jul 16, 2006, at 10:31 AM, Edward Reid wrote:

> At 07:43 07/14/06 -0700, Roberta L. Millstein wrote:
>> Are the current ratings more a reflection of who has gone to the
>> still-very-new website
>
> That's exactly what it is, and what it will always be. It's a self-
> selected
> sample. Self-selected samples are useless -- fun sometimes, but
> always useless.

I have to disagree with Edward while agreeing.

A self-selected sample will never be statistically valid and
therefore never interesting in a context where a statistically valid
sample is needed.

Pages such as are being discussed in this thread are potentially
interesting and potentially useful, and they can be interesting and
useful without being statistically valid. But they should be used
with the last part of that sentence in mind.

Note also that there really aren't surveys that don't have partially
self-selected samples. I'm sure I'm not the only person (probably
not the only subscriber to this list) who responds to any telephone
survey with "I don't do surveys". And I doubt that the three groups
I can think of most likely to do the same thing (irascible old
geezers [yours truly], busy young professionals who don't have time,
and parents in a child rearing crisis at the moment of the call) can
be partly removed from a sample and leave behind an equivalent sample).

   --John (whose only statistics training was MIT's (usually) junior
year Elementary Statistics one-semester course 47 years ago)

At 11:13 07/17/06 -0700, johnbaxterlistsmac.com wrote:
>I have to disagree with Edward while agreeing.

I'll return the favor. ;-)

>Pages such as are being discussed in this thread are potentially
>interesting and potentially useful, and they can be interesting and
>useful without being statistically valid. But they should be used
>with the last part of that sentence in mind.

I agree that the opinions of other individuals can be useful. Opinions of
individuals is essentially what you get in this case. To be useful, you
must have some idea of how much you can trust the person giving the
opinion. I don't see how you can accomplish that in the site as it's set up.

Should the site implement forums, and someone posts to those forums often
enough that you can glean some idea of their reliability, then the opinions
of such persons could be useful.

>Note also that there really aren't surveys that don't have partially
>self-selected samples. I'm sure I'm not the only person (probably
>not the only subscriber to this list) who responds to any telephone
>survey with "I don't do surveys".

Such surveys can count non-responders and factor them into the error
probabilities. (Of course, any survey is only as useful as the probability
of its accuracy.) Open surveys -- those where the contact is initiated by
the survey responder -- cannot count non-responders. Well, technically
that's not true -- it's just that they have to count every potential
responder -- every Mac user, for example, or every Internet user -- as a
non-responder, and that makes the error probability distribution
essentially cover the entire possible range.

> --John (whose only statistics training was MIT's (usually) junior
>year Elementary Statistics one-semester course 47 years ago)

I had a little more and not quite as long ago, which ain't saying much, but
perhaps have kept up with probability theory a wee bit more. I'm definitely
a fellow member of the Irascible Old Geezers Club.

Edward
Art works by Melynda Reid: http://paleo.org

The discussion of whether the sample is "self-selected" is
interesting. When I asked, "Are the current ratings more a
reflection of who has gone to the still-very-new website," I hadn't
thought much about who would be motivated enough to actually say "I
use this." The claim that the site lets you "organize your
applications" is by giving you a place where all the apps you use are
listed, with links to updates right at hand, so if you switch to a
new computer you can easily get everything back. So, someone might
be motivated to "respond" if they found that to be a useful service.
I'm sure there are other factors that motivate people to respond --
or not.

Rather, when I asked, "Are the current ratings more a reflection of
who has gone to the still-very-new website" I had a different sort of
sample bias in mind. Namely, I was wondering not so much if there
was a bias in who was responding, but in who was going to the site in
the first place. For example, is it a "Digg" crowd (whatever that
might be)? The numbers are still looking strange to me.

Roberta

Roberta L. Millstein wrote:
> The new web site at http://www.iusethis.com/ has got me thinking about
> how we choose software. Perhaps this would be a good topic for a
> TidBITS article? The folks at iusethis are claiming that how many
> people are actually using a particular piece of software is a better
> gauge than ratings. And yet, when I look at what pieces of software are
> popular, I'm surprised by some of the answers. It raises some questions
> in my mind: am I just an old fart who is not up on some of the cool new
> software? For example, I was very surprised to find that Retrospect, a
> piece of software often recommended in TidBITS, had not yet been listed
> (so, I listed it -- one of the features of the site is that anyone can
> list software that they are using). Are the current ratings more a
> reflection of who has gone to the still-very-new website, or will the
> current trends continue? Do sites like this just entrench software that
> is already popular without giving new kids on the block a chance? Most
> importantly, is this a good way to choose software? Of course, those of
> us who are partial to TidBITS like our software recommendations to come
> from intelligent power-users who will describe the pros and cons for us.
> So, is there something to be gained here for a TidBITS person, apart
> from voting for software that we'd like to see other people use and
> support?
>

I'll just talk about Retrospect. Alluding to the other responses, I
almost refuse to talk about which backup program is "better". The
problem is there's rarely any context as to goals, budget, operator
skills, etc... Without specifying these first, a discussion of what is
"best" for anyone is pointless. Retrospect is installed where it is a
great solution, a so so one, and a terrible one all because these issues
were not addressed before the program was chosen. The same is true of
all software.

Let's quickly switch to cars. If asked what car is best for "me" without
a proper context, I'd get very different responses from a mom with 5
kids, a traveling salesman, a home office worker who rarely saw clients,
a handyman, etc...

While sites like these are useful, they are rarely comprehensive.