Can Genealogy Software Be Rated Fairly? - Sun, 6 Jan 2013

As you know, I run the GenSoftReviews site that lists all genealogy software available and allows people to review and rate the programs. The ratings are from 1 to 5 stars. 1 is worst. 5 is best. They are subjective ratings, up to the user.

For the past 4 years, at the end of the year, I have listed all programs with at least 10 reviews (at least 1 that year) with a rating of 4.00 or more out of 5, and awarded them a “Users Choice Award”. For 2012, there were 19 programs earning the award.

Since then, I’ve seen many accusations, got emails, and even had “reviews” on GenSoftReviews stating that the awards are not fair. One person wondered about “a flurry of positive reviews”. Another said: “I cannot take the awards seriously. The numbers are clearly being gamed.” Someone even tabulated all the 5-star reviews in a row for one program and tried to prove that it couldn’t be different real people who are doing the ratings.

So I’ve had to think about what all this means. People don’t believe the ratings on GenSoftReviews either when a program they are using is not rated highly enough, or when another program they do not feel is superior to their own is rated higher than their own.

C’mon. Get a life, people. Realize that any user rating site is biased by definition. The same can be said for other sites that give user ratings for other products. Examples: TripAdvisor, electronics reviews online, or mobile app store ratings. It is only the people who find out about the site and take the time to go there, and decide that they have a reason to vote who do so. Their reason to vote may be that they have a product they really like or they tried a product but really didn’t like it. In statistical terms, all ratings sites of this format are a non-random sample.

So, no. The ratings really are not perfectly comparable between programs. But that does not mean they are not useful.

I would say that a program rated 4.4 out of 5 is not “better” than one rated 3.9 out of 5. What I would say is that the people who came to the site gave program A “a higher rating” than program B.

Ratings are subjective. But for most people, to give a 4 out of 5, then you must really like that program. To give a 5 out of 5, then you must love it and be a real fan of it. Once you get to 3 or less, then people have some qualms with the program. That is where the reviews will help you. You’ll see some of the issues that people have with the program.

So I’ve decided that the magic 4.00 out of 5 is the goal that all genealogy software developers should try to attain. Then they’ll be awarded a “Users Choice Award”. I hope the award becomes something that the genealogy community sees as a good thing to attain. And I hope vendors who earn it will want to display it on their website and literature, as it will indicate that users of the program really like their program – and the reviews are there to prove it.

Now what about the situation where a vendor tells its user base about GenSoftReviews and asks them all to rate and review the program. I think that’s great if they do that. They are confident enough in the quality of their program and the happiness of their users that they are not worried about getting bad reviews. If the reviews come in positive and they are rated above 4, they’ll get positive reinforcement that they’re doing a good job. But if they get a lot of poor reviews, then they’ll get important customer feedback that will help them improve their program in areas they may not have realized they needed to. Hopefully, the result of that will be that the program is improved and the next year the ratings for the program goes up. Hey – Isn’t that what you’d call a win-win situation?

Besides. Isn’t the most important thing about a program how much the users like it? This takes into account everything. Not just the features of the program, but how easy it is use, the support, the feeling of community, the marketing of the company, and other intangibles that are impossible to agree on.

If I had my druthers, I’d get 100 users from every one of the 702 programs to rate and review their program. If they got a rating over 4.00, then I’d say they’ve done a good job with their program and should continue their great work.

I think GenSoftReviews does the best job possible in providing you with other user’s opinions about programs. These are real people willing to stick their neck out and say they like or don’t like a program. I don’t know how you can get ratings that are fairer than that.

So is there anything wrong with my thinking here?