Login to participate
  
       
Register   Lost ID/password?
Louis Kessler’s Behold Blog » Blog Entry           prev Prev   Next next

Can Genealogy Software Be Rated Fairly? - Sun, 6 Jan 2013

As you know, I run the GenSoftReviews site that lists all genealogy software available and allows people to review and rate the programs. The ratings are from 1 to 5 stars. 1 is worst. 5 is best. They are subjective ratings, up to the user.

For the past 4 years, at the end of the year, I have listed all programs with at least 10 reviews (at least 1 that year) with a rating of 4.00 or more out of 5, and awarded them a “Users Choice Award”. For 2012, there were 19 programs earning the award.

Since then, I’ve seen many accusations, got emails, and even had “reviews” on GenSoftReviews stating that the awards are not fair. One person wondered about “a flurry of positive reviews”. Another said: “I cannot take the awards seriously. The numbers are clearly being gamed.” Someone even tabulated all the 5-star reviews in a row for one program and tried to prove that it couldn’t be different real people who are doing the ratings.

So I’ve had to think about what all this means. People don’t believe the ratings on GenSoftReviews either when a program they are using is not rated highly enough, or when another program they do not feel is superior to their own is rated higher than their own.

C’mon. Get a life, people. Realize that any user rating site is biased by definition. The same can be said for other sites that give user ratings for other products. Examples: TripAdvisor, electronics reviews online, or mobile app store ratings. It is only the people who find out about the site and take the time to go there, and decide that they have a reason to vote who do so. Their reason to vote may be that they have a product they really like or they tried a product but really didn’t like it. In statistical terms, all ratings sites of this format are a non-random sample.

So, no. The ratings really are not perfectly comparable between programs. But that does not mean they are not useful.

I would say that a program rated 4.4 out of 5 is not “better” than one rated 3.9 out of 5. What I would say is that the people who came to the site gave program A “a higher rating” than program B.

Ratings are subjective. But for most people, to give a 4 out of 5, then you must really like that program. To give a 5 out of 5, then you must love it and be a real fan of it. Once you get to 3 or less, then people have some qualms with the program. That is where the reviews will help you. You’ll see some of the issues that people have with the program.

So I’ve decided that the magic 4.00 out of 5 is the goal that all genealogy software developers should try to attain. Then they’ll be awarded a “Users Choice Award”. I hope the award becomes something that the genealogy community sees as a good thing to attain. And I hope vendors who earn it will want to display it on their website and literature, as it will indicate that users of the program really like their program – and the reviews are there to prove it.

Now what about the situation where a vendor tells its user base about GenSoftReviews and asks them all to rate and review the program. I think that’s great if they do that. They are confident enough in the quality of their program and the happiness of their users that they are not worried about getting bad reviews. If the reviews come in positive and they are rated above 4, they’ll get positive reinforcement that they’re doing a good job. But if they get a lot of poor reviews, then they’ll get important customer feedback that will help them improve their program in areas they may not have realized they needed to. Hopefully, the result of that will be that the program is improved and the next year the ratings for the program goes up. Hey – Isn’t that what you’d call a win-win situation?

Besides. Isn’t the most important thing about a program how much the users like it? This takes into account everything. Not just the features of the program, but how easy it is use, the support, the feeling of community, the marketing of the company, and other intangibles that are impossible to agree on.

If I had my druthers, I’d get 100 users from every one of the 702 programs to rate and review their program. If they got a rating over 4.00, then I’d say they’ve done a good job with their program and should continue their great work.

I think GenSoftReviews does the best job possible in providing you with other user’s opinions about programs. These are real people willing to stick their neck out and say they like or don’t like a program. I don’t know how you can get ratings that are fairer than that.

So is there anything wrong with my thinking here?

4 Comments           comments Leave a Comment

1. rjseaver (rjseaver)
United States flag
Joined: Wed, 5 Jan 2011
5 blog comments, 0 forum posts
Posted: Sun, 6 Jan 2013  Permalink

No, I don’t think that you are wrong. I think it’s the best genealogy software ratings service available. I really like that you weight recent reviews more heavily than earlier reviews.

However, the ratings are subjective - and therefore subject to the reviewers biases and experience. My guess is that most reviewers will rate the software that they own and have used. How can they judge it is a 3, 4 or 5 star program?

How many reviewers have actually used more than one software program to do a specific job? I think that I could rate RootsMagic 6, Family Tree Maker 2012, Legacy Family Tree 7.5 and Family Tree Maker 16 fairly because I have a lot of experience with each of them. There are a number of us that could do that, but have we done it? Perhaps a “select panel” of 10 or 20 reviewers could do that and come up with a better ranking of the programs.

My opinion is that some reviewers have given low ratings to some programs out of spite - Family Tree Maker 2008-2012 comes to mind. There is the possibility that a software provider could get a number of friends or employees to rate their software highly.

It is what it is. A software review site with hundreds of reviews each year for each program would be more respected than one with just a few reviews each year. The problem is getting software users to do the reviews.

Thank you for the review opportunity - you can only offer it and hope that it is used and is respected.

2. Louis Kessler (lkessler)
Canada flag
Joined: Sun, 9 Mar 2003
136 blog comments, 200 forum posts
Posted: Wed, 9 Jan 2013  Permalink

I think the key, Randy, is not to worry about comparative ratings between programs. I think the key is that each program must attempt to make its own users happy, 4 stars out of 5.

Louis

3. rodokmen (rodokmen)
Slovak Republic flag
Joined: Mon, 28 Jan 2013
1 blog comment, 0 forum posts
Posted: Mon, 28 Jan 2013  Permalink

Interesting point of view. What do you think about review by “Top ten reviews” (toptenreviews.com)?

4. Louis Kessler (lkessler)
Canada flag
Joined: Sun, 9 Mar 2003
136 blog comments, 200 forum posts
Posted: Mon, 28 Jan 2013  Permalink

rodokmen:

TopTenReviews are not user-based ratings. They are comparisons of programs based on features, and the rating is effectively the decision of a person from TTR. The rating may be influenced by the popularity of the program and possibly by the affiliate commission that TTR would receive. They only include a select list of 10 programs, and there are many more programs than that.

Sites like that are good to compare features and give you a good first idea of what programs you might want to try. But ultimately, you want the program that provides the best usability and benefits that you need.

Louis

Leave a Comment

You must login to comment.

Login to participate
  
       
Register   Lost ID/password?