Louis Kessler’s Behold Blog

It’s been just over a week since FamilySearch released the official version of GEDCOM 7.0. See my article about the announcement.

Now we probably will go into a period of silence, where nobody hears anything more about GEDCOM 7.0 for a while. The expectation is that all the developers are hard at work implementing the new standard so that they can release it as soon as possible.

But are they?

Not likely.

Motivation. Why?

Developers need a good reason to go through the work to implement the new version of GEDCOM. That’s true for any new feature they add to their program. It needs to be something useful and worth their time. The benefits must be better to them than any of the other features they are thinking of adding.

And unless it is an absolute must-have, then there’s no way a developer will put their current work aside and make GEDCOM a priority.

Something will need to motivate developers to implement 7.0.

Is GEDCOM 7.0 a Must-Have?

Everyone got very excited about GEDCOM 7.0 when it was announced, first before RootsTech as a release candidate. And then again a week ago with the official release. It’s because FamilySearch has not released a new version of GEDCOM in over 20 years. It seemed like something was finally being done. Maybe finally, our data would transfer properly between programs.

If you looked closely at the new spec, you’ll see a good number of small changes. The assumption is that each of these is addressing the transfer of genealogy data that GEDCOM currently doesn’t transfer properly.

Do I see one item there that makes GEDCOM 7.0 a must-have? To be honest, I can’t say that I do. There’s a whole lot of small changes, small new things added, and small things removed.

Okay, maybe there are 3 or 4 really good and useful changes that I see and I as a developer would like to implement. I won’t list them because every other developer will have 3 or 4 different items that they want. The point is, do you go through the work of implementing the hundred other changes required for the 3 or 4 things you want?

The developer likely already had implemented those 3 or 4 things into their own GEDCOM export and import in their own non-standard and custom way. What they’ve done already works for them. Why change?  They’ll need a good reason.

Why Your Data Doesn’t Transfer

GEDCOM 5.5.1 limitations are not and never have been the primary reason why your data does not transfer between programs.

The primary reason why your data does not transfer between programs is because the programmers have not implemented GEDCOM 5.5.1 correctly.

An example: Source information. This has been the number one complaint of people who use GEDCOM to transfer data is that their source information doesn’t transfer properly between two programs. The blame is often placed on GEDCOM being incapable of transferring sources details.

That is False. The problem is that developers were lazy and did not take the time to look to see what GEDCOM had. If they would have, they would have seen the PAGE tag and how to properly construct it. They could then have exported any source citation to GEDCOM in a manner that any other program could properly import it again.

I have seen few, if any, programs that have implemented the PAGE tag properly.

As I said over 10 years ago:

Maybe what’s really needed is an education program. So that developers will be able to study and learn what treasures are really hidden in the old GEDCOM standard. So that they’ll be able to learn how to implement the features correctly.

FamilySearch Needs to Be A Leader

Developers need a reason to use GEDCOM 7.0. If one developer is the first to implement GEDCOM 7.0, nothing is gained. There are no other systems to exchange data with. If two developers implement, they can exchange. If a dozen implement, they can all as well. (Assumption: they are all implementing it correctly, or we’re back to data loss.)

FamilySearch has emerged 20 years later from their abandonment of GEDCOM. They now want to lead the charge towards a new standard. For the others to follow, they really need to lead by example.

FamilySearch needs to show their commitment to their own GEDCOM 7.0 in a strong and demonstrative way. They need to show the rest of the genealogical community that GEDCOM 7.0 is required and they need to do so through their FamilySearch Family Tree.

You cannot export a GEDCOM from FamilySearch. The FamilySearch Wiki says:

Currently, a GEDCOM file cannot be exported directly from FamilySearch Family Tree. However, you can use partner programs of FamilySearch to get the data from FamilySearch Family Tree, and then create a GEDCOM file in those programs. Here is a list of the programs that are compatible with GEDCOM and FamilySearch.

What FamilySearch has done is developed GEDCOM X as their means for transferring data within their Family Tree and between Family Tree and partner programs. GEDCOM X is a programming interface to transfer the data directly. It does not produce an intermediate text-based file such as GEDCOM.

If FamilySearch really wants to commit to GEDCOM, they need to program into Family Tree a means for any user to export their tree to GEDCOM 7.0. In so doing, they’ll no longer be making a standard that they believe will work, but they’ll be putting the standard to test and see what works and what doesn’t and what needs to change. That will make the standard solid, and produce GEDCOM files from FamilySearch that other developers will want their programs to be able to input, and that alone will encourage developers to implement GEDCOM 7.0.

Supporting their own standard is not something new for FamilySearch. During the development of GEDCOM 2.0 to GEDCOM 5.5.1 from 30 to 20 years ago, FamilySearch had their program PAF (Personal Ancestral File).  Every time a new version of GEDCOM was released, a new version of PAF (usually with the same version number as GEDCOM) was released which used the new GEDCOM format.

PAF was free, and was an excellent and very popular program. And it still is being used by many people as it continues to work on Windows 10 even though FamilySearch dropped support of the program 8 years ago.

Maybe its time for FamilySearch to release PAF 7.0 to support and promote GEDCOM 7.0? That might do the trick.

Otherwise

Otherwise, I’m sadly quite pessimistic about GEDCOM 7.0.

I think FamilySearch could have done a much better job with this release. The goal needs to be to help developers do GEDCOM right. We don’t need “new expressivity”, “new flexibility” or “new compatibitlity”.  The old ways weren’t that bad. The developers just weren’t implementing them properly.

I was one of 10 genealogy developers and GEDCOM experts who worked during 2018 and 2019 to contribute our thoughts and ideas towards the GEDCOM 5.5.1 Annotated Edition and the GEDCOM 5.5.5 document that followed that were both edited by Tamura Jones.

Tamura is a well-known GEDCOM expert who posted scores of articles about GEDCOM over the past 15 years, including many detailed analyses and sets of best practices. He was encouraged to put these best practices together into a document to help developers. He did so in 2018 as the 5.5.1 Annotated Edition. Then a year later, Tamura released 5.5.5 as a “Maintenance release. Quality. Simpler & Stricter”. 5.5.5 has already been implemented by several different GEDCOM validator programs.

In my opinion, 5.5.5 is an excellent and important improvement over 5.5.1 for developers, without introducing anything new for developers to deal with. It is much superior to FamilySearch’s 7.0 which changed too much for no real apparent reason.

If FamilySearch is truly interested in advancing GEDCOM, they should be including what the genealogy programming community wants. The work and best practices of Tamura Jones are not something they should be ignoring. In fact, FamilySearch’s best move would be to contact Tamura and invite him to be a managing editor (or even THE managing editor).

Otherwise, I repeat, I’m sadly quite pessimistic about GEDCOM 7.0.

FamilySearch released the official version of GEDCOM 7.0 today. You can find it at https://gedcom.io selecting “Specifications” from the menu.

The standard is available as PDF or as HTML pages. The PDF is 96 pages, compared to Version 5.5.1 which was 101 pages. The specifications page also has a link to the Github Repository which is where the people on the team have been putting together the document. There’s also a Changelog, as well as the specifications for the previous version 5.5 and 5.5.1 of GEDCOM and for GEDCOM X which programmers primarily use to transfer data directly with FamilySearch’s Family Tree.

The FamilySearch GEDCOM Specification itself lists its version as 7.0.1.  This is likely due to a GEDCOM 7.0 being temporarily released as Release Candidate 7.0.0-rc1 back in February for a week during RootsTech before it was pulled, likely deemed not-ready.  So the 7.0.1 differentiates this release unambiguously. See my blog post GEDCOM 7.0 from February which explains what happened with that release candidate.

This version is called 7.0 and not 6.0 because there was a GEDCOM XML 6.0 draft issued on Oct 2, 2000, an updated draft on Dec 28, 2001, and a GEDCOM 6.0 XML Beta Version issued on Dec 6, 2002. The GEDCOM XML 6.0 specifications never became official and I don’t believe any program implemented it. It evolved into GEDCOM X which FamilySearch released instead.

Contributors

The Contributors are listed on page 91 of the specifications. They include a wide range of technical people and genealogy software developers. The contributors listed are:

Managing Editors

• Gordon Clarke, FamilySearch
• Luther Tychonievich, FHISO and University of Virginia

Taskforce

• David Pugmire, FamilySearch
• Jimmy Zimmerman, FamilySearch
• Larry Telford, FamilySearch
• Matt Misbach, FamilySearch
• Russell Lynch, FamilySearch
• Robert Raymond, FamilySearch
• Gaylon Findlay, Ancestral Quest
• Derek Maude, Ancestry
• James Tanner, The Family History Guide
• John Cardinal, Family History Hosting
• Albert Emmerich, GEDCOM-L
• Dave Berdan, Legacy Family Tree
• Evgen Zherebniy, Software Mackiev
• Jason Fletcher, Midlera Software
• Uri Gonen, MyHeritage
• Dallan Quass, OurRoots.org
• Tony Proctor, SVG Family-Tree Generator
• Bill Harten, Puzzilla
• Bruce Buzbee and Mike Booth, RootsMagic

Development Teams

• Tags team: Luther Tychonievich, Albert Emmerich, Russell Lynch, Tony Proctor, John Cardinal
• Extensions team: Luther Tychonievich, Tony Proctor, Jimmy Zimmerman
• Notes team: Dallan Quass, David Pugmire, Jason Fletcher, Russell Lynch
• External Media team: Dallan Quass, Jason Fletcher, Derek Maude

I know there are also many others not listed who contributed as well (myself included) as FamilySearch has been reaching out to all developers for their input.

Mondays with Myrt

This morning, Dear Myrtle (aka Pat Richley-Erickson), had her Mondays with Myrt session. I usually come to those. In her session, Tony Proctor was in attendance and while the MWM session was going on, Tony got an email announcement that GEDCOM 7.0 was released and mentioned it, so Pat invited both Tony and me onto the panel to talk about the specs from a developer’s point of view. This morning’s session is available on the Dear Myrtle blog:  ARCHIVED: Mondays with Myrt – 7 June 2021 including GEDCOM 7.0 info.

I should mention here (and I mentioned it live in the session) that Pat and her cousin Russ were the ones who originally started the BetterGEDCOM project about 10 years ago.
   

The BetterGEDCOM project got developers to discuss their likes, dislikes and recommended changes to the GEDCOM standard that was at that time over 10 years old. Too much data, especially sources, were not being transferred correctly or at all between different genealogy programs. A lot of excellent discussion took place, but Pat realized there needed to be an official organization to do this work.

So Pat spearheaded the creation of FHISO.

FHISO was created to be a standards-setting organisation for genealogical information. FHISO stands for Family History Information Standards Organisation. Their website is at: https://fhiso.org/

FHISO got buy-in from many genealogy software developers and solicited input from all developers. They had a Call for Papers that received dozens of submissions including two from me.

A number of people were on the FHISO Board and it had several chairs, the most recent being Luther Tychonievich, a Computer Science professor at the University of Virginia.  Luther over the past few years worked with FamilySearch as a managing editor to help get GEDCOM 7.0 released.

So it really is Pat Richley-Erickson and Russ Worthington who deserves a big thanks. It’s their initiative that ultimately led to this.

So Now What Happens?

Well, in my opinion, I think it will take a while for Version 7.0 of GEDCOM to be implemented by genealogy software developers. There are a lot of changes and it will not be simple to implement.

During the Dear Myrt session, Randy Seaver told us that RootMagic said they’ll have GEDCOM 7.0 implemented in a month. Personally I highly doubt it. They are currently still working to get RootsMagic 8 released and they are behind schedule. I’m sure they won’t want to mess with GEDCOM and delay their new version any more.

Changing GEDCOM input to support 7.0 would be relatively easy, if it weren’t for the new constructs needing support that a program may not yet have in their database. Export is trickier and needs to be carefully and precisely done, and avoiding custom tags where constructs are available so that other programs will be able to read their data.

FamilySearch is making a few tools available to help developers. Currently they have a v5.5.1 to v7.0 Conversion Code tool. Developers will be able to use this to convert their own GEDCOM exports and see what they will need to import and what their export will have to look like.

The Conversion Code Repository also includes parser/js, a simple program to read GEDCOM 7.0. It is written in JavaScript, so developers using other languages will have to translate it to their own language. But most developers already have their own GEDCOM 5.5.1 parser, so they’ll likely better off just carefully updating their own code.

I am impressed by the small size of the javascript parser. It is only 67 lines, and that include comments! Hard to believe it truly can do all the conversion. Has anyone tested it?

It won’t really help you if your genealogy program’s the first to support GEDCOM 7.0. You’ll need to wait until a number of different programs support it before you can use it to transfer your data from one program to another. More importantly, people today usually have a desktop program and an online systems they use. You’ll need to wait until both your desktop program and the online system (Ancestry, MyHeritage, WikiTree, Geni, etc.) support GEDCOM 7.0 before it become useful to you.

That might start happening in the next year or so. If more developers adopt GEDCOM 7.0, it will gain more traction as a standard, and that will encourage other developers to use it to replace 5.5.1.

My program Behold is currently just a GEDCOM reader. If GEDCOM 7.0 comes to be used by a number of different programs, then I’ll add GEDCOM 7.0 input into Behold. I’ll be looking at the various implementations of GEDCOM 7.0 export from the programs that first implement it and make sure I can handle them, as I’m sure there will still be significant differences in how different developers interpret and implement it.

I’ve been an Independent Software Vendor (ISV), i.e. single developer operation, for over 15 years. I have had my products Behold and Double Match Triangulator available for download for the many versions of each of them.

I never put adware, spyware, viruses or anything bad in my programs. I pay money each year to code sign them so that users and Windows itself can be assured that the program they are installing is the one that I distribute and has not been modified by anyone else.

From time to time, one specific antivirus tool has a false positive with one of my programs and declares it bad. When a user tells me about that, it’s usually not difficult for me to go to the antivirus company’s website and fill out a form to ask them to check my program. They’ve never taken very long to whitelist my program and the problem is solved.

Microsoft Smart Screen

Not too long ago, Microsoft made enhancements to it’s SmartScreen component that it uses on Windows 10 to help protect users against potentially malicious software. Up until a few months ago, the code-signing I added to my Behold and Double Match Triangulator setup programs was good enough for SmartScreen to accept them.

But over the past few months, the following has started to happen. Clicking on the downloaded setup program for either Behold or DMT now pops up the following window which is reminiscent of the dreaded blue screen of death:

Notice there is only one option: “Don’t run”.

You have to know enough and be willing to click on the non-obvious “More info” link to have it allow you to execute the program:

This will display the app information, and they know that I am the publisher because the program is code signed.

And now a “Run anyway” button is available.  Yeah, right. "Run anyway” sounds reassuring – not!

Clicking on “Run anyway” now takes you to the User Account Control window, which is the standard Windows safety procedure for any installation program, and at this point everything is normal:

(I had to take a photo of this window, because it takes over your screen until you answer Yes or No and you can’t screen capture it.)

Smart Screen and Microsoft Edge

Even worse, if you are using Microsoft Edge, then it gives you several additional ominous warnings. First, when you click the download link:

“behold-setup.exe was blocked because it could harm your device” – is not a very friendly download message.

If you hover your mouse over the download box, you get this:

It now shows a garbage can, so that you can delete the download, and three dots. This obviously is more suggestive that you click on the garbage can rather than the three dots.

If you do decide to click on the three dots, you get this:

Once again the first option is Delete. The second is Keep.

“Report this file as safe” takes you to a page where you can report the download as safe, either as a user, or as the developer (see below in this post).

“Learn more” takes you to a page about Microsoft Defender Smart Screen.

“See more” just lists your other downloads.

Clicking on “Keep” brings up yet another scary warning:

So how do you feel about installing a program which “might harm your device” when the only two options appear to be Delete and Cancel.

You have to know enough to click on “Show more” to proceed with the download. Doing so will give you this:

Notice they display my personal information. That is available in my setup program because it is code-signed. They pull out that information and don’t even display it well, showing my name twice, and giving my personal address (not nice of them) when my name, email address, company name (I use Behold Genealogy) and website address would have been much better.

You have to then realize that you must click on “Keep anyway”, and the setup file will finally appear in your Downloads with the ability to Open (i.e. run) it:

If you then try running it, you will get the blue “Windows protected your PC” box described earlier.

So that’s 3 additional ugly warnings that Microsoft Edge adds as obstacles to ensure that you really want to install my program. That even scares me and I know that my program is safe, so I imagine it must stop everyone else in their tracks preventing who knows how many people from wanting to try my programs.

Having my code signed used to be enough for Edge and it always had earlier allowed the download without problems. Try downloading in Google Chrome or Firefox and there are no obstacles. Installation still gives the blue “Windows protected your PC” box, but that’s a Windows thing, not a browser thing.

Report this file/app as safe

Clicking on one of the “Report … as safe” links takes you to this page:

There are two options, to report a program safe as a user, or as the developer.

Clicking the first button results in this:

Clicking on the 2nd option as the owner expands that form to ask more information:

I have submitted this several times in the past few months for both my programs.

After submitting this form as a developer, I get this email back:

And the next day, I get a similar email saying “Your file has been analyzed”. I click on the “View your submission” link in it and it takes me to this page:

If I read that correctly, they seem to be confirming that :

“behold-setup.exe has since established reputation and attempting to download or run the application should no longer show any warnings”.

They then say the signing certificate is still establishing reputation. That should be for other programs I sign it with. Behold and DMT should now be okay.

But they aren’t. Downloads still are going through all the above rigmarole.

I’d be okay if this procedure worked and cleared my program. Unfortunately it did not. I can keep trying, but this is more than frustrating I have to say.

Extended Validation (EV) Code Signing.

There does appear to be one way to get rid of these horrendous messages. That is to upgrade my level of Standard Code Signing to Extended Validation Code Signing.  For only $200 more a year, I can get an EV Code Signing certificate:

Supposedly from what I read, an EV Code Signing certificate will alleviate all these SmartScreen warnings as soon as I start using it.

So a week ago, I purchased a new EV Code Signing for 3 years. For some reason, they were taking a long time to validate my phone number. Yesterday, I found that EV certificates are only available for businesses, and they confirm your signer information by verifying that your organization is valid and that the phone number corresponds with the organization.

I call my software development “Behold Genealogy” but it is not an incorporated business. I declare my earnings on my personal income tax every year. For me to register Behold Genealogy as an official business involves many complications and is more than I want to do.

I’ve now sent in my EV Code Signing refund request.

So What to Do?

As an Independent Software Vendor (ISV), I am somewhat screwed here. Microsoft is putting up roadblocks to my users making them distrust my programs so that they’ll be reluctant to download and install them. And the reputation of my programs is only able to get better if there are users downloading my programs, A bit of chicken and egg here.

All I can really do is add some explanatory info at my two download links to help assure downloaders, try to get their trust, and give them instructions on how to avoid the obstacles and download my programs.

And I hope that Microsoft takes a closer look at the problems this is causing to ISVs like me, and at least mean it when they say my programs have now “established reputation”.

I am very relieved that this happened on its own in less than a month.  My Code Signing Certificates should now retain their reputation at least until I have to renew them which is 2 and a half years from now.

It would have been nice to have received notice when my “reputation” was achieved. I had been working on other ways to package my software so that Microsoft might approve it. e.g. build it as an MSIX package and submit it to the Microsoft Store, but even that wouldn’t have guaranteed anything.

None-the-less, this still was not a nice thing for Microsoft to do, and for the past month (and I’m not sure how many before that), it caused me a lot of anxiety and unnecessary effort to attempt to find a solution. If Microsoft’s “Report a Download” page resulted in a confirmation of established reputation and no more warnings that actually was true at the time, and not a month later, I would feel much better about it.