Login to participate
  
Register   Lost ID/password?
Louis Kessler's Behold Blog » Blog Entry           prev Prev   Next next

The Future of Genealogy – 6 Predictions - Tue, 7 Apr 2015

There’s been a lot happening the past few years. As I’ve developed Behold, I’ve tried to stay aware of the trends in the genealogy field and the expectations of genealogists of their genealogy software.

I was very inspired by James Tanner’s blog post yesterday titled: Expanding Our View of What is Possible in Genealogical Research. James correctly says that the old way of doing genealogy that we all did 30 years ago is gone. In other words, we no longer have to travel to the library, vital statistics office, or archives and laboriously track down all the bits of information we need to put our ancestors together piece by piece. No. Technology has fallen upon us. It allows us to sit comfortably in our house on our computers and search and find more records and connect with more people and more relatives that we ever could have imagined possible.

The world has changed. Here is my expectation of what is coming:

  1. More Interest in Genealogyimage

    Companies such as Ancestry, FamilySearch and MyHeritage have been claiming tens of millions of subscribers. I’ve heard that MyHeritage is adding thousands of new users each day. Over 20,000 people were at RootsTech in Salt Lake City this year. Who Do You Think You Are and Genealogy Roadshow are now regular programming on major networks.

    Why is this? Because technology has turned genealogy from a niche hobby for only the most studious meticulous researchers to one that can be done by anyone with an internet connection.

  2. Everything Digital

    One of the most tedious tasks 30 years ago was paper, and writing up your family information, and organizing it, and storing it.

    It’s becoming a digital world. Everything is getting scanned. It can be saved online, or shared in the cloud, and organized in folders and every word can be indexed so anything can be found.

    Genealogy software developers are learning as well that people want/need to record their assumptions and reasoning so programs are starting to make that possible and incorporate these features. The data is digitally transferred to your smartphone so you can take it with you. Your camera, scanner, social network, online browser, cloud data and genealogy tool is becoming one device that you carry around with you wherever you go.

  3. Online Data and Online Trees Ad Infinitum

    There are so many online repositories and so many online records, it is getting to the point that no one person has enough time in their lifetime to research all there is about their family.

    The online services now give you smart matches or similarly-named tools that match your data to potential family trees or records that may or may not be pertinent to you. You can easily get 10,000 of these “hints” thrust upon you. If you take only 10 minutes to thoroughly review, assess and if necessary incorporate the results of each smart link into your research, that will only take 2,000 hours of your time. By then, you’ll likely have 20,000 new links to check.

    This is obviously unmanageable and cannot persist. It means that new tools will be coming to identify and make the dissemination of this information easier. (I’m thinking deeply about this)

  4. Down with Standards. Up with APIs

    I’ve been a supporter for years of both the BetterGEDCOM and FHISO initiatives for a new genealogy data communication standard. But I’m now feeling the effort will not get anywhere unless it completely changes its emphasis.

    We don’t want to transfer just data anymore. We want to connect the information available at the online repositories and online services to what we have and make corrections, add conclusions and connect the conclusions to their evidence. In other words, we want our data AND our reasoning AND the evidence behind our reasoning to transfer and connect seamlessly with the online resources.

    I really think AncestorSync had the right idea. Connect to everything. Use the Application Programming Interfaces (APIs) of each company to connect your data to theirs. Hide the details of the API from the user and make it seemless to the user. It should just work like magic. Unfortunately, the implementation of that idea was much harder than the even the very smart people at AncestorSync thought, and the effort was abandoned.

    But it’s starting again. RootsMagic is connecting to MyHeritage and FamilySearch. FamilySearch has partner sites who interact with its data. And other sites are building public APIs as well.

    Once there is a company big enough that connects to everywhere by linking to all these APIs, it will becomes hugely popular, and the genealogical world will take another giant leap.

  5. My Data / My Research

    The concept of one world tree is fine. The concept of individual linked trees is also fine. Each has advantages and disadvantages.

    But everyone wants/needs to separate out the data that they “know” is true from all the other stuff/junk/conjecture/miscellanea that Joe Blow has put up on some online site. We want to know exactly what we have personally examined and verified and concluded.

    So there is still an extreme need for personal genealogy data. The best place for that is still and will always be on your own personal computing device so you can ensure that no one else will update or tamper and destroy what you worked so hard to produce. So desktop software is not dead and will never be (at least until the desktop itself dies).

    All you’ll need is that magic API program from #4 and you’ll be set.

  6. Genetics and DNA

    The elephant is in the room. The technological advances that made DNA testing affordable to the masses in the past 10 years has taken the world by storm. Millions of people have been tested at several different testing companies and a whole new science of genealogy has been born.

    It is really unbelievable what you can do with DNA results when a company has a million other tested people you can compare with. Genealogists are in the “still trying to figure this out” phase, but it’s really simple when you think about it.

    You have two genealogies. There’s your traditional genealogy of whom you, your relatives and the records think your ancestors were. Then there’s your genetic genealogy that says who your genetic ancestors were. These two genealogies are not the same. They may not even be close. The rate of genetic NPE (Not the Parent Expected) has been estimated at between 1% and 3% or higher. By the 6th generation, half the ancestors in your genetic tree might not be who you thought they should be.

    Genealogy will, by necessity, evolve so that people realize they have multiple ancestries, and will want to trace both their traditional family and their genetic family. People have to smarten up first and realize that there’s a reason why your grandfather does not have a DNA match with you. So don’t promote DNA research through your family until you are absolutely sure no one will get hurt by it!

    But this DNA thing is phenomenal. Take it. Embrace it. Use it if you dare.

12 Comments           comments Leave a Comment

1. arb (arb)
Australia flag
Joined: Wed, 25 Feb 2015
7 blog comments, 0 forum posts
Posted: Thu, 9 Apr 2015  Permalink

I agree that APIs are the future of genealogy software, however some standardisation is still neded. The genealogy tools I am writing will expose REST APIs to allow for data interchange between different tools and that’s great, but if you wanted to integrate Behold with my apps, we will still need to agree on how you will access the APIs and what format the data will be in. If every developer created their own API chaos will ensue. Developers need to agree on some set of (minimal) standards for the genealogy APIs so there is scope for an organisation like FHISO to do some valuable work - just as soon as they stop bickering over irrelevant issues like how many IDs should be catered for or what names to use for relationship types and start doing some real work on standards.

As it is, I decided a while back to do my own thing, and if FHISO or some other organisation actually gets somewhere with their standards work, I can worry about it then. (But I’m not holding my breath?)

2. Louis Kessler (lkessler)
Canada flag
Joined: Sun, 9 Mar 2003
213 blog comments, 224 forum posts
Posted: Fri, 10 Apr 2015  Permalink

Arb,

Unfortunately, the initial APIs must be created first by each developer to put in context the range of what’s required. Only then can the different APIs be analyzed and some standardization can be agreed on. And it usually takes a bit of chaos before there becomes a widespread realization that a standard is needed.

FamilySearch tried with GEDCOM X to claim it to be the new standard for APIs, but they were too early in the game, and their model was too complicated and very customized to their Family Tree internals. So the others, like you, started producing their own simpler APIs. This will probably continue for a few more years before we reach the Chaos stage.

Louis

3. jamestanner (jamestanner)
United States flag
Joined: Sat, 11 Apr 2015
3 blog comments, 0 forum posts
Posted: Sat, 11 Apr 2015  Permalink

Very good points. Well thought out. I certainly agree.

4. arb (arb)
Australia flag
Joined: Wed, 25 Feb 2015
7 blog comments, 0 forum posts
Posted: Sat, 11 Apr 2015  Permalink

While you are correct in how standardised APIs are generally created, I believe it can be done without so much chaos. FHISO has a great opportunity to be a key driver in this space, but the FHISO participants need to stop being distracted by the glare of their own brilliance. From what I can make out from the mailing list they have at least conceded that a GEDCOM replacement is not really the most ideal thing to be focussing on. Now all they need to do is stop trying to determine just how many unique IDs are needed and start working on some real, useful standards and initiatives.

I recently took a very interesting course on metadata and information systems, presented by the University of North Carolina at Chapel Hill on Coursera. This opened my eyes to a range of possibilities - thesauri, ontologies, Dublin Core, microformats, schema.org, etc. FHISO could take the discussion on relationship names and direct it into something useful like creating a standardised thesaurus or genealogical terms. They could also work on creating an ontology for genealogical software to use. Or they could create a micro format or schema.org schema for web sites and applications. All these things are (relatively) small, self-contained projects that would go a lot further towards helping create some unity in the API space and hopefully avoid the need for a “chaos stage”.

We don’t need to create a single, final, complete API - we can work on useful, useable elements that can contribute to a future API. Imagine how useful it would be to have a standardised microformat for websites that display genealogical data? This would be useful even for blogs, not just web apps - bloggers could wrap their genealogical data in blog posts in the microformat and other web apps (or even search engines like Google) would know how to interpret the data - genealogy comes to the semantic web!

As I said, I’ll be doing my own thing, but I would prefer it if FHISO (or some other organisation) could gets its act together and help with some of the heavy lifting.

5. Louis Kessler (lkessler)
Canada flag
Joined: Sun, 9 Mar 2003
213 blog comments, 224 forum posts
Posted: Sat, 11 Apr 2015  Permalink

Thanks, James.

Arb: I am VERY curious about “the genealogical tools” you mention that you are writing. Can you shed more light on them? Regarding FHISO, you make very valid points. Telling me won’t help them, though.

Louis

6. arb (arb)
Australia flag
Joined: Wed, 25 Feb 2015
7 blog comments, 0 forum posts
Posted: Sun, 12 Apr 2015  Permalink

My tools? Well, seeing as I don’t like the majority of the genealogical software I have seen to date I decided to try roll my own solution. When I was thinking about how to tackle such a large project, it seemed to make sense to break things down into a series of small, interconnectable tools that each focus on a particular problem. The first tool I am writing is a “simple” (ha! why did I ever think this would be simple?) genealogy research log - I’ve already written and decided to scrap my first effort. Other tools on the drawing board are a research planner (which will obviously dovetail with the log); source management & claim dissection tool; a claim analysis tool; something to assist with transcription; and finally a set of reporting tools.

I am not building a “family tree” tool, rather a suite of tools to help analyse sources and extract all their claims, and tools to help find connections between the claims with the aim of helping with the process of connecting claims to people and rooting out connections that may not be obvious at first glance. One of the reporting tools will produce a GEDCOM so the user can export data for use in “traditional” family tree software, but that side of things is not very interesting to me.

I must admit I hadn’t heard of Behold before I stumbled across your blog, but it seems we share some similar thoughts about turning genealogy software on its head. Traditional family tree software is great for beginners, but there’s nothing in the majority of the software available that really helps with the research process - the user still has to do too much work outside the software and I’d like to try change that.

7. Louis Kessler (lkessler)
Canada flag
Joined: Sun, 9 Mar 2003
213 blog comments, 224 forum posts
Posted: Sun, 12 Apr 2015  Permalink

Arb:

Now I know you. You’re Amos Bannister. I’ve been following your Geneatools blog as well as your Tracing the Footsteps of Amos blog for some time now. I may even have added a comment on your blog at some time to encourage you.

You have interesting ideas. Don’t get discouraged. Keep forging ahead!

Louis

8. Enno Borgsteede (ennoborg)
Netherlands flag
Joined: Wed, 9 May 2012
15 blog comments, 0 forum posts
Posted: Mon, 13 Apr 2015  Permalink

Louis, I agree with your vision, but it leaves me with a bit of pessimism. And that’s because the APIs still concentrate on persons, not evidence. There’s no good exchange of what you call source details, and part of that seems to be linked to the big companies seeing sources as their assets, and don’t provide more than a citation title in the API. Getting more fields, let alone personas, seems to be out of the question, so that source details and data can still only be accessed on-site, and that’s sad.

I would love to find a way to change that, but I don’t know how. Do you?

9. Louis Kessler (lkessler)
Canada flag
Joined: Sun, 9 Mar 2003
213 blog comments, 224 forum posts
Posted: Mon, 13 Apr 2015  Permalink

Enno,

I don’t have much experience with the APIs out there yet. But I am seeing that more and more of the online genealogy repositories and online family tree collections are making evidence through sources and documents more important parts of their systems. That has to mean that they will at some time in the future need to add references to these sources and documents into their APIs.

They will likely do them imperfectly at first, and each will do it their own way at first. And it will get into a bit of chaos for a while. But over time, this will evolve.

I would expect, no more 10 years from now, some standard system will have been built that will pull all this together.

Louis

10. Enno Borgsteede (ennoborg)
Netherlands flag
Joined: Wed, 9 May 2012
15 blog comments, 0 forum posts
Posted: Wed, 15 Apr 2015  Permalink

Louis,

I understand what you say, but my experience in technology is that once functionality is good enough for people that invest in it, like the providers of the unnamed APIs that you may be referring to, there’s not much progress to expect. It’s just like GEDCOM itself. If it sort of works for many, why upgrade for a few?

I am not developing for any API yet, but I occasionally use some through existing clients, like RootsMagic for the FamilySearch API, and a bit of My Heritage, and My Heritage’s own Family Tree Builder. In other words, I look at what the APIs expose through what the client software shows, and that’s not very advanced. For My Heritage it means that you get a title and a reference (URL), and that’s about it. It’s sufficient at the technical level, but it really does not look like serious genealogy.

I tried a few things yesterday, and at that time, I was unaware of the article that Randy Seaver wrote on his blog on the same day, which I will link to below. In my experiment, I created a small pedigree in FTB, and waited for the software to find record matches, which I then linked to my tree on site, because the client software can not do that yet. I have a free account on My Heritage, so the choice of records that I can personally link to is quite limited, but that itself is not the point.

The real issue is that when My Heritage finds a record match, and I approve, the citation itself is inadequate. When I link to a person on WikiTree, which I think should not be a record match at all, because it is a conclusion tree, My Heritage stores a title (WikiTree), and a URL. It used the page field for that, and I can live with that too for now, but that URL does not point to the original person on WikiTree, but to extracted data on the My Heritage site. And apart from that, there’s no readable data for you as a genealogist. The URL includes some sort of numerical record ID, and not the original WikiTree ID, like Borgsteede-3, so it’s pretty useless off-line. Moreover, there’s a quality 4 attached to it, which means direct and primary evidence. And that is utter nonsense, because this is a conclusion record. I know that it’s reliable, because I put it on WikiTree myself, but it is not direct nor primary. The extracted text is in a note, but there is nothing that even comes close to an adequate citation, i.e. one that mentions the referenced item in a human readable manner.

When I link sources on FamilySearch, and look at the information passed by their API to RootsMagic, I don’t see many more fields than offered by My Heritage, meaning that I get a title, URL, and citation text (end note), but their contents are way better. The title mentions Jan Andries Borgsteede in entry for Andries Borgsteede, “Netherlands, Births and Baptisms, 1564-1910″, which is quite descriptive, and I also get a full formatted citation text, that includes above title, film name, film number, record date, and more, like the URL (again). It is not perfect, because I really like to have citation elements, but with about the same amount of fields in the API, a handful, the data exposed by the API is way, way better than a similar API exposes on My Heritage.

You can see a detailed example of the same thing on Randy Seaver’s blog at

http://www.geneamusings.com/2015/04/one-document-four-databases-and-four.html

and I hope that it underlines what I mean. Both APIs provide references, but I consider the ones provided by My Heritage highly inadequate. They serve the site quite well, because you need to visit that to see what the reference really is about, but they absolutely don’t serve me as a genealogist. Between FamilySearch and My Heritage, the technical complexity of the API, i.e. number of fields, etc., is pretty much the same, but the information is not, and that’s what counts in the end.

Given the laws of progress that I mentioned as a start, where progress stops when things work for those who invest in them, I think that the state of the API as it is now, is the state of the API in the future. The formatted citations provided by FamilySearch are very close to the standards defined by EE. And quoting Randy: “That is a very specific source citation for this document, and is, I think, an Evidence Explained-quality citation. Except for the long URL.”. And when Randy writes a comment like this, being a RootsMagic user, I conclude that the state of the FamilySearch API as it is defined by the fields that it exports, and their contents as stored in the underlying public data model, is more than adequate for the English speaking market. And as long as that market rules, and I think it does, I don’t expect to see individual citation elements in an API anytime soon. I really wish for those to appear, but have no signs that they will, so right now, this is the state of the API, for now, and for the near future.

thanks,

Enno

11. Louis Kessler (lkessler)
Canada flag
Joined: Sun, 9 Mar 2003
213 blog comments, 224 forum posts
Posted: Thu, 16 Apr 2015  Permalink

Very interesting observations, Enno. I don’t share your negativism though. I think it is still early in the life of these APIs and they will evolve, maybe even into something quite different that we might not be able to foresee until it happens.

Louis

12. Enno Borgsteede (ennoborg)
Netherlands flag
Joined: Wed, 9 May 2012
15 blog comments, 0 forum posts
Posted: Thu, 16 Apr 2015  Permalink

Louis, I call it realism. The APIs that I see now serve a vertical market, where data exchange is centralized and controlled by the big companies and FamilySearch, and where, interestingly, when it comes to handling evidence, FamilySearch does a much better job than its commercial partners, as you can read in Randy’s article.

The big question is whether you want your data being handled by these, or rather prefer a peer-to-peer solution as was promised by AncestorSync. And although I use centralized trees on FamilySearch and Geni, my hope is that we, meaning independent developers, can figure out a way to exchange data between users, not companies.

 

The Following 5 Sites Have Linked Here

  1. Recommended Reads - Empty Branches on the Family Tree - Linda Stufflebean : Fri, 10 Apr 2015
    [...] The Future of Genealogy – 6 Predictions by Louis Kessler on Behold [...]

  2. Best of the Genea-Blogs - April 5 to 11, 2015 | Genea-Musings - Randy Seaver : Mon, 13 Apr 2015
    ... * The Future of Genealogy - 6 Predictions by Louis Kessler on Louis Kessler's Behold Genealogy blog. Louis looks into his crystal ball - I like the visions. ...

  3. Friday Finds and Follows | AncesStories: The Stories of My Ancestors | Miriam Robbins : Thu, 16 Apr 2015
    ... The Future of Genealogy - 6 Predictions by Louis Kessler at Behold Genealogy - Louis polishes his crystal ball. What do you think of his predictions? ...

  4. Why not Watson for Genealogy? | Canada\'s Anglo-Celtic Connections | John Reid : Thu, 16 Apr 2015
    ...The Future of Genealogy - 6 Predictions by Louis Kessler attracted my attention with the following in the 4th prediction "Down with Standards. Up with APIs." ... "That company does exist, its called..."

  5. [TSC-public] Elements of Genealogical Analysis - Enno Borgsteede : Sat, 16 May 2015
    "...see also Louis Kessler's blog on ... I'm still pessimistic about the state of the APIs, because they don't exchange the data that I like to see, but the vision itself is a very realistic one, I think."

Leave a Comment

You must login to comment.

Login to participate
  
Register   Lost ID/password?