Louis Kessler’s Behold Blog

One thing I’ve tried to do with Behold is optimize its speed. I want it to load your GEDCOM and display the Everything Report for you as fast as possible. I’ve worked hard to do this and have made it a program that is notable for its loading speed.

Every version prior to release, I check both the speed and memory use of that version of Behold to ensure any changes I’ve made have not degraded it significantly. Sometimes, when I add something major to Behold, it may slow down and use more memory. If I think that’s significant, I’ll do some program profiling with a program called AQTime that shows me which lines in Behold are taking a lot of time, and I’ll do some work to optimize and speed it up a bit.

I have several test files I use for benchmarking the speed and memory. One is a large GEDCOM from a friend of mine. Let’s call it the Lynn file. This file has 1.4 million lines, is 24 MB in size, includes 94,000 people and 33,000 families.

The first benchmarks I had for the Lynn file were for version 0.99.2 of Behold, when it took 11 seconds to load and used 265 MB of RAM.  Since then, I’ve added functionality to Behold and the time has reduced for version 1.0.5.1  to 8 seconds and memory to 164 MB.

The ratio of the RAM used to the GEDCOM file size for the Lynn file is 164 MB to 24 MB which is a 7 to 1 ratio. I know that’s way too much. One of the reasons is that Behold loads everything into memory. Behold does not yet use a disk-based database. I’ll be adding that in Version 1.5 in preparation for editing, since Behold will need its own native format to save your data. Once I am doing that, I’ll optimize Behold’s memory use and it should then not use more than a 1 to 1 ratio of its GEDCOM file size.

The second file I use is a huge one known as Good-Engle-Hanks (lets call it GEH). The copy of it that I have is a monster that is 333 MB in size, with 742,000 individuals and 267,000 families. This file challenged Behold. Version 1.0.1 loaded it and displayed the Everything Report after 94 seconds. But it used 1618 MB of RAM. Now that value of 1618 MB is near the limit of what any 32-bit program is capable of in a Windows Operating System. In Version 1.0.2, I added some new features to Behold that added to the memory use, and since then GEH causes Behold to run out of memory just before completing. That’s disappointing. But I do expect that when I get to version 1.5 and reduce the RAM to filesize ratio, Behold should once again handle GEH.

If youre interested to see how your genealogy program handles a huge file, I’m afraid that when I just looked to give you the link, I saw that Mr. Beeston took down his Good-Engle-Hanks GEDCOM from his Gedcoms page. His other files aren’t nearly as large. So might I instead suggest an even larger one that I sometimes test with which happens to be a very interesting file. It is the 2010 Catalog of Life Database converted to GEDCOM and contains 1,257,735 species that work out to 2.1 million individuals. The GEDCOM file is over 650 MB in size. You can get it at Paul Pruitt’s Famous Family Trees site. If you’re curious what program created this huge file, well it was Personal Ancestral File (PAF).

The last file I test with is actually a set of files created by Tamura Jones’ GedFan utility. This utility generates a number of GEDCOM files where each one has double the number of individuals that the previous had. By the time you’re up to the fan value 15, you’ve got over 32,000 individuals in it. The GEDCOM really has nothing but individuals who are connected to their mothers and fathers with very little data. But it is an excellent test of program capability. A fan value 16 file is twice as large as a fan value 15 file. If your program can handle one fan value, then try one higher. Eventually it must fail.

Behold currently can only handle fan value 19, which is about a half a million individuals. It is again the memory limit which Behold runs into. I expect that when I reduce the memory requirements in version 1.5, I’ll be able to increase Behold’s fan value somewhat. And then when I build the 64-bit version of Behold, that should increase Behold’s fan value even more.

The times I gave you above, e.g. 6 seconds for Behold to load my Lynn file, are very much dependent on the computer that Behold is run on. A slower computer will take more time. A faster computer will take less time. So I was wondering how much faster my new development computer was than my old one.

My old machine had an AMD Phenom 9650 Quad-Core with 4 GB RAM running 64-bit Vista. It was five years old. My new machine has an Intel i7-4770 Quad-Core with 12 GB RAM running 64-bit Windows 8.1. Running Behold with my Lynn file, it now took only 2.5 seconds. That was over 3 times faster than my old computer. Similarly, the fan 19 file also was over 3 times faster, reducing from 38 seconds down to 11.5 seconds. The memory used on both machines was, of course, the same, since memory is handled almost the same way between Vista and Windows 8.1. Now I have new benchmarks to compare the next versions of Behold to.

I’m almost through converting over to my new machine. It generally takes me about 2 weeks to get my web development and programming environment all set up again. I’m looking forward to the boost in productivity the new computer will offer me. That should help me achieve my resolutions to get the next few versions of Behold out this year.

I’ve saved all my important emails since 2002. This is my 2nd or 3rd computer upgrade since then. The other times, I was able to transfer my email easily into the mail program on my new computer.

But I ran into a slight glitch this time. My old computer was Windows Vista and my email program was Windows Mail. My new computer is Windows 8.1. It does not have Windows Mail. The recommended path is to go to Outlook and that’s fine by me. The problem was that Outlook  2013 (64-bit) on Windows 8.1 could not import Windows Mail emails.

The workaround was to import Windows Mail into an Outlook (32-bit) version, then export a .pst file from that and import it into the 64-bit version of Outlook. Well, I’m just lucky I had a 32-bit version of Outlook on my old computer because I had been using Outlook 2010 on it. I’m lucky it was 32-bit. It could have been 64-bit because my old machine was 64-bit. This just added one intermediate step, but that didn’t take too long, and now I’ve got my 12 years of important email in my new mail client and available for searching and reference.

What happened is I almost waited too long.

I almost waited until there was no longer a way to move from my old technology to a new one.

This reminds me to remind you of the big lesson you’ll learn the first time you lose something because you waited too long. So don’t do it:

1. Don’t wait until your hard drive fails before you decide backing up is a good thing.

2. Don’t wait until your computer is so old that you can no longer transfer your data onto a new computer.

3. Don’t use a non-supported genealogy program - think Personal Ancestral File (PAF) –and delay your switching to another program until there are no other programs available that can read in your data from that program. 

4. Don’t leave your 8 mm or VHS videos you recorded untransferred to digital unless you are sure you still have a device that will play them to allow you to transfer them.

5. Don’t expect in 20 years from now that anyone else will keep these transfer technologies around. Even if you’re willing to pay for the transfer service, there may be no one who can do it for you.

6. But most of all, and number 1 for genealogists: Don’t wait to interview your older relatives and get what they know. Drop everything else and do that now. Searching online can wait. But the information in people’s heads may be lost forever should they become incapacitated or pass away before you talk to them.

Which reminds me. I’m not sure I have a working cassette player. I’d better go look.

Ready for an adventure? Anything that should be easy usually isn’t, especially when it comes to computers and hardware.

A new computer is expected to result in a boost to productivity … but only after the dust settles. Why is there always so much dust?

The first step is setting up the hardware. I thought it would be easy to add a 240GB Solid State Drive (SSD) to my new computer to become the boot drive to contain the Operating System and program files, and use the 2 TB drive it came with as the data drive. After all, I did buy a kit to make it easy.

A Solid State Drive is made of memory like those in USB pens or SD cards for cameras. It has no moving parts and is about 10 times faster than a standard hard drive. Prices have come down and SSD sizes have increased so that it is now worthwhile and possible to set up an SSD drive as your boot drive. This kit cost $260 but I got it on sale for $160. It is one of the best ways to improve computer performance. Few computers come this way yet. Maybe in a few years.

So this is what happened:

First I had to physically install the SSD drive into my computer. This was difficult as there was no obvious place to secure the 3.5 inch tray that came with the 2.5 inch SSD. The video instructions showed mounting brackets, not a tray. The mounting brackets would have worked. Eventually, I found that I could slide the tray in place and use 2 screws to jam it into the frame so that it would be secure. That would be fine with not too much to worry about since and SSD doesn’t vibrate and gives off little heat.

Between getting my knobby fingers into the case and fiddling my way around, it only took 4 or 5 hours before I settled on that final solution. Only two minor cuts with slight bleeding.

Setting up the SSD required installing the installed Windows 8.1 operating system that was on the hard drive onto the SSD. They provide a bootable Acronis True Image software to do this. The tricky part was figuring out how to boot from the DVD with Windows 8.1. I tried going into the BIOS and changing it so that USB DVD was the first item. I did not realize at the time that it had to be a USB connected DVD (which I do not have) and not the DVD in the Desktop itself. So the second method I tried was in Windows 8.1 to access the Charm -> Settings -> Change PC Settings -> Update and Recovery -> Recovery -> Advanced startup -> Restart Now –> Use a device –> USB Floppy/CD. Of course that didn’t work either.

Well the whole point was to clone the disk. I found a free program: AOMEI Partition Assistant Standard Edition. I had to copy 4 partitions, one by one, but it knew it was an SSD and asked if I wanted to optimize for it, which I did. I set the BIOS to boot up off the SSD. When that rebooted, it asked me the Windows 8.1 first time startup questions again. But after some investigation, I realized it didn’t boot up off of the SSD at all, but still was booting up off the hard drive. I tried a number of things but decided to wait until the next morning to call Kingston Tech Support 1-877-546-4786 Mon - Fri, 6 am to 6 pm PT.

Kingston Tech told me that Windows 8.1 boots in UEFI mode meaning it can only boot from selected devices, and the computer’s DVD drive is not one of them. I would have to change to Legacy BIOS-compatibility mode to do that. I would have tried this, but after I got off the phone with them, there were all sorts of warnings in the BIOS of the consequences of going back to Legacy mode. So I didn’t try that.

The second thing Kingston Tech told me was once the drive is cloned, you have to disconnect the internal hard drive before the OS will make it the boot drive. That was the key thing that none of the other instructions anywhere told me. They also recommended Macrium Reflect to do the cloning which has a 30 day free trial and said it would do a better job to clone the main drive to the SSD.

So that’s what I did. This is something relatively simple but you’ve got to be quite meticulous to do it right. Then in the BIOS, I switched the boot drive to the SSD and disconnected the other drive. The machine started up and set the SSD as the boot drive. We are now at hour 12.

I reconnected the Sata cable for the other drive, but switched Sata positions so that the SSD would appear as Disk 0 and the other drive as Disk 1.

To clean up the old drive, I went into Windows Disk Management, but it wouldn’t allow me many operations on the old drive. Initially it looked like a bug because a right-click would only bring up a partial menu box with only “Help” visible. I tried the Macrium Reflect and AOMEI as well but they wouldn’t do it either. It seemed that HP must have protected the partitions of the hard drive. After some Google research, I found “How to delete a protected EFI disk partition with Windows 7 or 8″ from http://www.winability.com/delete-protected-efi-disk-partition/ and it showed me how to completely clean the drive, after which I could reformat it as one big data drive that would be my D: drive.

There. 16 hours of work later and its done. That was easy, wasn’t it? Probably only 100 steps to do this, but I tried 1,000 steps before I got to the right 100 in the right order.

If you get the least bit frustrated with trying to get computers working, then I would suggest that you don’t try this yourself. I had all weekend and from past experience I was expecting this.

Now I’m a bit closer to getting the new computer set up. I’ll still have to copy all my data over, install all my programs again, customize it the way I like it, and learn some of the nuances of Windows 8.1 so that I’ll be efficient at it.