So I was excited when Tamura Jones posted three GEDCOM torture tests along with how a good number of programs did against them. (Note: Internet Explorer can’t view Tamura’s site. Try a browser like Firefox or Google Chrome)

I was looking forward to Behold’s results. I was not happy when I saw Behold had only passed one of the challenges. I pride Behold in its ability to read all flavors of GEDCOM, and this indicated something needed to be fixed. Knowing that not a single program passed all the tests did not make me feel any better.

So as part of this version’s upgrades, I decided to include a few of the things on my “Tamura Jones” to do list - not the least of which was to easily read all the torture test files. That is now done. I hope Tamura will check it out and verify that I have actually done so.

I’ve made a number of improvements and fixes in this version. See the complete list on Behold’s Version History page.

I’m very pleased with my progress so far during the beta. I’m actually getting through all the bug fixes and changes almost as quickly as I had hoped I would. An April release date for Version 1.0 is still my goal.

So I figured I could reuse that routine and simply add a new option to write to the log file. The file information in the Everything Report was something that could then easily be sent also to the log file - so I thought.

It took about 2 hours to set up and then - yuk! It looked terrible.

The Everything Report, HTML and RTF are all nicely formatted and indented. Text for the search doesn’t matter since you don’t see it. But the log file is plain text with no formatting at all. The stuff was jammed together, not indented, had hugely long lines and basically was impossible to read. I spent another couple of hours trying my best to automate the formatting and make it look reasonable.

That wasn’t going anywhere. So I took it out, and in a half an hour, I added custom append to the logfile routine that now does it much nicer.

The sexier way is not always best, so it sometimes pays to take your lumps early and go on.

Basically, when Behold reads a GEDCOM, it may find problems with the GEDCOM file. I’ve now added the listing of the problems to the file information in the Everything Report. But what if Behold is interrupted or does not get to that point? How about more detailed information that isn’t appropriate for the file information (such as line references from the GEDCOM), or maybe just some warnings about the GEDCOM that you really don’t care about normally, but may want all the detail for.

Well this should all go into a log file that is created when a file is read. The fact that there is a log file at all will add to your security to know whether Behold is or isn’t detecting something.

I had two different interfaces for a log file. At first (way back in 2005) there was one I really liked. The top part had the GEDCOM files with a tabbed interface. The bottom was the log file. It was meant so that you click on an log message, and the line could be brought up in the pane above. But back then, loading all those GEDCOMs into TRichView windows was expensive in terms of performance. I didn’t have the Virtual TRichView back then - so I took it out and I put in a simple writing of the log to a file. That happened in version 0.98.8 alpha, 28 Jan 2007.

I then decided for version 0.98.9.5 alpha, 19 Nov 2007 to not print the log file altogether. This is the decision I am now reversing.

Putting it back in really isn’t too hard. All I have to do is compare the differences between version 0.98.9.5 and 0.98.9.6 to see what the log file code was. I started that two days ago, and it took about 15 minutes to get the code in again.

The harder part was my “AppendToLogFile” routine. It was written with Delphi 4 before Unicode and didn’t work with Unicode. I puzzled how to get that working and searched Google, but after two hours I wasn’t there. So I went back to my friends at StackOverflow and asked “How can I get this File Writing code to work with Unicode (Delphi)“. With that help, it’s now just two days later and the core code is working again.

There is no standard way to write a log file, so it’s up to me. Most genealogy programs do not do a good job with theirs and many don’t even produce one. Behold is already fairly decent at determining problems with GEDCOM files, so its log file should turn out to be quite useful when it’s needed.

So it’s really nice when someone else can write up what my blog is about even better than I can.

Dean Richardson at Genlighten did just that. In his January 22nd Follow Friday post, Dean writes about “Behold Genealogy and Louis Kessler’s Behold Blog”.

Dean writes:

I have a lot of admiration for Louis’ openness, persistence, and pride in his craft. You won’t find a lot of details of Louis’ genealogy research in his postings, but you can tell he’s channeling his love and respect for his ancestors into his software development efforts.

Well that makes me feel all warm and fuzzy inside. Thanks very much, Dean.

To read Dean’s entire post, pop over to the Genlighten Blog.

Not doing that will make it much easier to input and store the data and then later export it back out to GEDCOM. It will make the internal data structure of Behold a truly GEDCOM record-based structure. For some reason, some genealogy programmers think this is good. Personally, I see advantages and disadvantages to it, but the bottom line is “it’s convenient”.

It took a while (hey: a while is now just “a week”!!) to make these changes. Over a year ago, I had tried to implement them, but I bit off a bit more than I could chew, and the program was too complex to rebuild using that data structure so I had to abandon those improvements.

This time, I started with a working version and kept it working as I made the changes one by one. This method worked much better, and the knowledge of where I was going as well as having a prototype of the code already available made this second attempt a success. There are a few small things that I haven’t yet converted over, but those I’ll get done for the next beta. Most people won’t notice them.

Because of this major change, I do expect some bugs. The tests I have run are working, but I expect the real world will find others. You’ve all been great at uncovering new problems for me to fix. Really, do send me them. It’s the only way I can ensure that Behold becomes as bug-free as possible.

And that last version 0.99.5 did have some nasty problems that some people uncovered. I believe this version has fixed almost all of them. So I highly recommend upgrading to 0.99.6.

For a complete list of the changes in this version, see the Behold History page.

These system errors are called “exceptions”. The last alpha releases were very stable. I had over the years trapped or fixed every crash point I found. But the major changes between the last alpha and the first beta (virtualizing the Everything Report) have added some new ones. Also I’ve added an “exception” handler into Behold to catch not just the ones that Windows reports, but an even wider range of errors that previously went unreported or unnoticed. This is also allowing me to correct some code that I never knew was wrong, and that is good.

Then I also added the exception reporting mechanism, which brings up a custom screen and allows the user to send me the error message. At first it was quite onerous, but I think I’ve since fixed that and made it minimalistic and easier to use. The number of crashes I’ve been encountering have been going down, and so have the numbers users have reported to me (thank you all!).

Fairly soon (one or two beta releases from now), Behold should be back to being as stable as the alpha was. Once it’s there, I’ll finally feel confident enough in it to take the next step, send out the Behold News newsletter, and invite the world to try the beta (… and raise the price).

Yes there’s still lots of small things to fix, enhancements I want to make and unknown bugs left to find. I’m whittling away at those and will handle them rigorously when I start to redo the help file. We’re getting there.

I also had problems identifying why the recently used file list was not always getting updated. This was a component in the LMD ElPack that I was using for this. Rather than puzzling this through, I basically had the code without the component in the Find File and Forward/Back history lists. So I converted the recently used list to use my own code and that fixed the problem. It also changed a few minor appearance issues that now made that list more consistent with the other two. One minor byproduct of this change is if you upgrade to this version, you’ll lose your recently used file list in Behold, and will have to build it up again, which shouldn’t be too much of a bother.

All changes for this version are listed on Behold’s History page.

2 NOTE This is a note that is sp
3 CONC lit on two lines.

The note will be put together and the word “split” will be reformed as one word. That is the way it is clearly defined in GEDCOM. There will always be a word split in the middle, with the first part of it on the previous line and the rest of the word being completed on the CONC line.

Many programs follow that standard, but there are a lot of programs that missed that little nuance in GEDCOM and decided to implement their GEDCOM output in what would seem straightforward:

2 NOTE This is a note that is split
3 CONC on two lines.

Here the split occurs after the word. What this means for a program like Behold is that it now has to add a space before concatenating the two lines. In the previous example no space was added.

Unfortunately there is nothing in the GEDCOM file telling you which method the program used. I could write a procedure to scan the file and “guess” what it is or some other form of artificial intelligence. I might be able to make that 98% accurate, but never 100%. So something is needed to change this if the assumed method is wrong.

What I have done in Behold is made a list of programs that use the correct CONC that splits words and does not require a space added. I assume all others use the “bad” CONC that does not split words but requires the space.

My current list of programs that outputs CONC correctly to GEDCOM is not very long. It includes PAF, Brother’s Keeper, Legacy, The Master Genealogist (TMG), RootsMagic and only some versions of Family Tree Maker. I haven’t rigorously gone looking for them. If you know of any others, I’d be happy to find out about them and I’ll add them into Behold.

That’s all wonderful. But then I found out that The Master Genealogist added a user option to allow them to output CONC tags the incorrect way, presumably so that the file can be read by programs that don’t understand the “correct” way. So now, I can’t even rely on a specific program always doing it the same way. Boo to TMG for that.

What I did was on my Organize GEDCOMs page, I now have a CONC value for each GEDCOM. It specifies how many spaces to add after concatenating lines, either 0 or 1. It will be automatically be set to what I’ve assumed is its program’s default. But now you can change that value and save it in with your Behold file so it can be remembered. The next beta release will include this.

So if in your notes as displayed by Behold, you see spaces where they shouldn’t be or two words put together where they shouldn’t be, it may either be that the CONC setting needs changing, or maybe the GEDCOM you’re looking at has simply has tons of typos.

Unfortunately the web update feature of Behold in 0.99.1 through 0.99.3 beta of Behold was not working on Vista or Windows 7.

The web update downloads the behold-setup file to the Program Files directory where Behold is. Unfortunately, Vista and Windows 7 do not allow a program to add files to that directory, and that gives an error. To get around this, I now download that to the temporary files directory.

What this means is you may not be able to upgrade to the newest version of Behold with the web update. This now should be fixed in 0.99.4 but of course you won’t be able to try it until 0.99.5 comes out. For now, if Behold’s web update won’t work for you, then as I said in my last blog post, go to Behold’s download page and click the download link and run the setup from there.

There are a few other fixes in this version as well. See the version history.

Thank you for all your bug reports. Please keep them coming until I get rid of them all.

But first things first: If you try to do an auto-upgrade from the last beta, it may work, or you may get an “Error occurred” box, or it may update but when it tries to start up again you get an “Unable To Locate Component” error box. In the last two cases, please just go to the Behold download page, and click the download link and run the setup from there. Then everything should be okay. - No. I’m not happy about that either, but I’ll try to make sure it won’t occur again in the future. Unfortunately, this is what a beta is for - to get the kinks out.

None-the-less, this version should be better than the last. The hardest bug to fix was the scroll up and scroll down keys which would cause an exception everytime they reached a horizontal line with an ID number at the right. Those are tables in TRichview, and I wasn’t restoring the current selection correctly in my virtualization routine. I needed Sergey (the developer of TRichview)’s help, and just today it was solved. Then I thought I could get the new version out tonight, and I ran into the auto-upgrade problems when I tested it. But it’s too late tonight (2:45 a.m.) to see exactly why and maybe to fix it. So I’ll have to leave it.

Still some important things coming before I consider this beta functionally complete. I want to improve a few things before I set up the Behold News newsletter again and then try to stabilize and document towards version 1.

I tabulated the rating scores of the programs with 10 or more user reviews, and those with a rating of at least 4 out of 5 were given an award.

Hopefully this will generate a little interest in the genealogical community.

I couldn’t have been more pleased and surprised to read that yesterday Tamura Jones gave Behold an “honourable mention” for the 2009 GeneaAwards in the Most Improved Product of 2009. RootMagic 4 won that category and they definitely deserve the top honor with all the improvements they’ve made this year. See the Awards at: http://www.tamurajones.net/GeneAwards2009.xhtml (note: Internet Explorer cannot view the page. You’ll have to use a browser like Firefox or Google Chrome).

So what can I say about the next decade (and I know there are purists who don’t believe the next decade is starting, but personally, I find the “1″ in “2010″ quite significant). Well, if I might be so brash as to predict that by then, most people would have got rid of their “good enough” genealogy programs, and we’ll have had a significant number of computer genealogists migrate from their inefficient forms based systems to the friendlier and easier word processing model that Behold will introduce.

Ten years from now, I should have succeeded to have gone through my boxes of family data, and have them all entered (using Behold) into a complete and very useable personal family history. I can hardly wait.

I wish each and every one of you a healthy and happy New Year and many more after that.

And after a week of fiddling, I guess I’m happy with the way it looks now:

When I release the next beta, we’ll see if this implementation does not prove to be problematic like the last one. If so, I’ll have to either rethink it again, or remove it.

So it’s back to MailTo. We’ll try the new scheme in the next beta. If it doesn’t work out right, I’ll fix it or take it out.

But now there’s still a couple of important bugs still left to squash before the next beta release.

First I had to figure out how EurekaLog was calling up your email program. Two complaints about it was more than I wanted to hear:

Your debug routine fails to send the file, as it tries to start windows mail for some reason known to it’s self rather than using the default mail client (in my case thunderbird)

I tried to fill up the exception report but gave up when it asked for this server, that server! I then got really annoyed because it launched Outlook - I never, ever use it or want to use it.

So I found out that it was using Messaging Application Programming Interface (or MAPI). The problem is MAPI is not included with every email program, and will in such situations bring up a different program than the one you have as default. I also checked out the INDY (a Delphi package that provides Web services), and it has the same problem.

But there was a solution. The “mailto” command you find on websites brings up your default email program. It also has parameters that allow you to specify who it is going to, the subject, the body of the message and others. On a windows machine, that same function can be called by passing the mailto line to ShellExecute.

I thought I’d try it. It didn’t take too much to get a test going. And I was quite pleased to see that the web limitation of 256 characters was not there.

I had to figure out how to disconnected EurekaLog’s report and set up my own Exception Dialog form. Then I had to figure out how to pull all the info out of EurekaLog’s debug information for inclusion into my form. That went fairly smoothly until I got to the Operating System information. That was only in their log file, so I had to work out how to parse their log to find that information.

Wonderful. It was all set up and ready to go. I saw EurekaLog was attaching its log as a zip file. Hmmm. I don’t need to do that, since all the relevant error information was now in the body of the email message. But wouldn’t it be great if I could also (at the user’s option) automatically attach the GEDCOM and/or the Behold file and the future Log file?

Then I discovered (and this took over a day to realize it) that mailto did not support attachments with most email programs. I found a lot of people asking how to do this via Google searches, and there were several answers given. I tried them all and they didn’t work for me. Then I found that it only worked for certain email program such as Outlook, so that was no good.

But I was rolling. I knew there must be a some simple way to get the attachments to the email program. I posted a question on StackOverflow: How can a Delphi Program send an Email with Attachments via the DEFAULT E-mail Client?. The answers were coming in, but they were all telling me to go MAPI or INDY.

So I continued to do some research and I discovered some code that was not too complicated on Dragging and Dropping files from my application to Windows Explorer. Doing so would also allow me to list the files and allow the user to (relatively) easily drag them over to the email message that I’d create for them. It’s not the perfect solution, but it seemed like the best I could do if I wanted to ensure I properly accessed the user’s default email program.

I set up a Filelistbox on my Exception form, and got it all working. It was old code and required a few changes to work with Unicode strings. But it worked.

Next was setting up the Filelistbox to list the GEDCOM files and … oh oh. It required all the files be in the same directory because it was designed for an explorer-like directory listing. That didn’t work. After some thinking, I copied my ElTree from Behold’s FindFiles window and adapted that.

Once I had the file list box, then I discovered the GetFileListDataObject routine from the code I adapted also assumed everything was in the same directory. I had to convert that to figure out the data objects of each file no matter what directory they were in. This was code that I really didn’t understand. But I took a shot at it. I was extremely surprised when this worked the very first time - a rare event for me and for programmers in general.

I had to add “Select All” and “Select None” buttons to easily allow you to pick the files to drag. The selection of individual files had an interaction with the dragging, so that was a glitch that is still outstanding as I write this.

Then the cruncher came. My final tests and I get an exception while doing the mailto. It turns out that there is a limitation of 2048 characters that I can pass via mailto. Could I live with that? I don’t know. I was hoping to append the log file once I recreate it next version to the exception email, but that would now not be possible.

I was just about ready to take my lumps and maybe even back out of this whole thing, when surprise of surprises - another answer on my StackOverflow question appeared.

Instead of using “mailto”, mjustin pointed me to a way to used the Windows “sendto” command to send the attached files to the default email program and set up a new message with the files attached. This method will not require the user to do the extra step and drag and drop the files to the message and apparently it also has no limit in size of message either.

So just a bit more work to implement this now.

Yes. I did get carried away. This was a lot of work for something not part of the core functionality of Behold. But I’m not one to let something go half way. Best to finish it off while its in the moment.

And how was your holiday weekend?

There’s really only about 8 lines that are needed to get this working. I’ve just been trying hard to find those 8 lines, because they involve saving the selection and then restoring it. It’s easy in most of TRichView, but that horizontal line is a RichView Table, and I just haven’t been able to get that working. So I’ve got a question into Sergey on his Support Forums but I’ll have to wait until he gets back from holidays (Dec 27) before he can help me solve this.

Speaking of exceptions, a couple of people have complained about the exception handling that I added into the current beta. It causes problems interacting with their email system trying to set up an email to post back to me with the error.

That information has been tremendously beneficial to me. In about 4 bug reports, it led me directly to the source of the bug and saved me many hours of work. But I do admit that it is very intrusive and I don’t think I would like having a software product that tries to start my email program. So I’m going to work to simplify it considerably, and make it friendly and useful.

Basically, I’m going to have the box come up and just tell you the error that occurred (e.g. “Division by zero”) and the routine and line number where it thinks it happened. Then it will ask you to email that info to me to help me improve Behold.

I may decide to store that error in your Behold registry info, so that it can differentiate between the new errors and the ones that you’ve had before, and it can give you a count and ask you to email the info if it hasn’t happened before. When you upgrade to a new version, it will wipe out that registry info, because presumably those problems will have been fixed in Behold.

Then on my Future Plans page, I’ll list the errors, routines and lines where these exceptions have been reported. And I’ll mark them off as I get through them.

That’s what I’d call going from Exception-ally Bad to Exception-al.

1 NAME first middle /surname/ suffix

e.g. 1 NAME John Jacob Jingleheimer /Schmidt/ II

The name parts should be shown in the order given, but without the two slashes that surround the person’s surname.

To make this trickier, some East Indian names are in the form:

1 NAME /surname/ middle first

and should be displayed in that order.

I had Behold handling the East Indian names correctly, and I’ve now got it to handle the suffix automatically and correctly as well.

Behold is fairly unique in its ability to take the wife’s surname at birth and add the surnames of all the men she is matched to, ending with her final spouse, e.g. Mary Jane [Watson Osborn] Parker.

Now there are times you’ll want to override that. One example was a file that had under the marriage record a date of “NOT MARRIED”. So I check for that and do not add the husband’s surname in that case. I don’t know of any other instances where that sort of an indication is in the GEDCOM, but if I find any, I’ll add them. Once I get to Version 2.0 and editing, Behold itself will store this “override” into the file, so you’ll have complete control.

Some people have asked for the ability to turn off the generation of married names in Behold. While fixing the names, this turned out to be a very simple thing to do. So I’ve now added a Report option for this. Then Behold will give you less information than you’ll want just like other programs do.

I had a couple of those unexplainable programming occurrences happen. The worst was after I had done those two very difficult fixes, with the help of Uwe in the comments on my last blog post, and with the help of the LMD people on their newsgroup. I made those fixes and then was getting ready to go forward to the next item, and all of a sudden the files were loading much slower. I mean like 20 times slower.

It took me a while to trace this to where it was happening. First I tried using Eric Grange’s Sampling profiler that I was hoping to use in the future for optimization. I was terribly disappointed in how it couldn’t get me close to where the problem was. It was listing system routines with 90% of the time being string copies. Well I know that is going on at the low levels. What I want to know is from whence that is being called.

So I went back to AQTime. They just put out version 6.30 which finally includes integration into Delphi 2009, so it was worthwhile me trying it again. Of course, they’re still behind the times, because Delphi 2010 has been out for awhile so they’re over a year behind in their Delphi integration.

The AQTime Delphi 2009 integration didn’t help. Seemed to be buggy and I couldn’t get it to profile by line. So I did it externally and right away found the line in question. Problem was, there was absolutely nothing wrong with the line.

I was using Delphi PChars, and adding the length of the block to the starting character pointer to tell it where to stop. But somehow, it did not add correctly, and subtracted 4 instead. So the entire block was being Encoded once for each record, rather than once for each block. And the code was correct. I couldn’t find what was wrong.

I went back to my earlier beta version. It had no problem. I checked my next build. It was okay. Then the next build - that one no longer worked. There was nothing different codewise. I spent a day on it. Possibly some compiling option got changed, or some memory got filled somewhere? I don’t know. But my major changes had happened in the 5 builds since then. After I gave up trying to fix the problem, I had to go back to the working version and add all the changes since then and check step by step that the problem did not reoccur. It did several times along the way. So I had to go back and make fewer changes at a time. Doing so finally got me to my current state without the PChar problem.

So I’m totally perplexed and don’t know what this was. I have to stem my curiosity or I would spend the next two weeks or longer trying to track this down.

But the important thing is to continue on with the beta. So as long as I’ve rid myself of this problem that shouldn’t have occurred anyway, I better just forget about it, pretend it didn’t happen, and just cross my fingers that it doesn’t happen again. I don’t expect that it will, but if it does, then I’ll nail it down and put it away for good.

I guess I had entered the Programmer’s Twilight Zone. Let’s hope I’m out.

I’ve fixed a few already and I’m in the process of fixing about 6 others at the same time.

One involves something that Delphi itself seems to have a problem with. I’ve got a question on StackOverflow for it and one of the resident experts there is helping me out.

Another is a problem with my ElXTree component, and I’ve reported the problem to the LMD people on their newsgroup, and they are helping me with that.

Then I’m debugging some bugs that crash Behold and fixing those. One is simply using the arrow up key to scroll up the window. I had thought I had debugged my Virtualization of the Richview window, but I guess I missed that.

Then there is one that links to the child of a person, but that child is actually the user’s parents. That one is also surprising to me. I checked the GEDCOM thoroughly and it appears to be correct. My linking of this sort has been working correctly for years, and I’m very perplexed by this one. I’ll be interested in seeing the reason.

Then there was someone reporting something in Windows 7 that I can’t reproduce in Vista. So I’ve been loading both Windows 7 and Windows XP into Virtual PC windows so I can reproduce the error and ensure that Behold runs okay under the two systems, without needing multiple computers. (I absolutely love Virtual PC!)

So lots going on here and I’m making good progress. Hopefully in a week I’ll get out a new beta with most of the problems fixed.

Something is Behold was causing Avast to think that Behold and its install program had a virus, when I knew they didn’t. And that’s why I code sign Behold - to make sure nothing happens to it before it reaches you. Fieke checked and told me that Behold’s signature was okay.

You couldn’t imagine my horror upon hearing that. Having that happen to your program is as bad as Tiger Wood’s little escapade the other day. The damage is irreparable.

I went to Avast’s online scanner and submitted Behold myself. Sure enough, it reported that Behold.exe was infected with Win32:Delf-MZG [Trj].

I immediately shot off an email to Avast. I was pleasantly surprised to receive a prompt response that said:

It was a false alarm of Avast virus database 091203-0. It was fixed in the latest virus database update, please update virus database. We deeply apologize for the problem.

Well that certainly made me feel better. Fieke let me know when Avast next updated its new virus database, the reporting of a virus was fixed. Today I went back to the Avast’s online scanner and sure enough, Behold is now reported as clean.

So after a heart numbing affair, I now find that a search on Google for: win32:delf-mzg [trj] gives a good number of “false positives” with Avast reporting every program from Skype to Spybot to a host of others. Avast had to fix this, because the mistake would ultimately have caused them even more anguish than they did me. Here’s one of the typical reports on this by Andrew Wee just posted today.

This is what programming’s all about. There’s always something new creeping up that you never expect. Avast, me hearties.

The beta releases will come out more often than the alpha’s did. I hope to get an update every 1 or 2 weeks until the beta is complete. I’ll announce the updates on this blog.

If you are running Behold and you see on the menu line “New Version Available”, please click on it and update. You’ll find the update quite quick and painless. That way you won’t find any bugs that have already been fixed.

Tamura is one of the toughest graders of genealogy software you’ll find, with some reviews literally tearing programs apart and stomping on the pieces. I’m quite pleased with the review done of Behold. It’s an honest evaluation of a program whose beta is “definitely a serious improvement” but is still “far from finished”.

I’ve sent Tamura my comments and look forward to getting a response. I always manage to get excellent insights from our “discussions”. The whole virtualizing of the Everything Report was brought into being by some of Tamura’s ideas several years ago, and it is my intention to meet some of the challenges that Tamura has set upon the genealogy software industry. … One step at a time.

Well, I had changed the expiry from 90 days after the program was built to the new trial rule of 45 days after the key is sent for. I changed that number everywhere I could find. In the program, and in all the website procedures.

Then I uploaded it to the website. My upload compares differences in file size to see what files changed. In the one critical program, the only change was “90″ to “45″ and both of those are 2 characters and that did not change the file size. My comparison program did not report a difference and the file was not updated. Oooops.

All is fixed now. I’ve sent e-mails to those involved asking them sheepishly to try one more time.

Time to take a big deep inhale … and breathe.

Version 0.99.1 beta was released this morning, and version 0.99.2 beta came out this evening, after I fixed two minor issues with the “New Update Available” display and with text on the info box that still referred to the alpha.

If you are using the alpha, please do upgrade to the beta. It should be able to do everything that the alpha was able to do. If there’s anything you find that was working before that isn’t working now, please report it to me and I’ll fix it ASAP.

For large files, you should find it MUCH faster and use a lot less memory. Files of 100,000 people should load in 10 to 20 seconds, depending on the speed of your machine.

The help file has grown a bit old, and it will be updated during the course of the beta. I’ll actually be using the help file, as I update it bit by bit, to document my progress through the features of Behold as I ensure it all works as it should. Once everything’s checked and the help file is finished, version 1.0 will come out.

We’re off and running again!

I finished up and compiled the package. Then I had to make the install script. Whoops. I didn’t have it. Had to download InnoSetup and install it. Then I had to set up all my links again because everything on my new computer is in different directories. One very hidden setting prevented InnoSetup from working for me and it was an hour until I found that and fixed it.

So once I got the install set up, I had to test it. I installed the new version of Behold, and tested it. The welcome screen came up fine. Closed that. My Good Ideas box pops up. Close that and an exception occurs crashing Behold. What happened? I actually hadn’t tested my Good Ideas box since before I upgraded Delphi, and I had unchecked the option back then. Here was a new install and it appeared by default. But it was hooked into the old version of LMDTools, before I upgraded to their 2009 version. So it abended. Getting it linked back again wasn’t easy and that took another hour.

Then I noticed that the hyperlinks in the Good Ideas weren’t working. Actually, they may never have been working. This was obviously an easy thing to fix. But it ended up taking me 2 hours to fix it.

Despite all that, I still would have got the beta out today. But then it came to the code signing. I had to download and install Microsoft’s software development kit (oh, what a pain) and then get my old certificates and put them on my new machine. With that done, I attempt the signing and … it doesn’t work. I now have a ticket in with Comodo to help me get the code signing going again.

I suppose I can get the beta out without the code signing, but I might as well wait until tomorrow to see if I get an easy fix from Comodo.

Alas.

Examples of an exception is an array out of bounds, accessing an invalid memory location, or a division by zero. Often an error message pops up that has some fancy wording that is unintelligible to most people, but once leaving the message box, the program gets lost.

So with the help of a program called EurekaLog, Behold will now pop up a friendly box whenever one of these exceptions happens:

The beauty of this is that it will capture all the debug information and allow the person using Behold to easily send me an email containing mostly everything I need to find and fix the problem.

Now I’m sure you’re saying “I’ve seen this before”. Of course you have. Windows does this to you all the time. You often have things pop up like this:

I’ve often clicked on the “Send Error Report” button. But with Microsoft, I’ve never got any response back. So much so, that I almost feel I’m sending these things into a black hole that they just ignore.

To be fair, they probably get millions of them, all the same, for the same problem. And they most likely do fix the problem for the next release. They just can’t get back to every one of the millions who submit it.

What this also does is let Behold encounter its severe errors gracefully. Behold should not get stuck. If in that run it encounters 2 more errors, you’ll then get an  option to terminate the program.

I hope this will save me time and you from the annoyance of possible program crashes during the beta.

I was out yesterday at lunch for a walk and I passed by the Canadian Museum of Human Rights, a $260 million world-class museum that is currently in its initial stages of construction in Winnipeg. I took a picture:

It reminded me of Behold, now in alpha soon moving to beta.

Then I came to the display that said what was to come:

This reminded me of Behold version 2.0.

The race is on. Who will be first?

So those were the 6 major things I needed to implement for the beta. All done. I started them on October 14th, so they took just over a month.

Now there’s just 6 small things left:

1. Change the expiry date method to now use a 45 day trial from the date the key was downloaded.
2. Add a “New Version Available” link in the menu bar, to make it clear when a new beta is available.
3. Include an “Exception capturing ability” so that any bugs won’t cause Behold to crash, but will allow reporting of the bug encountered.
4. Recalibrate the progress bar now that I’ve re-twigged everything.
5. Change where files are installed to be the correct places on Vista and Windows 7.
6. Change the Behold web pages to announce the beta and build the new version and put it up.

The exporting has been going pretty well. Then I saw that the last alpha did not make URL’s or e-mail addresses work when exported to HTML for the web. So it wasn’t too hard to program those in.

What immediately became apparent to me is that if Behold’s HTML is then put up on the web, the email addresses are ready for harvesting by spam bots. Not nice, and not a thing that I’d like Behold to be responsible for.

Currently on all my web sites, I show my email address and allow a click on it to bring up your mail program seeded with my address. Try it with the email link at the bottom of this page.

Secretly, I use the javascript language to obfuscate (or hide) my actual email address from those automated spam harvesters, none of which know javascript. You can’t tell, because the email shown and passed to your email program looks fine. That’s because your browser processes the javascript for you. But those bots are not browsers and don’t get it. I have posted my email address on the web for years this way, and I get fewer than 10 spam a day compared to about 50 good emails. And I’ve used MailWasher (a great program) for longer than I can remember to check through my spam .

Unfortunately, the way Richview develops its email links, I can’t incorporate the Javascript code into Behold’s output. Instead I used a combination of URL character encoding and Hex encoding, which isn’t perfect, but should stop 95% of the spam bots.

So if you are a person that will be exporting Behold’s html output to put it on the web, I’m trying to be responsible to protect you as best as I can.

I’m pleased it wasn’t that much slower than the old way. For my very large test file, it previously took about 25 seconds to bring up the Print Preview. Now it takes 45 seconds. And there is optimization I can do sometime in the future to bring that 45 seconds down somewhat.

But remember this is a big file. This preview results in a 13237 page document. That size document does well to choke most programs.

It wouldn’t highlight some searches on the way back. This was tricky because the cause was not what I expected. Going backwards, I had to set it to start at the end of the current block of text. But the last item was sometimes my invisible text. In that case, the current location would not be set.

That’s basically working good enough for Beta now.’

On to number 5.

For my large test GEDCOM (24 MB, 100,000 people), it has reduced loading time and memory from 116 seconds and 994 MB down to 14 seconds and 260 MB. That’s a pretty good improvement in both time and memory.

But there is a tradeoff involved. The entire Everything Report is no longer resident in memory, so operations that need to go through the whole document need to have it generated.

That problem manifests itself in my last 3 tasks before Beta: Searching, Printing and Exporting. In each case, you may need to go through the whole file to get your result.

In the last alpha version, after waiting the 116 seconds for the file to load, if I searched for a text string that wasn’t in the file, it took less than 2 seconds to come back with “string not found”. In the new version, it takes about 10 seconds.

This is only a problem with very large files such as this one. It is something that I can optimize and make somewhat faster. It will never be as fast as searching though a pre-generated in-memory file. And the larger the file gets, the longer it will take. I may be able to get it to work fast enough for this size file, but one 3 times bigger will still take 3 times longer.

So for now, this will be the tradeoff. A few operations that will take longer than before in return for much faster loading and a big reduction in memory use.

The result is I will have to change the cursor to indicate when Behold is busy with these operations. For now, that will have to do.

But I found one other major glitch caused by the virtualization. There was my command that hid or displayed the tags you didn’t want to see, as well as the one that hid or displayed reference descriptions. They both still worked, hiding and displaying. But everytime this was done, the viewport would jump to some other place. Because of my virtualization, Richview was no longer there to maintain the location for me, so I had to program that in myself. It only took a couple of days, but was necessary.

Today I had the utmost pleasure in going to a lecture at the University of Manitoba by Jeffrey Rosenthal, a visiting professor of probability from Toronto. He is the author of the best seller: “Struck by Lightning: The Curious World of Probabilities” which I purchased and had signed after the lecture and am looking forward to read.

Jeffrey was extremely interesting and entertaining. He mentioned along the way the Monty Hall problem, you know: three doors, you pick one, they open another. Do you want to switch? I asked Jeffrey afterwards if he had heard about the Monty Fall, Monty Crawl and Monty Small adaptations. Well, to my surprise, he was the one who wrote that.

I had two questions I was going to ask him. The first was about coincidences. When we went to New York, there was a fellow I went to school with as a kid, who now lived in Toronto, who happened to be at the same Broadway Show as us in the same row! What are the chances?

The second question was on improbable events happening in sequence. We had a 15 inch rainfall one day in the Summer about 10 years ago. They said it was a 1 in 400 year event. We had the same 1 in 400 year rainfall 2 weeks later. Then one week later, slightly less but a 1 in 100 year rainfall. What are the chances?

So as Jeffrey was finishing up his talk, he ended with two last points. One about coincidences, and one about clumping of random events. He answered both my questions with his last two points before I could ask them. What are the chances of that?

Behold will hide all the tags you’re not interested in. The Show/Hide button will easily make them appear or disappear as you desire. When they are shown, they are shown in a light grey color so that you’ll know you normally want to hide these.

When you search and the hidden tags are shown, you want to search through them. But when they aren’t shown, you don’t want to search in them.

I had to go into the bowels of TRichView to find this and add a few kludges. It took almost all day today. I have one tiny bug left relating to highlighting that found item (it does it correctly 99.9% of the time) and this should be done.

But the test case I built up was somewhat amusing. I needed some text that I knew would alternate between hidden and displayed. Then I thought of the GEDCOM tag: “SEX” which is used to display whether the person is Male or Female. Normally that is obvious, and by default it is turned off so that it doesn’t add an extra line that you usually don’t want to every person in the Everything Report.

The simplest set of commands I could generate to obtain the error I was encountering was to open my test file with hidden items being displayed. Search for “sex” three times. Click the icon to hide the hidden tags. Search for “sex” three more times. Change the search direction to “up” and search once more for “sex”. Voila - Behold crashed.

The bottom line is that I must have searched for “sex” two thousand times today. And most of the time I found it! Sometimes I bombed out.

When discussing the humor of this with my family, my younger daughter said to me that maybe I should call it “gender” instead of “sex”. After all (as most comedians would say), the answer to give to “Sex?” is not male or female, but “yes”.

The official GEDCOM term for the tag is SEX. But there’s no reason why I couldn’t set the default text that is displayed for that tag to “Gender” instead. I believe the two terms are interchangable. Searching for “gender” doesn’t sound as dirty.

I hope this post doesn’t blacklist my site for all the “sex” talk in it.

I had built a lot of functionality into those forward/back keys over the years of the alpha. But they relied on checkpoints that were in the RichView document. Of course, since I’ve virtualized the document, those checkpoints no longer exist and I’ve had to redo all the coding relating to it.

I spent almost a whole week trying to debug why the menu items weren’t being transferred correctly between the back and forward menus, and that was the toughest task of this process. Other than that, it was just fixing each of 100 details up, one by one.

So now three major things still to reimplement to be able to release the Beta. That means I’ll have to do one a week and have a week to do the pre-release stuff if I want to get it out in November. That’s an ambitious task, but let’s see how it goes.

This is actually an initiative of Volkswagen. They’ve been very innovative in their advertising on TV lately (great commercials!) and this initiative may be an attempt for them to raise their profile on the Internet and get the younger generation interested in their cars again.

None-the-less, the idea is that people will do things more willingly if you make it fun for them to do it.

Entering your genealogy data into a genealogy program is usually NOT a fun thing to do. It is more often something you are compelled to do only because you, as a genealogist, want that data recorded. And despite the drudgery, you continue on to do it.

Maybe it should be like a pinball game, with the bings and the bongs and a score continuously updating as you add more people and events and dates and sources into the deep dark cavern of your genealogy tool … or maybe not.

But if nothing else, genealogy software should attempt to make your data entry less onerous, less un-fun, and less of a thing that you feel you have to do in spite of the program’s attempts to make it very difficult.

I know I’ve got boxes of data waiting to be entered. I cringe at the thought of entering it all into a forms-based entry system. I couldn’t think of a less desireable thing to do with several hundred (if not thousand) hours of my time. That is very un-fun.

What I want is a program that could allow me to enter my data very efficiently, attach my sources easily, and see the data in context as I enter it to allow me to easily come up with those revelations that make this hobby so enjoyable.

Will that be fun? Maybe?

Behold

I’ve been thinking about trying again for Windows 7. But I encountered a very eye-opening post about it at Successful Software, a blog by Andy Brice that I follow.

Andy writes why he won’t be bothering with the Windows 7 logo program. He did get certified for Windows Vista, but said when I asked, that he did not feel it was of any benefit at all. The bottom line is I found out that I wasn’t the only one finding that process difficult.

Scott K. says it well in the comments to the post: “I really can’t imagine any customer caring if you have some “Compatible with Windows 7″ logo. If you tell them it works (without the logo) I think they’ll believe you.”

What I will do is ensure Behold works and works well under all of XP, Vista and 7. And it will be code-signed guaranteeing you authenticity. I think those are what’s most important.

But until Microsoft makes the obstacle course a little less unattainable, then it’s not worth the setback in development time to get one more certification.

—

An aside: If you haven’t seen it, check out the Revenge of the GEDCOM monster merchandise for Halloween. Very cute.

First, when you right-click, the cursor needs to detect where you are in the document. This used to be done by what were called Checkpoints in RichView. I had previously set Checkpoints up throughout the document. But now the entire document is not loaded at once, and I needed to keep track of locations in the program but outside of RichView. So it was a matter of hooking up to these new locations, rather than Checkpoints.

Second was the Instant Organize history that is saved. This used to save the IDs and names of the individuals for each of the last 10 IO’s that was performed. The new locations were now both individuals and families, so changes were required here.

Instant Organize is one of the very unique functions of Behold. I don’t know of any other genealogy programs that can allow you in one click to quickly create all the family tree and all information about and ONLY about an individual or a couple.

This only took a few days to do. Next number two.

I’ve taken long enough, and it’s now time to take the expressway and make it happen. Beta in November. Version 1.0 in April, and Version 2.0 (and Editing!!) by the end of December next year.

Let’s do it!

Herb’s comment on my last blog entry made me rethink my Beta release plans. I was going to release it missing some functionality that was in the Alpha. But it would, as Herb says, appear “crippled”.

So I’m now going to add the back/forward, find, print/preview, and export to rtf/html back in before going Beta. I’ve updated my future plans to suggest a release date of November for the beta. Let’s see now if I can do that.

It has to do with the “virtual” rendering of the RichView window. I have perfected in getting it all to quickly render, scroll down, scroll up and resize. It is fast and works great. Except that every so often, there is one scroll up operation that causes a program abend and therefore a crash that happens in TRichView. I have isolated some precise examples that cause it to happen all the time, and I’ve traced it down to the exact lines in TRichView where the abend occurs … and it just doesn’t make sense.

In brief, prior to rendering the current view, I call a RichView routine to get the cursor location. I save that. I move the view, and I attempt to restore the moved location with another call to a RichView routine. I’ve checked the code 1000 times and there is nothing wrong with it. But the RichView routine crashes when it tries to restore some unneeded location internally with an index that is out of bounds.

I posted a question to Sergey on RichView’s user forum and Sergey responded that if the document is formatted, that should never happen. After sending Sergey the code, he agrees that my formatting has been done correctly, and he cannot help further unless I can isolate an example.

I’m still in debug mode, and my new tact is to compare step by step the difference in a PageUp that works and one that doesn’t. I’ve had a half a dozen surprises so far where I thought I found the answer, but it ended up not being it.

It is just a matter of time before I get the answer and fix it, or do something instead to get around it. But I am frustrated by the delay. So I’ve decided that as soon as this is fixed, I’ll then just ready the program for Beta and release it. It needs to be out there. The “Trial Download Temporarily Unavailable” must go.

So it will be missing a few additions I hoped to get in for the Beta. And it will be missing some things that were in the alpha, specifically Find, Print, Print Preview and Export. We’ll do those first and get full functionality back and it will be glorious.

Now back to that insect.

Hopefully the former.

After I removed the second screen and started up Behold, it showed up in the taskbar, but not on the screen. Ooops. Behold positions itself on the screen where you last had it. I last had it on my second monitor. No monitor, no Behold.

I didn’t like this, and I don’t think anyone will. So I made a quick change to ensure that Behold will always ensure it is visible on your screen when it starts up.

It took me four hours today to track that down. It was strange. It had been working flawlessly for almost a year. I didn’t remember doing anything to cause it, I couldn’t find any comments of other people having similar trouble with Hashcash. Working through it, I did find that the problem was both on the live site and on the copy on my computer. I worked through the code bit by bit, adding debugging statements and exits to try to track it down. I went through the settings and they seemed okay.

Finally I got lucky, and saw something I had missed the 3 other times I looked at the Hashcash plugin webpage. It said “WP Hashcash relies on the presence of two hooks in your theme, wp_head and comment_form.” I took a look at the GenSoftReviews header page and I noticed a change date of August 5th. There was no “wp_head” call in that header. I added it back in, and everything worked fine again.

I can’t for the life of me remember making that change back on August 5th, but obviously I must have, since it was the same both locally and online. But I’m really glad I got this working again. GenSoftReviews would be too onerous to operate without it. I turned it off overnight, and the site got 6 spam comments. Yuck. Those are what Hashcash gets rid of so perfectly. It required that the user has Javascript on and does a javascript test that only real browsers with real users using them can pass. The only spam that can get through is spam put there by someone typing it from their browser. That is a slow and non-productive way to spam (one blog at a time) so it is rarely done. It’s really a great test, and the only downside is that it requires javascript, which only the paranoid people turn off.

So that’s now fixed, with an apology at the top of the GenSoftReview pages that I’ll leave up for a month or so. I hope people come back and start adding their reviews again.

On a different note, earlier this week we bought a new Vista computer for my daughter to replace her now underpowered and slowing-to-a-crawl XP computer that was several years old. We attempted to use “Windows Easy Transfer” to copy everything and the settings over. I won’t get into that horror story. Just search: “Windows Easy Transfer” Problems on Google and look through the 1,550,000 results.

Re the Behold Beta, a few niggling things that need to be done before it’s out.

I’ve always been intrigued by tall buildings. My neck still hurts from looking up all the time. My city of Winnipeg has 700,000 people, so it’s not small. Our tallest buildings are about 30 stories and 400 feet tall. But the Doubletree Hotel at Times Square where we were staying was 43 stories and 470 feet tall - and it is listed as the 180′th tallest building in New York!

The cabs are something else. All yellow and 80% of the traffic. A $12 cab ride is like an amusement park ride reaching speeds of 60 mph and g-forces of a jet airplane. Every block has a Starbucks and a McDonalds.

Museums, Shopping, Empire State Building, Statue of Liberty, Central Park, Broadway Shows, Wow!

We had a great time.

I have a 24 MB GEDCOM with 100,000 people and a million lines that I like to use for my benchmark tests. Here’s what happened:

In the last alpha version, it took 116 seconds to load and ate up 994 MB of RAM.

The upcoming beta version takes 15 seconds to load and 230 MB of RAM.

That’s an 87% improvement in speed and 77% improvement in memory use.

In addition, certain operations that were slow before, such as resizing the window that took 25 seconds to complete, now happen instantly.

By comparison, the fastest program alive: GenViewer Lite takes just under a second to load this file, and uses 55 MB of RAM. These values will be hard for Behold to ever reach, because GenViewer is only a viewer, whereas Behold is loading the data so it can be edited.

But the current set of improvements should make Behold among the fastest programs, and eliminate the major performance problems for large files that Behold’s alpha has had. I have to thank Tamura Jones for persistently nagging me about this. I apologise that it has taken so long. I hope you’ll be pleased with the performance of Behold when you try the beta.

And it will rate right up there with the top programs speed-wise,* loading at 6500 INDI’s per second on my computer. Comparing the GenViewer loading speed that Tamura reports, it seems that Tamura’s testing computer was only 40% as fast as mine, so Behold should place at about 2600 INDI’s per second on Tamura’s analysis.

I know there are some major data structure and processing inefficiencies that are still with Behold. I did testing back in November and found I was able to get Behold to load data in times approaching that of GenViewer. To retain all of Behold’s functionality, I won’t be able to get all those gains. But in time, I expect that I’ll be able to get an additional 3 times speedup (about the speed of Legacy Charting) with half the memory use (about 5 KB per INDI).

This time, Tamura, I hope I won’t disappoint* this time.

* Note: Internet Explorer cannot view Tamura’s xhtml pages. Try a different browser.

I switched tasks and finished reimplementing the ANSEL and UTF-8 input so that those character sets will be read in automatically and converted to Unicode. It is so nice to input files and see the Russian with the English with no need anymore to select the proper character set. Thank you Unicode. I’m glad I ended up doing this earlier rather than later.

And as I test these things, I am cleaning the little loose ends that I find as I encounter whatever doesn’t work exactly right.

In order to hasten a beginning, I’ve moved the Find, Print and Export functions to be after the Beta is released. This is functionality that the last alpha had, but I’d sooner get the beta out and then let those be the first things that get added.

So I’ve only got the Back/Forward commands as the last major thing left before the beta. I’ll then check the program over with a number of different test files to ensure that there’s no important problems to fix, and then it’ll be out.

So I’m really relieved that I’ve got through this. Looking back in my blog, I see I started this back in mid January, so it’s taken over six months to get it working right. As always I never expected it to take this long, but I was advancing into unmarked territory. Several times along the way I wondered if I should continue, but the speed improvement in loading time that was even apparent in small test files implored me of the necessity to go on and complete this.

Next step is to reimplement the level 0 structure based GEDCOM and optimize the lookups with hash functions that I did late last year. But on January 12th, I had to step back because Behold itself had stopped working due to the changes.

This won’t be a problem. I have saved snapshots of all my major changes. So I’ll be able to add this easily again, while making sure that Behold continues working all along. I’ve found that when I’ve had to program something a 2nd time, I always end up with better code (because of the 2nd look) in half the time it originally took. Noone wants to program something a second time, but it’s never that bad when you have to.

Hmm. If I could program something the 2nd time all the time, then I should be able to get everything done in half the time, shouldn’t I?

In the article, Dick talks about the ways to convert data from one program to another (primarily through GEDCOM), and then talks about what genealogy programs will in business and supported years from now.

My take on this is that most of the current lot of desktop genealogy programs are pretty well the same in using forms-based input and mostly inadequate reports. So maybe you need a program that:

1. Can read everything your old program exported to GEDCOM.
2. Can record everything you wanted to record
3. Can export everything you recorded to GEDCOM for your next program or for someone else’s program.

If the program could also display your data in a useable way, that would be a bonus.

Having the security that your data will not be lost is truly the most important thing. So do this test right now: In your current program, export your data to GEDCOM. Now start a new database and import that GEDCOM. Now export it again. Compare the two GEDCOMs. If they are the same, then at least your program can read everything from its own GEDCOM. That will not guarantee that it is writing all your data to the GEDCOM file, but at least it will tell you if that author of the program is putting some thought into his GEDCOM input and output.

This gets exponentially more difficult for programs when you start talking about Citations and Sources. See what Mark Tucker at Think Genealogy says about this as he compares how Family Tree Maker 2009, Legacy 7 and RootsMagic 4 export their source data.

If you go to an online database to be your primary record, make absolutely sure that it will allow you to dump all your data out again. If it doesn’t have a GEDCOM export, you could be in trouble. An export function promised in the future means you should wait until the future before you consider them. And export your data to GEDCOM and save it on your computer at least once a month. You never know when the service might just close up shop and your data goes bye-bye.

Of course, if you really want your data preserved, don’t forget to manage your paper records well. All those boxes of material you painstakingly collected for many years have been summarized into names and dates in a computer file. The names and dates can be recreated from your records if absolutely necessary. But the names and dates will never replace your records should they get lost.

Hmmm. Forget everything else. Get a digital camera/scanner and go to it.