She got home the next day and I immediately worked with her to see what we could do to get the pictures back. We plugged the 2 GB SD Card into a USB card reader and … Vista couldn’t recognize the card. Windows Explorer froze when trying to identify the drive and only unfroze when the card was removed. We tried a number of utilities, including zar, disc doctor, and photorec. The best they could do was recognize that a 2 GB disk existed, but none of them could do anything with it. We also tried a USB image tool to try to copy it, but that could not get anywhere because of the freezing problem.

Researching on the Internet, I found out that memory cards have a controller on them, and if that fails, then a computer cannot recognize the card. That does not mean the data itself is missing so we still had hope.

There was no way I could do the next step, so I had to call on the “professionals”. We searched for “sd error recovery canada” on Google and that led us to ReStoring Data, a Vancouver company. I filled out their Free Quote Form and got a response by email within the hour. My eyes opened wide as I read that it would be $250 if the failure is logical, and $400 to $600 if the failure is physical. But there would be no charge if they cannot retrieve the pictures.

Now lets put this in perspective and find out how much memories are worth.

Coming back from a trip that cost several thousand dollars and were full of wonderful memories, only to horrifyingly lose a chunk of them on the last day, will not be something you’ll want to have weighing you down for the rest of your life. Paying even $600 to retrieve them – what is that the equivalent of – One airline flight? Three days hotel? A new computer you’d have to buy if yours blew up? Fifteen years ago, it would cost $10 for the film cost and development of a roll of 24 pictures. For 500 pictures, that would have cost over $200 then that we now get for free with digital cameras. No, even $600 is not too much to pay to regain these memories.

Of course these companies got you. They know how valuable your memories are and they make you pay for them – sigh. But I don’t discredit them for that. They provide a service that right now we deeply want. I’m just glad they don’t want $2,000, or $20,000.

Today I dropped the SD card off at their pick up location in Winnipeg. The “forensic” experts in Vancouver will look at it next week and we’ll get the diagnosis. We’ve got our fingers crossed for good luck.

Next on tap will be getting those darned recursive links finally implemented properly. I’ve finally figured out exactly how to do it, and it should only take a few days. Then theres a few niggly bugs to fix and then some of the last user interface improvements. That will be followed by a complete run-through of all my test GEDCOM files to make sure that Behold displays each file correctly. Then the Help file gets updated. Then Version 1 release. I’m excited.

This beta took a couple of weeks longer than expected. I got stuck at the end trying to implement the recursive links (links more than 1 level deep). Few GEDCOMs have these but it will be very important for Behold in the future. So I decided to release this beta with only level 1 links working and I’ll work to complete this first for the next beta.

There’s about 15 things left on the beta ToDo list. Last beta there were about 20. I did about 10 from the list, found 20 more to do, did 15 of them, and added 5 back onto the list. Thus I went from 20 down to 15. Well, at least that’s progress. I am getting closer.

Enjoy this version. There were a lot of internal changes this time. If you find anything not working that was before, please let me know and I’ll fix it ASAP. Same goes for anything else you find. Thanks for your help.

None-the-less, Thomas suggested that we genealogy bloggers share some information with the community about what we use in terms of technology to run our genealogy businesses or pursue our family history as a hobby. He provides a list of categories to follow.

So this one will be a little different. Because it will tell you what a genealogy software programmer uses. Here we go:

* Hardware: HP m9402f Desktop, AMD Phenom 9650 Quad-Core Processor 2.30 GHz, 7.00 GB RAM, 640 GB Hard Drive, NVidia GeForce 6150 SE Graphics card, Two LG L1970HR 19 inch monitors, Windows Vista 64-bit Home Premium with Service Pack 2.

* External storage: 2 HP Personal Media Drives, 250 GB and 750 GB

* Online storage: Don’t use it. After major changes, I copy my critical files that I don’t want to lose to a Pen Drive and leave them in a safe place far from my house.

* Backup: I use Beyond Compare 3 to copy my files to my External Storage. I do so about every month or so. I copy the directory with all my Behold work to another directory before I make a change that I may want to back out of.

* Firewall: Hardware firewall with router. Software firewall built into Windows Vista.

* Virus protection: Used to use AVG free for many years. Switched to Microsoft Security Essentials when it came out last fall. Love it. Don’t even know its there.

* Spyware: Microsoft Security Essentials.

* File cleaner: Don’t use one.

* Printer: HP OfficeJet Pro 8500.

* Phone: Land line. Cellphone begrudgingly. I find phone calls to be an interruption. Email is much nicer. You answer when convenient.

* Mobile media: Don’t use any.

* Music player: No.

* Car audio: I just listen to the latest songs on FM radio.

* eBook Reader: No. I still prefer to print out e-books and read them as paper copy.

* Browser: Internet Explorer 8 and I love it. I use Firefox only for xhtml pages, but I’m waiting for IE9 to come out which will read xhtml and then I won’t need Firefox anymore.

* Blog: Wordpress with my own custom modifications. I use bbPress for the forum and have integrated it into Wordpress.

* RSS: Internet Explorer 8. I love that the RSS feeds are integrated into IE on the favorites tab. I have about 100 feeds I keep up with and spend almost an hour a day doing so. IE does this very well and efficiently for me.

* FTP: I used to use WS_FTP. Now I use Beyond Compare 3. I compare my website on my computer with the site online and it will FTP all the changed files in one step.

* Text editor: An older version of UltraEdit. The new version has got too bloaty.

* Graphics: Years ago I used Paint Shop Pro. I got Photoshop Elements after that and hated it. Now I use Windows Live Photo Gallery. I sometimes simply use Paint.

* Screen capture: Used to use SnagIt. But with Vista, I found that the free Windows Snipping Tool does almost everything I need (except printing).

* Social media: That’s my wife and daughters’ domain. They keep me up to date since I’m too busy. But you can find me on StackOverflow where programmers get together to ask and answer programming questions.

* Social bookmarking: I don’t do it. From time to time, I’ll mention interesting sites in this blog.

* Social profile: lkessler at StackOverflow.

* URL shortener: Don’t use one.

* Office suite: Microsoft Office 2010

* E-mail: Windows Mail

* Calendar: No.

* Accounting: Used to use Quicken but quit it quite a while ago when I realized the amount of time I spent recording my expenses was 100 times as much as the amount I time I got info out of Quicken. Besides, my family and I spend everything we make, so once you know that, what else is there to know?

* PDF generator: Don’t use one.

* Genealogy database: Started with Reunion for Windows. Continued using it when it was bought by Sierra and changed its name to Generations. I was a beta tester for an extensive update of Generations. But when Sierra sold it to Ancestry, it was dropped. It no longer is compatible with Vista, and I haven’t updated my genealogy database in about 10 years. I’m awaiting my finishing Behold version 2.0 and then that will become my genealogy software.

* Genealogy tools: Behold. I use it whenever I have to look up my family information, or want to send out a family tree (just their relations) to a relative.

* Other tech stuff: Have used digital cameras for 10 years - a must for genealogists. Use it not only for pics, but as a “copier” in archives, or a “voice recorder” for interviews. Currently use a Canon SD600 which I carry in my pocket wherever I go. But I want to get a new camera with built-in GPS so the exact location (latitude and longtitude) can be written into the picture data when its taken. I’ll probably get one before the next trip I take somewhere.

Originally, my main task was to handle recursive links properly. I found I had to make some database changes to do this. That led to database improvements and a reworking of the internals. Then it caused me to look at the treeview, and all of a sudden I found ways to enable larger programs to be loaded. The new version shall be able to load up to about a 400 MB GEDCOM with close to a million people in it. It does so in about 70 seconds on my machine.

There are numerous changes and improvements coming. See Behold’s Future Plans page for the details. The new version beta 11 should be available in the next few days.

I’ve decided that there should be a few small useful things that need to be added before version 1. They won’t take too long, maybe a couple of weeks, but they will be things everyone can use. See the Future Plans page again for what will be in beta 12.

Then it’s ensuring Behold is bug free and documenting everything and out with Version 1.0. Don’t know if I can still squeeze that in before the end of July, but I’ll try for it.

So when the fine people who created StackOverflow offered to create other Q&A forums with the same technology and addictability, I couldn’t resist and had to propose a site for Genealogy Software and the Internet. But they won’t set it up until the proposal gathers enough interest. Several hundred people will have to get involved before it would go live.

The only way for it to get going is if (1) people are interested, and (2) people pass it on. I sent the message to a few people who may publicize it, but we’ll have to wait and see if it draws enough critical mass. There is another proposal for a general Genealogy Q&A site proposed by Dean of Genlighten, but I wanted something a bit more specific.

If it doesn’t work out, I won’t worry about it. I do need to concentrate on Behold.

Meanwhile with Behold, I’ve been quiet on this blog for the past two weeks because I got stuck with a problem in handling recursive links. I think I’ve solved that now and should implement the solution in the next few days and soonafter issue another beta release.

But it’s even more than that. I truly believe genealogy is something every single person should — no, must do.

My belief of this has been solidified 100% by the tremendous book I have been reading. It is Douglas Hofstadter’s latest book (2007): “I am a Strange Loop".

Hofstadter is a Pulitzer Prize winner for his amazing 1979 “Gödel, Escher, Bach: An Eternal Golden Braid” that tied together the Mind, Computers and Mathematics into a marvellous musical synergy.

But A Strange Loop goes further, defining what a “self”, a “soul”, a “consciousness” and an “I” really are. No matter what your religious beliefs, and whether you believe in an afterlife or not (Hofstadter doesn’t), the book makes one undeniable statement:

“In the wake of a human being’s death, what survives is a set of afterglows, some brighter and some dimmer, in the collective brains of all those who were dearest to them. And when those people in turn pass on, the afterglow becomes extremely faint. And when that outer layer in time passes into oblivion, then the afterglow is feebler still, and after a while there is nothing left.”

Do you want memories of your parents to fade away into nothingness? Would your parents want their parents to be remembered? How about your ancestors and relatives who are on their way to being forgotten?

It is up to you as a human being to preserve your memories of your loved ones. It is through preservation of the stories, pictures and memories in some organized form that you can ensure that their essence is preserved.

And that’s what’s so important to me about genealogy. It’s more than just collecting dates and places. I want to collect stories and memories – the things that will help take those names and faces and give you an understanding of how they lived, how they thought, and who they really were.

And it is your solemn duty to your loved ones, that you do the same.

I’m hoping to send out something once a month or so. The content will depend on what is relevant at the time, but I’ll try to make it information that is interesting and useful to you. Meanwhile, for the day to day stuff, this blog is the place to come.

I should say that after the mailings, about 10% of the emails bounced back to me. Most of these are due to invalid email addresses. If you purchased Behold, or sent for a Trial key, then you should have received a newsletter. If you didn’t, please contact me and give me your new email address so you can get future mailings.

I also finally sent a mailing out to all those people who filled in my Behold/GenealogySoftware survey that I had on my site from 1997 to 2004 prior to the release of the alpha of Behold. Over half of those addresses were bounces, but even so, I have now fulfilled my promise to let those who were interested in Behold know that the Beta is now available.

The Newsletter system works quite well. I should be okay for awhile, or at least until I’m mailing hundreds of thousands of people each month.

All outstanding problems I know of are fixed, and this beta version is the first that I can say that it finally produces as good (if not better) an Everything Report as did the last alpha version, 0.98.99 from 07 Oct 2008. It’s taken over a year and a half. The virtualization of the Everything Report was tough to implement, and changed a lot internally.

But the completion of the alpha also mean’t that functionally, Behold was ready. It was just flawed because it was too big and too fat, loading the entire report into memory. Where the alpha took almost two minutes and 1.35 GB of RAM to load a 100,000 person 25 MB GEDCOM file, the current beta takes under 8 seconds and uses 180 MB of RAM.

So the beta has been an attempt to now deliver that functionality in a nicely packaged well-performing program with a standard user interface that looks and feels more like a word processor than a genealogy program.

Now onwards towards the version 1.0 release.

Of course I am working to fix those, but I also realize that a 100% perfect program is impossible. Microsoft itself doesn’t strive for perfect, but only tries to be “good enough”. That model is not a bad one. Yes, it does frustrate a lot of people who enjoy the hobby of complaining all the time about the glitches they run into. That can’t be helped. The bigger and more complex the software is, the more glitches it will have.

But at this point I am getting happy again the way I was at the end of the alpha phase. More and more of the bugs reported and that I’m finding are turning out to be fairly obscure. This means that it is unlikely that two people with different GEDCOMs will run into the same bug. This is unlike the bad bug I added in the previous to the last beta which broke the source references for everyone.

I’ve got a few more user-reported problems to fix and I’ll issue another beta release. Then I’ll work to get the error checking and log file to be rock solid, followed by a rigorous checking of all my test datasets and corrections of any obscure bugs that show up from them. Then its the help file and version 1.0. If I can do that, that will be “good enough”.

The complete list of changes in this version is on the Behold History page.

Interestingly, the “best” version of Behold as far as how well the report came out was the last Alpha version 0.98.99. The work to virtualize the Everything Report really changed the way Behold’s internals work and I still haven’t quite finished replacing everything that this alpha did properly. So next on the agenda is to simply get the new version displaying all my test GEDCOMs as well as that last alpha did. This is something I knew I’d have to do sometime before version 1.0 was released. The good news is that now I’m finally at that point where I figure its time to do it.

But it’s not a good beta because I added a bad bug. It seems that all the Source references don’t display correctly any more. I hadn’t noticed that before I released it. I was shocked when I encountered it yesterday.

I found the cause. It was one single statement that was outside an “ELSE” statement rather than in it. It was hard to find but very simple to fix.

I’ll update a new beta in a few days. It will have this fix and several others that are fixes of bugs that were reported to me.

I’d put this fix up right away, but I want one or two days to see if I can first solve a special problem. Tamura Jones was looking at the newest beta and liked the new log file. But Behold failed to open one of Tamura’s GEDCOMs. I haven’t yet been able to exactly replicate that problem and I’ve tried over 50 of my GEDCOM collection. I’ll try more to hopefully make that failure happen and eliminate it. Tamura’s GEDCOM is a large one and seems to work up to a point and then Behold starts filling the logfile up with incorrect messages. I’m guessing that it may be at the 4 MB buffering point and the current record location is not being transferred properly. I’m sure when it comes down to it, the fix will be changing just one or two lines in the program.

Times like this, I’m glad Behold is still in beta. Even though that’s an excuse, I really hate to cause problems for people trying to use Behold. I’ll get this fixed and a new release out ASAP.

This one fixes up a lot of niggly things, and I think the the virtualization of the Everything Report is finally rock-solid. I can’t seem to break it any more. Just about every abend I’ve been able to find has been eliminated. And the abend error display no longer attempts to email a message, but instead will connect you to Behold’s Feedback page with instructions.

Of significance, I’ve segmented the internal structure to separate the data loading from the organizing. Now if you only do some sort of reorganize function, the data is not reloaded. I’ve eliminated memory leaks and have again reduced memory use, although it’s still about 7 times the GEDCOM size for large files. That will go down significantly once I implement saving and create an on-disk database file format for Behold to save its information.

Then I’ve put back in a new and improved version the log file that I once had many versions ago. So now Behold reports data problems and incorrect GEDCOM syntax into a log file named the same as the .ged file but with a .log added on the end. There is a “log file” icon on the toolbar, which will open the log files for you. Behold actually does quite comprehensive checks on the family connections and it’s log file information can help you fix data problems that your program allowed you to add. It even detects people-loops (e.g. I am my own grandpa) and will include in the log file, a very clear message showing you the problematic links so you can fix it. Behold doesn’t do logic checks on dates yet, but that will come.

A complete list of changes this version is available on the Behold Version History page.

This last set of changes took longer than I expected. I’m now scheduling Version 1.0 to be out in July. I’m leaving the price at $20 for the rest of the beta, but will raise it to $40 when Version 1.0 is out.

Also there’s the latest addition: elimination of the “Incorrectly Linked Families” section with a simple message in the log file instead that now correctly identifies the people who have themself as their own ancestor and listing every person in the chain. I am really excited about that last improvement. It is finally something useful that can be used to identify and fix incorrectly linked people in your GEDCOM.”The Incorrectly Linked Families” was supposed to do this, but it didn’t cut the mustard as it was too difficult to follow the chain from that.

But I’ve got a list of about 10 bugs that I created during the course of these changes. I’m working through them and will release this version as soon as those are fixed. I really don’t want to issue a release that introduces new problems if I already know about them and can fix them first.

So just a few more days …

It went back again to my 25 MB test GEDCOM with 94,000 people in it. That file now loads in only 8 seconds but still uses 226 MB of memory. That’s 9 times the file size.

So I did a bit of tracing and got a surprising result. Loading the data only uses 138 MB. Since Behold uses Unicode characters, the 25 MB input file immediately gets doubled in size to 50 MB. Add on the data structures and all the derived data, links, pointers and indexes for them and that gives the 138 MB.

Then where are the other 88 MB coming from? I was very surprised that it is from the TreeView. 35 MB were from the text and link in each of the 160,000 lines of the TreeView. But the other 53 MB were from the data structure of those 160,000 nodes in the Treeview. That means each node was using about 330 bytes of memory. That’s a bit much, compared to Virtual TreeView which says it only uses 60 bytes per node.

Is this really important? For files up to about 600,000 people, everything can still be stored in under 2 GB and most computers have that and more. But Behold can’t go much higher, since 32 bit Windows programs can only address 2 GB (or 3 GB with a special trick). And extra memory allocations do take time as well.

I still do want Behold to be able to handle super-large files and be a potential winner of the Confucious Cup. So although this isn’t really important now, I’ve got a few future solutions available to reduce the memory hogging burden that Behold now runs into for large files:

1. Use an online database to store the GEDCOM data. This will be done in Version 1.1
2. Replace or optimize the TreeView to use only what it needs to use. One day I’ll do this, but not important now.
3. Make a 64-bit version of Behold to eliminate the 3 GB limit. Can’t do this yet. Embarcadero hasn’t finished their 64-bit Delphi compiler. They are working on it (mid-2010?).

Again I ask the question: Is this really important? For most of you with files that I now consider “small” for Behold (say up to 100,000 people), then no. Behold will be fast and 250 MB of memory is really unnoticeable on computers today.

But for big files … I’ll get there.

—

Addenum: I reported the excess memory use by ElXTree to LMDInnovative. They’ve added it to their bug tracker and may have some improvements for me in a few months - without me necessarily needing to hack the tree routine for myself.

Then I noticed the Instant Organize was not working properly in all cases. This was a bug that got in there when I changed to the virtual Everything Report at the onset of the beta. It even could cause a program crash in certain cases. That is now fixed. Instant Organize is one of the star features of Behold, and I apologize to those of you who tried it and got less than satisfactory results. Unfortunately nobody reported it, or I would have found it and fixed it earlier, but it was a rather subtle problem in most cases, and you would have had to be quite familiar with how it worked to notice it.

It’s been almost two months since the last beta. I’m a bit hesitant to release this beta right away. The splitting of the processing into two parts was major surgery, and I want to do some more testing first to ensure that this version will be at least as stable as the last one. So hopefully in a week or so, I’ll get the next beta out.

The next beta release will include the log file, elimination of memory leaks, and a new location for the sample files in the user’s AppData directory as Windows Vista and 7 desire and other things. So once that is out, then I’ll send out a newsletter.

The price of $20 was a special price for Behold during alpha phase. I was intending to raise it to $30 for the beta and then to $40 for version 1.0. But I wanted to send a newsletter out announcing “a month until the price increase”, but it took so long to get the newsletter going that now the beta is getting too near to be done. So I’ll continue the special price of $20 for the remainder of the beta.

The real circle was the memory leaks. I planned not to do them yet, and they led me on a wild goose chase for awhile. I had to ask a question on StackOverflow, and I just couldn’t track down one of the leaks. It took until a few days ago, and then it all fell into place and I got rid of them all.

Partway through trying to figure out the leaks, there was something in my taglists leaking. My tags were based on a hash table system. I thought if I could temporarily replace the hash tables, then I could see if the leak was related to my use of them. So up came Generics that were just implemented in Delphi 2009. I created a TagDictionary Generic to replace the tag hash table. I spent a week learning about Generics and then another episode with StackOverflow to try to sort them.

Generics were neat. They simplified code. They were supposed to be fast using Hash tables, and for Behold I need fast. But then when profiling with AQTime, I was surprised to find they were over 3 times slower than the GPStringHash that I was already using. Since I also found out that GPStringHash was not the source of the memory leaks, I went back to GPStringHash.

So I went around a bit in circles, and am sort of where I was about 2 weeks ago, but a whole bunch of things have been done in the process. It will be hard to make my April goal for Version 1.0, but hopefully any delay won’t be too long.

When html e-mail is turned off, the person sees the e-mail as text only. No images will show. No formatting will be done. And the textual parts of the newsletters are usually translated by most e-mail programs into text that is formatted in a very ugly manner.

Well I don’t want my newsletters to look like that. The solution is to send both an html and a text version of the newsletter in the one e-mail. That is a standard way of making sure that both ways of viewing the e-mail will have it look good.

Of course, I don’t want to write two versions of each newsletter, so I need an automated way of pulling the text out of my html and formatting it nicely.

Unfortunately my Simple PHP Newsletter System did not include an automatic Html to Text conversion in it.

Fortunately, there was one in PHPlist.

Unfortunately, it did not produce results looking as nice as I wanted it to.

Fortunately, the code was not very long and I could probably customize it.

Unfortunately, it was using Regular Expressions which are a beast to work with and would take me a week to get good at.

Fortunately, there were other Html2Text routines on the web, and I found and implemented a pretty good one.

So now its up to 3 weeks for the Newsletter system. We’ll see if I can send it out by the weekend.

I cancelled Dewahost (they were nice about it - but I still wonder why they never responded to my queries) and signed up with LuxSci’s High Volume SMTP service.

Then I went about trying to set up some newsletter templates for phpList to work with it. I complained about the user interface of phpList a few blog posts ago. Now I needed to customize the inner workings of phpList to work with my list my way. Then I realized it wasn’t just the user interface of phpList that was terrible. The code itself was just as bad. There was no way I wanted to work with it. Sometimes, this is what happens to Open Source tools, and it’s often better to buy one that one person develops and has a stake in.

So a bit of research and I came up with Simple PHP Newsletter System (SPNS). It’s smaller and simpler and has code I can read that I can customize where needed.

I converted its database over to use my user list, and hooked it up to use LuxSci. It took about a week to get that working. Apparently I hadn’t fully registered the Behold bounce email address with LuxSci, but once that was done, test newsletters started flowing.

Now I had to redo a template for the look of the Newsletter. This would serve double-duty and also be the template for Behold’s help file. SPNS pointed me to a very nice template that I looked at and fell in love with. I purchased it. But when I tried it out, the background and graphics file was made up of big graphic files. I wanted to minimize those in my emails. And I didn’t have the graphic skills to convert my website header into something that would work with that design.

So I took that template and used parts of it and ended up with a cross between the old newsletter, the help file and the new website. It’s simple and clean and keeps my light blue and yellow pastel backgrounds - which add a bit of softness. I think that is better than grey text which does the same thing but is harder to read.

Finally I had to change my subscribe, purchase and unsubscribe scripts to work with the new system. Then I tested the newsletter system and see that the bounce handling works.

It seems close to ready to go. Next I’ll write up my newsletter and send it out and I’ll be off and communicating again.

Sorry, Dewahost, but this is not going to be a nice post for you.

When I purchased the service, I sent them a note with the purchase asking them for the SMTP setting so I could send out the Behold News newsletter. I waited a day with no response. I then sent an email to them and waited 3 more days. Nothing.

Being very patient, and assuming nothing, I submitted a support ticket on their support site. Someone there responded on the ticket only 9 minutes later that he’s forwarding my ticket to their sales team who will be able to assist me better. I then responded that it was not a sales issue, but a technical issue. Three days later. No response at all on that ticket. I then went to their online chat.

I had a pleasant talk and was told that they’d give my ticket a higher priority. That night, they responded to my ticket. They said: “We didn’t receive your order. What’s the order reference number?”.  Now I’m starting to get frustrated and I posted my whole Plimus receipt from what was already 10 days earlier.

Two more days went by and I went on chat again. The friendly fellow apologised and added a new ticket for me that was quite forceful and said that if they cannot handle this order, to let me know so I can look into other providers and a refund.

That was yesterday. Today I got no response on either ticket. So I’m now looking at other providers of a non-spam bulk email SMTP service. I found a few possibilities:

Authsmtp does not appear too bad.  They charge about 0.2 cents per address and that goes down to about 0.13 cents as the volume increases.

Sendblastersmtp looks good. They’re a little more starting at 0.3 cents (which is the same level that Dewahost starts at) and going down to 0.2 cents. But at the high end they have a dedicated server offer for $300 a month. They have bulk emailing software that looks good and might be useful for those who need it. But I expect I’ll be happier with phplist running from my ISP, rather than trying to maintain my list myself on my own PC.

Dewahost has an unexpected pricing scheme. It starts out high at 0.3 cents per address and decreases until for $40 a month, you have unlimited emails. *Alarm Bells* - The word unlimited never means unlimited. There’s got to be something that they’ll impose if you start using “too much”. Also, they do not have dedicated servers for this “unlimited” plan, so there is an uncertain future there when the list gets large.

Of all the companies, the one that now seems the best to me is LuxSci. They seem very professional. Their prices surprisingly start lower, at 0.1 cents per address but grow to 0.18 cents as the volume increases. But then they have a dedicated server offering at only $200 per month which I can move to once Behold News starts going out to 100,000 or more people. They also have a free bounce analysis that will promptly remove invalid email addresses from my list. I was planning on doing that with phplist, but it would be better to offload that where it won’t slow down the Behold website.

If I don’t hear back from DewaHost by tomorrow night (and maybe even if I do), I think I’ll sign up for service with LuxSci. Then hopefully I can restart the Behold News newsletter and send it out to everyone over the weekend.

In the meantime, I have been making good progress on the log files and probably will have another beta out in a couple of weeks.

Well, the beta of Behold is available, and I’m finalizing the Newsletter so I am just about ready to send out that letter. Just one thing to do: verify that those email addresses still exist. I do not want to send out 600 emails and have a lot of them bounce back. It could blacklist me and/or make my ISP very unhappy with me.

So then the journey to do that began. I found out that there is no perfect way to do so. Even sending an email may bounce back because an account is just suspended or a mailbox is full. Without sending an email, it is even trickier. The main method used that does the best job under the circumstances is to set up an SMTP call to the mailserver at the domain of the email address, and send it a HELO and RCPT command. You can sort of see how this works at a Free Email Address Verifier.

So I found a number of articles on how to do this and set up PHP code to do the verification. That took a couple of days. But when I ran it, it didn’t work. It took me another couple of days before I found out that the reason was my website host blocked port 25 where SMTP goes as a spam prevention policy. The research told me there’s no way around that. Find a host who doesn’t block it. So next I tried running it from PHP on my own machine. That would go through my ISP here in Winnipeg who gives me my internet access. Nope. Port 25 blocked here as well.

Without doing the SMTP checks, only 15 of the 685 addresses could be proven wrong. The rest were unknown. I knew there were a lot more than that. So I needed something with the SMTP checks. Downloading an email verification program didn’t help. They need port 25 as well.

Finally I was able to write a routine that accessed an online email checker and check the emails one by one. I was willing to pay for this service, but I could not find one that would do it in a batch online manner for me. So here goes, a one shot check of 685 addresses. Painful but necessary.

As I write this, I’m about half way through. Looks like about 45% of the email addresses from back then were good. That will make about 300 of them. Was all this effort worth it? Now that I look back, possibly not. But I didn’t know it at the time.

I have to wait to hear back from the SMTP mailing service I signed up for to know how to configure phplist to use them for my newsletter, so it will still be a few days before the newsletter is sent.

So tomorrow it’s back to work on Behold - let’s get that log file working.

I needed something more robust, and after the change to a PHP/MySQL backend on the new Behold website last year, the logical choice seemed to be the Open Source mailing list program: phplist.

Phplist seemed to have what I needed: The ability to send out a large number of mailings, batch sending and throttling, html/text newsletters and bounce management.

But gawd - it has the most horrendous user interface with no thought to managing the users or the mailing lists themselves. And to make things worse, my Netfirms account limits the capabilities of phpmyadmin so just getting my current user list into phplist took me 3 days of puzzle solving.

But its almost there. Hopefully I’ll be able to send out the first Newsletter in the next few days, and then get back to moving Behold closer to version 1.

So I was excited when Tamura Jones posted three GEDCOM torture tests along with how a good number of programs did against them. (Note: Internet Explorer can’t view Tamura’s site. Try a browser like Firefox or Google Chrome)

I was looking forward to Behold’s results. I was not happy when I saw Behold had only passed one of the challenges. I pride Behold in its ability to read all flavors of GEDCOM, and this indicated something needed to be fixed. Knowing that not a single program passed all the tests did not make me feel any better.

So as part of this version’s upgrades, I decided to include a few of the things on my “Tamura Jones” to do list - not the least of which was to easily read all the torture test files. That is now done. I hope Tamura will check it out and verify that I have actually done so.

I’ve made a number of improvements and fixes in this version. See the complete list on Behold’s Version History page.

I’m very pleased with my progress so far during the beta. I’m actually getting through all the bug fixes and changes almost as quickly as I had hoped I would. An April release date for Version 1.0 is still my goal.

So I figured I could reuse that routine and simply add a new option to write to the log file. The file information in the Everything Report was something that could then easily be sent also to the log file - so I thought.

It took about 2 hours to set up and then - yuk! It looked terrible.

The Everything Report, HTML and RTF are all nicely formatted and indented. Text for the search doesn’t matter since you don’t see it. But the log file is plain text with no formatting at all. The stuff was jammed together, not indented, had hugely long lines and basically was impossible to read. I spent another couple of hours trying my best to automate the formatting and make it look reasonable.

That wasn’t going anywhere. So I took it out, and in a half an hour, I added custom append to the logfile routine that now does it much nicer.

The sexier way is not always best, so it sometimes pays to take your lumps early and go on.

Basically, when Behold reads a GEDCOM, it may find problems with the GEDCOM file. I’ve now added the listing of the problems to the file information in the Everything Report. But what if Behold is interrupted or does not get to that point? How about more detailed information that isn’t appropriate for the file information (such as line references from the GEDCOM), or maybe just some warnings about the GEDCOM that you really don’t care about normally, but may want all the detail for.

Well this should all go into a log file that is created when a file is read. The fact that there is a log file at all will add to your security to know whether Behold is or isn’t detecting something.

I had two different interfaces for a log file. At first (way back in 2005) there was one I really liked. The top part had the GEDCOM files with a tabbed interface. The bottom was the log file. It was meant so that you click on an log message, and the line could be brought up in the pane above. But back then, loading all those GEDCOMs into TRichView windows was expensive in terms of performance. I didn’t have the Virtual TRichView back then - so I took it out and I put in a simple writing of the log to a file. That happened in version 0.98.8 alpha, 28 Jan 2007.

I then decided for version 0.98.9.5 alpha, 19 Nov 2007 to not print the log file altogether. This is the decision I am now reversing.

Putting it back in really isn’t too hard. All I have to do is compare the differences between version 0.98.9.5 and 0.98.9.6 to see what the log file code was. I started that two days ago, and it took about 15 minutes to get the code in again.

The harder part was my “AppendToLogFile” routine. It was written with Delphi 4 before Unicode and didn’t work with Unicode. I puzzled how to get that working and searched Google, but after two hours I wasn’t there. So I went back to my friends at StackOverflow and asked “How can I get this File Writing code to work with Unicode (Delphi)“. With that help, it’s now just two days later and the core code is working again.

There is no standard way to write a log file, so it’s up to me. Most genealogy programs do not do a good job with theirs and many don’t even produce one. Behold is already fairly decent at determining problems with GEDCOM files, so its log file should turn out to be quite useful when it’s needed.

So it’s really nice when someone else can write up what my blog is about even better than I can.

Dean Richardson at Genlighten did just that. In his January 22nd Follow Friday post, Dean writes about “Behold Genealogy and Louis Kessler’s Behold Blog”.

Dean writes:

I have a lot of admiration for Louis’ openness, persistence, and pride in his craft. You won’t find a lot of details of Louis’ genealogy research in his postings, but you can tell he’s channeling his love and respect for his ancestors into his software development efforts.

Well that makes me feel all warm and fuzzy inside. Thanks very much, Dean.

To read Dean’s entire post, pop over to the Genlighten Blog.

Not doing that will make it much easier to input and store the data and then later export it back out to GEDCOM. It will make the internal data structure of Behold a truly GEDCOM record-based structure. For some reason, some genealogy programmers think this is good. Personally, I see advantages and disadvantages to it, but the bottom line is “it’s convenient”.

It took a while (hey: a while is now just “a week”!!) to make these changes. Over a year ago, I had tried to implement them, but I bit off a bit more than I could chew, and the program was too complex to rebuild using that data structure so I had to abandon those improvements.

This time, I started with a working version and kept it working as I made the changes one by one. This method worked much better, and the knowledge of where I was going as well as having a prototype of the code already available made this second attempt a success. There are a few small things that I haven’t yet converted over, but those I’ll get done for the next beta. Most people won’t notice them.

Because of this major change, I do expect some bugs. The tests I have run are working, but I expect the real world will find others. You’ve all been great at uncovering new problems for me to fix. Really, do send me them. It’s the only way I can ensure that Behold becomes as bug-free as possible.

And that last version 0.99.5 did have some nasty problems that some people uncovered. I believe this version has fixed almost all of them. So I highly recommend upgrading to 0.99.6.

For a complete list of the changes in this version, see the Behold History page.

These system errors are called “exceptions”. The last alpha releases were very stable. I had over the years trapped or fixed every crash point I found. But the major changes between the last alpha and the first beta (virtualizing the Everything Report) have added some new ones. Also I’ve added an “exception” handler into Behold to catch not just the ones that Windows reports, but an even wider range of errors that previously went unreported or unnoticed. This is also allowing me to correct some code that I never knew was wrong, and that is good.

Then I also added the exception reporting mechanism, which brings up a custom screen and allows the user to send me the error message. At first it was quite onerous, but I think I’ve since fixed that and made it minimalistic and easier to use. The number of crashes I’ve been encountering have been going down, and so have the numbers users have reported to me (thank you all!).

Fairly soon (one or two beta releases from now), Behold should be back to being as stable as the alpha was. Once it’s there, I’ll finally feel confident enough in it to take the next step, send out the Behold News newsletter, and invite the world to try the beta (… and raise the price).

Yes there’s still lots of small things to fix, enhancements I want to make and unknown bugs left to find. I’m whittling away at those and will handle them rigorously when I start to redo the help file. We’re getting there.

I also had problems identifying why the recently used file list was not always getting updated. This was a component in the LMD ElPack that I was using for this. Rather than puzzling this through, I basically had the code without the component in the Find File and Forward/Back history lists. So I converted the recently used list to use my own code and that fixed the problem. It also changed a few minor appearance issues that now made that list more consistent with the other two. One minor byproduct of this change is if you upgrade to this version, you’ll lose your recently used file list in Behold, and will have to build it up again, which shouldn’t be too much of a bother.

All changes for this version are listed on Behold’s History page.

2 NOTE This is a note that is sp
3 CONC lit on two lines.

The note will be put together and the word “split” will be reformed as one word. That is the way it is clearly defined in GEDCOM. There will always be a word split in the middle, with the first part of it on the previous line and the rest of the word being completed on the CONC line.

Many programs follow that standard, but there are a lot of programs that missed that little nuance in GEDCOM and decided to implement their GEDCOM output in what would seem straightforward:

2 NOTE This is a note that is split
3 CONC on two lines.

Here the split occurs after the word. What this means for a program like Behold is that it now has to add a space before concatenating the two lines. In the previous example no space was added.

Unfortunately there is nothing in the GEDCOM file telling you which method the program used. I could write a procedure to scan the file and “guess” what it is or some other form of artificial intelligence. I might be able to make that 98% accurate, but never 100%. So something is needed to change this if the assumed method is wrong.

What I have done in Behold is made a list of programs that use the correct CONC that splits words and does not require a space added. I assume all others use the “bad” CONC that does not split words but requires the space.

My current list of programs that outputs CONC correctly to GEDCOM is not very long. It includes PAF, Brother’s Keeper, Legacy, The Master Genealogist (TMG), RootsMagic and only some versions of Family Tree Maker. I haven’t rigorously gone looking for them. If you know of any others, I’d be happy to find out about them and I’ll add them into Behold.

That’s all wonderful. But then I found out that The Master Genealogist added a user option to allow them to output CONC tags the incorrect way, presumably so that the file can be read by programs that don’t understand the “correct” way. So now, I can’t even rely on a specific program always doing it the same way. Boo to TMG for that.

What I did was on my Organize GEDCOMs page, I now have a CONC value for each GEDCOM. It specifies how many spaces to add after concatenating lines, either 0 or 1. It will be automatically be set to what I’ve assumed is its program’s default. But now you can change that value and save it in with your Behold file so it can be remembered. The next beta release will include this.

So if in your notes as displayed by Behold, you see spaces where they shouldn’t be or two words put together where they shouldn’t be, it may either be that the CONC setting needs changing, or maybe the GEDCOM you’re looking at has simply has tons of typos.

Unfortunately the web update feature of Behold in 0.99.1 through 0.99.3 beta of Behold was not working on Vista or Windows 7.

The web update downloads the behold-setup file to the Program Files directory where Behold is. Unfortunately, Vista and Windows 7 do not allow a program to add files to that directory, and that gives an error. To get around this, I now download that to the temporary files directory.

What this means is you may not be able to upgrade to the newest version of Behold with the web update. This now should be fixed in 0.99.4 but of course you won’t be able to try it until 0.99.5 comes out. For now, if Behold’s web update won’t work for you, then as I said in my last blog post, go to Behold’s download page and click the download link and run the setup from there.

There are a few other fixes in this version as well. See the version history.

Thank you for all your bug reports. Please keep them coming until I get rid of them all.

But first things first: If you try to do an auto-upgrade from the last beta, it may work, or you may get an “Error occurred” box, or it may update but when it tries to start up again you get an “Unable To Locate Component” error box. In the last two cases, please just go to the Behold download page, and click the download link and run the setup from there. Then everything should be okay. - No. I’m not happy about that either, but I’ll try to make sure it won’t occur again in the future. Unfortunately, this is what a beta is for - to get the kinks out.

None-the-less, this version should be better than the last. The hardest bug to fix was the scroll up and scroll down keys which would cause an exception everytime they reached a horizontal line with an ID number at the right. Those are tables in TRichview, and I wasn’t restoring the current selection correctly in my virtualization routine. I needed Sergey (the developer of TRichview)’s help, and just today it was solved. Then I thought I could get the new version out tonight, and I ran into the auto-upgrade problems when I tested it. But it’s too late tonight (2:45 a.m.) to see exactly why and maybe to fix it. So I’ll have to leave it.

Still some important things coming before I consider this beta functionally complete. I want to improve a few things before I set up the Behold News newsletter again and then try to stabilize and document towards version 1.

I tabulated the rating scores of the programs with 10 or more user reviews, and those with a rating of at least 4 out of 5 were given an award.

Hopefully this will generate a little interest in the genealogical community.

I couldn’t have been more pleased and surprised to read that yesterday Tamura Jones gave Behold an “honourable mention” for the 2009 GeneaAwards in the Most Improved Product of 2009. RootMagic 4 won that category and they definitely deserve the top honor with all the improvements they’ve made this year. See the Awards at: http://www.tamurajones.net/GeneAwards2009.xhtml (note: Internet Explorer cannot view the page. You’ll have to use a browser like Firefox or Google Chrome).

So what can I say about the next decade (and I know there are purists who don’t believe the next decade is starting, but personally, I find the “1″ in “2010″ quite significant). Well, if I might be so brash as to predict that by then, most people would have got rid of their “good enough” genealogy programs, and we’ll have had a significant number of computer genealogists migrate from their inefficient forms based systems to the friendlier and easier word processing model that Behold will introduce.

Ten years from now, I should have succeeded to have gone through my boxes of family data, and have them all entered (using Behold) into a complete and very useable personal family history. I can hardly wait.

I wish each and every one of you a healthy and happy New Year and many more after that.

And after a week of fiddling, I guess I’m happy with the way it looks now:

When I release the next beta, we’ll see if this implementation does not prove to be problematic like the last one. If so, I’ll have to either rethink it again, or remove it.

So it’s back to MailTo. We’ll try the new scheme in the next beta. If it doesn’t work out right, I’ll fix it or take it out.

But now there’s still a couple of important bugs still left to squash before the next beta release.

First I had to figure out how EurekaLog was calling up your email program. Two complaints about it was more than I wanted to hear:

Your debug routine fails to send the file, as it tries to start windows mail for some reason known to it’s self rather than using the default mail client (in my case thunderbird)

I tried to fill up the exception report but gave up when it asked for this server, that server! I then got really annoyed because it launched Outlook - I never, ever use it or want to use it.

So I found out that it was using Messaging Application Programming Interface (or MAPI). The problem is MAPI is not included with every email program, and will in such situations bring up a different program than the one you have as default. I also checked out the INDY (a Delphi package that provides Web services), and it has the same problem.

But there was a solution. The “mailto” command you find on websites brings up your default email program. It also has parameters that allow you to specify who it is going to, the subject, the body of the message and others. On a windows machine, that same function can be called by passing the mailto line to ShellExecute.

I thought I’d try it. It didn’t take too much to get a test going. And I was quite pleased to see that the web limitation of 256 characters was not there.

I had to figure out how to disconnected EurekaLog’s report and set up my own Exception Dialog form. Then I had to figure out how to pull all the info out of EurekaLog’s debug information for inclusion into my form. That went fairly smoothly until I got to the Operating System information. That was only in their log file, so I had to work out how to parse their log to find that information.

Wonderful. It was all set up and ready to go. I saw EurekaLog was attaching its log as a zip file. Hmmm. I don’t need to do that, since all the relevant error information was now in the body of the email message. But wouldn’t it be great if I could also (at the user’s option) automatically attach the GEDCOM and/or the Behold file and the future Log file?

Then I discovered (and this took over a day to realize it) that mailto did not support attachments with most email programs. I found a lot of people asking how to do this via Google searches, and there were several answers given. I tried them all and they didn’t work for me. Then I found that it only worked for certain email program such as Outlook, so that was no good.

But I was rolling. I knew there must be a some simple way to get the attachments to the email program. I posted a question on StackOverflow: How can a Delphi Program send an Email with Attachments via the DEFAULT E-mail Client?. The answers were coming in, but they were all telling me to go MAPI or INDY.

So I continued to do some research and I discovered some code that was not too complicated on Dragging and Dropping files from my application to Windows Explorer. Doing so would also allow me to list the files and allow the user to (relatively) easily drag them over to the email message that I’d create for them. It’s not the perfect solution, but it seemed like the best I could do if I wanted to ensure I properly accessed the user’s default email program.

I set up a Filelistbox on my Exception form, and got it all working. It was old code and required a few changes to work with Unicode strings. But it worked.

Next was setting up the Filelistbox to list the GEDCOM files and … oh oh. It required all the files be in the same directory because it was designed for an explorer-like directory listing. That didn’t work. After some thinking, I copied my ElTree from Behold’s FindFiles window and adapted that.

Once I had the file list box, then I discovered the GetFileListDataObject routine from the code I adapted also assumed everything was in the same directory. I had to convert that to figure out the data objects of each file no matter what directory they were in. This was code that I really didn’t understand. But I took a shot at it. I was extremely surprised when this worked the very first time - a rare event for me and for programmers in general.

I had to add “Select All” and “Select None” buttons to easily allow you to pick the files to drag. The selection of individual files had an interaction with the dragging, so that was a glitch that is still outstanding as I write this.

Then the cruncher came. My final tests and I get an exception while doing the mailto. It turns out that there is a limitation of 2048 characters that I can pass via mailto. Could I live with that? I don’t know. I was hoping to append the log file once I recreate it next version to the exception email, but that would now not be possible.

I was just about ready to take my lumps and maybe even back out of this whole thing, when surprise of surprises - another answer on my StackOverflow question appeared.

Instead of using “mailto”, mjustin pointed me to a way to used the Windows “sendto” command to send the attached files to the default email program and set up a new message with the files attached. This method will not require the user to do the extra step and drag and drop the files to the message and apparently it also has no limit in size of message either.

So just a bit more work to implement this now.

Yes. I did get carried away. This was a lot of work for something not part of the core functionality of Behold. But I’m not one to let something go half way. Best to finish it off while its in the moment.

And how was your holiday weekend?

There’s really only about 8 lines that are needed to get this working. I’ve just been trying hard to find those 8 lines, because they involve saving the selection and then restoring it. It’s easy in most of TRichView, but that horizontal line is a RichView Table, and I just haven’t been able to get that working. So I’ve got a question into Sergey on his Support Forums but I’ll have to wait until he gets back from holidays (Dec 27) before he can help me solve this.

Speaking of exceptions, a couple of people have complained about the exception handling that I added into the current beta. It causes problems interacting with their email system trying to set up an email to post back to me with the error.

That information has been tremendously beneficial to me. In about 4 bug reports, it led me directly to the source of the bug and saved me many hours of work. But I do admit that it is very intrusive and I don’t think I would like having a software product that tries to start my email program. So I’m going to work to simplify it considerably, and make it friendly and useful.

Basically, I’m going to have the box come up and just tell you the error that occurred (e.g. “Division by zero”) and the routine and line number where it thinks it happened. Then it will ask you to email that info to me to help me improve Behold.

I may decide to store that error in your Behold registry info, so that it can differentiate between the new errors and the ones that you’ve had before, and it can give you a count and ask you to email the info if it hasn’t happened before. When you upgrade to a new version, it will wipe out that registry info, because presumably those problems will have been fixed in Behold.

Then on my Future Plans page, I’ll list the errors, routines and lines where these exceptions have been reported. And I’ll mark them off as I get through them.

That’s what I’d call going from Exception-ally Bad to Exception-al.

1 NAME first middle /surname/ suffix

e.g. 1 NAME John Jacob Jingleheimer /Schmidt/ II

The name parts should be shown in the order given, but without the two slashes that surround the person’s surname.

To make this trickier, some East Indian names are in the form:

1 NAME /surname/ middle first

and should be displayed in that order.

I had Behold handling the East Indian names correctly, and I’ve now got it to handle the suffix automatically and correctly as well.

Behold is fairly unique in its ability to take the wife’s surname at birth and add the surnames of all the men she is matched to, ending with her final spouse, e.g. Mary Jane [Watson Osborn] Parker.

Now there are times you’ll want to override that. One example was a file that had under the marriage record a date of “NOT MARRIED”. So I check for that and do not add the husband’s surname in that case. I don’t know of any other instances where that sort of an indication is in the GEDCOM, but if I find any, I’ll add them. Once I get to Version 2.0 and editing, Behold itself will store this “override” into the file, so you’ll have complete control.

Some people have asked for the ability to turn off the generation of married names in Behold. While fixing the names, this turned out to be a very simple thing to do. So I’ve now added a Report option for this. Then Behold will give you less information than you’ll want just like other programs do.

I had a couple of those unexplainable programming occurrences happen. The worst was after I had done those two very difficult fixes, with the help of Uwe in the comments on my last blog post, and with the help of the LMD people on their newsgroup. I made those fixes and then was getting ready to go forward to the next item, and all of a sudden the files were loading much slower. I mean like 20 times slower.

It took me a while to trace this to where it was happening. First I tried using Eric Grange’s Sampling profiler that I was hoping to use in the future for optimization. I was terribly disappointed in how it couldn’t get me close to where the problem was. It was listing system routines with 90% of the time being string copies. Well I know that is going on at the low levels. What I want to know is from whence that is being called.

So I went back to AQTime. They just put out version 6.30 which finally includes integration into Delphi 2009, so it was worthwhile me trying it again. Of course, they’re still behind the times, because Delphi 2010 has been out for awhile so they’re over a year behind in their Delphi integration.

The AQTime Delphi 2009 integration didn’t help. Seemed to be buggy and I couldn’t get it to profile by line. So I did it externally and right away found the line in question. Problem was, there was absolutely nothing wrong with the line.

I was using Delphi PChars, and adding the length of the block to the starting character pointer to tell it where to stop. But somehow, it did not add correctly, and subtracted 4 instead. So the entire block was being Encoded once for each record, rather than once for each block. And the code was correct. I couldn’t find what was wrong.

I went back to my earlier beta version. It had no problem. I checked my next build. It was okay. Then the next build - that one no longer worked. There was nothing different codewise. I spent a day on it. Possibly some compiling option got changed, or some memory got filled somewhere? I don’t know. But my major changes had happened in the 5 builds since then. After I gave up trying to fix the problem, I had to go back to the working version and add all the changes since then and check step by step that the problem did not reoccur. It did several times along the way. So I had to go back and make fewer changes at a time. Doing so finally got me to my current state without the PChar problem.

So I’m totally perplexed and don’t know what this was. I have to stem my curiosity or I would spend the next two weeks or longer trying to track this down.

But the important thing is to continue on with the beta. So as long as I’ve rid myself of this problem that shouldn’t have occurred anyway, I better just forget about it, pretend it didn’t happen, and just cross my fingers that it doesn’t happen again. I don’t expect that it will, but if it does, then I’ll nail it down and put it away for good.

I guess I had entered the Programmer’s Twilight Zone. Let’s hope I’m out.

I’ve fixed a few already and I’m in the process of fixing about 6 others at the same time.

One involves something that Delphi itself seems to have a problem with. I’ve got a question on StackOverflow for it and one of the resident experts there is helping me out.

Another is a problem with my ElXTree component, and I’ve reported the problem to the LMD people on their newsgroup, and they are helping me with that.

Then I’m debugging some bugs that crash Behold and fixing those. One is simply using the arrow up key to scroll up the window. I had thought I had debugged my Virtualization of the Richview window, but I guess I missed that.

Then there is one that links to the child of a person, but that child is actually the user’s parents. That one is also surprising to me. I checked the GEDCOM thoroughly and it appears to be correct. My linking of this sort has been working correctly for years, and I’m very perplexed by this one. I’ll be interested in seeing the reason.

Then there was someone reporting something in Windows 7 that I can’t reproduce in Vista. So I’ve been loading both Windows 7 and Windows XP into Virtual PC windows so I can reproduce the error and ensure that Behold runs okay under the two systems, without needing multiple computers. (I absolutely love Virtual PC!)

So lots going on here and I’m making good progress. Hopefully in a week I’ll get out a new beta with most of the problems fixed.

Something is Behold was causing Avast to think that Behold and its install program had a virus, when I knew they didn’t. And that’s why I code sign Behold - to make sure nothing happens to it before it reaches you. Fieke checked and told me that Behold’s signature was okay.

You couldn’t imagine my horror upon hearing that. Having that happen to your program is as bad as Tiger Wood’s little escapade the other day. The damage is irreparable.

I went to Avast’s online scanner and submitted Behold myself. Sure enough, it reported that Behold.exe was infected with Win32:Delf-MZG [Trj].

I immediately shot off an email to Avast. I was pleasantly surprised to receive a prompt response that said:

It was a false alarm of Avast virus database 091203-0. It was fixed in the latest virus database update, please update virus database. We deeply apologize for the problem.

Well that certainly made me feel better. Fieke let me know when Avast next updated its new virus database, the reporting of a virus was fixed. Today I went back to the Avast’s online scanner and sure enough, Behold is now reported as clean.

So after a heart numbing affair, I now find that a search on Google for: win32:delf-mzg [trj] gives a good number of “false positives” with Avast reporting every program from Skype to Spybot to a host of others. Avast had to fix this, because the mistake would ultimately have caused them even more anguish than they did me. Here’s one of the typical reports on this by Andrew Wee just posted today.

This is what programming’s all about. There’s always something new creeping up that you never expect. Avast, me hearties.

The beta releases will come out more often than the alpha’s did. I hope to get an update every 1 or 2 weeks until the beta is complete. I’ll announce the updates on this blog.

If you are running Behold and you see on the menu line “New Version Available”, please click on it and update. You’ll find the update quite quick and painless. That way you won’t find any bugs that have already been fixed.

Tamura is one of the toughest graders of genealogy software you’ll find, with some reviews literally tearing programs apart and stomping on the pieces. I’m quite pleased with the review done of Behold. It’s an honest evaluation of a program whose beta is “definitely a serious improvement” but is still “far from finished”.

I’ve sent Tamura my comments and look forward to getting a response. I always manage to get excellent insights from our “discussions”. The whole virtualizing of the Everything Report was brought into being by some of Tamura’s ideas several years ago, and it is my intention to meet some of the challenges that Tamura has set upon the genealogy software industry. … One step at a time.

Well, I had changed the expiry from 90 days after the program was built to the new trial rule of 45 days after the key is sent for. I changed that number everywhere I could find. In the program, and in all the website procedures.

Then I uploaded it to the website. My upload compares differences in file size to see what files changed. In the one critical program, the only change was “90″ to “45″ and both of those are 2 characters and that did not change the file size. My comparison program did not report a difference and the file was not updated. Oooops.

All is fixed now. I’ve sent e-mails to those involved asking them sheepishly to try one more time.

Time to take a big deep inhale … and breathe.

Version 0.99.1 beta was released this morning, and version 0.99.2 beta came out this evening, after I fixed two minor issues with the “New Update Available” display and with text on the info box that still referred to the alpha.

If you are using the alpha, please do upgrade to the beta. It should be able to do everything that the alpha was able to do. If there’s anything you find that was working before that isn’t working now, please report it to me and I’ll fix it ASAP.

For large files, you should find it MUCH faster and use a lot less memory. Files of 100,000 people should load in 10 to 20 seconds, depending on the speed of your machine.

The help file has grown a bit old, and it will be updated during the course of the beta. I’ll actually be using the help file, as I update it bit by bit, to document my progress through the features of Behold as I ensure it all works as it should. Once everything’s checked and the help file is finished, version 1.0 will come out.

We’re off and running again!

I finished up and compiled the package. Then I had to make the install script. Whoops. I didn’t have it. Had to download InnoSetup and install it. Then I had to set up all my links again because everything on my new computer is in different directories. One very hidden setting prevented InnoSetup from working for me and it was an hour until I found that and fixed it.

So once I got the install set up, I had to test it. I installed the new version of Behold, and tested it. The welcome screen came up fine. Closed that. My Good Ideas box pops up. Close that and an exception occurs crashing Behold. What happened? I actually hadn’t tested my Good Ideas box since before I upgraded Delphi, and I had unchecked the option back then. Here was a new install and it appeared by default. But it was hooked into the old version of LMDTools, before I upgraded to their 2009 version. So it abended. Getting it linked back again wasn’t easy and that took another hour.

Then I noticed that the hyperlinks in the Good Ideas weren’t working. Actually, they may never have been working. This was obviously an easy thing to fix. But it ended up taking me 2 hours to fix it.

Despite all that, I still would have got the beta out today. But then it came to the code signing. I had to download and install Microsoft’s software development kit (oh, what a pain) and then get my old certificates and put them on my new machine. With that done, I attempt the signing and … it doesn’t work. I now have a ticket in with Comodo to help me get the code signing going again.

I suppose I can get the beta out without the code signing, but I might as well wait until tomorrow to see if I get an easy fix from Comodo.

Alas.

Examples of an exception is an array out of bounds, accessing an invalid memory location, or a division by zero. Often an error message pops up that has some fancy wording that is unintelligible to most people, but once leaving the message box, the program gets lost.

So with the help of a program called EurekaLog, Behold will now pop up a friendly box whenever one of these exceptions happens:

The beauty of this is that it will capture all the debug information and allow the person using Behold to easily send me an email containing mostly everything I need to find and fix the problem.

Now I’m sure you’re saying “I’ve seen this before”. Of course you have. Windows does this to you all the time. You often have things pop up like this:

I’ve often clicked on the “Send Error Report” button. But with Microsoft, I’ve never got any response back. So much so, that I almost feel I’m sending these things into a black hole that they just ignore.

To be fair, they probably get millions of them, all the same, for the same problem. And they most likely do fix the problem for the next release. They just can’t get back to every one of the millions who submit it.

What this also does is let Behold encounter its severe errors gracefully. Behold should not get stuck. If in that run it encounters 2 more errors, you’ll then get an  option to terminate the program.

I hope this will save me time and you from the annoyance of possible program crashes during the beta.

I was out yesterday at lunch for a walk and I passed by the Canadian Museum of Human Rights, a $260 million world-class museum that is currently in its initial stages of construction in Winnipeg. I took a picture:

It reminded me of Behold, now in alpha soon moving to beta.

Then I came to the display that said what was to come:

This reminded me of Behold version 2.0.

The race is on. Who will be first?

So those were the 6 major things I needed to implement for the beta. All done. I started them on October 14th, so they took just over a month.

Now there’s just 6 small things left:

1. Change the expiry date method to now use a 45 day trial from the date the key was downloaded.
2. Add a “New Version Available” link in the menu bar, to make it clear when a new beta is available.
3. Include an “Exception capturing ability” so that any bugs won’t cause Behold to crash, but will allow reporting of the bug encountered.
4. Recalibrate the progress bar now that I’ve re-twigged everything.
5. Change where files are installed to be the correct places on Vista and Windows 7.
6. Change the Behold web pages to announce the beta and build the new version and put it up.