Login to participate
  
Register   Lost ID/password?

Louis Kessler's Behold Blog

What About 64 Bit? - Sun, 12 Jul 2015

Yes, I do now have a 64-bit executable of Behold that works exactly the same as the 32-bit version. But I’m not going to release it yet.

What I found out in my testing was that it was slower than the 32-bit version and used more memory. So for most people, there’s no real reason to make it available because it will not help you.

64-bit does allow larger files to be loaded. Tamura Jones provides a wonderful capacity test with a program called GedFan that generates successive GEDCOM files each larger than the previous with double the number of individuals. Behold 32-bit has a fan value of 19 meaning it can load a file of about a half a million people, but doubling that it runs out of memory.

Without modification, Behold 64-bit failed at fan value 22 (four million people) ironically because the memory reporting function call I was making placed the result into a 32-bit integer. I changed the variable from an Integer to an Int64 and ran it again.

With that change, Behold now does load fan value 22. But this extra capacity is not going to be of benefit to too many people yet. It took 99 seconds to load (a good amount of that time was checking ancestral loops, which I’ll have to put into its own thread) and the internal data structures Behold needed caused it to red-line both my computer’s RAM and swap file and that’s with 12 GB RAM and a 40 GB swap file!

Red-lining memory and swap file

The trouble is that this version of Behold loads everything into memory and builds all its data structures and indexes and links in memory. A doubling test like this will make it run out of memory before anything else shuts it down.

What will make 64-bit useful for Behold and will increase both its capacity and speed will be writing the data to a real disk-based database, rather than keeping it in memory. I’ll be using SQLite as the database. This is the database that RootsMagic uses and Tamura Jones found RootsMagic to be one of the faster GEDCOM readers with one of the highest fan values. SQLite is a very fast database, and even though it is disk-based, it doesn’t lose much to the overhead required for page swapping when you keep everything in memory.

For now, I’m not releasing the 64-bit version.

When the database work is done, it will come with both 32-bit and 64-bit versions of Behold. I’m starting the database work tomorrow.

image

Seven Years of Improvements - Sun, 12 Jul 2015

I know my last two posts were not written in English for the normal person. But I felt I had to document a bit of the work involved in just upgrading a development environment. It is complex process full of pits and valleys. Getting through it needs a machete and a torch. We programmers struggle with our programming tools as much as our users struggle with the programs we create.

Now it is time to reap the rewards. Delphi and TRichView and ElPack and EurekaLog each have gone through seven years of improvements. I can run through the change logs of each version, and see what changes are notable and will provide benefits to Behold users.

 

Delphi improvements

Lots has been added since Delphi 2009.

The most important improvement for the future of Behold is FireDAC with SQLite support. I will be using this for Behold’s database.

Another improvement is 64-bit capability. Now I will be able to make Behold available both as a 32-bit program and as a 64-bit program.

There are new online data access protocols now available including REST web services, and libraries I may be needing. The newer versions of Delphi include improved JSON low level processing and an improved XML engine.

There were supposedly style improvements that take on the Theme of the O/S (Vista, Windows 7, WIndows 8), etc., but other than it adding a blue background to indicate that a toolbar item was selected, I haven’t noticed too much difference. It’s also touch-enabled. XE8 added support for Windows 10 and Windows 10 styles, so it will be ready when that comes.

Delphi has really enhanced its Generics library, which is a suite of data structures I can use in Behold. They are well-written, generalized and fast. They should speed my coding time, and over time I’ll replace my older data structures with these. Delphi 2009 included an early version of Generics, but there were some bugs so I only made limited use of them.

A Parallel Computing library has been added that will allow me to optimize Behold with background processes and to keep it running as fast as possible.

They also made lots of fixes and improvement. One item that really pleased me was a fix that helped Behold’s Find box. When you bring it up, it stays on your screen, but it is called a non-modal box because it does not take control of your screen. So you can find something and then go to the Everything Report and work on it, and then go back to the Find box. The default action in the Find box is of course to find the text. In Behold 1.1 and earlier, going back and forth between the Find Box and the main program would somehow lose the default action. I would press Ctrl-F to go back to the Find box and then press enter to get the next search, but it wouldn’t work. I tried in the past to find an easy fix to this, but it was a Delphi/Operating System thing. What pleased me is apparently, whatever was causing this loss of the default was fixed sometime between Delphi 2009 and XE8. Hopefully more subtle bugs like this were also fixed by going to XE8, without many new ones creeping in.

 

RichView improvements

The Everything Report is built on RichView, so any new features and improvements to RichView can benefit Behold.

I was using RichView 10.1.4 from 2008 and have now installed version 15.7.

Now it can export as a word document (.docx files). I’ve added DocX as an export option in Behold and it will be available in the next release.

Some fixes/improvements include improved rendering of bidirected text, optimizations in calculating text width (which is probably the main reason why Behold loads 30% faster when compiiled with Delphi XE8 and RichView 15.7), touch screen support, e.g. handles when selecting with a touchscreen:image

Also improvement to the line breaking algorithm, 64-bit compilation support, style templates (which I may or may not use). Hidden items were added, but I already had my own customized implementation of hidden items which I liked better, so I kept my own.

 

ElPack improvements

I use ElPack in Behold for the TreeView, the grids in the Organize pages, and a few other components. I had the 2009 Version and I upgraded to the 2015 Version to get a version that would work with Delphi XE8.

ElPack is not LMD Innovative’s major product. They acquired it quite a while ago from ELDOS who I initially purchased the package from because I needed a treeview and grid with Unicode support. So LMD is maintaining it, and is slowly merging it with their core LMD products, but they are not doing much to enhance it.

They’ve added 64-bit and touch-based features. They fixed a number of bugs. And they’ve included a few new packages with ElPack like a DialogPack that I could make use of in the future.

 

EurekaLog improvements

EurekaLog traps errors that would otherwise crash Behold. It lets me display a message to inform the user there is a problem, and provides some information that might allow me to find and fix the problem. I also use it when I’m developing since it can help me find and fix memory leaks.

I first purchased EurekaLog Version 6 back in 2009. I initially used EurekaLog’s default dialogs and this allowed the users to email the problem to me. It caused me problems. Users complained. So I changed it:

I came up with my own form after careful thinking. I did not want a generic box. I wanted something that the user would know it was my program self-detecting the problem, and that it was not Windows telling them that my program had a problem. EurekaLog’s generic dialogs look too much like Windows. I attempted to fix the email problems with a "mailto" email. But even with that, the emailing was problematic, so I changed it to my current form, where I instead ask the user to drop off the bug information on the Behold feedback page:

image

EurekaLog’s improvements going from Version 6.x to 7.x involved a complete rewrite of their product. My old code wouldn’t work. I placed a support ticket with EurekaLog and Kevin of their support team worked with me and provided me a framework to continue to trap errors and provide my own form the way I had been doing.

Upgrading to Version 7 will provide 64-bit error trapping and better information for me for debugging purposes.

 

Overall

Upgrading to my new development environment that includes the latest versions of Delphi, TRichView, ElPack and EurekaLog took 14 days and involved about 400 recompilations of Behold.

Now I’ve got the tools available I’ll need to go forward and create a native Behold database to store your data. Then, with a place to store your edits, I’ll be able to add editing to Behold.

Getting Behold Running in Delphi XE8 - Mon, 6 Jul 2015

Last post, I updated my development tool and installed new versions of my three 3rd party applications. Now it was time to get Behold working again in the new Delphi XE8 environment.

Theoretically, if all updates made to an environment are backwards compatible, then old code would just run unchanged in the new environment. But they seldom are. Work needs to be done to convert.

To get started, I included the 3rd party packages without my customizations to see if what was left would still run. Then I compiled, and knowing that there was little likelihood of it working, I got:

2 Errors, 5 Warnings, 1 Hint

One error was an incompatible type between Integer and TRVTag. This was a change I remember reading that RichView had made a few years ago to use strings instead of integers for their tags. The second was the error aborting the compile because it couldn’t compile the unit. For now, I just changed the declarations to eliminate the error and proceed.

image  image

That doesn’t sound so bad for a first attempt, to have only 1 error. But I know how these compilers work. You fix one error and the compiler can then look at more code. Sure enough, after I fixed it, I now had:

16 Errors, 5 Warnings, 2 Hints

EurekaLog was now the culprit. The errors indicated major changes to that package, and indeed when I checked out their site, I found that Version 7 was almost completely rewritten.

image

I wasn’t trying to get EurekaLog to do much. It was passing information from unexpected errors to a special form I had set up for that purpose. I looked to see what changed, and it wasn’t simple. I always give myself a couple of hours to see if I can figure out any problem. I go through their documentation and try various things. If stumped I might check StackOverflow and maybe ask a question there about how to do it. But I could not make any headway on this, so I went to the next step and submitted a support ticket on the EurekaLog site with my problem and requested a method to fix it. I got a response quickly, within a couple of hours (on a Saturday!) and that answer led me to some good examples, but they weren’t quite enough to help me. So I added to my ticket and while waiting went back to fix the rest of what wasn’t working.

I commented out the offending EurekaLog statements and that allowed Behold to compile and run. It was nice to see Behold running from XE8. It ran fine, looked fine (except for a minor problem with the toolbar icons), and did not have any obvious problems.

There were still lots of warnings and hints to fix. These included fixing declarations, replacing a deprecated symbol or routine with another, and new messages about possibly uninitialized variables that the XE8 compiler was picking up that 2009 didn’t.

I worked my way through the most important compiler warnings and hints. In doing so, I must have recompiled the program 200 times over 2 days, finding and fixing things that worked under the old environment but not in the new one. I finally managed to get through them, leaving only a few hints about variables that are only valid on some operating systems.

With everything finally working with 32-bit processing, it was time to try 64-bit.


On my first 64-bit compile, I was quite surprised that there were new errors. The 64-bit compiler must have different checks in it. For example, one warning that came up was for code that I’ve had in place for as long as I can remember. It had never given me a message before. It was:

N := MaxN;    { Execute the loop at least once }
while N = MaxN do begin
   …
   N := …;
   …
end;

The warning message on the first assignment statement was that the value assigned to N was not being used. 32-bit compilation never gave me this. 64-bit must optimize differently. So to eliminate the warning, I changed this to:

repeat
   …
   N := …;
   …
until N <> MaxN;

There were a bunch of little fixes like this I could make. But then I was getting errors again. The 64-bit compilation wanted me to address the RichView change of their tags from integer to string values. I was going to try to use a setting in RichView that allowed using the old style, but then I found the 64-bit compiler was not accessing those settings. I posted a query on the RichView forum. While waiting for a reply, I found that using the old style was not recommended, so I decided to go through the work and change it. It took me a few hours to update what was needed here. Once I did, Behold was able to compile and run in 64-bit.

The last thing to do was to add my customizations back into the RichView and ElPack routines. I had customized a 3 RichView units and 4 ElPack units to enhance them to provide some functionality I needed for Behold and to fix some bugs in them. There were changes to 7 different units. They represent dozens (maybe even hundreds) of hours of work I spent many years ago to isolate problems or limitations in these packages and figure out how to get them to work the way I needed.

I had delineated all my changes clearly with comments in my customized units, so they were easy to find. One of my typical changes looked like this:

{ LK - Jul 15, 2012 : Start of code to allow invisible text }
const
  rvInvisible = 21; {= rvInvisible in _links.pas}
{ LK - Jul 15, 2012 : End of code to allow invisible text }

Now the question was, how much did these units change in 7 years? Fortunately, these were mostly low-level units with core functionality. The new version of these units did not change much from the old version, and I was able to simply slot my customized code back in, occasionally with minor modifications. I had the RichView units working in just 15 minutes.

But then it took me 2 hours to get the 64-bit version compiling. Some sort of strange dependency in the use of one unit by another was only revealing itself one unit at a time. And this was just in the 64-bit compile, not the 32-bit compile. It was very strange. I never did figure out what this was, but I thankfully reached the end of the loop after going through only 60 of the 200 units.

I stumbled a bit while updating the ElPack units with my custom code. In the ElIni unit, which handles reading and writing the Behold Organize file, the ElPack unit did not read ASCII files properly. I had the customized code to do this correctly. But it did not seem to be working. I installed, tested and removed that code 3 or 4 times. Behold was not reading any of the information from the customized Behold file at all. Then then it hit me. My code wasn’t what was wrong. The customized Behold file I was trying to read was empty. There just wasn’t anything to read. I kicked myself about this oversight. Only cost me 2 hours.

So that was it. Behold is now all updated and is now pretty well working under my new Delphi XE8 environment. I’m still waiting for the EurekaLog support message and I’ll get back to that later.

At this point, I couldn’t help myself. I just had to do run some comparisons of Delphi 2009 performance versus Delphi XE, and then of 32-bit performance versus 64-bit performance. The results were interesting:

Compared to the current Behold Version 1.1 release (which was compiled with Delphi 2009), the new 32-bit compilation under XE8 is about 30% faster and uses about the same memory.

Then I found that the 64-bit compilation is about 12% faster than Version 1.1, but is about 18% slower than 32-bit speed. And 64-bit uses almost 50% more memory.

This surprised me. My computer has a Windows 64-bit operating system. I would have thought 64-bit processing should have run faster than 32-bit processing on my machine, but I guess the manipulation of all that extra memory the 64-bit program takes also adds a time penalty.

Once I convert Behold’s in-memory data structures to a true disk-based database, that will eliminate much of its memory use. Then time will be more dependent on 32-bit versus 64-bit database access. Once that’s done, I’ll recheck these results and won’t be surprised if 64-bit happens to emerge as the winner.image