Huge UFO Database in DOS - Someone's labour of love, can it be saved?

I frequently see suggestions that someone should build a database of UFO reports.

Well, this has been done - several times.

The biggest effort is probably UFOCAT, sold by CUFOS (which I'll discuss more some other time).

Probably the second biggest effort was the *U* database compiled by ufologist Larry Hatch over a period of about 20 years.


Sadly, Larry Hatch had serious health problems a few years ago and dropped out of contact with those in the UFO field. His website has disappeared, save for via the Wayback Machine's Internet Archive:

web.archive.org...

As stated on that webpage:

"As of June 2006, *U* holds 18,552 carefully filtered UFO events, distilled from hundreds of books, major journals, catalogs, correspondence and other sources. Scope is world-wide, for all dates from antiquity to the present."

The database "now a large stand-alone DOS application, written in C-language for PC, with supporting files. *U* is unique. The only part 'off-the-shelf' was the Borland C-language compiler"

I had Larry Hatch's database working on one of my computers a few years ago (which I bought on a floppy disc - an item that older members of ATS may remember...), but since it was written to work under Microsoft DOS it has not worked on any of the versions of Windows released in several years.

While quite a few of the leading UFO researchers have written positive comments about the *U* database over the last couple of decades, I have not heard about anyone managing to get it working, or extracted the data from it, in Windows Vista etc.

I'm spending a bit of time trying to get it to work under one of the DOS emulators currently available (e.g. DOSBox)

I have raised the topic on the forum of one of the DOSBox emulation software, in a thread at the link below:
vogons.zetafleet.com...

I don't expect the experts on the DOSBox forum will give this a high priority - at the moment I'm the person that has indicated to them that I have an interest in getting that database working again.


It seems a considerable shame that someone dedicated countless hours to compiling a database of UFO records yet we are now not able to easily access that data.

If anyone on ATS has any thoughts on what can be done, I'd love to hear about it here on ATS or in that forum.

All the best,

Isaac

Edit - In case it helps anyone that wants to look into these issues, I've found a downloadable copy of a demo of the database (with only some of the relevant data and some of its functions) in the Wayback Machine's archive of Larry's website at the webage below :

web.archive.org...


[edit on 21-6-2010 by IsaacKoi]

I think you should discuss this with ATS management.

I would hazard a guess that ATS has the necessary technology to at least host such a database.

Extracting it may prove trickier but from what I see of ATS, SO knows his way round computer tech.

The man power to do this could be an issue though...

Maybe members can assist in that regard?

[edit on 21/6/10 by Chadwickus]

reply to post by IsaacKoi


Hey Isaac, do you happen to know the format of the data base DB4 was popular a while ago or something similar. Could it be a simple delimited file, can you ask the questions. It should be possible.

MJ2

Originally posted by majestictwo
Hey Isaac, do you happen to know the format of the data base

While the UFOCAT database sold by CUFOS uses the Microsoft Access format, I fear Larry's *U* database seems to have been completely home-grown and to his own format.

Larry's documentation says the database is "a large stand-alone DOS application, written in C-language for PC, with supporting files".

After the floppy was installed, the relevant directory contained : one .exe file ("u.exe"), one .rnd file and one .vce files (plus a couple of Word documentation files).

Due to Larry's health problems, it hasn't been possible to ask him questions for a few years.

It _may_ be possible (but I'm not very optimistic) to get a bit more information from other UFO researchers and I'm trying this via various routes at the moment.

All the best,

Isaac

reply to post by IsaacKoi



Yeah it is a real shame that the data is not readily available in its compressed format for use by the general public.

Good thread
Thanks for bringing this up, im not very technical and doubt I could help in any way but im glad other people are on this.

From that forum post, it looks like the application does direct disk access, which means that just copying the files from the floppy disk isn't enough -- their location on the disk is important, too.

So first thing you want to do is create an image of the floppy disk:
www.softpedia.com...

for archive, then attempt to mount that as a physical drive emulation in one of various emulators.

Edit: And if you can find the C source code for that application, it would greatly help someone port it. The author or current copyright-holder's permission to make it open source, under something like the GPL or BSD license, would be a real contribution to the community.

[edit on June 21st 2010 by Ian McLean]

Originally posted by Ian McLean
From that forum post, it looks like the application does direct disk access, which means that just copying the files from the floppy disk isn't enough -- their location on the disk is important, too.

But, from memory (and I'm probably going back 3 or 4 years) I don't think the floppy disk had to be in the drive when using the database application.

I hope I don't need to image the floppy disk since I no longer have a floppy drive and also am not too sure where my original floppy disk is. I only have immediately at hand the files copied from the directory they were installed into.

However, thanks for the link to floppy disk imaging software - if it comes to it, I can forward that link to a colleague that does have the floppy disk at hand and I'll see if I can persuade him to give it a try.

And if you can find the C source code for that application, it would greatly help someone port it. The author or current copyright-holder's permission to make it open source, under something like the GPL or BSD license, would be a real contribution to the community.

I have emailed someone that was in touch with Larry's brother a couple of years ago - but I'm far from optimistic that I'll get any more information or permissions.

All the best,

Isaac

Absolutely right. It is a shame that someone devoted so many man hours collating a database only to let it go to waste. I've subscribed to this thread and starred it.
I know this sounds like a total noob-shot in the dark, but you mentioned current running windows platforms havent been able to run the dos application/database.

Is it possible for you to find an older computer/system that you could run this on? Then once you have access copy over that information onto a newer platform or database.

For something like this if your wanting public access it would be good to have a format that is completely universal in its entirety, so that if it was to be accessed to the public it wouldnt require any special changes or software to access it?

I might have totally the wrong end of the stick in what your asking here, so apologies if that is the case, best of luck with this.
In friendship
Richard

Most likely, once you getting running, you'll only have something that allows you to view the data, not perform any type of export.

Ideally it sounds like someone familiar with the programming language and methodology would need to decompile the code and data... which may be somewhat difficult.

Perhaps you could send us the relevant files and we could investigate the costs involved with hiring an expert to extract the data into a modern/open format. If it's not costly, we might be in a position to get it done, then make it freely available to anyone who wants it (assuming we can get permission to do so).

Tough one. Looks like a general incompatibility issue built into the original EXE. One of those unfortunate things that just can't be forseen in the wonderous technological age. I've had enough problems trying to get my own DOS app to run again, and it was only because I could get to the code that I was able to make it work - with massive problems still.

With the C+ source files though, it could potentially be remodelled into a Windows application.

The only other thing I can think of is trying an exe decompiler, or one of those programs that lets you mess around with the contents of an EXE file - though they may only respond to win32 applications and not DOS. But I'll ask around, see if anything can be done.

But yes, shame.

Did he intend to sell it?

reply to post by IsaacKoi


How big is the database in total mate?

Is there any possibility that the database may be on an embedded file that you may be able to access with a decompiler?

IRM

Originally posted by IsaacKoi
I'm spending a bit of time trying to get it to work under one of the DOS emulators currently available (e.g. DOSBox)

For a few dollars you can buy an old Windows 98 PC that will run DOS, I think there was one at the local salvation army store for $20. I happen to have one that I just never got rid of. I haven't turned it on in quite a while but the last time I did it worked great. It just seems like it would be easier to use DOS than to knock yourself out trying to work the bugs out of a DOS emulator.

Edit - In case it helps anyone that wants to look into these issues, I've found a downloadable copy of a demo of the database (with only some of the relevant data and some of its functions) in the Wayback Machine's archive of Larry's website at the webage below

Your link to the wayback machine works, but on that page, the link to the demo file doesn't work. I just tried it.

Originally posted by SkepticOverlord
Most likely, once you getting running, you'll only have something that allows you to view the data, not perform any type of export.

That would be my guess too.

[Perhaps you could send us the relevant files and we could investigate the costs involved with hiring an expert to extract the data into a modern/open format. If it's not costly, we might be in a position to get it done, then make it freely available to anyone who wants it (assuming we can get permission to do so).

Wow, what a fantastic and generous offer, to at least spend a little time to look into the feasibility. If that worked out to be economically feasible it would be a great asset!

[edit on 21-6-2010 by Arbitrageur]

Seriously I have an old trash 80 that still works great somewhere in storage. I will see if I can find it. I am quite certain it would be able to run it no problem.

For what it's worth I'd open the .rnd and .vce file in a hex editor (UltraEdit, Emacs, Vim / XXD) to see if there are any text blobs that correspond to UFO cases. If it's not compressed, Larry's using simple serialization to encode the data. That's imminently more reversible than trying to figure out non-standardized compression algorithms and map out the various equivalence classes between compressed and uncompressed symbol sets. If he's using terminating sequences for each datum it's even easier. If not, having a running version of the program would be an absolute necessity.

But if it is that easy, then all that's required is a little macro/shell-script to export the data into a more standard format like a CSV. From there, if the data integrity is good enough, someone could create a few SQL tables and import the contents; ultimately allowing others to perform more robust queries (using aggregate functions and such).

I also have more than just a passing interest in how this turns out. I've been helping MUFON develop and migrate to the new CMS2 system (with what little spare time I have) and it would be interesting to see how Larry organized his database as compared against the CMS design. Also I think it it would be worthwhile to see if I could integrate what he has into the new case management system.

At the very least it would help ensure the data would be around for a long time to come.

[edit on 21-6-2010 by Xtraeme]

If the data is within the files, it is most lily using some kind of hash procedure to store the data.

Do you have the application available for me to look at and dissect?

I have a degree in Comp Sci and may be able to extract the data, if possible.

Thanks,
-x

I have installed VMWare with Windows 95 on it. I've just tried the demo and it works well although many of the features have been taken out. The software has a "print out" option that, only perhaps, it could export the cases to TXT files. If not, I'm sure a good old fake printer-to-pdf diver could do the trick.

VMWare is not free although VMWare Player is, but I'm not sure if that software is capable of creating virtual disk drives. In any case, there are free alternatives such as VirtualPC and VirtualBox. Once you have this you'll need to create a new virtual disk and install Windows 95/98 on it (of course you'll need the original copies). It's really simple and it sounds to me like you used old PCs on the past so you'll manage, however if you need detailed instructions just send me a U2U and I'll walk you through it.

Many thanks for all the quick responses above.

I'll try to respond to some of the points and questions.

One point I'd stress - the copyright holder has not given his permission to sharing the application/data. I've emailed a few people about this, but I would not count on any joy in that direction. But if we can at least work out how to access the data using modern systems, those researchers that paid for a copy of Larry Hatch's database application (including me) can continue to build on his work and share, at least, the results of analysis of that data (and preserve it for the future).

Due to the copyright issues, I'm a more than a bit hesitant about emailing the relevant files to anyone - although I'm grateful to SkepticOverlord and others for offering their help in this way. Of course, if I or others can get the Larry Hatch (or the current copyright holder, if different) to consent then I'd be more than happy to pass things on to ATS for them to try to make the material more widely available.

It looks like there may be an easier solution for those of us that already have the right to use the application:

Originally posted by Xtraeme
For what it's worth I'd open the .rnd and .vce file in a hex editor (UltraEdit, Emacs, Vim / XXD) to see if there are any text blobs that correspond to UFO cases. If it's not compressed, Larry's using simple serialization to encode the data. That's imminently more reversible than trying to figure out non-standardized compression algorithms and map out the various equivalence classes between compressed and uncompressed symbol sets.

Mmm. This looks VERY promising.

Following up on your post a few minutes ago, I quickly downloaded the first free hex editor that I found via Google, i.e. HxD:

mh-nexus.de...

HxD installed in seconds. Opening the .vce using that program just resulted in gibberish being displayed, but the .exe and RND files contain readable text. The data seems to be in the latter file (i.e. the .rnd file).

Xtraeme - Since your basic idea looks promising, could you expand on where to go next?

While (as I mentioned above) I'm more than a bit hesitant to start emailing the relevant data files due to the copyright position, if anyone wants to help on this exercise then there is a downloadable copy of a demo of the database (with only some of the relevant data and some of its functions) in the Wayback Machine's archive of Larry Hatch's website at the webage below :


web.archive.org...

I presume that the file structure etc is going to be identical, so if a method can be found for accessing that data then that method can be shared with the researchers that already have the data (and hopefully, if we can resolve the copyright issues, more widely).

At least one member of ATS, k3rm1t, has already downloaded the demo version and accessed some of the files. Those include a file called UDATA.DOC which includes the following testimonials:

" Larry Hatch's UFO Database is remarkable as a source of documentation which encompasses most of the existing compilations in a single, easily researchable file. As a research tool it clearly surpasses all previous efforts at cataloging carefully screened UFO data. And, the graphic and mapping facilities are simply superb. "- Jacques Vallee 23 NOV 1994

" .. blows everyone else out of the game! " - Dr. Bruce Maccabee 1996

I think it would be rather good for SkepticOverlord to be able to send a message to Dr Vallee and Bruce Maccabee saying that the members of ATS have found a way for them to access the data on their old version of Larry Hatch's database so they can continue working with it (or at least access the data and possibly even query/manipulate it using some other platform)...

Let's see how far we can get.

All the best,

Isaac

[edit on 21-6-2010 by IsaacKoi]

Originally posted by Radiobuzz
I have installed VMWare with Windows 95 on it. I've just tried the demo and it works well although many of the features have been taken out.

While I'm currently attracted more by the approach of extracting the data directly from the relevant .rnd file (partly since that approach may result in my being able to query the data using a more modern software interface), I'm certainly interested in trying out this approach as well.

Can you let me know precisely which VMWare product you used to run the demo?

I've looked at the vmware.com website and they seem to have a fairly bewildering array of products.

On the up side, VMWare seem to have free trials of many of their products:

www.vmware.com...

All the best,

Isaac

This is the free one:

www.vmware.com...

But I use VMWare Workstation so I don't know if the player is able to create a virtual disk, which you'll need in order to run the software. VirtualPC on the other hand is free:

www.microsoft.com...

I'm currently looking into MZ DOS executables and trying to find a resource dumper without much success, but I'll tell you if I find something.

A few more responses to questions/points raised above:

Originally posted by InfaRedMan
How big is the database in total mate?

The various files total 2.68 MB, including the manual.

U.RND = 1.98 Mb
U.exe = .47 Mb
Manual (.doc file) = 0.16 Mb
WM.VCE = 0.07 MB
USOURCES.DOC = 0.05 Mb
README.DOC = 0.01 MB

Originally posted by InfaRedMan
Is there any possibility that the database may be on an embedded file that you may be able to access with a decompiler?

Er, pass...

Is this different to accessing the database application and/or other files using a Hex editor??

Originally posted by Arbitrageur
For a few dollars you can buy an old Windows 98 PC that will run DOS, I think there was one at the local salvation army store for $20.

That's one possibility - but ideally I'd like a solution that doesn't involve my wife saying looking at yet another computer in our household and questioning my sanity...

Also, I'd prefer to be able to cross-refer quickly to that database (e.g. to find a reference) without having to power up some old machine (which is what I did for the first year or so after buying the first computer that wasn't compatible with this database).


All the best,

Isaac

[edit on 21-6-2010 by IsaacKoi]