Huge UFO Database in DOS - Someone's labour of love, can it be saved?

reply to post by LightFantastic


OK, well, if Isaack makes the 'full version' data available then it is a job done...

Just a thought... Is there a standard Db schema for UFO for recording UFO recordings? If not, shouldn't there be?

Imagine being able to take all sighting in N different Dbs and putting them all together in a super Db so that they can all be x-referenced with each other.

Perhaps the 'UFO community' should be prioritising this - the use of standards.

Maybe it has already been done in which case this data should be converted to that form (i.e. table and field names)...

The community needs to develop a canonical dictionary for classifying events and associated attributes so that conformance allows the x-correlation of data to identify patterns and even identify fakers and hoaxers.

Mind you, the paranoia brigade might see that as a NWO influence...

Originally posted by NASA is ASAN
Excellent, nitpick one comment and disregard the message of the rest of the post, you're a stand up guy.

Wasn't the message based on the point I "nitpick", besides other points?

Isn't this database from the 1980s and the creator of it dead? So that's it, you're worried about a 30 year old copyright owned by a deceased person? *facepalm*

As far as I understood it, the database stopped being updated in 2006, the executable for demo version is from 1999 and I don't know if the author is alive or not.

I'm sure he cares if his work lives on or not, so help it live on. How old are you, by the way? You have a very outdated mindset, I'm going to guess 40+.

I suppose he cares about his working living for ever, but I don't know if he does or not. And you don't need to guess, my profile shows my birth date, and my opinion about the rights of authors of anything has nothing to do with my age.

And one last thing, if the owner of the copyright is against this database living on forever, then that guy is extremely selfish and has no business owning the copyright.

It's his work, he has the right to do what he wants with it, regardless of what we think or what the law says.

He doesn't deserve to own it if he's going to so @ss backwards in his thinking.

How can you (or anyone else) decide how much backwardness is need for someone to lose the right of doing what he/she wants with his/her works? What someone sees as backwardness someone else may see in a different way.

The fact of the matter is, no one is going to buy it ever again, it's in the public domain now.

Why is it in the public domain?

PS: as this is becoming too off-topic I think we should keep on discussing it via U2U, if you don't mind.

FYI @ All

I haven't had much time to look at this but so far I have produced the following entities (which ATs will screw up in display terms but ignore the "obvious" errors in XML data:






























I know... looks a mess. However, the next step is to clean the text values (i.e. remove trailing null characters from the 'Description' fields) and convert the "Unknown" fields into their atomic constituents that represent numeric data (i.e. dates, coordinates etc). I just need to work out the encoding sequence for the numerics but the rest is pretty straightforward.

The only other possible issue is 'enumerated' values. That is to say that some 'byte' values may represent foreign keys to be looked up in enumerated tables.

[edit on 28-6-2010 by SugarCube]

Many databases allow you to save the data in a text format file.

If nothing else you might be able to get it to 'print' out the data whole or in batches in a text format to a file.

If worse comes to worse writing a 'c' function to save the raw data that way can't be too terribly difficult,

When you are saving the data just make sure you do it from a copy of the database, in case of any accidents.

Once that is done it can be easily, fed to virtually any database system.

[edit on 28-6-2010 by slank]

reply to post by slank


As I understand it, that is the problem! If the data could be exported to text this would not be an issue at all. However, I do not believe that the data can be exported as text...

There is no database definition as such (i.e. field content definitions) only the raw data and by examining the code, some print masks which indicate the fields that are available (view the compiled code in a good text editor and go near the end).

The data can be grabbed, but it then has to be interpreted correctly whether C or your fingers. That interpretation definition is the next key process. As indicated, I can already 'grab' the data, but some fields need to be split up into atomic constituents (i.e. Unknown fields) and then read as bytes to determine values.

Some of the values are foreign keys but we do not have the lookup tables to identify the data being looked up, this would have to be extrapolated from the compiled code.

Isaac, I can make a few calls and see if Jacques Vallee and Bruce Mac still have an interest in this. Better yet, they may know how to get to the copyright holder and get us permission to publish it.

I have zero problem doing a revenue share (if it generates any revenue) with CR Holder. Heck, for this, I'd happily give them 100% of the revenue after whatever expenses we incur in publishing it.

Let me know your thoughts...

Springer...

If some numbers are stored in numeric binary format [floatingpoint, integer] you just need to translate/convert/calculate them into character text.

If you can view the values on screen/display then somewhere in the software it is already doing that.

Basically you want to pull up each record, and output all the fields [including blank fields] & convert/translate any that need it.
There is comma separated data or full maximum character/digit field length [maybe a couple of others].

ie.
Joe,Smith,750-281-2500, . . .
Kate,Jones,532-429-7707, . . .

I have an old copy of GW BASIC that I use from time to time and I'm running Vista. I have a copy of cmd.exe that use to run the basic program. If you go to C:\Windows\System 32 you'll find a copy of cmd.exe, it should work.

reply to post by slank


Yup, after all, its all 0s and 1s at the end of the day.

For example, one of the records has the hex values as follows:

a00710071d3cff0a00bc4a391902

What does it mean? (Rhetorical question)...

As an example, in order to decipher the hex as numerics, we need to know what the 'type' is, maybe whether they are lo/hi bytes or hi/low bytes - we need to know what data values they are supposed to be representing. In order to do that you need a file definition otherwise it is all guesswork.

You can work out an algorithm by 'en clair' data comparison however, in order to do that you have to have data in the readable form which, at the moment, does not appear to be available per se. The 'demo' Db is available but you need to be able to run it to see the data in readable form in order to work out the algorithm/presentation.

In the example above, I am pretty sure that a007 is lo/hi byte for 1952, which would suggest that the following bytes are month/day/time in some form, however, I have not experimented enough to work out the conversion.

The hex characters above 'probably' represent the data:

a00710071d3cff0a00bc4a391902

1952/7/29 10:0 10 106:17:20W 35:52:20N 3333 NAM USA

Some will be numeric representation (e.g. the year), some will be foreign keys (e.g. NAM). The character delimiters can be ignored.

Fill your boots...

Originally posted by SugarCube
Just a thought... Is there a standard Db schema for UFO for recording UFO recordings? If not, shouldn't there be?

Excellent idea, why not kick off the schema and we can all add and modify?

Originally posted by Radiobuzz
One small tip: you can drag files from your host OS (Windows Vista) into the desktop on your virtual PC (Win 95) to copy files between them, you don't really need to create an ISO file.

VirtualPC 2007 doesn't allow that with Windows 95 (Virtual Machine Additions doesn't install on Windows 95), I think that is only possible with the Virtual Machine Additions from VirtualPC 2004.

Hi all. Sorry I didn't read the whole thread, but I would like to offer my services. I have reverse engineered file formats before, including databases. I have also reversed DOS applications in the past. I have 20+ years programming experience, in a variety of languages. I'm no super-guru, but I am certainly competent.

This kind of challenge is right up my street. I'd prefer to help on something that is going to be "open", but that is not a necessity, the challenge would be enough

I see a few helpful guys are volunteering so far, but if you need any more manpower/eyeballs, please U2U me with details.

Originally posted by SugarCube
Just a thought... Is there a standard Db schema for UFO for recording UFO recordings? If not, shouldn't there be?

There has always been talk of standardizing UFO sightings databases, but there's never been any agreement made about it. There's never been any money available, so most of the databases are done by individuals either using custom-made programs or with programs that inevitably become old and creaky. There's also a bit of a problem about people maintaining databases and purposely keeping them secret, either because the collector sees it as something valuable or because they're a little paranoid. Comes with the territory.

That's the situation even without coming to an agreement about which coding system to use, whether it's Hynek's original "Close Encounters" system, or Jacques Vallee's expanded sighting description grid.

It's a bit discouraging. People are always looking for ways to apply strict scientific methodology to the study of UFOs, and the databases are a great way to do that. Unfortunately, the disorganization within the field often leaves very good data just rotting on the vine, while people go chase crappy videos and photos.

reply to post by Blue Shift


Well, methods for beating that "rot", open source, and good documentation. There's no reason to keep making the same mistakes on something that doesn't really need to be hidden away.

Just a note about copyright -- likely, it only applies to the software program and 'hand drawn maps' that the author made himself. Unless the author interviewed all these people on his own, and had them sign a rights transfer agreement?

OP, posting the exact copyright and license text from the product would be helpful, especially if ATS or members are to make an 'official' effort.

Quite probably, the database can be extracted and secondarily published without infringing on anything the software author or his subsequent assignees might claim.

reply to post by harpsounds


U2U'd U...

I'll send across the extracts I have produced against the demo Db...

Thanks, I'm just about to go out for the evening, but wanted to make a few notss.

I tried to get the demo program from archive.org, but it failed to download for me. I'll try again later.

On the subject of legalities, here is my understanding of the laws in my locale.

I could legally reverse the database, and make a compatible version under "interoperability" and "data lock-in" rules. That would allow me to publish a new program that can read the data. BUT, it may not be legal to actually distribute the actual data myself depending on the copyright issues.

An example of this law in action, is many open soiurce clones of old games. They are allowed to distribute the new game binary, but the user must have their own copy of the data files needed.

This would be a kind of worst case scenario if no permission can be found, but at least it would allow those who bought a legit copy to keep accessing, and extract the data for their own purposes.


[edit on 28/6/2010 by harpsounds]

Originally posted by Springer
Isaac, I can make a few calls and see if Jacques Vallee and Bruce Mac still have an interest in this. Better yet, they may know how to get to the copyright holder and get us permission to publish it.

Hi Springer,

Sounds good to me. Give me a day or two to get some additional information for you.

All the best,

Isaac

U2U me an e-mail address if you want the data extracts that I have produced from the demo Db so far. I haven't managed to spend much time on this but it just needs some manpower on it to derive the algorithms/structures used to store the original numeric data by correlating the hex dump values against readable forms.

Anyway, let me know if you're interested...