Huge UFO Database in DOS - Someone's labour of love, can it be saved?

Originally posted by Radiobuzz

Originally posted by IsaacKoi
I think the next step is to try using the "print" option to "print" to a text file. I should be able to try this in the next day or two.

That's what I was going to suggest. You can use a PDF printer like this one:

sourceforge.net...

Which apparently works on Win 95.

I'll try a PDF printer tomorrow evening, time permitting.

This evening I've tried (and succeeded) in installing a "print to file" driver and managed to print a test page (as part of the installation process) and a test document from Wordpad. Both generated a .prn file which was easy to find on the virtual hard disk and could be opened to see text.

However, when I tried printing all the *U* Database application records after setting the "print to file" printer as the default printer, the application took about an hour to run through 18,000 records - after which I could not find any .prn or other file containing the relevant records.

After that, I've tried printing just a couple of records at a time - with no success.

It seems that I can "print to file" from any of the Windows applications, but not from the DOS database.

A bit frustrating really.

I thought I'd cracked it using this method, since the "print" option in the *U* Database seems to involve more fields than the "export ASCII" option.

So, unless anyone has any ideas about getting the "print to file" option working, I'll try installing one of the PDF creation "printers". I hope I don't have the same problem that they only work with the virtual Windows 95 rather than the DOS application running under it...

All the best,

Isaac

Originally posted by ArMaP
VirtualPC 2007 doesn't allow that with Windows 95 (Virtual Machine Additions doesn't install on Windows 95), I think that is only possible with the Virtual Machine Additions from VirtualPC 2004.

Humm, you're right. I've googled it and apparently the additions that came with Virtual PC 2004 do work on Win95 though:

www.microsoft.com...

reply to post by IsaacKoi


It's weird... Are you positive you are able to print from within any Windows software (like Wordpad)? You should go to control panel and check the driver preferences under "printers" option.

I am done for the night, but If anyone else is playing with this, here is what I have so far incase it helps. The format of each record is like this:

2 bytes : year
1 byte : unknown
1 byte : month
1 byte : day
1 byte : time
1 byte : unknown
1 byte : duration
1 byte : unknown
2 bytes : longitude
2 bytes : latitude
1 byte : unknown
4 bytes : location
8 bytes : attributes
78 bytes : text
1 byte : reference
1 byte : unknown
1 byte : strangeness/credibility

notes:

the month/day fields have some other unidentifed data in them, I think the "known/unknown" information, in the higher bits. so the month only uses the 4 low bits, and the day uses the lower 5.

the time is stored as (hour * 6) + min, where min is reduced to just 00, 10, 20, etc

the strangeness/credibiliity byte is split into two nibbles.

the longitude is stored with the top bit marking E/W, and the value stored like s + (m * 3) + (h * 3 * 60) the seconds are stored only in 3 values, 00, 20, 40

the latitude is the same, just with the top bit marking N/S

The attributes are 8 bytes, making 64 bit flags, seemingly laid out in the order they appear on screen.

reply to post by harpsounds


Fantastic, I'll be able to have another look at this in the evening!

Originally posted by Radiobuzz
Humm, you're right. I've googled it and apparently the additions that came with Virtual PC 2004 do work on Win95 though:

I can confirm that they do work, I tried it yesterday.

Originally posted by harpsounds
The format of each record is like this:

2 bytes : year
1 byte : unknown
1 byte : month
1 byte : day
1 byte : time
1 byte : unknown
1 byte : duration
1 byte : unknown
2 bytes : longitude
2 bytes : latitude
1 byte : unknown
4 bytes : location
8 bytes : attributes
78 bytes : text
1 byte : reference
1 byte : unknown
1 byte : strangeness/credibility

Given the amount of progress you have made Harpsounds, I'm almost tempted to go back to trying this "reverse engineering" approach...

In the last few days, I've been concentrating on getting the full application to run in a virtual machine (which, as I mentioned above, now works) and then trying the "export ascii" option offered by the database (which, unfortunately only exports a few fields) and trying to use the print option to print all the records to a file.

I keep running into obstacles in relation to printing.

I can now "print to file" using Windows 95 drivers I've installed _but_ only from Windows applications, including Notepad and Wordpad.

If I try printing from the DOS database or, incidentally, run the readme.bat file and try printing that (from what looks like a DOS environment) then nothing seems to happen - no .prn file is createdm nor do I get any sort of error message.

I've found a utility online called prn2file. Although it seems to install (and when I run it from the command prompt it tells me that printing is redirected from LPT1 to the specified filename), the file that is created is empty. Again, when I try to print from the DOS database or after running a readme.bat file, then nothing happens.

I found a discussion online which suggested running an echo command to >LPT1, but the relevant text did not appear in the relevant file - which apparently means that it is necessary "to look at compatibility settings in your command prompt":
www.experts-exchange.com...


Incidentally, while I'm grateful for the offers from several people to look at the full database files themselves, I remain very concerned about the copyright issues involved in emailing that around the place. I don't mean to be difficult, but I am a lawyer after all...

All the best,

Isaac

[edit on 29-6-2010 by IsaacKoi]

Originally posted by Radiobuzz
reply to post by IsaacKoi

 

It's weird... Are you positive you are able to print from within any Windows software (like Wordpad)? You should go to control panel and check the driver preferences under "printers" option.

I'm positive - that has worked several times.

A generic print to file driver shows as being the default (and only) printer under the "printers" option.

All the best,

Isaac

reply to post by IsaacKoi


I've just tried a "print to file" utility with success. This is the one I used:

www.uvpafug.org...

I don't know which one you are using but I think the problem might be that you are trying to print the text from a DOS console within Windows. What you should do is restart in DOS mode and do it from there directly. Just in case, this utility I'm using works like this: you boot in DOS, go to prn2file folder and type "prn2file c: file.txt" and that's it. I've used EDIT to test it and while it produced a blank file within Windows, when I went to DOS it was able to print it right.

Originally posted by Radiobuzz
www.uvpafug.org...

I don't know which one you are using

I'm trying prn2file as well.

I think the problem might be that you are trying to print the text from a DOS console within Windows. What you should do is restart in DOS mode and do it from there directly.

Thanks - I'm afraid I don't know how to "restart in DOS mode".

I've accessed the command prompt from the Windows 95 Start menu and run the prn2file command and it has acknowledged that printing is redirected, but when run the *U* database and try printing from it the relevant text file remains at 0 bytes.

Can you clarify what you mean by "restart in DOS mode" - it's been a while since I've used Windows 95 and/or DOS...

I feel like we are almost there!

All the best,

Isaac

There are two ways, either go to shut down option (from Windows) and choosing "restart in MS-DOS mode) or just press F8 while the PC is booting and choosing DOS from the menu there. I can confirm it works so you should be able to get it .

Originally posted by Radiobuzz
There are two ways, either go to shut down option (from Windows) and choosing "restart in MS-DOS mode) or just press F8 while the PC is booting and choosing DOS from the menu there.

Many thanks for the step-by-step instructions. I need them...

I can confirm it works so you should be able to get it

.

Let's hope so - unfortunately, I will now be away from home for a couple of days so it will probably be the weekend before I can try it and report back.

All the best,

Isaac

Double post - please ignore.

[edit on 29-6-2010 by IsaacKoi]

reply to post by IsaacKoi


Hi Isaac

I would guess Harpsounds is only a couple of hours work away from being able to export the entire full db.

Why not save yourself some trouble and let him finish the job. I'm sure he will delete all the copies of the full data he has if you ask him to.

Originally posted by harpsounds
reply to post by Blue Shift

 

Well, methods for beating that "rot", open source, and good documentation. There's no reason to keep making the same mistakes on something that doesn't really need to be hidden away.

It has certainly become easier over the past few years to work across various platforms. Not even all that long ago, that wasn't the case. Maybe it's about time to start working on that consolidation and standardization.

(Oh, no. The New World Order is making inroads even here! )

Good evening all. I'm just about to go for food and fun for a while, so I figured I'd dump the progress I made today for anyone else wokring/playing with this:

2 bytes : year
1 byte : unknown
1 byte : month

That unknown is the "terrain", and corresponds to a look up table that appears to be as follows:

[ _
"METROPOLIS",
"RESIDENT'L",
"TOWN &CITY",
"FARMLANDS ",
"PASTURE ",
"OIL & COAL",
"TUNDRA ",
"DESERT ",
"MOUNTAINS ",
"WETLANDS ",
"FOREST ",
"RAINFOREST",
"COASTLANDS",
"OFFSHORE ",
"HIGH SEAS ",
"ISLANDS ",
"IN-FLIGHT ",
"SPACE ",
"MIL. BASE ",
"UNKNOWN ",
"ROAD+RAIL "
]

@@@@

1 byte : time
1 byte : unknown
1 byte : duration

That unknown is the YMDT column, the values like 3333 and 3331. it's stored as 4 lots of 2 bit values for the values 1-3

@@@@

2 bytes : latitude
1 byte : unknown
4 bytes : location

This unknown correlates well to the CON/CTR fields, although I'm not sure just how it works exactly yet. This appears to be another look up table, but I have not yet built one to match it.

@@@@

So, it's coming along well, just needs a couple more unknown bytes figured out, the CON/CTR to be solved fully, sanity checking and so on.

Isaac says he's going away for a couple of days, I think that will be enough to solve these issues.

I haven't been able to get the actual program to run myself, it fails under dosbox/dosemu, and I don't have a w95/w98 anymore. If worst comes to worst, I could install something like freedos on a VM, but it would be nice if anyone who's got this working could post a few more screenshots from the "examine data records" screen, like these that were posted

img39.imageshack.us...
img576.imageshack.us...

They were a great help to me.

On the subject of getting the full version done. if we can come to some kind of agreement that would be fine. When companies contract out work like this, they make the workers sign things like NDA/Destruction orders, so that they cannot disclose any details/data, and must destroy their copy once the work is done. So I think it's certainly possible within the law to do this, but I don't know fine details. I've stuck to what I know are legal methods in my locale so-far, and will continue to do so.

The full version database may be slightly different, but it's unlikely to be much different, just maybe a few extra fields, or a few spare bits utilised. The work so far is highly unlikely to be a waste of time.

I may be able to spend another hour or two later on this, if not I will be back tomorrow.

Originally posted by IsaacKoi
I've found a utility online called prn2file. Although it seems to install (and when I run it from the command prompt it tells me that printing is redirected from LPT1 to the specified filename), the file that is created is empty. Again, when I try to print from the DOS database or after running a readme.bat file, then nothing happens.

TSRs (terminate and stay resident) stopped working because Windows 95 did not let them remap interrupts. With a an older DOS version it will work, as in DosBox.

Originally posted by Radiobuzz
just press F8 while the PC is booting and choosing DOS from the menu there.

Ok, I'm back and (despite the quite remarkable progress being made by Harpsounds) tried Radiobuzz's suggestion to complete my attempt to print to a file. It only took a few minutes and I didn't want anyone doing a lot more work if it could be avoided.

Some good news and some bad news:

The good news : Radiobuzz's step-by-step instructions meant that I got prn2file to work and have used it to print (to a text file) in relation to the entire database.

The bad news: The print option only prints some of the fields. Probably about 60 or 70% of the fields are printed, but without the additional fields it limits the ability to search.

The first couple of examples are below in case anyone is interested:

1: -593/01/01 00:10 60 46:10:00 E 31:00:00 N 3111 MDE IRQ CHL 8:4
CHALDEA:EZEKIEL:FIERY SPHERE LANDS/4 SUPPORTS:TAKEN FOR A RIDE:see Bible acct.
Ref# 2 VALLEE:UFOS IN SPACE:Anatomy/phenon. Page No. 2 : PASTURE

2: -322/01/00 00:00 3 35:13:00 E 33:16:00 N 3111 MDE LEB TYR 9:6
TYRE =SUR,LEBANON:SIEGE/ALEXANDER the GREAT:FLYING SHIELD BEAMS^:WALLS CRUMBLE
Ref#160 MUFON UFO JOURNAL (by issue no.) Issue No. 320 : MIL. BASE

Basically, this gives a reference number, the date, the duration, the co-ordinates, a measure of the accuracy of the date information (which I'll explain in a separate post in a moment in reply to a post by Harpsounds), the region/country, strangeness and probability ratings using Hynek's system, text in relation to the sighting, a number for the reference source, the name of the author and relevant book, and the type of terrain.

This is useful in future-proofing quite a bit of the data in the database - but does not include (for example) all of the data in the matrix of attributes which appears on the top left of the screen.

I think we have now eliminated the "Export to ASCII" option in the database (since that only exports a few fields) and can see the limitations of the "print" option. However, I still have over 1,200 pages of reference material as a result of using this method (currently as text in a Microsoft Word document, but - if the method being used by Harpsounds does not work - it should be fairly quick and easy to put the at least the fields above into the format of a table and/or export it to a spreadsheet or database application).

So, I think we can now get back to concentrating on the approach being pursued by Harpsounds.

Onwards and upwards,

Isaac

Originally posted by harpsounds
I figured I'd dump the progress I made today for anyone else working/playing with this

You seem to be making quite remarkable progress, particularly given that you don't actually have a copy of even the demo working!

While I think you and I are both very reluctant (due to the copyright issues) for me to share the full software even for the current very limited purpose, I don't think there is any problem with sharing explanations of certain codes supplied in the accompanying documentation. Quite a bit appeared in the material accompanying the demo and/or in Larry's material online over the years.

That unknown is the "terrain", and corresponds to a look up table that appears to be as follows:

Larry's explanation was:

TERRAIN: (somewhat ambiguous) Use METROPOLIS for events within the
largest cities, TOWN for smaller cities/towns, Residential for
Suburbs and housing districts/compounds. Barracks are MILITARY.
Farms are FARMLANDS. Ranches, sheep stations etc. are PASTURE.
Generally these are indicate population density and/or predominant
land usage. OIL & COAL are petrochemical recovery or refining.

TUNDRA is any treeless, sparsely populated polar region.
DESERT is just that, unless its a CITY in the desert.
MOUNTAINS: Any mountainous area, except as noted above.
WETLANDS: Swamp, estuary, marsh .. except as noted above.
FOREST: Thickly wooded areas except as above.
RAINFOREST: Tropical jungles and forests with high precipitation.
COASTLANDS: Within a few miles of the shore, except as above.
OFFSHORE: Coastal waters .. within a few miles of shore.
HIGH SEAS: Away from land. Ships at sea.
ISLANDS: Smaller islands. NOT Britain, Australia, Taiwan..
IN-FLIGHT: Observer in aircraft, especially if buzzed or paced.
SPACE: Any UFO event beyond earth's atmosphere.
MILITARY: Any military Base, Field, installation, camp..
UNKNOWN: Environment cannot be determined from source data.
ROADS/RAILS: Highways (NOT streets) railways, generally between
towns, where none of the above clearly apply.

You said:

1 byte : time
1 byte : unknown
1 byte : duration

That unknown is the YMDT column, the values like 3333 and 3331. it's stored as 4 lots of 2 bit values for the values 1-3

This is a measure of the accuracy of the date information provided in other fields:

ACCURACY: For YEAR, MONTH, DAY and TIME, enter accuracy rating:
0 = DOES NOT APPLY/irrelevant. 1 = UNKNOWN/Unspecified.
2 = APRX: within 1 or 2 Years or Months; within 3-4 Days, Hours.
3 = KNOWN: EXACT year, month, day: Time within one hour.

You said:

2 bytes : latitude
1 byte : unknown
4 bytes : location

This unknown correlates well to the CON/CTR fields, although I'm not sure just how it works exactly yet. This appears to be another look up table, but I have not yet built one to match it.

If it is the CON/CTR these are (as you may have worked out) CONTINENT and COUNTRY fields. In case it provides any help with building a "look-up table" (new expression to me...), I'll send you a U2U with the relevant codes.

it would be nice if anyone who's got this working could post a few more screenshots from the "examine data records" screen, like these that were posted

I'm happy to provide a few, if someone can provide step-by-step instructions of the process of producing them within the virtual machine and exporting them - but it would almost certainly be simpler for Radiobuzz (who has already provided some) to be imposed upon to provide a few more. I'll send him a U2U later.

I've stuck to what I know are legal methods in my locale so-far, and will continue to do so.

Absolutely fine with me. I'm concerned about potential copyright issues as well.

The full version database may be slightly different, but it's unlikely to be much different, just maybe a few extra fields, or a few spare bits utilised. The work so far is highly unlikely to be a waste of time.

Exactly. There are a few extra bytes per record in the full database - I posted a brief comparison of the relevant data in a post above at:
www.abovetopsecret.com...

All the best,

Isaac


[edit on 2-7-2010 by IsaacKoi]

Sorry that it took me so long and that the print thing hasn't worked out... Here are more pics:








EDIT: BTW, to take these pics I just press the "prn scr" button on my keyboard, paste it on Paint and cut the screen .

[edit on 2-7-2010 by Radiobuzz]

reply to post by IsaacKoi


Once you get the data exported to a file of some sort, what do you plan to do with it?

it should be fairly quick and easy to put the at least the fields above into the format of a table and/or export it to a spreadsheet or database application).

In order to export your data to a modern spreadsheet or database engine you will need to put it in "comma delimited" file. The first line of text in that file should be your field names. All fields, expecially the date data will have to be in a standard date format (mm/dd/yyyy).

Your best hope for resurecting this application is to put it into a SQL based database engine (access ofr SQL server), and then build a search engine around the data.

Best of luck in your endevors ...