Huge UFO Database in DOS - Someone's labour of love, can it be saved?

a reply to: ArMaP

Ok great.

Now does you or anyone have something to complain about the output structure. I mean should some columns be renamed, re-ordered, split in multiple ones or merged, should some values re-formatted another way, etc. ?

Still searching how to decode the remaining bits in the meantime... (any help appreciated)

originally posted by: javarome
a reply to: ArMaP

Ok great.

Now does you or anyone have something to complain about the output structure. I mean should some columns be renamed, re-ordered, split in multiple ones or merged, should some values re-formatted another way, etc. ?

Still searching how to decode the remaining bits in the meantime... (any help appreciated)

No complaints, it looks great!

I note that you are resolving the link reference for external look up tables to the actual data itself, which makes it more versatile as a stand-alone dataset. I think you're right not to do this for the "ref" field; a new text field x 18,000 records would significantly push up the file size.

Those remaining bits are a pain in the arse, it must be said

originally posted by: javarome
a reply to: ArMaP

Ok great.

Now does you or anyone have something to complain about the output structure.

No, I think it's good.

Hi there,

I've published an online version which can be queried.
It's not optimized as Bob plans to do with a real database ; it just loads the binary file in memory (which can take a few seconds) then allows to query it from memory.
For now the query syntax is the same as the uDB command-line tool : a set of field=value expressions, which can be joined by '&' or '|' for "and" and "or" restrictions, respectively.

a reply to: javarome

Nice, but shouldn't there be a horizontal scroll bar?

originally posted by: javarome
Hi there,

I've published an online version which can be queried.
It's not optimized as Bob plans to do with a real database ; it just loads the binary file in memory (which can take a few seconds) then allows to query it from memory.
For now the query syntax is the same as the uDB command-line tool : a set of field=value expressions, which can be joined by '&' or '|' for "and" and "or" restrictions, respectively.

Looking good. I think it is very handy to have an easily-distributed version.

As far as progress, I've got a (local) mysql database in place and I'm currently playing around with queries and ways to make the search forms understandable, while retaining the flexibility of the original. The search can potentially get quite complex and is not very user friendly yet. At this point it's a UI issue more than a data issue, but I suspect this will feed back into to another revision of the SQL queries.

I've also finished my reverse geocoding script, but this will be a last step and entirely "additional" to the main data. It will provide a location reference in line with the modern ISO codes. A limitation of this will be incidents located at sea, which won't return data from the google api. I'll cross that bridge when I get to it.

originally posted by: ArMaP
a reply to: javarome

Nice, but shouldn't there be a horizontal scroll bar?

A scroll what ?
No kidding, I can see one.

Hi there,

Just to let you know about some updates of the online query form:

- More operators are now supported, so that you can now ask for crebility>5 for instance.
- The case sources display have been improved. Notably, some sources numbers were actually referring to secondary tables (periodicals, blogs, etc.) which are now correctly listed. So sources descriptions are now displayed instead of just reference numbers.

originally posted by: javarome
Hi there,

Just to let you know about some updates of the online query form:

- More operators are now supported, so that you can now ask for crebility>5 for instance.
- The case sources display have been improved. Notably, some sources numbers were actually referring to secondary tables (periodicals, blogs, etc.) which are now correctly listed. So sources descriptions are now displayed instead of just reference numbers.

Excellent. Any chance of a rundown of how you've programmatically made that distinction? I'll be honest, I don't recall coming across any records that directly reference anything but the primary reference table.

a reply to: EvillerBob

Hi Bob,

Any chance of a rundown of how you've programmatically made that distinction?

Sure! I was still searching a clue about where to find the page versus volume/issue distinction info, and suddenly remembered the sources text file: it does contain a classification of sources (primary, periodicals, blogs, etc.)! So I hoped that source references could be classified by reading this but... no, the source ref numbers were all primary ones, as far as I found.

Anyway, looking at the classification, I found that some sections were beginning by "Ref. xxx = some description". For example "Ref. 093 = NEWPAPERS and FOOTNOTES.", "Ref. 096 = OTHER DATABASES and WEBSITE addresses.", etc. That meant that when primary ref was 93, refIndex was actually a secondary reference pointer to the "NEWPAPERS and FOOTNOTES" list. And so on.

Programmatically speaking, see here.

originally posted by: javarome
a reply to: EvillerBob

Hi Bob,

Any chance of a rundown of how you've programmatically made that distinction?

Sure! I was still searching a clue about where to find the page versus volume/issue distinction info, and suddenly remembered the sources text file: it does contain a classification of sources (primary, periodicals, blogs, etc.)! So I hoped that source references could be classified by reading this but... no, the source ref numbers were all primary ones, as far as I found.

Anyway, looking at the classification, I found that some sections were beginning by "Ref. xxx = some description". For example "Ref. 093 = NEWPAPERS and FOOTNOTES.", "Ref. 096 = OTHER DATABASES and WEBSITE addresses.", etc. That meant that when primary ref was 93, refIndex was actually a secondary reference pointer to the "NEWPAPERS and FOOTNOTES" list. And so on.

Programmatically speaking, see here.

Well spotted!

On another note, I've not abandoned the project, I just have a bigger real-world one I need to complete for money within the next two weeks - which my wife insists should take priority

Any news?

originally posted by: ArMaP
Any news?

God, I hope not.

Abra, Kadabra, Alakazam, by the powers not invested in me, may a dead thread stay dead.

Save for the following criteria being met: If a thread is older than one year, can subsequent resurrection replies be required to provide an outline summary of the entire thread up to that point?

I have no idea how far this project got, and would love to help, but I'm not inclined to read 10 pages of dead project. If you help me out, I'll fire up DOSbox and see if I can help you out.

Must have missed this back in the day.
Javarome's database is still running.

originally posted by: ArMaP
Any news?

I left it be as I thought javarome beat me to the punch.

I just checked the website I set up and it doesn't exist anymore?!? Thought I set the domain to autorenew. It was simple but functional.

Hi all, some of us members of SCU were wondering-
What is the Latest on the availability status of the Hatch database?
particularly I'm trying to search descriptions for statements of witness emotional state. Great work I see from many board members.
Thanks in advance