Digital Storage Dangers - Digital Dust Could Ruin Your Files!

I am just going to make you folk who care about music or file storage in general aware about a phenomenon which could ruin your stored files.

This is caused by a 1 going turning into a zero spontaneously or due to environmental factors like temperature entropy generation.

I have a PhD in Digital Music Conservation from the University of Texas at Austin. I have to stress that the phenomenon known as "digital dust" is the real problem regarding conservation of music, and any other type of digital file.

Digital files are stored in digital filing cabinets called "directories" which are prone to "digital dust" - slight bit alterations that happen now or then.

Now, admittedly, in its ideal, pristine condition, a piece of musical work encoded in FLAC format contains more information than the same piece encoded in MP3, however, as the FLAC file is bigger, it accumulates, in fact, MORE digital dust than the MP3 file.

Now you might say that the density of dust is the same. That would be a naive view. Since MP3 files are smaller, they can be much more easily stacked together and held in "drawers" called archive files (Zip, Rar, Lha, etc.) ; in such a configuration, their surface-to-volume ratio is minimized. Thus, they accumulate LESS digital dust and thus decay at a much slower rate than FLACs.

All this is well-known in academia, alas the ignorant hordes just think that because it's bigger, it must be better.

So over the past months there's been some discussion about the merits of lossy compression and the rotational velocity density issue. I'm an audiophile myself and posses a vast collection of uncompressed audio files, but I do want to assure the casual low-bitrate users that their music library is partially quite safe.

Being an audio engineer for over 18 years, I'm going to let you in on a little secret. While rotational velocity density is indeed responsible for some deterioration of an unanchored file, there's a simple way of preventing this. Better still, there have been some reported cases of damaged files repairing themselves, although marginally so (about 1.7 percent for the .ogg format).

The procedure is, although effective, rather unorthodox. Rotational velocity density, as known only affects compressed files, i.e. files who's anchoring has been damaged during compression procedures.

Simply mounting your hard disk upside down enables centripetal forces to cancel out the rotational ruptures in the disk. As I said, unorthodox, and mainstream manufactures will not approve as it hurts sales (less rotational velocity density damage means a slighter chance of disk failure.)

Also slower hard drives like 5400 suffer less due to a lower rotational speed.

I'd still go with uncompressed .wav myself, but there's nothing wrong with compressed formats like flac or mp3 when you treat your hardware right.

Now digital dust could make your compressed files unreadable and even uncompressed files flawed as the decoders stutters or stumbles over the digital dust.

In other words the files even on your hard drive are subject to it including flash memory types and may explain the high failure rate of flash memory to spontaneous flipping of a 1 to a zero and vice versa.

I know error correction but this limited and the cascade effect has turned many flash memory computers useless as well as loss of stored data.It is little known that flash memory is less reliable even than hard disks,some only lasting a year up to 18 months before failing.

What this means is digital files will suffer storage losses just like paper files suffer from decay.

So....This is the reason my Beatles White album plays backwards at random times?



Thank you for the information. Time to back up all my back-ups!

This may be the most informative topic on ATS in ages.

Quite interesting. Any sourcing for this?

All my files are flac and wav. I've never had any of this happen to me personally, but it doesn't really come as a surprise either. Consumer available digital storage media hasn't matured yet.

I'm not so sure I buy the upside down hard drive trick, but slower rpm hard drives do last longer usually, and have greater stability. There's always a trade-off when it comes to speed.

Great post OP.

To keep it simple for everyone else, this is basically entropy at work.

EVERYTHING decays over time, so the magnetic domains on a medium such as disk will decay over time, or be interfered with by another physical process, and the atom that was positioned to represent a 0 will lose it's magnetic potential causing it to break free from it's alignment but not it's atomic bond. That's when it will become a 1 or vice-versa.

No magnetic media is safe from this, but CDs and DVDs are not prone to it.

~Namaste

reply to post by SonOfTheLawOfOne

No magnetic media is safe from this, but CDs and DVDs are not prone to it.

Good point. But cheap cd/dvd media can be prone to data loss. This is why archival quality media is best for long term storage. Japanese media comes to mind. Of course the writing process is also part of the equation. We don't see as many failed copies as we used to, because the quality of writers is better.

Originally posted by nobodysavedme
This is caused by a 1 going turning into a zero spontaneously or due to environmental factors like temperature entropy generation.

Does it also happen when a zero turns into a 1?

I have a PhD in Digital Music Conservation from the University of Texas at Austin.

Good for you.

Digital files are stored in digital filing cabinets called "directories" which are prone to "digital dust" - slight bit alterations that happen now or then.

Never happened to me, and I have (working) disks more than 10 years old.

Now you might say that the density of dust is the same.

No, I wouldn't say that, as I don't know what's your definition of "density of dust".

That would be a naive view. Since MP3 files are smaller, they can be much more easily stacked together and held in "drawers" called archive files (Zip, Rar, Lha, etc.) ; in such a configuration, their surface-to-volume ratio is minimized.

MP3 files do not compress much when archived as zip or rar. And what do you mean by "surface-to-volume ratio"?

All this is well-known in academia, alas the ignorant hordes just think that because it's bigger, it must be better.

Forgive us our ignorance.

So over the past months there's been some discussion about the merits of lossy compression and the rotational velocity density issue.

What's "rotational velocity density"?

Being an audio engineer for over 18 years, I'm going to let you in on a little secret. While rotational velocity density is indeed responsible for some deterioration of an unanchored file, there's a simple way of preventing this.

What's an "unanchored file"?

Better still, there have been some reported cases of damaged files repairing themselves, although marginally so (about 1.7 percent for the .ogg format).

Smart files...

The procedure is, although effective, rather unorthodox. Rotational velocity density, as known only affects compressed files, i.e. files who's anchoring has been damaged during compression procedures.

I could agree with you if you could explain all those words, like "rotational velocity density" and "anchored files".

Simply mounting your hard disk upside down enables centripetal forces to cancel out the rotational ruptures in the disk.

What are those "rotational ruptures"?

It is little known that flash memory is less reliable even than hard disks,some only lasting a year up to 18 months before failing.

Only the ignorant hordes ignore that.

What this means is digital files will suffer storage losses just like paper files suffer from decay.

You could use something like parity files.

reply to post by nobodysavedme


Could I ask maybe why part of this post is an exact copy of another post on a different forum, but by someone claiming to be from the university of South Carolina?

Not sure if I would get t and c'd for posting the link or not, any mod advice armap?

reply to post by nobodysavedme

I am not sure this is a hoax thread, as what you are talking about is based on some old hoax that is not true at all.
If there is no bits that change and crc is okay then there will be no degradation in quality at all.
There wont be any errors unless the media has degraded, which is why it is recommended to transfer it over to new media from time to time or have several backups to make sure this does not happen.

If you do this then the music should be just as intact even in a century.

So you are either a little troll or you fell for a hoax, which is kinda weird since you are supposed to be a audio engineer.

reply to post by juleol


I think you may be right on this. "Digital dust" usually refers to extraneous data that is unusable, but was saved along with usable data. There's more to it, but that's the simple definition.

Also, I don't know any sound techs that refer to this specifically. We all talk about data loss, but flipping 1's and 0's? Meh, maybe, but unlikely without influence of some kind. And turning a hard drive upside down these days makes no difference. In the old days, yes. But not now.

I thought this was an april fools post, until i looked up the date posted. What on earth are you talking about?

Modern file systems have multiple layers of error correction, for instance hash checksums for every file and copy-on-write. If a bit-flip you are talking about would happen, they immediately notice this and correct it.

Also, most modern audio formats have inbuilt error correction as well, which can be used to restore the original file.

If you google it, you can find how many % of an audio CD is error correction data, since read errors happen a lot with optical media. Even more so in digital TV there is a lot of error correction going on where the original information can be recreated even if bits (or even packets) are lost.

If you are using some ancient file system from decade(s) ago, i am not sure how stuff works out (like Windows NTFS), that also has some kind of checkdisk functionality that can recover damaged files, but i am not aware to what extent.

The only thing i can see as a hazard for your stored data is using severely outdated technology or disk failure. The way to protect you from this is to use RAID1. Nowadays RAID can be done at a file system level, so you do not even have to worry about setting up RAID volumes and stuff, like in the old days (and under Windows today)

Although this sounds like an April Fool's post, I'll give it the benefit of the doubt for two reasons: It's several days too late, and I know from experience that stored files deteriorate.

I would request a little more information on how it matters what plane the physical drive operates in. And, of course, some description of exactly what "rotational velocity density" is.

Still, it's good to have this for reference and educational purposes--assuming it doesn't turn out to be a hoax....

reply to post by nobodysavedme


Is this for real?

How come i cant find any info on 'digital dust' anywhere on the net? Do you have any sources for the info you are presenting?

Originally posted by PhoenixOD
reply to post by nobodysavedme

 

How come i cant find any info on 'digital dust' anywhere on the net? Do you have any sources for the info you are presenting?

The same post with the exact same wording appears all over the web, google
"Digital files are stored in digital filing cabinets called" or any other line from the post with quotes and you will find at least a dozen or so exact copies the only difference being the university the poster claims to have studied at

reply to post by davespanners


Ok thanks. But when i search for 'digital dust' there seems to be nothing. At least nothing in the context of what the OP is talking about.

Im not even sure there is a PhD in Digital Music Conservation.

Originally posted by PhoenixOD
reply to post by davespanners

 

OK. But when i search for 'digital dust' there seems to be nothing.

I would assume that is because it isn't true, as far as I can see it's just a hoax email / post that floats around the web getting re posted over and over

Well, I just went to the UT Austin website. I don't see anything resembling a "PhD in Digital Music Conservation," even taking into account any similar wording or ambiguous concepts. Taken together with the results other posters have gotten in their searches, I withdraw my above conclusion.

In other words: Seems to be a copycat-type hoax....

reply to post by davespanners


I agree with you there, this seems total BS.

reply to post by PhoenixOD

So, what happens to these massive blocks of data? Well the majority of it sits on servers, collecting digital dust. Even after the data is run through analytical software that groups commonalities and segments it into useful information, most of the data goes to waste. Is there a way companies can reduce “data waste”?

blog.eloqua.com...

It has little to do with the OP's definition. But as I said above, this is what I refer to as digital dust.

Forgot to add. Rotational velocity refers to the speed the platters inside the hard drive spin. While density refers to storage capacity. I've never heard these three words used together before though.

reply to post by Klassified


Yeah i found reference to digital dust also but its not a phrase used in the same context as the OP was using it

Digital Dust , digital dust meaning , definition of digital dust , what is digital dust - When you download a song on your ipod or computer and listen to it for like a week and never again, but leave it on there anyway.
Some examples : I really used to like Three Days Grace, but now all their songs are just collecting digital dust on my ipod.

Seems it just means data that lays around unused forever.

I've kept all of my MP3 files since the days when they had to be uncompressed in DOS since there was no GUI for the decoder yet and I have not had a single file become corrupt. What are the chances of this "digital dust" occurring?