Understanding hard disk's and how partitions/file tables work

Ok, guys this is pretty much required reading if you ever want to be able to install Linux and understand what your doing.
I know its dry stuff so I'll try to keep it short.

How a hard disk works(NOT SSD) -- You'll find lots of different explanations, most are correct, just different ways of thinking about it.
A hard disk has platters that look much like a cd, but smaller, they are usually double sided. There is a head much like the one on old record players that moves back and forth(straight and back), while the platter spins. The head essentially magnetizes(1) or de-magnetizes(0) a small small portion of the disk to write data. This could get a lot more technical, but I'm a software guy so I can't honestly delve into the specifics, but for our purposes this works.

What are partitions?
We've learned how a hard disk works. Now lets imagine a hard disk like a room. We can partition that room into several more rooms. It's the same with hard disks. On the first(or couple) of the hard disk is the information for how a hard disk is partitioned. ie Partition one starts at sector 5 and goes to sector 10912121, partition 2 starts at 10912122(one sector ahead of the last partition) and ends at 30092322. When you look in Windows you will see these partition labels as C: and E: (the next available alphabet). So partitioning is just separating out areas of the disk which Windows actually looks at like separate disks. So you could have just a C: partition and save say 100gb of data for a Linux partition in the future. It is also possible to resize a Windows partition. Windows 7 and above typically have 2 partitions one is the boot partition, that is hidden and the other is the system partition, which is visible as C:. In Windows you can view your partitions in two ways. Go to My Computer and right click, then click on Manage. Then click on Storage and Disk Management. Those are your hard disks/partitions. Here you can resize your partitions for a future Linux install. I'd recommend at least 50GB for Linux.

How does the Operating System know about files?
When the computer looks at a hard disk, it only see's block's(boxes) of data. It knows nothing about file names or files. The operating system creates something that is called the File Allocation Table(remember FAT?). This starts shortly after the first sector on the partition. Using the above example, partition 1 started at sector 5, so the FAT would start at sector 6 or 7, or very close. The FAT table can be thought of as a table of contents. It literally states that file "Josh.txt" starts at sector 20.
Example
Josh.txt 10
John.txt 20
Win.ini 30
You get the idea. The operating system uses this table to identify files and where they start. When the operating system reads a file, it reads until it hits a "0". Also when a file takes up more than one box, the end of the sector will tell the operating system which sector is the continuation of the file until the last sector of that file is read. That is why the FAT table doesn't contain an end sector. It should be noted that different filesystems call this table different things. Master File Table for NTFS. FAT for FAT16/32 and exFAT. It should be mentioned that most filesystems have two copies of their table of contents just case one gets corrupted. Again things can get much more complicated here, but this gives you inside glimpse of how the computer see's things. So essentially a filesystem is a way for the operating system to manage where files are and give them names.

What happens when I delete a file?
When you delete a file the operating system crosses it's name off the table of contents. It doesn't actually erase the data. If we delete Josh.txt and then asked for data at sector 20, the information that Josh.txt had inside of it, is still there, that is why data recovery is possible. Now when files are copied back to the system there is a good chance sector 10 will be overwritten with the new data, at that point Josh.txt is gone. So if you accidentally delete a file that you need, its best to turn off the computer, get a copy Linux on a boot cd with testdisk on it, and immediately start the recovery process to another hard disk. If you wait and copy new files to your computer, you could overwrite your accidentally deleted file. This is also the reason file shredders are used. They will go to the sector that the file is on and overwrite it several times to make sure that data is not recoverable without seriously expensive tools.

What is defragmenting or fragmentation?
Imagine John.txt takes up 1 box(sector) and to the left and right of John.txt is Josh.txt and Win.ini. Now lets delete John.txt. There is an empty box(s) left. Now lets say we have a file that is twice as big as John.txt. The operating system copies a portion of the file to the old John.txt(sector 20) and the rest to sector 185(next free box). Now the file is fragmented because it is not in consecutive order on the disk. As you can imagine the old record player, it slows down reading the file because the drive has to make at least 2 passes(circle). One pass to get sector 20 and one pass to move the head to the correct location for sector 185. Now imagine this happening hundreds of times. Now you have a badly fragmented filesystem. The head has to reposition it self several times just to get one file, but if a file where on sector 2,3, and 4 the disk head could read all the data in one pass. Making it work less and read faster. The same goes for writing a file. So when you defragment the software looks for free space and moves everything it can there, then it slowly rearranges the files so all their boxes are right next to each other. Much like you would organize a room. If like items aren't together than you have to walk across the room.

So a typical Windows partition setup looks like this:
Boot 350MB - Hidden - Contains the Windows boot files. It is hidden so that you can't accidentally delete essential files to booting.
System Drive C: - This drive contains all Windows files, Programs, and Documents. It is always the C: drive except in rare cases where someone has reinstalled and made a mistake.

A typical Linux partition setup looks like this:
/boot - 500MB or less.
/ - this is the system partition which contain all applications and system files. It varies in size but is typically 50+GB
/home - this partition contains all user files.
Now I know I just confused you because Windows referred to partitions as C: D: E: etc. Linux doesn't do that. It refers to partitions as / or commonly called ROOT. Partitions are mounted in Linux(possible in Windows to). I'll do up a second tutorial on how to understand Linux filesystem/partitions.

A typical Windows + Linux setup
Boot - 350MB - Hidden Windows boot partition
C: - 100GB - System partition
/boot - 500MB - Linux kernel and boot files
/ -50gb - Linux system files
/home - 100gb - your files

A boot loader such as Grub would be used to choose Linux or Windows.

So you can see that partitions are the trick to understanding how to get Linux and Windows to play together.

Hopefully you have a better understanding of Partitions and how the computer looks at files now. For the techies there are lots of stuff I didn't address I'm not addressing this to you guy, I'm trying to make it short and understandable for those who don't already know.

I'll throw up a Linux+Virtualbox tutorial next to get people comfortable in Linux.

a reply to: lobograndemalo

I must be really tired. When I saw this thread I was thinking, "Huh. I was just talking to someone about this on another thread." It was this one: www.abovetopsecret.com...

Then, I went over to the other thread and saw that you wrote it. I had a really good laugh at myself over that one. Thanks for writing this. I'll be looking forward to your next tutorial.

a reply to: lobograndemalo
Star and flag for the excellent, thorough information. Too many people are far too reliant on computers and technology, without having the slightest inkling of how they actually function. This information is not new to me, but it's always good to have a refresher. I look forward to more from you.

originally posted by: Skid Mark
a reply to: lobograndemalo

I must be really tired. When I saw this thread I was thinking, "Huh. I was just talking to someone about this on another thread." It was this one: www.abovetopsecret.com...

Then, I went over to the other thread and saw that you wrote it. I had a really good laugh at myself over that one. Thanks for writing this. I'll be looking forward to your next tutorial.

Were you thinking about your own thread on freaky coincidences when you say the topic!

Good OP btw.

Just to add

click click

click click

click click

Thats the click of death, get new hard drive straight away and copy your old data emediately!!!

Its always fascinating to understand the basics of computers. Essentially, they move, read and sort long streams of ones and zeros which are nothing more than magnetic spots and the spaces between them on media such as a hard drive surface.

The magnetic head places, and reads these tiny magnetic spots (positive charge) and the spaces between them (no charge) as "Ones" and "zeros".

An analogy is a light switch. On is a one and off is a zero. On an oscilloscope the one is a peak and the zero is an off on a digital sine wave.

In magnetic terms (on the hard drive surface) the positive magnetic spot or charge is a one and the space between the charges is a zero.

Computers read these strings of ones and zeros and translate them to higher and higher languages (from machine language to code to ascii and finally the letters appearing on your screen.Your input by keyboard (human interface) translates all the way back down to inside the computer hard drive and chips as the presence and absence of these little magnetic charges and absence of charges (1's and 0's) that we call "Data".

The streaming data and the speeds at which this data is read, moved and stored by the computer is like people flipping wall light switches off and on billions of times a second.

Dumb as dirt. They will never know that they know.

I have been out of the field for a while, I could have skewed that a bit.

a reply to: lobograndemalo

Can Linux run google, you tube, games etc. Im wondering what practical use it has to the average computer user.

a reply to: lobograndemalo

Nice Tutorial Lobo! Thanx for this!!

Thanks guys!

These are all theoretical concepts, so if you don't get my explanation check out someone else's.

Many (I think it was almost 23) years ago, knowing how FAT works allowed me to recover a school work my sister had on a floppy disk.

I just changed the FAT so it looked like the whole floppy had only one file, from the first block to the last, and that allowed the computer to read that disk-sized file with all the other files inside it.

As my sister's work was in several text files it was easy to find them and get them back on their original individual files.

Knowledge is always a good thing.

a reply to: ArMaP

Perfect example of using knowledge and hex editor/debug back in the day!

originally posted by: AthlonSavage
a reply to: lobograndemalo

Can Linux run google, you tube, games etc. Im wondering what practical use it has to the average computer user.

It can run most the same as windowz. Google youtube etc are run by the browser and there's plenty of browsers for linux, the version I use comes bundled with firefox.

"practical use it has to the average"
Its FREE.
It doesnt report to microsoft every time its switched on!
Its far more efficient at doing most things the average user may want to do.
Hardware just works when plugged in! On windows that usually involves finding and installing drivers...that may or may not work!
Install Wine and it can run a lot of windows software.
Thousands of free and tested, adware and spyware free apps!

The only possible negative (at the moment) is gaming, but thats improving by the day and now even the game makers are getting on board and programing for linux.

For anyone considering linux, download Linux Mint. Install it onto a usb stick and you can test it without altering anything on your existing pc. All the details are easilly found via google.

a reply to: VoidHawk

You're getting crazy with the Linux zealotry man. Linux is cool man, but it isn't for everyone. It's certainly for the more technically inclined. Most older hardware does just work, but often times newer hard doesn't and laptops can be a real pain to get everything working properly. Which is much harder to fix than just Googling a driver for Windows. It's great that you love Linux so much man, but please don't misrepresent things. When you work in the IT Field you don't do that because you'll have to support your own recommendations, which can get very very time consuming if you put everyone on Linux. You quickly learn that Mac is your friend when recommending computers for people otherwise you'll spend every weekend holding their hand. For those who are a little more savy Windows is always my recommendation, and for those who are hardcore techies Linux is always the way to go. A good word of advise is: There's a time and place for everything, but not everyone.

Linux is for technically inclined computer nerds. Always has been always will be. The day that's changed is the day Linux will die. So be careful what you wish for.

originally posted by: lobograndemalo
a reply to: VoidHawk

You're getting crazy with the Linux zealotry man. Linux is cool man, but it isn't for everyone. It's certainly for the more technically inclined. Most older hardware does just work, but often times newer hard doesn't and laptops can be a real pain to get everything working properly. Which is much harder to fix than just Googling a driver for Windows. It's great that you love Linux so much man, but please don't misrepresent things. When you work in the IT Field you don't do that because you'll have to support your own recommendations, which can get very very time consuming if you put everyone on Linux. You quickly learn that Mac is your friend when recommending computers for people otherwise you'll spend every weekend holding their hand. For those who are a little more savy Windows is always my recommendation, and for those who are hardcore techies Linux is always the way to go. A good word of advise is: There's a time and place for everything, but not everyone.

Linux is for technically inclined computer nerds. Always has been always will be. The day that's changed is the day Linux will die. So be careful what you wish for.

I'm sorry but I dissagree!
For the average person distro's such as Mint are just as simple to use as windows.
One click brings up a window listing 70 thousand+ free and tested apps! Just click an app and it installs in seconds! No payment! No adware! and virus free!

It comes with the following already installed: Libre office (easily rivals ms office) that many corps have now switched to, email software, video editing software, music software, graphics software, Internet sofware etc etc..
With windowz you might get very basic versions of SOME of those, or versions that stop working after 30 days and then request payment!!!

Linux installs happily on most environments, be it an old desktop or a win8 laptop, I've installed on most with no problems at all. Maybe the problems you are having are of your own making!

It was people making comments such as yours that kept me away from linux for so long, but now that I've tried it I see such comments have no grounding in reality!

Also, you'll notice I suggested people install it to a usb stick as that would allow them to try it out without making ANY alterations to their existing setup, so to accuse me of giving bad advice seems a little bit suspicious, especialy when you admit that your in the IT business! Whats the matter dude, worried people wont need your expensive services if they switch to Linux?

a reply to: VoidHawk

Voidhawk you can't be serous. Yes I've been doing it for almost 20 years. I've done all sides from helpdesk to enterprise network administrator. So I'm quite familiar with OS's from Novel, to NT, to Irix, Sun Spark etc. At one time or another I have worked on most of the systems out there. I even worked on a Cray supercomputer in the 1990's. No I'm not afraid that anyone won't need my services after switching to Linux. The truth is I would make even more money as a consultant if people switched to Linux. Even you know that. You grossly over estimate the population. As someone who works in the IT field for non IT people I didn't give you my opinion in my post I gave you the facts. Fact is partitions are to complicated for you average people. Hell files are hard enough for them to understand and if it weren't for Windows autosaving files in My Documents they would have no idea where they where. If I could transfer most of my clients over to Linux I would A. save them a ton of money and B. be able to pocket more money, because I could charge more for my services and still be cheaper than say purchasing a Windows Server which is $600+ not counting CAL's. But for lots of reasons that won't work. #1 being that customized software for most businesses simply won't run on Linux. Linux is great for tech's and some advanced power users, but recommend it to everyone, no. It's fine if you disagree, but you can't argue with the facts Linux is more complicated than any Windows system. If this weren't so, why are so many PC's running Windows? I guess because of comments like mine.

Your trolling my threads.....