Cat and Sort command

I have a need for cat sort commands that can help sort unique lines and sort by unique number and letters.
I have run out of room on my disk to sort this 148gb file and am looking for suggestions to sort the file b numbers then alphabetical removing duplicate lines. I have used the sort commands individually but would like to combine them into one command that can do everything I need at one shot.

Yes I only have 250Gb Disks and am limited to 16Gb Ram.

What kind of database are you using to manage these files? Are you trying to do it with a file manager? If you want to get into sorting stuff, check out SQL. There are a few different versions of this Structured Query Language around. Starting with something like mySQL is an easier path into it.

a reply to: kwakakev

Flat flie database. It is mostly a word list. One file combined multiple dozens of .txt files into a gigantic .txt file

a reply to: kwakakev

The applications I use are direct to the file system so using a database like MySQL Isn't really an option. They read from .txt files directly sorted on a disk. Something like "Wifite --dict /Home/ghost/ *

Where i have combined word list like cat -u *.txt alpha.txt and another command like-n counts numerically then alphabetically unique structure.

What else can use and expect from sort and cat commands.

Maybe check out Native Classes if you are trying to share functionality between different applications and programming languages. Sounds like you have quite a job there trying to tidy up and make sense of all that data.

It reads like you are using the command line to perform all your functions. Have you got a good reference manual for your operating system that clearly defines just what functions and options are available. To tidy the data up it will help being able to program. If you stick to the same language the operating system and cat sort is in it it will get all this sorted how you want in one shot. Likely C++.

If this is the first time doing something like this, it will take a while to work it all out exactly how you want.

a reply to: shaneslaughta

Sorting by unique characters sounds like you might want to cat the file and pipe it through a grep command that outputs to the sort command.

Nawk and sed can be very useful commands to manipulate the data.

Usually in Unix / Linux, there are various ways to herd the data processing cat; a lot of it boils down to the preferences of which tools one chooses to use.

The uniq command can be used to get rid of repeating instances while providing a count of how many times they occurred.

Cheers

originally posted by: shaneslaughta
I have a need for cat sort commands that can help sort unique lines and sort by unique number and letters.
I have run out of room on my disk to sort this 148gb file and am looking for suggestions to sort the file b numbers then alphabetical removing duplicate lines. I have used the sort commands individually but would like to combine them into one command that can do everything I need at one shot.

Yes I only have 250Gb Disks and am limited to 16Gb Ram.

In this tutorial you will learn:
How to remove duplicate lines from file when sorting
How to count the number of duplicate lines in a file
How to remove duplicate lines without sorting the file

linuxconfig.org...

If you have the individual commands already, just pipe them to the next.

command | command | command

a reply to: shaneslaughta

If you're trying to do it in a text editor, I've used UltraEdit in the past...not sure if there's a file size limit though.

Here's some text editor tools:
Text Editors

You may also want to see if a cloud based NoSQL tool can do what you want...or create a cloud db instance, import the data and manipulate it however you want. While there will be a cost, it'll only be while the vm or db is active.

Thanks everyone. I appreciate the input.
Seems the most efficient way for me at this point is piping commands into each other using stdout.
I also created a small raid 0 array to take advantage of striping speed increases and am looking into multi threaded applications to take advantage of extra clock cycles that seem to not get utilized on all cores during certain command executions.

I also noticed that now that I have combined multiple text files together I have somehow added a line count that is being counted as the data in my application and needless to say is polluting the data. So now my new goal is to remove the line count.