Domestic Surveillance Technology: Can they really watch everything?

I'm really curious about this topic. It both frightens and intrigues me. I'm hoping some ATS Geeks chime in here and school me, with supporting facts.

I'm dubious on if it's even possible. The fantasy I suspect, is being pushed on us in our current media. Every TV show or movie I watch; some agent is doing miracles with computer networks. I'm finding it all a little hard to swallow. And myself being so far down the learning curve, the aspect of actually researching and understanding if such networks are possible would take me the better part of a decade. So here is where I tap the geeks, the coders, the network admins, and the professors... are these systems even possible?

Let's start with telephones/cellphones. While I don't know the exact number of people that are talking on a cell phone this very second, I'm willing to assume it's somewhere in the low millions. I'm told there's a system, a great big dreamy Echelon-like mother ship somewhere that's monitoring every word spoken into every phone and is cross referencing those words with 'flagged' words.

The computer power required just to convert the speech to text for this claimed task would be astronomical. From what I understand, the current (publicly researched) speech-to-text programs take hours/days to properly calibrate to one specific voice. Also worth mentioning, most of them are bulky resource hogs (~400meg RAM from quick google research). Could the government/military have honed in the programming on such a complex (and again I note, publicly researched) program? And even if they did, could they have the computing power to run millions of different processes at once, and then have computing power to actually screen each of the translated words (or dare I say, phrases) with a database full of flagged words/phrases? I find it hard to imagine, short of a super computer stationed at every cellphone tower in the nation (which I'm pretty sure, isn't the case), that they can't possibly monitor every spoken word muttered into a cellphone. Let alone, every word spoken near a cellphone, as some would believe.

I do want to note that my knowledge of just how powerful a super computer could be, is lacking.. so maybe someone can fill this hole with some numbers. Also I'm not denying the tech isn't there to monitor specific groups. I'm denying the notion that they're "always listening!" both on and off the phone.

Cue the Internet. This is where my lack of knowledge of exactly how "things work" on a computer leaves me looking pretty dumbfounded. I started poking around and did some reading into the Internet Archive. 100 Terabytes of stored webpages. And grows at a rate of roughly 70 megs per second. But after doing more reading, I discovered the exclusions and suddenly the numbers didn't look so big. Then I found a groundbreaking new study which forecasts that a staggering 988 billion Gigabytes (988 exabytes) of digital information will have been created by 2010. Still no number for exact data amount per day/hour/sec but I'm assuming it's pretty large. I need some geeks here to tell me if it's even possible to crawl all the websites, irc channels, p2p, online gaming, etc.. ALL the traffic, at once.. and then cross reference all that with flagged words. I don't think it's possible. Not in real-time; and probably not even in week/month-time.

I'm a skeptic. I keep hearing people referencing these massive data mining/surveillance systems, yet I can't fathom to believe it's possible with our current technology. Perhaps I'm much farther down the learning curve then I imagined.. but from what I can research, and even with a little imagination; I can't swallow it. Any of it. Aside from specific targets, I'm pretty sure if I utter threatening revolutionary words into my cellphone; I'll be just fine (Don't text though!). As for the internet; that's a much bigger beast and what you write doesn't necessarily go away.. but I'm sure without word-of-mouth; it would take them a very long time, if ever (depending on location), to find my naughty words.

So do you buy it? Do you buy into the echelon-ike surveillance, or do you think it's just all fairy tales and popcorn?

It's been said before here, but social networking sites are a real danger to privacy.

Most people blindly upload their profile, name address, all of it, without realizing that this is public information, anyone can view it, including government, law enforcement.

You're just giving them everything they could ever want to know about you.

Theres no reason to be skeptical about it; yes they CAN WATCH YOU

it's not even a secret that they are!

However I don't think this means that they are watching every single thing at all times. Complete omniscience is something only God is capable of...

But they have THE ABILITY to watch you no matter what you are doing. But that doesnt mean they are 24/7!

You have to be a practical target. Anyone can find out someone's IP address and spy on them, it isn't very hard. But why would they want to waste their time? Give someone a reason to watch you, and they will.

Consider how many government agencies, corporate entities, and "special
interests" are in the US alone? They ALL have some sort of surveillance system of collecting information about the common man. Most of it is not of malevolant intent... Or is it?

Now consider that the web is WORLDWIDE... Meaning that every country out there has groups that are not only spying on their own peoples, but also are keeping an eye on the others!

In this day and age, the real wars are fought in the silent domain of INFORMATION!!

Finally consider that a global website such as this one, containing tons of information that many groups would want to keep hidden... Is full of spies, disinfo agents, and people with agendas from all over the globe...

using cellphones for surveillance: positiv
computers and email and messengers: positiv
TV: negativ
email reading: positiv
voice recognition: positiv and quick and easy now
listenning cellphones radio frequencies: damn pretty easy now

reply to post by QuietSoul


The technologies beyond public access is believed to be easily fifty years more advanced. If they can space travel, what can they NOT do?

If you thing back 3-4 years ago to the Scott Peterson trial. Months after his wife disappeared they had his cell phone conversations from the time she went missing. So where did they get them. I got rid of my cell phone at that point.

From what I can tell most people here have no idea about surveillance so I wouldn't take their word for it. In my opinion neither the CIA or NSA have the capability or manpower to run a constant surveillance program that completely monitors the US at all times. You have to understand that politics have crippled and hamstrung the CIA but I don't know about the NSA.

Do you have any idea how many personnel (staff, support, officers, assets) it takes just to mount a large operation? And compared to the "complete surveillance" theory that would be a krill lining up against the Megamouth. The alphabet agencies have a long running feud and don't always like to work together or share info so a joint effort wouldn't happen unless they suddenly had a change of heart.

If they can find a target or a network of targets than they can run surveillance. Other than that I severely doubt they watch millions of people all the time. Too much money, too few people and too much politics. When it comes to Facebook or Myspace yeah it would be easy to pick out targets and since most people are stupid and put all of their information on those sites it wouldn't take a genius to find out about a person. I know they are tracking targets on Facebook...

well, it might be possible but the amount of data crunching would be tremendous!
i would say they cant for the moment, but in the next 5 to 10 years it should very well be possible.
this is all to say that they havent developed a supercomputer capable of doing all of this and havent told the public about it, the best person to ask in my opinion would be bill gates.

This is an area I find real interesting.

My comments are based on the USA only. Other nations are much tighter. Britain just recently announced about keeping copies of all internet based information for one year time frames. source info

First we need to define "they" and then how much do "they" really know.

To have anyone watch you in the means of actually tracking your every move means you have done something to bring an incredible amount of attention to yourself.

This is totally different than the usual surveillance that is performed on the general population. Everyone leaves a certain amount of digital trail.

There are a couple ways to track someone, real time and backtracking after an act or event happens.

Phone usage is tracked in a couple of ways. There will be a record of times all calls are made and received. A specific number will be monitored if it has reason to be watched. Cell phones are nothing more than radios and these are even easier to listen to. General records have a very important function for tracking after an event, generally not useful for real time.

VOip and cell phone frequencies do have monitoring performed with word filters in place. These listen to for specific words or groups of words before the whole conversation is recorded for listening by a person. Land lines require a specific court order for this to happen (yes it really does, not to be confused with the patriot act and illegal tapping of cell phones).

Texting is incredibly easy to track and messages stored for later retrieval. SMS (short message service) is just that short and to the point. Detailed records are kept on all servers and routers the system uses to send message from point A to point B.

Let's not forget GPS enabled cellphones. Not much more needs to be said about this.

Email has a copy made of the message along with all routing from using header information on each E mail server handling the message. This can be monitored in real time.

Internet usage can be and is somewhat monitored. Later retrieval of information can be performed to see what internet sites parties used or visited. This is a normal function of any website for tracking unique hits or visits.

ATM and banking functions are monitored as a normal function. Once again, data gathered for later retrieval can provide detailed records of movement by individuals. Usually not performed in real time unless you have an active case working on you. ATM, credit card, and other forms of direct digital transactions can be traced in real time to find a person due to kidnapping, ID fraud, or missing persons type cases.

Now for cameras. Cameras everywhere, or so it seems. There is a certain amount of information stored for different systems using facial identification that is used in some parts of the country. Las Vegas has used this form of surveillance for years in finding known card counters and people who have been accused of beating there gambling systems over the years. Airports are just the latest players in looking for known people of interest at public airports.

A person can make themself somewhat hard to find and follow in the digital world. It does take an effort.

Echellon System dont require any human help to function, its totaly automated.
the NSA is actualy called Never Say Anythings for an obvious reason.

So,believe me, its totaly functional and been perfected again recently.
You dont really know what their telling you about the technology they're using.

Best solution to keep ppls quiet: tell them you only have a recent technology like you and me are usualy using.

The technology is there as far as I'm concerned. If you want to test it out just call someone up and start talking about some things that you know would set out a huge flag to the NSA/Big Brother. Repeat these things over and over again. If two men in suits appear at your door and no one ever hears from you again then it's true!

Part of a recent thesis I did dealt with this exact issue - yes they can - yes they do.
MI6 has the ability and authority to hack into your computer, if on, without any need for warrant.

Even better and absolutely true, latest mobile phones CAN HAVE the microphone, gps and video switched on remotely and silently (no light). I was shocked at this but is totally true.

The UK is already building a data base to track hold and store every single piece of data transmitted - truly terrifying.

"spies in the Sky" by Norris, "Wired for War", P.W.Singer, "Spies for Hire", Tim Shorrock, are a good start.

Just to be clear Echelon isn't a question. It's a reality. I thought I was ahead of the curve because I was aware of it back in the mid-to-late '90s when people first started to flip out about Carnivore, but as it turns out it's supposedly been operational since the '70s.

For a good several decades intelligence agencies were largely restricted to keyword searching email, faxes (using OCR), and other written communications. Outside of that a person needed to sit down and manually sift through raw SIGINT. What's somewhat new is the ability to machine-transcribe speech to text. For more on this read Suelette Dreyfus's, Ph.D, articles on the 1997 NSA patent or for a more directed search dig around for Julian Assange's 2000 AUCRYPTO mail-group postings.

As for your logistical questions, I'll start by commenting that the NSA operates the worlds largest computer, communication centers, and is the owner of the single largest group of supercomputers (1, 2). The scale of the NSA's efforts is hard to determine from unclassified data, but one clue is the electricity usage at Fort Meade. The NSA's budget for electricity exceeds $21 million per year, making it the second largest electricity consumer in the entire state of Maryland.

Storage Considerations

Back in 1999 the NSA contracted Cluster File Systems to tailor Lustre (lovingly referred to as the "inter-galactic filesystem") to theirs and the tri-lab needs (Los Alamos, Livermore, Sandia). It's safe to assume they're working in exabytes (($220 mil / 2 [dev&it]) / $50 [1TB HDD] = 2.2 exabytes!) if not zettabytes (1,000,000,000 1TB HDDs!) of data at this point (don't forget they probably also use robotic tape backup solutions too). If they have a reliable means of converting speech to text there's very little reason to keep the raw audio-file unless the file holds some significance.

There is considerable debate about how much storage would be required to retain every word spoken by every human in all of history. The guesstimates range from 5 exabytes to 400 zettabytes.

The point being even if the NSA doesn't have the ability to store everything in it's raw format they do have the capacity to retain everything once it's converted to text (give or take compression) or they can transcode digital conversations to a more lossy format using something like Sony's DVF codec. The compression ratios for voice encoded data are amazing. I recorded an hour long meeting (1:02:09) and using DVF it only consumes 8.62 MBs! When confronted with huge blocks of video / binary data they very likely just hash TCP streams to reduce duplication of commonly recurring blobs. Furthermore with rapidly increasing hard-drive sizes and speeds (SSDs are getting very affordable) their ability to increase capacity gets a lot easier. Consider just a year and a half ago the first production 1 TB drive was released and now companies like Western Digital are manufacturing 2 TB HDDs. At some point space will be so readily available storage will no longer be the problem. Rather write and read speeds will be the issue.

Next lets consider our ability to passively capture data.

Passive Signal Collection

Back in the 60s geostationary communications satellites presented opportunities to intercept international communications. The report to the European Parliament of 2001 states, "If UKUSA states operate listening stations in the relevant regions of the earth, in principle they can intercept all telephone, fax and data traffic transmitted via such satellites." (source) Unfortunately for the NSA the role of satellites in voice and data communications has largely been replaced by fiber optics. As of 2006, 99 percent of the world's long-distance voice and data traffic is carried over optical-fiber. (source)

Intercepting data off optical relays is a bit more tricky as it requires splitting light. However as evidenced during the Bush-warrantless wiretapping scandal, in AT&Ts room 641A, the NSA has technology in place to do just that.

This leads to questions about processing power needed to sift through the huge amounts of data flowing across the backbone. According to a 2002 UC Berkeley study the world produces 1 to 2 exabytes of information annually or 2.7 to 5.4 petabytes / day.


[edit on 17-4-2009 by Xtraeme]

Processing Considerations

Are you aware that there are fully functional 110 GHz ASIC chipsets (often referred to as Silicon Packet Processors) already on the market? They're used in Cisco's CRS-1 routers.

Before you get too excited, there's a big difference between a special-purpose ASIC network processor and a microprocessor like the one used in your computer. However the underlying technology, silicon germanium, is finding its way to in to more generic semiconductor research.

The point being through specialized equipment (anyone remember EFF's Deep Crack?) it's not only possible to transmit 40-100 Gbps of data, but to process and act on it!

This is just processing power as it relates to network operations like packet filtering. Another consideration is decryption and the computational requirements to perform cross-references or, in SQL-ese, joins.

To address this issue the NSA is betting its money on quantum (petaflop scale!) computing. What blows my mind is as of Sep. 2007 a private company, D-WAVE, has publicly demonstrated an operational (but still somewhat basic in the sense that it's not designed to execute Shor’s algorithm - the holy-grail for the NSA) 28-bit quantum computer.



This must have been somewhat to the NSA's chagrin as they spent $60 million researching quantum-computing semi-conductor fabrication in 2007 only to have a private company beat them (or so we're told) to the punch. As of 2008 the NSA, NSF, & DOE have been funding research efforts at Lawrence Berkeley National Laboratory. (1, 2)

Once this research pans out cracking modern day encryption will be a joke.

In the mean-time they have to make do with FROSTBURG aka "The Thinking Machine" (ca. 1991-1997, operating @ 65.5 gigaflops, ~2 TBs of physical memory), the newer Cray XT Jaguar (operating @ 1.64 petaflops potentially coupled with a 2.5 TB SSD RAM disk ) and their distributed network of lesser machines probably configured to work together using something like Beowulf, OpenMosix or BOINC.

Cooperative data-sharing

Most of the questions asked by the OP relate to feasibility of centralized voice-processing and whether or not all data can be sniffed and acted on instantaneously.

First I'd like to point out that there's no reason this can't be a distributed task. When the FBI wanted to monitor email via Carnivore they required all ISPs and the telecommunication providers to install the DCS 1000 hardware at the central office (CO). In the case of the Bush/NSA-warrantless wiretapping same thing. The DHS/NSA approached AT&T, compelled them to help and then armed with a Patriot Act authorized National Security Letter prevented AT&T from divulging the operation.

Once field deployed equipment breaks down signal intelligence in to manageable pieces higher priority messages are bubbled up to the NSAs database at Fort Meade or to temporary data-warehouses or colocation centers, for additional processing or storage. I imagine lower priority data probably sits on the telecommunication companies network for upwards to several weeks / months before it's eventually overwritten.

Also realize that phone conversations are largely digitized at this point.

Conclusions

Basically the NSA in collusion with other countries and US agencies can capture just about all signal intelligence. Total Information Awareness if you will. What they can't necessarily do is cross reference everything other than sorting by aggregate data (like IP / geographic region / behavioral patterns).

The scary part is it will only get easier with better hardware. The only thing helping privacy is the increased growth of transmissions and relatively strong public encryption. However if my predictions are correct human signal growth is logarithmic. Which is to say once quantum computing becomes a reality, if quantum encryption isn't as strong as we're led to believe (*cough* *cough*).

Then we better pray our laws are enough to prevent others from invading our privacy.

[edit on 17-4-2009 by Xtraeme]

Well they know what i am doing 24/7, and i have no privacy, fact.

So yes they can watch people 24/7, it just depends on how many people they can do this too at any given time. I think an awful lot of people are watched 24/7, and i think gang stalking, is one way of them learning how to control people etc... with these things.

The stuff they are telling you in the news is bull, it is far more advanced than they say. They used electronic mind controlon me in 1992, and just last year the press talked about this tech.

So the control grid is far more advanced than the press say.

Originally posted by darthStar
....
this is all to say that they havent developed a supercomputer capable of doing all of this and havent told the public about it, the best person to ask in my opinion would be bill gates.

haha that is funny, if Gates was involved, the thing would crash all the time, and attract viruses. It would be a superflop. speaking of Gates, is he trying to make biological software now?

Originally posted by oneinthesame
The technology is there as far as I'm concerned. If you want to test it out just call someone up and start talking about some things that you know would set out a huge flag to the NSA/Big Brother. Repeat these things over and over again. If two men in suits appear at your door and no one ever hears from you again then it's true!

It's funny you mention testing it. I was in a heated political discussion with my wife just 2 days ago. Bitching and moaning about all kinds of # that's currently going down. I ended the conversationg jokingly saying I wish the US Gov would collapse; and my wife made a snark remark about them listening for key words.. I laughed and said "Like what, BOMB and REVOLUTION?"

After that I spent the evening on the /i/ chans, organizing for the 4/20 protests.

No white vans outside.

Originally posted by hinky
This is an area I find real interesting.

All pertinent information, and I intend to followup with the J.Bamford books you u2u'd me, however; I understand (well enough) the current powers of tracking individual targets. And I'm sure there's a whole slew of surveillance methods on specific targets that I couldn't even imagine. If you do something stupid, or illegal; then I'm quite sure their methods of gathering information (both past and current) are quite effective. I understand we're "tied in". I don't buy that we're recorded AND blindly monitored through some super computer database for threat patterns.

But, I do believe I made this clear in my OP, I'm talking about overall Herd surveillance. I don't buy that muttering a few words into my phone or having a heated "rebel!" type discussion on the internet is going to miraculously land me on a target list. If you've spent more then 15 minutes outside of ATS, you understand completely what I mean just by the millions of "flagged" comment discussions littering every corner of the internet. And that's not even including the cesspools like 4chan.

Originally posted by OTTOKARMA
Echellon System dont require any human help to function, its totaly automated.
the NSA is actualy called Never Say Anythings for an obvious reason.

So,believe me, its totaly functional and been perfected again recently.
You dont really know what their telling you about the technology they're using.

Best solution to keep ppls quiet: tell them you only have a recent technology like you and me are usualy using.

Believe you? You didn't give me anything to believe other then a dreamy scenario I could watch in X amount of movies from Hollywood. I asked in my first paragraph for people to retort with supporting FACTS. You've (and a few others) simply spun a fairy tale and appended a "believe me" to it.

Originally posted by ctjctjctj
The technologies beyond public access is believed to be easily fifty years more advanced. If they can space travel, what can they NOT do?

30 years ago that may have been the case. But considering how easily it is for me to talk to someone in a skilled field in say, Japan; at a moments notice I could solve a mathematical problem or a programming flaw. There are less R&D hurdles and infinitely more scholarly based networks today.

I understand that Military complex makes a habit of tracking down the brightest and most skilled individuals in the world. But I also understand the reality that not everyone that's a genius in this world is devoted to the military goals and/or morals.

The 'gap', I believe is 5-10years; at best. Believing it's 50+ years is embracing ignorance.

Posted by Xtraeme
Tons and tons of supporting facts and reading material!

When I asked for constructive supporting facts, THIS is what I meant ATS, not dreamy theories! Thank you Xtraeme, I have some reading to do! I'll follow up in a few days.








[edit on 4/18/2009 by QuietSoul]

reply to post by QuietSoul


No, the media hype is only for those who usually deny the existance of those types of things because they do not actually believe what is broadcast, or it goes over their intellectual levels of intelligence and comprehension.

The World-Wide Web of Surveillance of Your Online/Offline Activities



[edit on 18-4-2009 by SpartanKingLeonidas]