www.whitehouse.gov/ Disallowed Files.

Please forgive if this has been discussed before and if this is in the wrong section.

Today I was searching google, using its brilliant advanced search strings.
I came across a rather interesting and disgusting find. It was a robots.txt file for the whitehouse www page www.whitehouse.gov.... it contained a list of forbidden (403) directories that bore rather intriguing names such as these for example:

User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /360pics/iraq
Disallow: /360pics/text
Disallow: /911/911day/iraq
Disallow: /911/911day/text
Disallow: /911/heroes/iraq
Disallow: /911/heroes/text
Disallow: /911/iraq
Disallow: /911/patriotism/iraq
Disallow: /911/patriotism/text
Disallow: /911/patriotism2/i

Another way to look at the file and not directly show up on the whitehouse log is this:

64.233.167.104...:tCfemw3M-aUJ:www.whitehouse.gov/robots.txt+filetype:txt+inurl:gov&hl=en&ie=UTF-8
(props go to Decimalz)
Any comments?

[edit on 22-6-2004 by Ezekial]


edit to correct long subject in recent posts format

[edit on 22-6-2004 by SkepticOverlord]

[edit on 22-6-2004 by John bull 1]

There doesnt appear to be anything too shocking in that list.

Disallow: /barney/iraq
Disallow: /barney/photoessay/iraq
Disallow: /barney/photoessay/text

What could these be about?

[edit on 22-6-2004 by TPL]

I don't understand. Whats are they disallowing?

They are disallowing search engines such as Google look through these directories of the whitehouse site. Why would anything to do with "Barny" or 911 be kept secret?

The Spider (Search) is not allowed to search within those files. Basically what a robots.txt is suppose to do. It flags what a Search Spider can do when it hits a site to go search in.

Oops little late on my post.

The Barney name could just be what the person used to create a temp directory. I know a couple places that use any type of name instead of using "temp".

[edit on 22-6-2004 by DJSpellBound]

[edit on 22-6-2004 by DJSpellBound]

It's quite simple. If they mistake and put up some false information or even something that shouldn't be out in public, by that it won't be cached by Google and Co.

Example:

Press release about an Event, false dates or somebody realised they gave too much info that could help "terrorists" or whatever...so they just change the text without having to fear that the old version is cached by Google.

Though it's a bit strange they included nearly every folder + the folder with "iraq" at the end, it looks like they want to ban the whitehouse.gov page from the search engines for searches that deal with "iraq".
Lame

Sick people who work there, I wonder what made them that weired.

Disallow: /ghosts/iraq
Disallow: /ghosts/text

Sorry couldn't resist putting those two up.

Quite interesting really if you know what you're looking for.

All of these files were on the WhiteHouse site and could be viewed.

I think it's particularly interesting that so many press releases have been disalowed.From the beginning of the Bush Presidency past 9/11 and up to date all on Iraq.Examples:

Disallow: /news/releases/2001/01/images/iraq
Disallow: /news/releases/2001/01/images/print/text
Disallow: /news/releases/2001/01/images/text
Disallow: /news/releases/2001/01/print/iraq
Disallow: /news/releases/2001/01/print/text
Disallow: /news/releases/2001/01/text
Disallow: /news/releases/2001/02/images/iraq
Disallow: /news/releases/2001/02/images/print/text

Now that's what I call News Management! Just wipe what doesn't look good in hindesite off the site and it can't be verified.


Now you find a reference to Bin Laden or Al Qaeda in there pre-9/11 and you've hit conspiracy gold!!!



[edit on 22-6-2004 by John bull 1]

Originally posted by John bull 1
Now that's what I call News Management! Just wipe what doesn't look good in hindesite off the site and it can't be verified.

Sounds a little bit too much like George Orwell's 1984 to me. Remeber they would just change the news releases to make everything fall in line with what Big Brother said. Could we have Big Dubya on our hands????

Originally posted by BlackJackal
Sounds a little bit too much like George Orwell's 1984 to me. Remeber they would just change the news releases to make everything fall in line with what Big Brother said. Could we have Big Dubya on our hands????

HA! Well said BJ With a touch of Animal Farm.
Have a dig around this link and see what happens...you basically get 'sorry 404 403'
(deep dig) You may well get the page once, but if you refresh...?!
www.gov.au...

Animal Farm...
en.wikipedia.org...
Sanc'.


[edit on 22-6-2004 by sanctum]

I think these two are interesting:

Disallow: /agencycontact/iraq
Disallow: /agencycontact/text

Oh, to read some of the files on that server....

There is nothing disgusting about blocking spiders from accessing those folders. If you have several directories on your webserver, it can be quite annoying for spiders to crawl through all of your directories cataloging information. Locking down the directories keeps the spiders focused on the primary content for the site.

WhiteHouse.gov is not keeping YOU from going to those directories. They are keeping the spiders from accessing the directories. I've been able to find links to most of the directories listed just by browsing the WhiteHouse.gov site. If you want to view the files, go to www.whitehouse.gov/"THE DIRECTORY YOU'RE SO CONCERNED ABOUT."

Also, please realize that if the directory you want to view says, "CANNOT BE FOUND" when you try to access it...it doesn't mean they're hiding it from you. It's been almost 3 years since 9/11. Websites change constantly. It just might not be on the server anymore...or they may have moved the same information to a different directory. If they don't update the robots.txt file, these (re)moved directories will still show up.

Originally posted by FiberOptik
I think these two are interesting:

Disallow: /agencycontact/iraq
Disallow: /agencycontact/text

Oh, to read some of the files on that server....

Disallow: /agencycontact/text = www.whitehouse.gov...

That was a very defensive reply tkeax1088 .

I did a little digging on this.A few people have looked at this already.Below is the best one I could find.I've just quoted the intro but it is worth the read for those interested.

PS.And no it's not normal or acceptable.It appears to be quite systematic as concerns Iraq files.

In this "digital age", the White House online public documents could be the primary, authoritative record of public information released by the Executive Branch. Those documents should be managed in such a way that their historical integrity is transparently verifiable by the public to whom those documents are addressed.

However, the document management system that has been put in place by this White House involves changing a substantial portion of the source documents on at least a monthly basis, completely unnecessarily, in order to update unrelated information that is displayed on each page along with the original source material.

Whether by design, or as merely an unintended consequence, the White House has thereby made it extremely difficult for those who value truth and integrity to determine the extent to which this administration, that has shown reckless disregard for truth and integrity, has been changing its own written record as it goes along.

home.pacbell.net...

[edit on 22-6-2004 by John bull 1]

Originally posted by TPL
There doesnt appear to be anything too shocking in that list.

Disallow: /barney/iraq
Disallow: /barney/photoessay/iraq
Disallow: /barney/photoessay/text

What could these be about?

[edit on 22-6-2004 by TPL]

They could be about Barney Bush, the Bush family dog.

www.whitehouse.gov...

Disallow: /barney/photoessay/text = www.whitehouse.gov...

Originally posted by Ezekial
Another way to look at the file and not directly show up on the whitehouse log is this:

64.233.167.104...:tCfemw3M-aUJ:www.whitehouse.gov/robots.txt+filetype:txt+inurl:gov&hl=en&ie=UTF-8
(props go to Decimalz)
Any comments?

I doubt you're able to log a visit from a text file. I might be wrong, but I'm not aware of any way in which visitors are logged on the server side.

Also, relating to the file itself, can someone please explain why the folder name "iraq" is tagged onto 90% of the folders on that list? Example:

Disallow: /omb/reports/print/iraq
Disallow: /omb/reports/print/text

90% of the folders on there that have a folder named "/text" has a "/iraq." What's even more bizzare about this would be:

Disallow: /president/winterwonderland/iraq
Disallow: /president/winterwonderland/text

How does the desert of Iraq relate to "Winter Wonderland"? @_@

Here's a suspicious directory as well:

/president/worldunites/iraq

World unites and Iraq...hmm...

[edit on 22-6-2004 by Blackout]

Just to save you going through them all tkeax1088.

Point #4: At least two pages, the transcript of Bush's "[Major] Combat Operations Have Ended" speech from May 1, and a Condi Rice press briefing from Oct. 2001, are confirmed to only exist in an 'iraq' subdirectory, and include the robot blocking instructions. Practically every other document in the News Releases section exists in both "standard" and "iraq" versions. In these cases, the "standard", search engine indexable, versions of those particular documents were either deliberately or inadvertently deleted. The effect of those deletions is that searches on the text of that document return nothing. Whether those deletions were deliberate or accidental, a current or future web search using excerpts from either the Bush speech or the Rice press briefing will not turn up the full, official White House transcripts.

I don't want to post any more as it is copyrighted so go check it out yourselves.

Originally posted by John bull 1
Just to save you going through them all tkeax1088.

Point #4: At least two pages, the transcript of Bush's "[Major] Combat Operations Have Ended" speech from May 1, and a Condi Rice press briefing from Oct. 2001, are confirmed to only exist in an 'iraq' subdirectory, and include the robot blocking instructions. Practically every other document in the News Releases section exists in both "standard" and "iraq" versions. In these cases, the "standard", search engine indexable, versions of those particular documents were either deliberately or inadvertently deleted. The effect of those deletions is that searches on the text of that document return nothing. Whether those deletions were deliberate or accidental, a current or future web search using excerpts from either the Bush speech or the Rice press briefing will not turn up the full, official White House transcripts.

Is there something sinister about this? Possibly. Possibly not. As a web developer, I really don't see the big deal. I've done similar things, intentionally and unintentionally, on projects.

Maybe it's me, but I didn't think it particularly hard to find the President's remarks from May 1, 2003. www.whitehouse.gov...

I also don't see a problem with a redirect from the "standard" template to an Iraq theme on issues that deal with Iraq. Oh well!

Oz Gov' privacy clause,
"records your visit and logs the following information for statistical purposes -your server's address; the name of the top-level domain from which you access the Internet (for example, .gov, .com, .au, .nz etc); the type of browser you use; the date and time you access the site; the pages you have accessed and the documents downloaded and the previous Internet address from which you linked directly to the site.
We will not identify users or their browsing activities, except when where a law enforcement agency may exercise a warrant to inspect the service provider's logs.
This Website contains links to other sites.
* The use of the term "Personal Information" in this privacy statement means any information from which your identity is apparent or can be reasonably ascertained.
www.gov.au...
(hot-links were edited)
Sanc'.

Originally posted by Blackout
I doubt you're able to log a visit from a text file. I might be wrong, but I'm not aware of any way in which visitors are logged on the server side.
[edit on 22-6-2004 by Blackout]

Servers typically log any requests, whether for a text file or a media clip. The software I use reports the type of browser in use, the operating system, the IP address, the user's domain (aol.com, msn.com, etc), where they came from, what they searched for (if they came from a search engine), their monitor resolution setting, etc. Web Admins can find out a lot of stuff about the people who visit their sites.

robots.txt files are no big deal. MANY sites use them. You could use it to keep search engines from indexing old news. Why have a search engine update 2001 or 2002 news? Its a waste of your bandwidh. If you want to keep people AND bots out of a directory you use htaccess to set password protection on a directory.