The robots.txt from the http://www.whitehouse.gov/robots.txt

KTPR said that it had over 200 disallows, here they are

www.whitehouse.gov...



[Edited on 27-10-2003 by Netchicken]

All I can say to this is NIIIIICE Glad to know exactly what their hiding from us.

Let me ammend your statment NC. The robot.txt file has something like 1,550 dissallows. Check the directories. It is my understanding that two years ago they had 150 dissallows.

edit-- This has been throughly disscussed at slashdot.org: yro.slashdot.org...
weblog.siliconvalley.com...
www.bway.net...


[Edited on 27-10-2003 by ktprktpr]

That's an awful lot more than 200 Disallows, webfowl and acronym.

In the scheme of things, what does it mean? And why are searches for the first lady so popular and so unallowable?

Disallow: /firstlady/asia/iraq
Disallow: /firstlady/asia/text
Disallow: /firstlady/behindthescenes/iraq
Disallow: /firstlady/behindthescenes/text
Disallow: /firstlady/healthystart/iraq
Disallow: /firstlady/healthystart/text
Disallow: /firstlady/images/iraq
Disallow: /firstlady/images/text
Disallow: /firstlady/initiatives/education/iraq
Disallow: /firstlady/initiatives/education/text
Disallow: /firstlady/initiatives/healthystart/iraq
Disallow: /firstlady/initiatives/healthystart/text
Disallow: /firstlady/initiatives/images/iraq
Disallow: /firstlady/initiatives/images/text
Disallow: /firstlady/initiatives/iraq
Disallow: /firstlady/initiatives/text
Disallow: /firstlady/iraq
Disallow: /firstlady/media/iraq
Disallow: /firstlady/media/text
Disallow: /firstlady/news-speeches/iraq
Disallow: /firstlady/news-speeches/releases/iraq
Disallow: /firstlady/news-speeches/releases/print/iraq
Disallow: /firstlady/news-speeches/releases/print/text
Disallow: /firstlady/news-speeches/releases/text
Disallow: /firstlady/news-speeches/speeches/iraq
Disallow: /firstlady/news-speeches/speeches/print/iraq
Disallow: /firstlady/news-speeches/speeches/print/text
Disallow: /firstlady/news-speeches/speeches/text
Disallow: /firstlady/news-speeches/text
Disallow: /firstlady/photoessay/bookfestival/iraq
Disallow: /firstlady/photoessay/bookfestival/text
Disallow: /firstlady/photoessay/iraq
Disallow: /firstlady/photoessay/salutingservice/iraq
Disallow: /firstlady/photoessay/salutingservice/text
Disallow: /firstlady/photoessay/text
Disallow: /firstlady/photoessay/welcometowh/iraq
Disallow: /firstlady/photoessay/welcometowh/text
Disallow: /firstlady/photos/2003/01/iraq
Disallow: /firstlady/photos/2003/01/text
Disallow: /firstlady/photos/2003/02/iraq
Disallow: /firstlady/photos/2003/02/text
Disallow: /firstlady/photos/2003/03/iraq
Disallow: /firstlady/photos/2003/03/text
Disallow: /firstlady/photos/2003/04/iraq
Disallow: /firstlady/photos/2003/04/text
Disallow: /firstlady/photos/iraq
Disallow: /firstlady/photos/text
Disallow: /firstlady/recipes/iraq
Disallow: /firstlady/recipes/text
Disallow: /firstlady/text
Disallow: /firstlady/whitehouselife/iraq
Disallow: /firstlady/whitehouselife/text

A lot of those seem to have "iraq" involved at some point, although I have no idea why they do.

I would have imagiend that is they have a website then the would WANT it to be indexed in the search engines, and their side of the story to be on the net.

This file stops the bots from spidering those pages.

Its very strange .....

www.abovetopsecret.com...
same thread? hehe