ATS Website Performance...

According to two sources I know here in NY, ATS was mentioned on MSNBC during a science segment on the hurricane. The first mention was Wednesday (at the start of last week's traffic troubles), where my coworker only heard the URL at the end of the segment... and didn't get the context. The second mention was a late-night repeat of the segment, and my son only had the TV on in the background, and thought they mentioned theoretical weather control.

If anyone else has heard these segments, please let us know what was said!

In any event, ATS is breaking into the mainstream... and we're addressing ways to deal with the traffic. Our biggest problem is Google, which has automatically elevated our status from "website" to "news-site"... which means the GoogleBot spiders all of ATS every week, or more (including our member blogs!)

The first step is the installation (and a difficult one too) of new software that speeds up the way PHP programming is executed. So far, it seems to have helped a great deal (it was installed late yesterday). Please let us know if you notice an increase in overall site performance.

We're going to watch the performance over the next few days and see how things progress. But simply dealing with current traffic isn't enough. Next, we'll be looking at either improving the database server of our existing machine, or, upgrading to a Xenon dual-processor Ultra-Wide-SCSI machine.


More updates later...

William

That is good news!

I don't see MSNBC as being the likely prime source for the sudden spike in the graph (nor do I see the immediate relevance of the ATS site to the MSNBC subject matter, although I haven't read whatever articles are in our main web page content on theoretical weather control).

But I hope it is as innocent as that. I have read it differently.

ATS seems to be running smoother than ever. No problems at all

Like Masked, I am having a hard time wrapping my mind around a 3 second verbal mention on T.V. causing this great amount of traffic...

When we spoke on the phone I thought you meant ATS was "mentioned" in a news article on MSNBC.COM... THAT would make total sense to me...

Has anyone figured out if ATS was indeed mentioned/linked on MSNBC.COM? If you got the "double whammy" of T.V. mention coupled with a link on the website (MSNBC.COM) then you would certainly get a stampede similar to what's been happening around here...

Just my $.02.

PEACE...
m...

I have made several websites, and struggled with the traffic thing before.

My guess is this. With the content being spidered more often, and hopefully the meta tags have been tweaked for the engines, more page links will be loaded into the search results databases. Google, probubly the best, I understand upgraded your listings to a better catagory. Means even more pages. Right now "abovetopsecret"=19000+ hits and "above top secret" =2250000 page hits. With the keywords now spidered on a viriety of topics this may mean many more.

Overture results "above top secret"=407 & "top secret"=3899
actual search terms this month.

marketleap link popularity for the domain name registers = 18345 on the verge of being a "Contender"
the top end of "above average presence"

Basically I see the traffic continuing to grow, as these numbers indicate that the web presence is growing. If the meta tags are pampered, as well as google, this could mean continuous exponential growth, as search results draft more traffic based on keywords that are relevent.

With the current political environment, people are searching for more news, which will bring more traffic, which will bring more posts and members, which will bring more spyder activity, wich will increase ranking, which will bring more traffic....

The site numbers have all crossed the tough line, that defines the difference between a website, and a popular website. You dont cross the line, you never really get noticed, or considered viable.

Good Job !

ATS was again down, just a minute ago.. for 30min-1hour..

If that traffic is real (I was out of the house, so I didn't get a look at what was hitting the server), there's not much any software tweaks can do. We peaked at 249 simultaneous users.

Originally posted by MaskedAvatar
I have read it differently.

Really? How different?

From all the data I have at this end, we're being overwhelmed with a high number of unique users.

'Only'

249 simultaneous users.

And the site is out of the game..

That isnt good..

As that is quite 'few' still..



ATS needs to get better sponsors..

And more bandwidth..

I doubt that any DOS attack would be suggested, as IP pings in the last few months have been very high, continuously, across the board, everywhere, even on roadrunner lines.

I suspect that the software can easily handle 250+ logged in users simultaneously, but you have to guess that is less than one tenth of actual pageviews.

And if the search engines have been kind like you would want, the ratio may even be higher.

I believe spiders are also bandwidth consuming, and may appear as a page hit, for each page indexed.

[Edited on 21-9-2003 by smirkley]

From what I've seen, this isn't a DDoS of any kind. A well-designed DDoS will focus on a few key pages... I'm seeing a logical spread of interest between the main site's HTML pages, the board's "viewthread.php" pages, and the blogs.

The GoogleBot is currently responsible for about 40% of the site's traffic right now and was also very active earlier.

A highly-dynamic site (database-driven, like ATS) is a slave to the performance of the database/hard-drive combination. We're on medium-fast 7200rpm IDE drives. mySQL is currently attempting to service almost 30 queries per second... that's a lot of data per-second. Anything more than 100 simultaneous users is a very high-load for any dynamic site.

We're also close to serving 100,000 page-views a day... which places us very firmly in the big-time.

I was reading a scientific news site the other day and it made referrence to ATS. I guess it's going to get much bigger in the future.

I hope William's HDD can survive the pace. That damnes google bot makes up a significant portion of the traffic on the internet. It is possiblie to hide from it by inserting a specific meta tag into the pages, but that's probably not what you want to do.

We're on medium-fast 7200rpm IDE drives.

No wonder this happens..

This is something that is good for game comp, but server would really need SCSI or S-ATA RAID..

And also DUAL processors..

Indeed... and it costs twice as much, both in montly payments and set-up fees.

Originally posted by smirkley
and hopefully the meta tags have been tweaked for the engines,

The only meta tags that search engines pay attention to any more are description and title... anything else is pointless.

My comment "I read it differently" was in reference to my disbelief that the number of unique users would increase radically as a result of this single mention on MSNBC, but Springer and others may have cast more light on the subject.

I only hope that the mentions bringing about this increase in lurking have the ultimate effect of good posting from quality members, and not nonsense and waste of space.

I'd suggest adding most of the forum to your robots.txt exclusions. While google may be spidering your entire forum, it's probably only indexing your archive page(s) anyway, right? Most php forums don't get indexed beyond the home page. Or do you have some type of spider friendly hack?

[Edited on 9-23-2003 by Satyr]

Google is spidering every page... and when it finds 'em, even the member profiles.

Over the past two years, the technology team at Google (I still know two people there) have been working on tweaking their bot to understand dynamic URL variables so that full sites can be understood. About 9 months ago, they completed the process so that the bot can utilize full URL strings as unique pages.

Two reasons I don't want to exclude robots (Google):
1) We want people to find what we have to say
2) The targeted advertising (Google ads) need to be able to spider the page they're on

Accomodating traffic from Google bots is the cost of doing business online these days. (Google IP's are not used to calculate our traffic numbers though)

Originally posted by Zzub
I was reading a scientific news site the other day and it made referrence to ATS. I guess it's going to get much bigger in the future

Do you have a link Zzub?

Originally posted by William
Google is spidering every page... and when it finds 'em, even the member profiles.

Over the past two years, the technology team at Google (I still know two people there) have been working on tweaking their bot to understand dynamic URL variables so that full sites can be understood. About 9 months ago, they completed the process so that the bot can utilize full URL strings as unique pages.

Two reasons I don't want to exclude robots (Google):
1) We want people to find what we have to say
2) The targeted advertising (Google ads) need to be able to spider the page they're on

Accomodating traffic from Google bots is the cost of doing business online these days. (Google IP's are not used to calculate our traffic numbers though)

Interesting. I wonder why it won't spider vBulletin forums still? Same type of strings, but they rarely get spidered beyond the main page, without some type of archive. I'd like to know what the difference is. Although I have many pages excluded, I would like my forum to be indexed a bit. I'll have to look into this and see if I can find the answer.

What I was getting at more, though, was excluding pages you don't need indexed. For example, these are all portions of a vB forum that google would be wasting it's time attempting to index, since they're not really pages that can be indexed. This can save you some bandwidth, if spider bots become a problem. All of your topic pages will still be indexed.

Robots.txt:

User-agent: *
Disallow: /attachment.php
Disallow: /avatar.php
Disallow: /calendar.php
Disallow: /editpost.php
Disallow: /member.php
Disallow: /member2.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /private2.php
Disallow: /register.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendtofriend.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /admin/
Disallow: /images/
Disallow: /mod/

[Edited on 9-24-2003 by Satyr]