GC is unbelievably slow right now
 

Is anybody else having issues with GC being incredibly slow right now or is it me? It's taking forever for any page to load. When I go to any forum, it takes a long time for the threads to load. I can watch them load almost one by one. Was there a change? Denial of Service Attack going on? Or is it me?

I think it may be you? I'm not experiencing any issues at the moment.

Oddly, none of my other web browsing seems to be affected. I can barely read GC though. Which might be ok since I should be reading 7 chapters for school right now...

GC is slow for me.

I'm good. Though given the 2 papers I am supposed to be writing, this may be hazardous to my attention span.

It's doing fine now. It was crazy slow earlier. No idea what was going on.

I did a bit of investigating on the server but couldn't determine what may have caused the issue. I didn't see any excessive bandwidth, CPU or memory usage in the logs for approx 3pm or any indication of a DoS. No traffic spikes as well.

It's possible that we just happened to have a bunch of simultaneous site-wide searches which slowed things down. I doubt that was the issue, though.

If it happens again look down to the bottom of your browser window in the status area and see if there's a domain that appears stuck and not loading. That might give a bit more detail of what's going on.

Hopefully it's not a server hardware issue, which might be another possibility.

How long was the slow down going on for?

It seemed to be at least an hour.

Yeah, it was about an hour for me too, I think. I'm glad there was no indication of a DoS. Thanks for checking though, John. My first thought was that it was one of the ad sites because those were taking a long time to load too. Maybe one of them had a glitch which probably wouldn't have shown up in your server logs.

Disturbing email received today at 7:59AM, indicating possible trouble on the horizon...

The server load averages in the quote above are the 1 minute average (14.19), 5 minute average (17.13) and 15 minute average (11.04).

A server load average of 1 generally means the server is running at 100% capacity, per server core. The server GC is hosted on has 4 CPU cores, so 4.0 would be the 100% capacity load average. Staying below 70% capacity is ideal, which would be a max server load average of 2.8. Going over 20.0 (5.0 per cpu core) starts setting off serious alarms and might indicate impending doom for the server (depending on what's causing the issue).

As seen in the email, 17.13 was the 5 minute average. The peak server load may have been much higher than the average. This is pretty bad.

Today's issue + the issue this thread was started about does have me concerned. I've been through this several times before, 3 times resulted in major server crashes (twice about 10 years ago and once around 4 or 5 years ago). Twice it cost me a crazy amount of work with very little sleep over a period of days. Once it was the same + nearly $2k in direct expenses to recover from the crash. Believe me, I don't look forward at all to getting that phone call at 3:00AM again.

Speaking of server crashes, they sometimes come unexpectedly so it might be good for whoever is on Twitter to follow GreekChat on that site: http://twitter.com/#!/GreekChat In the event of a major server crash, I'll use that account to post updates & to announce when GC is back online.

Dealing with this stuff takes time & money. I can make the time available, but money is definitely an issue. One of the reasons I don't work all that much further developing GC anymore is simply because I need to constantly chase the bucks in order to get the bills paid. Times are tough and I know that it's this way for many of us.

3 million people were served by GreekChat in the last 12 months alone. I'm sure GreekChat has enough members & lurkers wanting GC to thrive that the support will be there when it's needed the most.

So, with all that in mind, I am going to:

1. Try to sort out the site finances by making premium paid subscriptions available on GreekChat for those among us who choose to support GC in that way. Free access will remain available just as it is. Paid site memberships will simply be an alternative option for those who would like to help financially support the GreekChat community. I'll do whatever I can to make premium accounts as valuable as possible, including whatever extra account functionality that I can provide. The price I have in mind is approx 10 cents per day, $39.95 per year. I'll also guarantee a 100% refund within 60 days (PayPal's refund time limit) for anyone who gets buyer's / GC supporter's remorse. More details coming soon.

2. Server rebuild is on the way with much higher capacity hardware. I already have all but 1 part of the server hardware that I'd like for this rebuild. I purchased all the parts for a server rebuild a while back, but decided not to spend so much time doing all the required legwork since things were going reasonably well with GC's current rented server. The time saved enabled me to focus on other work, but upgrading/rebuilding the server really shouldn't wait too much longer.

3. Depending on the level of support received in #1, I might be able to cut back on other work and start spending a lot more time working on GreekChat again. A very serious time commitment from me would come if support from the community membership is there. As I mentioned in #1 above, I would do whatever I can to make paid memberships as valuable as possible and would offer a 60 day 100% money back satisfaction guarantee. That last part sounds like a good sales pitch... I just want premium GC members to be happy with their choice of helping to support the site and would want to give a refund to anyone who isn't satisfied.

On the topic of advertising sales, it's easier said than done. It also might be that I enjoy developing community websites so much more than wheeling & dealing trying to sell ad space to other companies every day. I still work on it all the time, though. Advertising sales is a necessary component, but it's also difficult making enough ad sales to replace the need to ask for members to become paid supporting members. I mention this because we've discussed the topic before on GC some time ago.

Ad networks often put low quality ads on GC with no option to pre-approve only good quality or better ads. I maintain contact with 70 or so companies, but it's really tough getting even a very small fraction of them to buy ads on GC each month. I really appreciate GreekChat's advertisers, though, and it would be good for anyone who is interested to check out their offers (ads are usually up at the top of pages or on the left side). GreekYearbook has probably been GreekChat's most consistent long-term site advertiser/sponsor over the past few years and they really deserve the extra mention.

During discussions in the past it's been mentioned that I should look for a major site sponsor. I'm still looking. :) Figured I would mention, though, if anyone knows of a company that wants to become a superstar to GreekChat's audience of 3 million visitors each year, please have them contact GC.

Back to the main point of this message. It's apparent that GreekChat needs upgraded server hardware again, which means I need your support if you are able and willing to help. Please just think about it at this point and in a few days or so when I setup premium GC memberships I would hope that you have decided to become a GC supporting member.

This is a serious problem and the message needs to reach all readers. Is there a way to keep this thread at the TOP of the recent threads page?

Is there an option available for people to help out even with $5, 10 at a time? I don't care about "premium services" in exchange for it, I just would like to be able to help out a little when I can. I don't want to speak for others, but I am guessing that there are some others here who feel the same. I mean, if you got 200 people to donate $10 each, that would cover a lot of your cost, right?

This sounds like a good idea. One of the sites I frequent has a PayPal donation link. Once a year I donate $20-25. You could do it that way without having to add "premium services".

When premium account access is available for GreekChat supporting members I'll be promoting it throughout the site in a very noticeable location.

Yes, every contribution does help make a difference and would be very appreciated. A few years ago I held a donation drive, raised approx $500 from around 20 people. It sort of feels too awkward in a way to be asking for donations month after month so I only did it that one time. If I could provide a lot more value for premium/paid account holders (I'm sure I could) then I'd feel right about promoting that regularly as opposed to always asking for donations. I guess I'd feel as though I earned it by providing additional value beyond the free level of access to GC.

Just to put it into perspective, I probably have $50k+ into this (plus thousands of hours of work) since launching the site years ago and have only recovered a fraction of that. I joke sometimes and refer to GC as my expensive hobby. It wasn't envisioned to turn out that way, but I still try to make it work and look forward to when I finally figure out the recipe for success.

Covering base costs and dealing with the more urgent current server issue is very important, but I'd like to try and get GC beyond that, if possible, so that site development doesn't remain stagnated.

Anyhow, I can setup an option for $5 or $10 contributions as you mentioned. I think the way I'd like to accommodate for that is to make monthly premium account access available (in addition to the annual plan) with the price being $5 per month. This way anyone wanting to contribute $5 would still receive the bonus account features during the month.

I'm thinking more GCers would be interested in helping to support the site if there were some fringe benefits of doing so. Maybe things like virtually unlimited private messaging storage or ability to use avatars. I might still setup a donations link, though, if there are members or site visitors who would like to support GC without the upgraded premium account access or in addition to it.

Thanks for all the comments.

:)

Yes.

Downsizing GC in that way would help some, but there would be negatives which probably outweigh the benefit.

There's plenty of hard drive space, so that's not an issue. The hard drives do take quite a beating, running 24/7 for years, which might be related to the current server issue (running over 2 years now on non-enterprise drives).

Reducing the database size would likely make the database slightly faster. But at the same time, if old messages were purged there would be value lost (to whoever would have found that info to be useful) and there would be less people visiting GC by finding links to all the old messages in search engines, etc.

More people using GC is always a good thing, IMO, as that's what I really like to see. The site has served 3 million visitors in the past year and I would love to see GC serving that many or more each month. But at the same time it's more people using GC which has always generally necessitated server upgrades over the years. The number of people using GC is ultimately what it comes down to for the most part. Probably of most impact is the number of concurrent visitors using the site at the same time. That could easily result in things going smoothly most of the time and at other times site speed nearly screeching to a halt.

If things got really bad regarding server capacity (not considering any potential hardware issues) and the resources were not there to deal with it, I would probably make GC available only to those with accounts (not talking paid accounts there, just any account). If that wasn't enough I would then turn off new member account registrations for a while, until it could be dealt with. I'd rather do all that, if the circumstances necessitated it, rather than having GC be unusable/unresponsive due to overload. I think we're a relatively safe distance away from that kind of scenario, though.

Comparing that solution to purging the database of old messages, by just turning some access off it would be temporary and could be turned back on. Purging would be permanent and I think there are some good reasons to keep older messages around.

A good example with what GC is sort of facing would maybe be a highway that usually has a great traffic flow, except for rush hours. During peak times traffic would slow to a near standstill with it taking long periods of time to get anywhere. With the server, my last resort fix would be to close off the on ramps to anyone who is not a GCer. Far from an optimal solution, but it would work if there was no better way to quickly fix it. There's also an option have the site stop serving pages when the server is overloaded, over a specific load average, but that would apply to everyone and not just those without site accounts.

Back in the day I remember only a few thousand visitors daily with another forum software system resulted in a rather miserable summer in terms of GC site speed. Upgrading the server & software helped a lot, but I've had to upgrade a few times since then. The upgraded hardware in 2007 was enough up until around late summer 2009 and now the server hardware upgraded to 2009 has been running ok for just over 2 years as well. The capacity of the first 4 or 5 servers GC has run on, combined, probably could not handle the site traffic of today.

Maybe some time from now, depending completely on how everything goes, I can envision having a setup where I just build a new server & add it to the cluster when GC needs another capacity boost.

Anyhow, I've probably turned this into much too long of a message so I'll cut it here. Hopefully I didn't complicate the issues with all that.

I'm not a fan of Avatars and hate cluttered private message boxes so those wouldn't appeal to me personally. However, I would probably do periodic donations if there was a link.

I think it was a screwed up ad trying to load that day that I was having issues.

I would like to say thank you John for everything you've put into GC over the years. It is definitely appreciated by the users.

Thanks for the update, John.

I don't know if it's possible, but I'd happily pay a membership fee if it included a mobile version of GC. I'm on another vBulletin board (www.hfboards.com) that has at least four different browser options including a "Lite" and "Mobile" choice. Very user-friendly and so nice when using an iPhone/iPad/Blackberry/etc.

There would be an option to turn avatars off for anyone who does not want to see them. Other fringe benefits would be worked on as well. Basically I think I would just ask for suggestions/requests from GC supporters and implement what I can. Ability to turn off ads might be another idea..

Maybe what I'll do with a donations link is make it available in addition to the premium GC memberships. Seems there might be enough interest in that.

That's another good idea for GC supporting member fringe benefits. Definitely something I would consider spending some time working on.

The mobile thing, now that would be awesome since I can't GC from work except from my phone :)