when the site goes down,

[Anonymous]

you make me do actual work, Jeff. Unacceptable.

[Anonymous]

Seconded! Very troubling.

[jsteele]

It means I have to work too, so I share your feelings. It is totally unacceptable. New servers are due Wednesday. So, hopefully by the end of the week, none of us will be working.

[Anonymous]

Suddenly the web site is loading crazy fast. What was the last thing you did? I'd say it started performing in the last five minutes.

[jsteele]

Anonymous wrote:Suddenly the web site is loading crazy fast. What was the last thing you did? I'd say it started performing in the last five minutes.

The reality is that I don't think I've ever done anything to actually fix the site. I change everything I can think of and nothing works and then -- poof -- it starts working. The next time, I try to redo everything I did before but nothing will work and then -- poof -- again it works.

I'm convinced now that this is a result of a distributed spambot network. Because the requests are for real pages (for the most part) and a variety of user agents are used, it is almost impossible to identify which connections are real and which are from spambots. However, after a week of chasing this, I have started to recognize their patterns. But, I'm always a step behind them and by the time I block them, the damage is done. I'm not even sure they are the cause of the problem, but right now I don't have any other culprits in mind.

[Anonymous]

What are the symptoms as the site grinds down?

[jsteele]

Anonymous wrote:What are the symptoms as the site grinds down?

We have a database cluster of two nodes. Both nodes are underpowered and relatively low on memory. I built the cluster for redundancy rather than performance because additional performance was not previously necessary, but redundancy is important. Beginning about a week ago, the load on the database nodes suddenly gets very high. We start seeing a combination of the cluster dropping connections and connections timing out. Also, the cluster starts reporting some tables as being "read-only" even though they are "read-write".

Two new servers will arrive tomorrow. They are built for performance with loads of memory and solid-state drives. That should eliminate any performance issues with the database nodes. In the meantime, I'm doing everything I can to monitor the cluster to try to find settings that can be tuned for better performance. But, there are literally thousands of settings, of which I probably understand about 2 or 3. So, that is pretty challenging.

[Anonymous]

jsteele wrote:

Anonymous wrote:What are the symptoms as the site grinds down?

We have a database cluster of two nodes. Both nodes are underpowered and relatively low on memory. I built the cluster for redundancy rather than performance because additional performance was not previously necessary, but redundancy is important. Beginning about a week ago, the load on the database nodes suddenly gets very high. We start seeing a combination of the cluster dropping connections and connections timing out. Also, the cluster starts reporting some tables as being "read-only" even though they are "read-write".

Two new servers will arrive tomorrow. They are built for performance with loads of memory and solid-state drives. That should eliminate any performance issues with the database nodes. In the meantime, I'm doing everything I can to monitor the cluster to try to find settings that can be tuned for better performance. But, there are literally thousands of settings, of which I probably understand about 2 or 3. So, that is pretty challenging.

Intermittent "read only" sounds like table locks are being held and the app layer is calling it read only. I'd focus on your inserts because those still seem slow. When they queue up it could tie up your dh connections, then eventually even your reads start failing. Not sure if inserts are failing due to some corruption, permission issues, or unusual post volume and type.

I remember you ran into a similar problem a year or two ago when you were trying to do some spam filtering.