Tonight, after DCUM being down for a while, I can't start a new topic. When I hit submit, it tells me that the "topic does not exist."

I didn't find anything unusual (other than DCUM appears to be very popular with the Dept. of Education).

Love this!

and too much traffic - the downsides of success ...

Anonymous wrote:I have no idea what you just said, Jeff, but thanks for the explanation.

I think he just said he forgot to pay the elec bill last month... and that's somehow causing problems.... or something like that.

I saw a repeat spam post on the family forum the other day and I assumed that was tied to the error message - maybe being inundated with spammers?

I have no idea what you just said, Jeff, but thanks for the explanation.

Dear PP,

In DC, DOE usually means dept of energy. "Ed" is the abbreviation you are looking for.

jsteele wrote:I haven't identified the root of the problem. We have a software stack that has apache web server on the front end, Tomcat application server in the middle, mysql database server also in the middle, and then a mysql ndb cluster on the backend. We have two nodes of all of this which are load balanced.

There have been problems at almost every level except with Apache. Tomcat used up all of its connections to mysql, so I increased the number. Ndb didn't have enough memory, so I gave it more. Then, the servers started running out of memory. I have two more servers ordered that will be here next week. I will separate some cluster components so the memory requirement per server will go down. I think that will fix it.

What I haven't been able to identify is what is causing the increased load that is behind all of this. It could simply be increased usage of DCUM. I have gathered a ton of debugging data, but it is really too much to be able to analyze. During a rather short outage yesterday, I had 50,000 lines of access logs to dig through trying to find a culprit that was generating the traffic. I didn't find anything unusual (other than DCUM appears to be very popular with the Dept. of Education).

Really? Meaning that there are a lot of hits from users from DOE? Interesting.

Anonymous wrote:Don't be naive, Jeff. It's Ted Cruz.

Just wait and see what happens once he tanks net neutrality in the name of freedom.

Well, I have seen my text being deleted one line at a time. Perhaps I'm being attacked by Sharyl Attkisson's hacker?

(though my cat was walking on my keyboard at the time).

Don't be naive, Jeff. It's Ted Cruz.

Just wait and see what happens once he tanks net neutrality in the name of freedom.

Thanks for the explanation, Jeff! I didn't understand a lot of it but I got the gist -- and even a "gist" is helpful when trying to deal with the site doing down. Thanks for all the work you do.

I haven't identified the root of the problem. We have a software stack that has apache web server on the front end, Tomcat application server in the middle, mysql database server also in the middle, and then a mysql ndb cluster on the backend. We have two nodes of all of this which are load balanced.

There have been problems at almost every level except with Apache. Tomcat used up all of its connections to mysql, so I increased the number. Ndb didn't have enough memory, so I gave it more. Then, the servers started running out of memory. I have two more servers ordered that will be here next week. I will separate some cluster components so the memory requirement per server will go down. I think that will fix it.

What I haven't been able to identify is what is causing the increased load that is behind all of this. It could simply be increased usage of DCUM. I have gathered a ton of debugging data, but it is really too much to be able to analyze. During a rather short outage yesterday, I had 50,000 lines of access logs to dig through trying to find a culprit that was generating the traffic. I didn't find anything unusual (other than DCUM appears to be very popular with the Dept. of Education).

s

Yeah, I get the 503 all the time. It's better for my productivity.

I think it must be because Kim Kardashian broke the internet.

Not a complaint, just wondering.