Why is DCUM always down lately?
![[Post New]](/jforum/templates/default/images/icon_minipost_new.gif){kind=icon}
Anonymous
| Not a complaint, just wondering. |
Anonymous
| I think it must be because Kim Kardashian broke the internet. |
Anonymous
| Yeah, I get the 503 all the time. It's better for my productivity. |
shhanrahan
Member Offline
Member Offline
| s |
jsteele
Site Admin Offline
Site Admin Offline
|
I haven't identified the root of the problem. We have a software stack that has apache web server on the front end, Tomcat application server in the middle, mysql database server also in the middle, and then a mysql ndb cluster on the backend. We have two nodes of all of this which are load balanced.
There have been problems at almost every level except with Apache. Tomcat used up all of its connections to mysql, so I increased the number. Ndb didn't have enough memory, so I gave it more. Then, the servers started running out of memory. I have two more servers ordered that will be here next week. I will separate some cluster components so the memory requirement per server will go down. I think that will fix it. What I haven't been able to identify is what is causing the increased load that is behind all of this. It could simply be increased usage of DCUM. I have gathered a ton of debugging data, but it is really too much to be able to analyze. During a rather short outage yesterday, I had 50,000 lines of access logs to dig through trying to find a culprit that was generating the traffic. I didn't find anything unusual (other than DCUM appears to be very popular with the Dept. of Education). |
DC Urban Moms & Dads Administrator
http://twitter.com/jvsteele
https://mastodon.social/@jsteele
http://twitter.com/jvsteele
https://mastodon.social/@jsteele
Anonymous
| Thanks for the explanation, Jeff! I didn't understand a lot of it but I got the gist -- and even a "gist" is helpful when trying to deal with the site doing down. Thanks for all the work you do. |
Anonymous
|
Don't be naive, Jeff. It's Ted Cruz.
Just wait and see what happens once he tanks net neutrality in the name of freedom. |
jsteele
Site Admin Offline
Site Admin Offline
Well, I have seen my text being deleted one line at a time. Perhaps I'm being attacked by Sharyl Attkisson's hacker? (though my cat was walking on my keyboard at the time). |
Anonymous
Really? Meaning that there are a lot of hits from users from DOE? Interesting. |
Anonymous
|
Dear PP,
In DC, DOE usually means dept of energy. "Ed" is the abbreviation you are looking for. |
Anonymous
I have no idea what you just said, Jeff, but thanks for the explanation. 
|
Anonymous
| I saw a repeat spam post on the family forum the other day and I assumed that was tied to the error message - maybe being inundated with spammers? |
Anonymous
I think he just said he forgot to pay the elec bill last month... and that's somehow causing problems.... or something like that. |
Anonymous
|
I didn't find anything unusual (other than DCUM appears to be very popular with the Dept. of Education).
Love this! and too much traffic - the downsides of success ... |
Anonymous
| Tonight, after DCUM being down for a while, I can't start a new topic. When I hit submit, it tells me that the "topic does not exist." |