DCUM FORUMS CRASH - Sunday's posts lost

[jsteele]

It appears that during the nightly database cluster backup (a process that runs every night) the database cluster crashed. I have partially restored it, but it seems that almost an entire day's worth of posts was lost.

I do not know the cause of the crash other than it occurred during the backup process. The cluster is not completely restored yet. I have no idea why so many posts were lost or whether they will be recovered.

It is possible that while I try to correct the current issues the forums have to be taken out of service for brief periods of time.

[Anonymous]

Thanks for the update Jeff! I work nights and am on the west coast so I have been checking back to see when it’d come back!

[Anonymous]

Tell us more, what relational database system are you using?

[Anonymous]

What? I posted my finest work yesterday! And I probably gave somebody some sage cooking advice! Lost...like tears in the rain....

[Anonymous]

Introducing SnapDCUM! A reward for those of us on the site continuously!

[jsteele]

Anonymous wrote:Tell us more, what relational database system are you using?

I am using MySQL NDB Cluster. On Saturday morning I upgraded from version 8.0.20 to version 8.0.22. The problem may be related to this, but not necessarily. I run a backup of the data at 12:55 am. That run Sunday morning without problem. But, this morning (last night), the cluster crashed during the backup.

Of course I didn't know about the crash until this morning. I was able to start one node, but a day's worth of posts have disappeared. The second node will not start, so I am working on that now.

[Anonymous]

I wondered. Thanks Jeff.

[Anonymous]

Anonymous wrote:What? I posted my finest work yesterday! And I probably gave somebody some sage cooking advice! Lost...like tears in the rain....

Ha ha. This!

[Anonymous]

Did you check the hryroid? EBF would have prevented this.

[Anonymous]

What a relief! Sunday was pure acrimony with no value. Good riddance.

[Anonymous]

Anonymous wrote:What? I posted my finest work yesterday! And I probably gave somebody some sage cooking advice! Lost...like tears in the rain....

Kisses to you

[Anonymous]

Hey Jeff, if you could just wipe out allllll of 2020 that would be great. Thanks.

[Anonymous]

Anonymous wrote:Hey Jeff, if you could just wipe out allllll of 2020 that would be great. Thanks.

Funny because it is true. God.

[jsteele]

For those who are interested, I think I have figured out what is going on. I did a minor upgrade to the database cluster on Saturday morning. At 12:55 am Sunday morning, a backup ran. I think this backup crashed the primary database node (the cluster has two storage nodes). But, the cluster kept running on the secondary node. I didn't notice that the first node had crashed. After the crash, the primary node no longer received new data since it was dead.

At 12:55 am this morning (last night), the backup process ran again and crashed the secondary node, causing the cluster to completely crash and putting the forums out of service. This morning, when I learned that the cluster had crashed, I started the primary node. Since it had not been collecting data since 12:55 am Sunday, all of Sunday's posts were gone. Those posts probably exist on the secondary node, but that node won't start. It probably won't start because the data between the two nodes is inconsistent and the solution will probably be to wipe out all data on the secondary node. But, I am still working on this.

[Anonymous]

Ugh, sorry you’re in such a mess.

I am glad I wasn’t blocked though, which is what I thought happened at 1 am last night.