Tuesday 31 October the server had a hardware failure. Two banks of defect memory. After replacement of the memroy we still had the same issues and we replaced the whole server. Unfortunately this new server also had a memory error. We needed to take serious action and quickly moved to a different server we had to configure from scratch.
In order to prevent that a hardware or data-center problem can stop the channel.me service we have prepared two different servers in separate area's. This allows us to easily switch from server A to server B if one server fails or when there are problems in the area where the server is located.
But it even got worse, 9 November the power went down at our (DNS) domain provider. So our backup plan of two different servers that we could easily switch didn't work, why? Even if our new servers are running, they where not reachable because the power failure at our DNS domain provider. A DNS domain provider is like a phone book, it tells everybody where to send traffic for channel.me needs to. Server A or server B.
How can you assure this will not happen again?
- We have replaced the current server (from 2013) with two newer servers.
- Servers will be in the same data center AMS-01 and located in Hall3 and Hall4. This means they are in separated area's protected from fire, power outages, and network problems.
- There will be an automatic synchronization from master,- to the fallback server. - Server to server sync will be implemented next week. This means that we can quickly resume normal operation on the fallback server.
- When one server goes down we will receive an SMS so we can instantly switch to the other (online) server within one hour.
- We will move to a DNS provider with more backup name servers.
Is there any data lost?
No, we had a good backup plan and could rebuild a new server from scratch.
Follow us on twitter:
We will also start posting updates on twitter:
Any other questions or feedback, let us know below: