The first step in automating my system was to create a new server cluster. A highly redundant (N+1) server cluster with mega power.
What I ended up with was an 8 server cluster with 8 cores per server. 8 GB of ram per server. Continuous real-time offsite backup. 24/7/365 proactive monitoring with proactive hardware and software fixes (in other words, things are fixed before the server actually goes down). RAID 5 arrays throughout the cluster. Gigabit dedicated switch for EACH server.
Datacenter that is multi-homed, tier 1, N+1 bandwidth, redundant temperature control, and redundant generator power backups.
Before I had this setup, I had 11 servers, all with different configurations, none of them connected to each other. I had no way to load balance except by looking at the stats and trying to put new accounts on the server with the lightest load. Not exactly scientific.
With the new setup, everything was connected, mirrored, redundant, load-balancing, with dynamic dns failover.
In other words, I couldn’t make your site go down if I tried. The only thing I’m still vulnerable to is widespread outages (as in ICANN blows up, or a terrorist takes out a fiber ring).
More to come tomorrow!