IspCP from HELL!

I wanted to update ispCP on one of our test server’s – Normally all the updates go painless buy was I wrong this time.
After the upgrade, no pages were working, mails were blocked same as the ftp server was done. Tried various bug fixes but nothing seemed to work. The error occurred 23.25 29/12-08 and I first got completely done now 01:20 31/12-08.
The fix was; installed a new virtual server, reinstall ispCP – Import the backup, and to my biggest surprise the backup was corrupted. Recovered as much data as possible from the old server, tried another import, moved mail users to the right directories, gave the folders the right permissions and owners gid/uid. I gave the ispcp_daemon a test run, which resulted in all web-pages having Internal Server error 500, checked the log and saw all uid’s were wrong – Somehow the ispcp_daemon managed to change all uid’s on all users. Fine, started changing the uid’s again – Which the same result internal server error 500. Hmm, well I checked a bit around and mod_fcgi can be a bitch said in a mild way, since I had forgotten to change the ownership for each php5-fcgi-starter for each user. That done, I had to enter the DB and fix all the uid/gid’s manually, since they got changed in the first run of the ispcp_daemon.
Okay, gave the ispcp_daemon another run and now it started to look bright. All pages worked, though no mail came through, nor did the ftp work. I started checking log files for the mail server, which I used around 1 hour on, trying different things – When it certainly hit me, I have a firewall in front which also acts like a spam-filter, checked it and of course I had forgotten to change the IP for where the mails should be delivered afterwards.
Now I only had one thing left, to get the ftp server up and running – That was probably the simplest part, so glad I solved it for last – It was as simple as a inconsistency in the password for the ftp’s MySQL user, easy solved and now everything is running.
I’m sorry for any inconvenience this might have caused those users being on the test server, but I can promise anything as bad as this won’t happen again.