Exchange Server 2013 – Blank ECP page

I was asked to look into an issue with Exchange 2013. Report was that users were not able to send emails internally or externally. This had been happening since around 2PM and it was now 5PM when I was asked to intervene. I asked my usual questions such as what changes had there been. Specifically anything in Active Directory such as user accounts, groups and so forth, asked if there were any network changes or virtualization reconfiguration. The answer given to all of these was no. The exchange servers had already been rebooted multiple times.

Opened OWA just fine, yet no mail since around the affected time, outlook was connecting on the client PC. Then tried to connect to the Exchange Administration Console website ECP and no dice, just a blank screen with no error. Logged into the client access server, all services are running, no major errors in the event log, no errors in the IIS log for the default site.

Next up was to try the Exchange powershell console. When launched it came up a bit slow, after a minute it gave errors of  “…winrm client cannot process the request access denied…” against all three servers in the organization. Tried the exchange console on the other two servers, and one of them worked the other had the same error of access denied. Verified that AD was syncing properly, and that DNS was working. I looked into this error online which lead me down many paths, none of which were of any luck.

I called my contact back and probed some more. Turned out that the network time was changed on the primary domain controller role holding server, early in the morning, to adjust for the users complaining the time on their computer was not close enough to their cell phones. With this new information I logged into all 5 domain controllers and found 2 of them had time that was  10 minutes off due to the earlier time change. Fixed the time on the domain controllers, rebooted all 3 exchange servers and now email was flowing and the ECP page worked fine.

Office 365 connection, calendar and delegated account issues

I have seen many issues lately where Office 365 has connection issues affecting setting up new Outlook profiles, odd certificate popups, seeing delegated email accounts and shared calendars.  Turns out it is due to Outlook resolving to the root domain such as domain.com rather than autodiscover.domain.com.  This is an issue when hosting providers have mail servers that are listening on that domain and it causes Outlook to get the response and fails the connection.  The quick way to test this is to import a registry file that will change this behavior.  Copy the data below, save to a text file, rename to rootdomainfix.reg and import into the registry.

 

Exchange 2007 update HELL

In what should have been a simple 30-45 minute task to update 2 Exchange Server 2007 SP3 severs to Cumulative Update 10, ended up being many hours of hell. One server upgraded just fine, first shot no issue. The second server on the other hand would get most of the way though, then roll back. Ultimately it was having problems finding files during the install process which seemed very odd. The fix was in the Windows Environmental Variables settings. While %SystemRoot%\Temp should equal C:\Windows\Temp but in this case it did not work. After changing that setting for both TEMP and TMP the update installed just fine.

Just looking at it from this screen is deceiving:

image001

If you edit one of them, you find they are using %SystemRoot%\Temp like:

image002

Changing it to C:\Windows\Temp fixed the issue in this case

image003