I was investigating an issue with a customer’s deployment where their on-premise Exchange users couldn’t see the free/busy information of the pilot users in Office 365.
After looking through the logs on all of their servers I found event id 4002 on one of them:
So, the EWS request was being blocked by an authenticating proxy, in their case TMG.
After a little investigation on the TMG it was identified that their default web filtering policy was restricted to Authenticated Users and as this request was coming as the Exchange server account itself it was being denied.
Now, they already had a policy in place on the TMG with a set of addresses that could by-pass the authentication and go straight out, so simply adding *.office365.com to this list allowed Exchange to connect to the EWS of Office 365 and get the free/busy information.
Clients were now working as expected and all should have been good, however the Exchange servers should have had direct internet access and shouldn’t have been trying to go through the proxy in the first place. The customer wanted to know why, so I had to get it to work without entering the url in the by-pass list.
I tried disabling the proxy options within my Internet Explorer settings on the Exchange servers and confirmed I was able to get out onto the internet without going through the proxy. Now, why wasn’t the Exchange server itself able to do this?
I set about trying to work out how to run Internet Explorer as the system account, but how do to do this? Well the answer was to use psexec from the sysinternal tools (available from: https://technet.microsoft.com/en-us/sysinternals/bb545027)
I started on the server that had generated the error, the command I needed was:
psexec.exe -is "c:\program files\internet explorer\iexplore.exe"
the switch -is set the program to run as the system and in interactive mode (without the i you’d not see the browser window as I found on first attempt)
Upon opening the Internet Settings in the newly viewable browser window I discovered that there were proxy settings enabled. Disabling these options allowed free/busy to work, interesting!
I ran the same test on the other Exchange servers but none of them had proxy information entered. Strange.
Until I came across: http://blogs.msdn.com/b/askie/archive/2013/05/09/user-proxy-settings-showing-up-in-local-system-account-correct-way-to-apply-proxy-settings.aspx which talked about how Group Policy settings, if not correctly configured could result in what I was seeing.
So I had a check, all Exchange servers had the same policy applied specifying the Internet Explorer proxy options but it was set to “Authenticated users” only, which shouldn’t include the service and system accounts that Exchange was running under.
But on the server that was reporting this error I did find proxy information located in the registry key: HKEY_USERS\.DEFAULT. \Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections and after I replaced this with the blank key from another server everything worked and Exchange was no longer going through the proxy.
Not having access to their GPO I wasn’t able to confirm its configuration but I suspect it was changed at some point but the server in question had already applied the incorrect settings which are not superseded by the correct configuration.
So the lesson here is, check GPO and the registry for proxy settings and ensure that Exchange servers (and clients) do not go through a proxy when trying to reach Office 365 services and if they must, ensure that it does not require authentication for the Office 365 addresses.
Microsoft does call this out on their Office 365 URL and IP address webpage – https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2