Adding Exchange ICS files via Outlook on the Web (Office 365 OWA)

I often access my Exchange Online mailbox via my browser when working from home rather than firing up the Outlook client on the work laptop and recently had cause to add an appointment that was in an ICS file to my calendar.

Now this didn’t used to be possible via OWA but I’ve discovered it now is, by going New > Add calendar > from file:

open calendar

then browsing to the file in question

add calendar file

and then clicking on save, and low and behold, the appointments in my calendar, no full fat client required!

Not sure how long the feature has been there for but I’m glad it has been added.

Exchange Hybrid Free/Busy and Proxy issue

I was investigating an issue with a customer’s deployment where their on-premise Exchange users couldn’t see the free/busy information of the pilot users in Office 365.

After looking through the logs on all of their servers I found event id 4002 on one of them:

exchange 4002 tmg error

So, the EWS request was being blocked by an authenticating proxy, in their case TMG.

After a little investigation on the TMG it was identified that their default web filtering policy was restricted to Authenticated Users and as this request was coming as the Exchange server account itself it was being denied.

Now, they already had a policy in place on the TMG with a set of addresses that could by-pass the authentication and go straight out, so simply adding *.office365.com to this list allowed Exchange to connect to the EWS of Office 365 and get the free/busy information.

Clients were now working as expected and all should have been good, however the Exchange servers should have had direct internet access and shouldn’t have been trying to go through the proxy in the first place. The customer wanted to know why, so I had to get it to work without entering the url in the by-pass list.

I tried disabling the proxy options within my Internet Explorer settings on the Exchange servers and confirmed I was able to get out onto the internet without going through the proxy. Now, why wasn’t the Exchange server itself able to do this?

I set about trying to work out how to run Internet Explorer as the system account, but how do to do this? Well the answer was to use psexec from the sysinternal tools (available from: https://technet.microsoft.com/en-us/sysinternals/bb545027)

I started on the server that had generated the error, the command I needed was:

psexec.exe -is "c:\program files\internet explorer\iexplore.exe"

the switch -is set the program to run as the system and in interactive mode (without the i you’d not see the browser window as I found on first attempt)

Upon opening the Internet Settings in the newly viewable browser window I discovered that there were proxy settings enabled. Disabling these options allowed free/busy to work, interesting!

I ran the same test on the other Exchange servers but none of them had proxy information entered. Strange.

Until I came across: http://blogs.msdn.com/b/askie/archive/2013/05/09/user-proxy-settings-showing-up-in-local-system-account-correct-way-to-apply-proxy-settings.aspx  which talked about how Group Policy settings, if not correctly configured could result in what I was seeing.

So I had a check, all Exchange servers had the same policy applied specifying the Internet Explorer proxy options but it was set to “Authenticated users” only, which shouldn’t include the service and system accounts that Exchange was running under.

But on the server that was reporting this error I did find proxy information located in the registry key: HKEY_USERS\.DEFAULT. \Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections and after I replaced this with the blank key from another server everything worked and Exchange was no longer going through the proxy.

Not having access to their GPO I wasn’t able to confirm its configuration but I suspect it was changed at some point but the server in question had already applied the incorrect settings which are not superseded by the correct configuration.

So the lesson here is, check GPO and the registry for proxy settings and ensure that Exchange servers (and clients) do not go through a proxy when trying to reach Office 365 services and if they must, ensure that it does not require authentication for the Office 365 addresses.

Microsoft does call this out on their Office 365 URL and IP address webpage – https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2

Testing SMTP connectivity between On-premise Exchange and Exchange Online

Having recently been troubleshooting a customers hybrid Exchange environment I thought I’d write about my experience.

The customer had an Exchange 2010 deployment and the Hybrid Configuration Wizard had already been run but they were having mail flow problems from their On-Prem to Online mailboxes.

First I decided to perform the basic SMTP connectivity test to Exchange Online Protection to make sure their address was allowed to send mail through. To do this from a command prompt I ran:

telnet [tenantname].mail.protection.outlook.com 25
ehlo
mail from: onprem.mailbox@domain.com
rcpt to: cloud.mailbox@domain.com

Now, normally you should get a “250 2.1.5 Recipient OK” message back however we received:

smtp blacklist

Our IP address is blacklisted, opps! So we proceeded to resolve the blacklist issue, upon investigation I discovered that the Exchange servers were actually connecting externally as the default IP address on their internet connection, which client devices also used.

I always recommend that mail servers should be presented externally via an IP address that isn’t used by other services just to prevent issues like this.

Once removed from the blacklist we were able to perform the test again successfully, all good, or so we thought!

Email was still not flowing and building up in the queue, at this point I turned on verbose logging on the outbound SMTP connector and went to check the SmtpSend folder for the logs.

Upon opening the logs I found the following, repeated

#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2015-09-29T00:08:44.653Z,Outbound to Office 365,08D2B5B9B93409D6,0,,213.199.154.23:25,*,,attempting to connect
2015-09-29T00:08:44.653Z,Outbound to Office 365,08D2B5B9B93409D6,1,[INTERNAL IP]:14101,213.199.154.23:25,+,,
2015-09-29T00:08:44.669Z,Outbound to Office 365,08D2B5B9B93409D6,2,[INTERNAL IP]:14101,213.199.154.23:25,<,"220 AM1FFO11OLC001.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Tue, 29 Sep 2015 00:08:43 +0000",
2015-09-29T00:08:44.669Z,Outbound to Office 365,08D2B5B9B93409D6,3,[INTERNAL IP]:14101,213.199.154.23:25,>,EHLO [HOSTNAME],
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,4,[INTERNAL IP]:14101,213.199.154.23:25,<,250-AM1FFO11OLC001.mail.protection.outlook.com Hello [IP ADDRESS],
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,5,[INTERNAL IP]:14101,213.199.154.23:25,<,250-SIZE 157286400,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,6,[INTERNAL IP]:14101,213.199.154.23:25,<,250-PIPELINING,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,7,[INTERNAL IP]:14101,213.199.154.23:25,<,250-DSN,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,8,[INTERNAL IP]:14101,213.199.154.23:25,<,250-ENHANCEDSTATUSCODES,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,9,[INTERNAL IP]:14101,213.199.154.23:25,<,250-STARTTLS,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,10,[INTERNAL IP]:14101,213.199.154.23:25,<,250-8BITMIME,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,11,[INTERNAL IP]:14101,213.199.154.23:25,<,250-BINARYMIME,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,12,[INTERNAL IP]:14101,213.199.154.23:25,<,250 CHUNKING,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,13,[INTERNAL IP]:14101,213.199.154.23:25,>,STARTTLS,
2015-09-29T00:08:44.685Z,Outbound to Office 365,08D2B5B9B93409D6,14,[INTERNAL IP]:14101,213.199.154.23:25,-,,Remote

Now, I like to play spot the difference with logs from working deployments to try and spot things that are not working or incorrect, so I got a log from a working deployment:

2015-04-11T16:24:07.627Z,Outbound to Office 365,08D21A598CD8BB94,0,,213.199.154.23:25,*,,attempting to connect
2015-04-11T16:24:07.643Z,Outbound to Office 365,08D21A598CD8BB94,1,[INTERNAL IP]:30731,213.199.154.23:25,+,,
2015-04-11T16:24:07.659Z,Outbound to Office 365,08D21A598CD8BB94,2,[INTERNAL IP]:30731,213.199.154.23:25,<,"220 AM1FFO11FD013.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Sat, 11 Apr 2015 16:25:42 +0000",
2015-04-11T16:24:07.659Z,Outbound to Office 365,08D21A598CD8BB94,3,[INTERNAL IP]:30731,213.199.154.23:25,>,EHLO mail.[DOMAIN].com,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,4,[INTERNAL IP]:30731,213.199.154.23:25,<,250-AM1FFO11FD013.mail.protection.outlook.com Hello [IP ADDRESS],
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,5,[INTERNAL IP]:30731,213.199.154.23:25,<,250-SIZE 157286400,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,6,[INTERNAL IP]:30731,213.199.154.23:25,<,250-PIPELINING,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,7,[INTERNAL IP]:30731,213.199.154.23:25,<,250-DSN,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,8,[INTERNAL IP]:30731,213.199.154.23:25,<,250-ENHANCEDSTATUSCODES,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,9,[INTERNAL IP]:30731,213.199.154.23:25,<,250-STARTTLS,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,10,[INTERNAL IP]:30731,213.199.154.23:25,<,250-8BITMIME,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,11,[INTERNAL IP]:30731,213.199.154.23:25,<,250-BINARYMIME,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,12,[INTERNAL IP]:30731,213.199.154.23:25,<,250 CHUNKING,
2015-04-11T16:24:07.690Z,Outbound to Office 365,08D21A598CD8BB94,13,[INTERNAL IP]:30731,213.199.154.23:25,>,STARTTLS,
2015-04-11T16:24:07.706Z,Outbound to Office 365,08D21A598CD8BB94,14,[INTERNAL IP]:30731,213.199.154.23:25,<,220 2.0.0 SMTP server ready,
2015-04-11T16:24:07.706Z,Outbound to Office 365,08D21A598CD8BB94,15,[INTERNAL IP]:30731,213.199.154.23:25,*,,Sending certificate
2015-04-11T16:24:07.706Z,Outbound to Office 365,08D21A598CD8BB94,16,[INTERNAL IP]:30731,213.199.154.23:25,*,"CN=mail.[DOMAIN].com, OU=PositiveSSL Multi-Domain, OU=Domain Control Validated",Certificate subject
2015-04-11T16:24:07.706Z,Outbound to Office 365,08D21A598CD8BB94,17,[INTERNAL IP]:30731,213.199.154.23:25,*,"CN=COMODO RSA Domain Validation Secure Server CA, O=COMODO CA Limited, L=Salford, S=Greater Manchester, C=GB",Certificate issuer name
2015-04-11T16:24:07.706Z,Outbound to Office 365,08D21A598CD8BB94,18,[INTERNAL IP]:30731,213.199.154.23:25,*,[CERTIFICATE SERIAL NUMBER],Certificate serial number

As you can see, after the STARTTLS we should see the “220 2.0.0 SMTP server ready” response followed by the establishment of the TLS connection, but in our case we weren’t and presumably the “Remote” entry was to say it had timed out.

Now I had to find a way to manually establish a TLS SMTP connection, I’m not aware of a way of using the standard telnet based approach to do this so had to find an alternative.

That alternative was to use openSSL to establish the connection, to do this you can use the following command:

openssl s_client -starttls smtp -crlf -connect [tenantname].mail.protection.outlook.com:25

When run work a working machine the connection should establish and look similar to the below (full output is much longer):

smtp TLS successful

However from the customers Exchange server it would not connect

smtp TLS unsuccessful

The customer raised it with their firewall team who advised that there was some “app awareness configured and it only allowed smtp over tls” not sure how it worked from a telnet client or what they changed to resolve it but that’s the dark art of firewall appliances!