Well, I have to admit I didn’t think people would actually like what I’ve been writing. I guess I enjoy playing a part in helping make others better. I’ve been to many conferences and it kills me how it’s so agenda-driven and doesn’t actually give people the information they need to be successful. I mean come on people, how many times can we hear about WorkspaceONE?! I was pleasantly surprised to see them do a “single” troubleshooting session at VMWorld on email. I get it, we all LOVE WorkspaceONE, but it’s not the Chuck Norris of mobility! Now here’s an amusing montage of Chuck Norris meme’s for your pleasure.
So, for my ActiveSync deep-dive, my main focus is going to be on a few areas that you should know:
- ActiveSync Commands you need to know
- What is an ActiveSync Partnership?
- Direct Push
- Optimizing ActiveSync
- ActiveSync Troubleshooting
ActiveSync Commands you need to know
I know you’re thinking, c’mon nerd who cares about these commands! Well, you FRIGGING SHOULD! If you don’t know things work you can’t fix problems. Do you have any idea how long it takes to get a support person at your MDM vendor? Then, they go “sorry not our problem!” Then you call Microsoft, then you consider jumping out the window! Sometimes it’s just easier to try to understand and figure out what is happening on the device.
PING
Ping is the lifeblood of ActiveSync. A PING tells the server: “Watch this stuff and tell me if anything changes.” We will cover PING more in the direct push section as its a intricate part of direct push.
FOLDER SYNC
Folder Sync, get ready for it……Syncs your Folder List! Yeah I know it’s a shocker. The key is that it will perform this command when you first provision ActiveSync and return a sync key that allows your device to keep the folder list up-to-date and listen for changes.
SYNC
Before I explain Sync, I’ll explain the collection element because its important. You see collection in two places. One, is in Get Item Estimate which determines how many items need to be synced. The collection groups together the request and response. It is used in Sync in the same fashion.
I think it’s important to highlight how this works as it can be a bit confusing:
- A user selects a folder for synchronization
- The request sends a sync key of zero. This establishes the client-server relationship and initializes the sync state.
- The server responds back with an initial value of the sync key, which is used by the client to request the initial set of items for that folder.
- Every request going forward will require sending the sync key to the server from the previous request.
- One big key here, is until the entire contents of a folder at the time of sync are synchronized it will NOT synchronize any changes to that folder.
- If not changes occur, the server sends back an empty response which tells the client to send another request which forces the server to resume the previous sync. The server caches that to optimize that action.
This diagram highlights how it works:
OPTIONS
The options command is one of the first commands you will see. It simply shows what the server supports to the client. It’s similar to the OPTIONS commands you see in CORS. The main importance behind this is that it shows you the protocols and commands supported by the server.
RESOLVE RECIPIENTS
I wish Apple used Resolve Recipients for what it’s really worth as it’s one of the most powerful commands. We use it resolve a list of recipients, get user availability, and retrieve encryption certificates.
PROVISION
This command happens during the original email activation process and will also happen at set intervals based off your ActiveSync policy. It’s simply a client query for your security policy settings set on ActiveSync. I will highlight a specific issue around this in the troubleshooting section.
ITEM OPERATIONS
Item operations is an interesting one that most people don’t know about. It’s basically a container for FETCH, EMPTY ITEM CONTENTS, and MOVE commands. Fetch is used to get search results, mailbox items, and attachments. Empty item contents is used to empty a folder of its items and move is used for moving a conversation to a different folder. Item operations is useful because it groups these commands together and processes them in a specified order for optimized batching.
MOVE ITEMS
Move items is pretty simple in that it moves items from one folder to another. The interesting thing is that it actually does a delete on one folder and an add on the other folder. They also write the source and destination IDs to the request. Once it is completed, a new server ID is assigned to the item.
SMART REPLY/SMART FORWARD
I get a ton of question about these two commands. Basically, they are used to reply or forward messages without receiving the full message from the server. One of the primary uses of it is to optimize reply and reply all. These commands identify the message that is being replied/forwarded and ties it to the new message, including the text of it. The cached version of the original message provides a big savings in network bandwidth. These commands also list the message recipients as well. This should basically be thought of as a smarter more effective version of the send mail feature.
SETTINGS
Settings is another value command that has its hand in many of the ActiveSync workflows. It is used to get/set Out of Office, get/set global properties, and setting the email password. Other settings it impacts are device information, IRM, and user information.
ActiveSync Partnerships
ActiveSync Partnerships are best thought of as the relationship between the client and server. It describes what information is created and updated between the two. It specifies device information, some basic mailbox information, and user information. One way to think of would be a mobile information store. These objects are written to a user’s Active Directory objects and are often a source of problems.
One thing that we often see with partnerships is a challenge with maintaining them. A user could delete their EAS profile and re-add it which resumes that partnership. That sometimes will carry bad behavior over to their device. We’ll cover stuff around that more in the troubleshooting section. If we are being realistic the only real good that these partnerships do for us is show when someone last synchronized and it is the lifeblood of the Add/Block/Quarantine ACLs that we leverage in some environment to ensure only trusted devices are able to access email.
Direct Push
Microsoft has tons of documentation on Direct Push as you will see. I know you can read it and everything is lovely, but I prefer to explain it really simply. Basically, your device’s ability to get new email is based off this technology. We can explain this simply in a few sentences so that people don’t get confused by some of the Microsoft jargon.
Your device will go out to ActiveSync and open a long-lived connection AKA PING. This lasts for 15m and “should” send back a HTTP/200 OK if no new email is received. If it doesn’t receive anything back, it decides that something is wrong and will send a new request in a shorter interval. It does this because it thinks something is breaking that connection. If it does get back a HTTP/200 then it will restart the process.
When new mail arrives, the server tells the client to go sync the folder with new items in it.. The client sends the SYNC command and after new email is received it will send a new PING based off the last heartbeat interval. This technology can become a real sore spot quickly because if your environment is not properly configured/optimized you will run into performance issues quickly. This is especially true with Android which has much more aggressive heartbeat requirements.
ActiveSync Optimization
This is a topic near-and-dear to my heart. I’ve been working on optimization for years in multiple environments. The reason it’s so difficult is the information out there is VERY misleading and is constantly changing. I’m going to tell you what the actual truth is on this.
YOUR EXCHANGE SERVER MUST CLOSE THE CONNECTION
Okay now that we got that out of the way! I can explain this in a very simple way. As you saw in the last section, direct push DOES NOT LIKE when you screw with its heartbeat. It gets very very unhappy with you. You will run into issues like sync failures, email delay, poor performance, and battery life issues. We can make this pretty simple and basic to understand.
I put together a simple little diagram to show how this is supposed to be done below. The key is that you need to be really thorough. As an example, on your F5 you must make sure that EVERY timeout that touches your EAS BIG-IP is set to 35m+. I’ve seen a lot of stuff overlooked where someone misses something silly like SNAT timeout. As long as the Exchange Server is what closes the connection, you will have a much better experience with EAS as whole. I think is also goes without saying is: USE LEAST CONNECTIONS! You would be amazed over the course of my career at BlackBerry how many people were using Round Robin….I know people like Batman but jeez!
Also I can’t stress enough to read the deployment guide for F5.
AirWatch-Specific Optimization
One item that I have been running down for a month or so was a really pesky issue that I talk about later as well. It looks like this in your logs, if you use a Secure Email Gateway (SEG), and you SHOULD be unless you are on Office 365 you might see it:
2017/09/30 15:30:31.256 SERVERNAME 46e485ee-c661-4aaa-b59e-a26f27bd7033 [0000000-0000000] (1409) Warn AW.Eas.Web.Listener.ProxyGateway.GetMailServerResponse *** EXCEPTION ***2017/09/30 15:30:31.256 WDZPRDWINASEG01 46e485ee-c661-4aaa-b59e-a26f27bd7033 [0000000-0000000] (1409) Warn AW.Eas.Web.Listener.ProxyGateway.GetMailServerResponse *** EXCEPTION ***System.Net.WebException: The request was aborted: The request was canceled. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at AW.Eas.Web.Listener.ProxyGateway.EndGetMailServerResponse(IAsyncResult asyncResult)Diagnostics Context
I’ve run this issue down in about 15 different ways. I finally found the solution for this problem. As I mentioned earlier, when you optimize Exchange for mobile connectivity, Exchange MUST close the connection. I found something that is undocumented and no one really tells anyone about. In the Web Listener Web.Config File you will find two entries.
#1: This field specifically called easRequestTimeOut will close any connection that surpasses the property value in minutes. This by DEFAULT is set to 15 minutes aka NICE JOB AIRWATCH ON NOT KNOWING EXCHANGE, So, this should be set to 45 minutes in most environments given the need to start at 30m at Exchange and growing outward.
#2: This field called executionTimeout MUST be longer than easRequestTimeout. I set it to 3000 aka 50 minutes.
After setting these fields, you will no longer experience aborts in your logs. I strongly URGE everyone to check your AW.Eas.Web.Listener.Log and search for abort to see if you have this issue. Many clients especially Android do not like their connections being closed abruptly.
One other item that I recommend are home-made monitoring scripts to ensure your SEG can hit Exchange and talk to your AirWatch environment at all times. You can use this code below, save it to a PS1 file, and then create a TXT file with the URLs you want to hit. You would then use a SysLog/Log Scraping to alert your team based on the Event IDs that my script creates:
Code:
## The URI list to test
$URLListFile = “C:\Monitoring_scripts\URLList.txt”
$URLList = Get-Content $URLListFile -ErrorAction SilentlyContinue
$Result = @()
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12
Foreach($Uri in $URLList) {
$time = try{
$request = $null
## Request the URI, and measure how long the response took.
$result1 = Measure-Command { $request = Invoke-WebRequest -Uri $uri }
$result1.TotalMilliseconds
}
catch
{
<# If the request generated an exception (i.e.: 500 server
error or 404 not found), we can pull the status code from the
Exception.Response property #>
$request = $_.Exception.Response
$time = -1
}
$result += [PSCustomObject] @{
Time = Get-Date;
Uri = $uri;
StatusCode = [int] $request.StatusCode;
StatusDescription = $request.StatusDescription;
ResponseLength = $request.RawContentLength;
TimeTaken = $Time;
}
}
#Prepare email body in HTML format
if ($result -ne $null)
{
Foreach($Entry in $Result)
{
if ($Entry.StatusCode -eq “200” -or $Entry.StatusCode -eq “407” -or $Entry.StatusCode -eq “401”)
{
$Message1= “The request to connect to” + ” ” +$Entry.uri + ” ” +”is successful. The response returned is” + ” ” +$Entry.StatusCode+ ” ” +$Entry.StatusDescription
Write-EventLog -LogName Application -Source “SEG Accessibility” -EntryType Information -EventId 1128 -Message $Message1
}
else
{
$Message2= “The request to connect to” + ” ” +$Entry.uri + ” ” +”has failed. The response returned is” + ” ” +$Entry.StatusCode+ ” ” +$Entry.StatusDescription
Write-EventLog -LogName Application -Source “SEG Accessibility” -EntryType Warning -EventId 3345 -Message $Message2
}
}
}
ActiveSync Troubleshooting
So…mobile email is one of the worst things to try to deal with period. The point of this entire article is to equip people with the knowledge they need to be able to troubleshoot it effectively. That’s why you should understand how the flow works, the commands that are issued, and overall how it works. For troubleshooting this stuff, you need the proper tools. I recommend the following:
- Notepad++
- Wireshark
- Fiddler
- MFCMapi
- EWSEditor
- Sometimes…LogParser Studio
So…how do we troubleshoot ActiveSync? Well it’s the process of elimination. First what are people reporting? Let’s use a few examples.
Example 1: User is not receiving email
My methodology:
- First I check the ActiveSync partnership and verify it doesn’t say “Access Denied” How do you solve that you ask? You need to set the ActiveSync Mailbox Policy attribute PolicyRefreshInterval to unlimited: Set-ActiveSyncMailboxPolicy -DevicePolicyRefreshInterval unlimited
- Next I check to make sure ActiveSync is enabled and what their last sync time was. If I see that their sync time isn’t recent, then I know its a localized issue or an issue on your email proxy.
- Next, I enable Mailbox Logging via OWA or via Powershell:
Set-CASMailbox alias -ActiveSyncDebugLogging:$true
You then reproduce the issue. You can then run the command below to collect logs
Get-MobileDeviceStatistics -Mailbox alias -GetMailboxLog:$true -NotificationEmailAddresses "admin@contoso.com"
- For localized issues, I check mail settings and make sure the password is in there. Sometimes there have been bugs where the password doesn’t save properly or a network issue. Reset Network Settings is a great feature to use if its an iOS device. I will also make sure the device can browse to their ActiveSync or Mail Proxy URL. If you don’t see that blue IIS splash screen, then we have a problem!
- I will also try to send a message from the device and see if it can be received. Sometimes if email can be sent but not received, it means that the folder sync operations failed and as I said earlier if you don’t finish that initial sync then you will never see anything new.
- Next, I will check the IIS Logs to see if requests are coming in and make sure I’m not seeing any 401s or 500s. Apple for example has logic in their client if they see numerous 500s (3 I believe) it will actually reload the mailbox entirely. That is where your Exchange optimization comes in. Connection aborts can result in 500s.
- If all that is clear, then we go deeper! If this is an Apple device and you have a developer account, you can use the Sync Diagnostics Profile found here to capture device logs that will help with ActiveSync issues.
- One other option I may look at if everything else is inconclusive is running a wireshark network capture between my mail proxy and my F5. This will help show if traffic is flowing as expected. Some load balancers will not give you source IP addresses so it’s limited in what you have access to, but you can decrypt it and understand if traffic is flowing at all.
- Follow my steps in Example 2 for a fresh re-provisioning of their mail account if all else fails.
Example 2:Sync Performance is inconsistent and email constantly stops working
- If you have already tried the stuff in my previous example, a clean re-enrollment is always a good idea.
- First, delete the mail account from the device or push deletion via MDM
- Delete ALL ActiveSync Partnerships via the Exchange Control Panel. If you do run into any issues deleting orphaned or corrupt partnerships, you will need to use MFCMapi to delete them.
- Perform a reboot or a “Reset Network Settings” for an iOS device
- Re-add the mail account or re-push it via MDM
- This process will ensure you are working with a clean slate and are not carrying over any bad behaviors.
Example 3:Deletions, reads, or moved emails are not synchronizing properly
- Before you do any testing or validation, enable the mailbox logging I referenced earlier.
- Log into OWA and make sure the issue is present there also. Everyone always forgets this! Outlook is typically in cached Exchange mode and can be inaccurate. OWA is a true representation of the mailbox.
- Remember what I covered in the command section? All of these things basically work more or less the same. Changes in folders sync. As long as it knows where the new location is, then it should work!
- Perform the old school test: Send 3 emails from Outlook to yourself, Send 3 emails from your mobile device to yourself. Delete #1 in Outlook, Delete #2 on the mobile device, Move #3 to a folder from your mobile device, Move #4 to a folder from Outlook, Read #5 on Outlook and Read #6 on your mobile device. This will show you where the actual breakpoints are.
- Once you reproduce it, you can review the logs to see where the issue is. Remember just like with calendar you can always force a re-sync by either removing the account or just disabling it and re-enabling it to force a resync.
- Also remember, that any hard-deletions (i.e. deletions that are purged and not sent to deleted items will take 45-60m to synchronize). ActiveSync relies on a source and destination folder so it is much harder to handle that. This is also the same for batch operations like deleting a bunch of emails at once.
Note: iOS 10 has a bug that will be fixed in iOS 11 around read synchronization
A few issues you might see in the logs and how to resolve them
I wanted to share a few issues that I have found in logs and how you fix them for everyone’s benefit.
TCP Idle Session Timeout:This means that something in your flow is killing the connection before ActiveSync does. This will cause a ton of performance issues. I always recommend checking all of your F5 connections. F5 has some really nice tools for troubleshooting RST packets you can leverage. Don’t forget you have to rely on the ActiveSync idle session timeouts and the Mail ones as well (i.e. your Outlook VIP which usually what ActiveSync has as its member)
Connection Abort:You might see in your logs something like “The request was aborted: the request was cancelled.” This is typically caused by a F5 bug where it isn’t sending the connection close after a 15m PING. This will create battery and sync issues if not addressed. You can fix this by applying the iRule below and applying it to your App Pools:
when HTTP_RESPONSE {
if { [HTTP::status] == 401 and [HTTP::header exists “Content-Length”]} {
HTTP::header insert “Connection” “Close”
}
}
I think that’s likely it for now. I expect to continually update this as time goes on. ActiveSync continues to evolve and I do wonder if it will soon transition to MAPI over HTTP, but only time will tell there. The challenge with ActiveSync is that vendors constantly try to push its limits and interpret it as they see fit. This can often require us as mobile engineers to be creative and determine what they’re actually doing. Email is a vital part of every organization and if you don’t understand it you will certainly drown.
