Real life Tales of Server Side Sync

The idea of Server Side Synchronization for Microsoft Dynamics 365 is very exciting. It can enable the syncing you know and love to run even when Outlook (with CRM for Outlook) is not running. 

In my organization, we have teams who are scheduling appointments for other users. So it is important that Outlook reflects the most up-to-date information. If there are overlapping appointments that need to be rescheduled or if a user is not prepared for an appointment this is lost revenue.

So we enabled Server Side Sync to get past some of the limitations of CRM for Outlook (primarily that there needs to be a primary syncing computer and that syncing only occurs when Outlook is running). Unfortunately this did not run as smoothly as intended. Appointments were taking 12 hours to sync if they synced at all!

Things are now running smoothly (mostly) but it took time to get everything right and a lot of help from Microsoft support. Let's talk about the changes and items that helped us.

Background
We have been using Dynamics in my organization since CRM 4.0 and were on D365 8.2.2 on-premise when launching Server Side Sync. CRM for Outlook was in use for most users for the whole implementation. Users are syncing Calls, Tasks and Appointments. We are also using Exchange on-premise.

When enabling Server Side Sync we found that many (40,000+) duplicate calls and tasks were being created. We believe this was due to the history in Outlook and potential disconnects with the data (for example, the task knew it synced at one point but the Outlook profile was rebuilt and lost the connection so a new task was created).

Getting Started
When we launched Server Side Sync we learned several things we should have looked at prior to turning on for all users:

• We enabled SSS for all users and all options at once (incoming email, outgoing email, and Appointments/Contacts/Tasks). When we were having the performance issues we changed incoming email back to CRM for Outlook.
• The Email Server Profile was configured as part of our upgrade to 2016 so the "Process Email From" setting was set to October 2016. We updated this to the date SSS was turned on so older emails would not be processed.
• Outlook sync filters were also configured to sync items after October 2016. We also updated these to the same date SSS was enabled.
• Power: Throughout the troubleshooting we added more power to everything related to our asynchronous services servers. More CPU, more memory and additional processor cores.
• Also verify that the Organizer field is on the form for appointments and being set correctly. Verify this is set to a real user who will be syncing For example we had this set to a service account for some appointments (set via workflow) and these did not sync because that user did not have a calendar to sync to.
• Review your mailbox records and clean up any you do not need. If the user does not need to sync, do not turn it on to sync. If it is a service account or placeholder account of some kind, turn it off and disable it.

Nitty Gritty Settings
There are several settings that Microsoft recommended we update. I will explain the changes we made. When you look at these options you should look at your environment to see which values make sense.

Max Concurrent Connections
There is a suspicious setting in the Exchange Server Profile called "Max Concurrent Connections". This is related to the throttling policy in Exchange. We increased this setting to 100 initially (default is 10) and then increased to 1000. This means you also need to be working with your exchange team to determine what setting they are comfortable with.

AsyncMailboxesPerEvent
By default AsyncMailboxesPerEvent is set to 5. This means that 5 mailboxes will be attempting to process at once on the same thread. We were concerned that if there was an error in one mailbox it could delay the other 4 mailboxes in that group. So this setting was updated to 1. The default has been updated to 1 in some newer versions.

AsyncLongJobThreasholdInSeconds
The AsyncLongJobThreasholdInSeconds was updated to 86400. This value is typically lower for online environments where there are concerns about long-running async operations. The increase was recommended in our case to avoid any issues because of throttling (on the async server) because the sync was taking a long time.

X-Anchor Header and v8.2.3
In our fiddler tracing on the async servers we found that the X-Anchor mailbox header was not included. This is recommended as a best practice when working with exchange (source).

To add this header we needed to upgrade to the latest version of D365 on-premise 8.2.3. The X-Anchor change is not included in the release notes and is not enabled by default. To enable, after upgrading all servers you must update the Organization DB settings to set UseXAnchorMailboxInExchangeRequestHeader to true.

After upgrading to 8.2.3 and turning on the X-Anchor header we saw immediate improvements in the SSS behavior. Additionally some backlogs (SubscriptionSyncEntryOutlookBase table) of items to sync that we were monitoring began to improve after weeks of minimal change.

However we did have some issues with the upgrade. Check out my Upgrade from 8.2.2 to 8.2.3 blog for more information.

Related to SSS, the mailboxes processing during the upgrade needed to have "Test & Enable" repeated to keep functioning and the message sent from this process changed. Formerly the "Test & Enable" email message had a subject of "Test Message". Now it has a subject of "Your Mailbox is now connected to Dynamics 365". Post upgrade, we no longer saw issues with duplicate tasks being created but instead we did see cases of old (completed) activities being updated to a status of open. We monitored these cases via Advanced Find to clean up.

"Understanding" the Mailbox Details
When reviewing a specific Mailbox inside of Dynamics 365 there is a button on the ribbon to "Download Mailbox Details". This gives you a lovely file of information you do not know how to read. So here is what I learned about this file through the process.

1. Current Date and Time in UTC - All the times in the file are in UTC. If you don't feel like doing the conversion (me!) then you can just check the file name to compare current time to the time in the file.
2. Processing status - 1 means currently processing. 2 means not processing right now.
3. Time of last Processing attempt - also stored in CRM "Date Processing Last Attempted"
4. Next Processing for ACT - ACT = Appointments/Contacts/Tasks. This is the next time a sync will be attempted. This should be just a few minutes after the last attempted processing and should be in the future (i.e. if it's a past time then for some reason it didn't process when it was supposed to).
5. Last Successful Sync - Time of last successful, completed sync
6. Last Error on Sync - The last time a sync encountered an error
7. Processing details - number of items successfully processed on last sync and number of items that failed during the last sync
8. Error details - full details of all recent errors. These are also visible from the Alerts area of the mailbox. This will show you the full error details if it was not all visible in the interface.

Other Stuff 
There were a few things I skipped over. Throughout the process of working with Microsoft support we provided many logs, traces, mailbox statistics, mailbox details, memory dumps, perfmon details, etc. Some of these were from D365 but also some additionally from Exchange. We also used this tool to look at mailbox details but this did not lead to actionable steps.

After all of the steps above we are still monitoring progress and making sure things are processing as expected. To do this you can review the Mailboxes view and add the "Date of Last Processing Attempt" as well as the Status field. If it has not processed in a reasonable amount of time or is in a failed status, it is probably time to review the Alerts on that mailbox and to run "Test & Enable" again. Keep in mind you can also review all alerts from the Email Server Profile record as well (all Alerts for mailboxes linked to that profile).

Finally don't forget that Server Side Sync for appointments will send out an Appointment invite or changes to the appointment to all attendees including those external to the organization. This was not acceptable in our business so we used a workflow to add text to the appointment that was captured by Exchange to prevent send as recommended in this Community Post.

So has anyone else used Server Side Sync successfully? What has been your experience? What other settings or tips to improve performance have you tried?

Appendix: Queries
There were several queries our consultant worked on with Microsoft to gather more details. I take no credit for their creation, just for seeing the results.

Backlog Status
Shows items currently waiting to be synced.


select sq.*
from (
select sub.FullName, eo.ObjectTypeCode, e.name as ObjectName, eo.SyncState, count(*) reccount
from SubscriptionSyncEntryOutlookBase eo with(nolock) inner join
(select distinct e.name, e.ObjectTypeCode from metadataschema.Entity e with(nolock)) e on eo.ObjectTypeCode=e.ObjectTypeCode inner join
subscription s with(nolock) on eo.SubscriptionId=s.SubscriptionId inner join
systemuserbase sub with(nolock) on s.SystemUserId=sub.SystemUserId
where eo.SyncState<>0
group by sub.FullName, eo.ObjectTypeCode, e.name, eo.SyncState
) sq
order by sq.reccount desc;

Mailbox Status
Show status and details for a particular mailbox. For part of our troubleshooting we actually set this up to run on a schedule and capture the details for specific mailboxes every 30 mins. Update the names list at the end to the users you want to capture for.

select sub.FullName
,m.ProcessingStateCode
,m.ACTDeliveryMethod
,m.UTCConversionTimeZoneCode
,m.PostponeMailboxProcessingUntil
,m.OutgoingEmailDeliveryMethod
,m.Username
,m.ProcessEmailReceivedAfter
,m.ModifiedOnBehalfBy
,m.NoEmailCount
,m.MailboxId
,m.IncomingEmailDeliveryMethod
,m.EnabledForIncomingEmail
,m.NoACTCount
,m.EmailAddress
,m.OutgoingEmailStatus
,m.EWSURL
,m.OrganizationId
,m.PostponeSendingUntil
,m.EmailServerProfile
,m.ModifiedOn
,m.OwnerId
,m.LastActiveOn
,m.TimeZoneRuleVersionNumber
,m.LastAutoDiscoveredOn
,m.TestEmailConfigurationScheduled
,m.IsEmailAddressApprovedByO365Admin
,m.ModifiedBy
,m.Password
,m.TestMailboxAccessCompletedOn
,m.ExchangeSyncStateXml
,m.ProcessAndDeleteEmails
,m.ReceivingPostponedUntil
,m.RegardingObjectId
,m.EntityImageId
,m.CreatedOn
,m.EmailRouterAccessApproval
,m.StatusCode
,m.ReceivingPostponedUntilForACT
,m.StateCode
,m.PostponeTestEmailConfigurationUntil
,m.UndeliverableFolder
,m.MailboxProcessingContext
,m.Name
,m.IsForwardMailbox
,m.TestEmailConfigurationRetryCount
,m.CreatedOnBehalfBy
,m.AllowEmailConnectorToUseCredentials
,m.HostId
,m.LastMessageId
,m.OwningBusinessUnit
,m.EnabledForACT
,m.IncomingEmailStatus
,m.CreatedBy
,m.TransientFailureCount
,m.IsACTSyncOrgFlagSet
,m.ACTStatus
,m.EnabledForOutgoingEmail
,m.RegardingObjectTypeCode
,m.RegardingObjectIdName
,m.OwnerIdType
,m.LastSuccessfulSyncCompletedOn
,m.ProcessedTimes
,m.AverageTotalDuration
,m.LastSyncErrorOccurredOn
,m.ItemsFailedForLastSync
,m.LastSyncErrorMachineName
,m.LastSyncErrorCount
,m.LastSyncErrorCode
,m.ItemsProcessedForLastSync
,m.LastSyncStartedOn
,m.LastSyncError
,m.OrgMarkedAsPrimaryForExchangeSync
,m.LastDuration
,m.ForcedUnlockCount
,m.ProcessingLastAttemptedOn
,m.LastMailboxForcedUnlockOccurredOn
,m.IsServiceAccount
,m.VerboseLoggingEnabled
,m.FolderHierarchy
,m.OfficeAppsDeploymentCompleteOn
,m.PostponeOfficeAppsDeploymentUntil
,m.OfficeAppsDeploymentScheduled
,m.MailboxStatus
,m.OfficeAppsDeploymentError
,m.OfficeAppsDeploymentStatus
from mailboxbase m with(nolock) inner join
SystemUserBase sub with(nolock) on m.OwnerId=sub.SystemUserId
where m.statecode=0 and sub.FullName in ('Last1, First1','Last2, First2','Last3, First3');