Real life Tales of Server Side Sync

The idea of Server Side Synchronization for Microsoft Dynamics 365 is very exciting. It can enable the syncing you know and love to run even when Outlook (with CRM for Outlook) is not running. 

In my organization, we have teams who are scheduling appointments for other users. So it is important that Outlook reflects the most up-to-date information. If there are overlapping appointments that need to be rescheduled or if a user is not prepared for an appointment this is lost revenue.

So we enabled Server Side Sync to get past some of the limitations of CRM for Outlook (primarily that there needs to be a primary syncing computer and that syncing only occurs when Outlook is running). Unfortunately this did not run as smoothly as intended. Appointments were taking 12 hours to sync if they synced at all!

Things are now running smoothly (mostly) but it took time to get everything right and a lot of help from Microsoft support. Let's talk about the changes and items that helped us.

Background
We have been using Dynamics in my organization since CRM 4.0 and were on D365 8.2.2 on-premise when launching Server Side Sync. CRM for Outlook was in use for most users for the whole implementation. Users are syncing Calls, Tasks and Appointments. We are also using Exchange on-premise.

When enabling Server Side Sync we found that many (40,000+) duplicate calls and tasks were being created. We believe this was due to the history in Outlook and potential disconnects with the data (for example, the task knew it synced at one point but the Outlook profile was rebuilt and lost the connection so a new task was created).

Getting Started
When we launched Server Side Sync we learned several things we should have looked at prior to turning on for all users:

• We enabled SSS for all users and all options at once (incoming email, outgoing email, and Appointments/Contacts/Tasks). When we were having the performance issues we changed incoming email back to CRM for Outlook.
• The Email Server Profile was configured as part of our upgrade to 2016 so the "Process Email From" setting was set to October 2016. We updated this to the date SSS was turned on so older emails would not be processed.
• Outlook sync filters were also configured to sync items after October 2016. We also updated these to the same date SSS was enabled.
• Power: Throughout the troubleshooting we added more power to everything related to our asynchronous services servers. More CPU, more memory and additional processor cores.
• Also verify that the Organizer field is on the form for appointments and being set correctly. Verify this is set to a real user who will be syncing For example we had this set to a service account for some appointments (set via workflow) and these did not sync because that user did not have a calendar to sync to.
• Review your mailbox records and clean up any you do not need. If the user does not need to sync, do not turn it on to sync. If it is a service account or placeholder account of some kind, turn it off and disable it.

Nitty Gritty Settings
There are several settings that Microsoft recommended we update. I will explain the changes we made. When you look at these options you should look at your environment to see which values make sense.

Max Concurrent Connections
There is a suspicious setting in the Exchange Server Profile called "Max Concurrent Connections". This is related to the throttling policy in Exchange. We increased this setting to 100 initially (default is 10) and then increased to 1000. This means you also need to be working with your exchange team to determine what setting they are comfortable with.

AsyncMailboxesPerEvent
By default AsyncMailboxesPerEvent is set to 5. This means that 5 mailboxes will be attempting to process at once on the same thread. We were concerned that if there was an error in one mailbox it could delay the other 4 mailboxes in that group. So this setting was updated to 1. The default has been updated to 1 in some newer versions.

AsyncLongJobThreasholdInSeconds
The AsyncLongJobThreasholdInSeconds was updated to 86400. This value is typically lower for online environments where there are concerns about long-running async operations. The increase was recommended in our case to avoid any issues because of throttling (on the async server) because the sync was taking a long time.

X-Anchor Header and v8.2.3
In our fiddler tracing on the async servers we found that the X-Anchor mailbox header was not included. This is recommended as a best practice when working with exchange (source).

To add this header we needed to upgrade to the latest version of D365 on-premise 8.2.3. The X-Anchor change is not included in the release notes and is not enabled by default. To enable, after upgrading all servers you must update the Organization DB settings to set UseXAnchorMailboxInExchangeRequestHeader to true.

After upgrading to 8.2.3 and turning on the X-Anchor header we saw immediate improvements in the SSS behavior. Additionally some backlogs (SubscriptionSyncEntryOutlookBase table) of items to sync that we were monitoring began to improve after weeks of minimal change.

However we did have some issues with the upgrade. Check out my Upgrade from 8.2.2 to 8.2.3 blog for more information.

Related to SSS, the mailboxes processing during the upgrade needed to have "Test & Enable" repeated to keep functioning and the message sent from this process changed. Formerly the "Test & Enable" email message had a subject of "Test Message". Now it has a subject of "Your Mailbox is now connected to Dynamics 365". Post upgrade, we no longer saw issues with duplicate tasks being created but instead we did see cases of old (completed) activities being updated to a status of open. We monitored these cases via Advanced Find to clean up.

"Understanding" the Mailbox Details
When reviewing a specific Mailbox inside of Dynamics 365 there is a button on the ribbon to "Download Mailbox Details". This gives you a lovely file of information you do not know how to read. So here is what I learned about this file through the process.

1. Current Date and Time in UTC - All the times in the file are in UTC. If you don't feel like doing the conversion (me!) then you can just check the file name to compare current time to the time in the file.
2. Processing status - 1 means currently processing. 2 means not processing right now.
3. Time of last Processing attempt - also stored in CRM "Date Processing Last Attempted"
4. Next Processing for ACT - ACT = Appointments/Contacts/Tasks. This is the next time a sync will be attempted. This should be just a few minutes after the last attempted processing and should be in the future (i.e. if it's a past time then for some reason it didn't process when it was supposed to).
5. Last Successful Sync - Time of last successful, completed sync
6. Last Error on Sync - The last time a sync encountered an error
7. Processing details - number of items successfully processed on last sync and number of items that failed during the last sync
8. Error details - full details of all recent errors. These are also visible from the Alerts area of the mailbox. This will show you the full error details if it was not all visible in the interface.

Other Stuff 
There were a few things I skipped over. Throughout the process of working with Microsoft support we provided many logs, traces, mailbox statistics, mailbox details, memory dumps, perfmon details, etc. Some of these were from D365 but also some additionally from Exchange. We also used this tool to look at mailbox details but this did not lead to actionable steps.

After all of the steps above we are still monitoring progress and making sure things are processing as expected. To do this you can review the Mailboxes view and add the "Date of Last Processing Attempt" as well as the Status field. If it has not processed in a reasonable amount of time or is in a failed status, it is probably time to review the Alerts on that mailbox and to run "Test & Enable" again. Keep in mind you can also review all alerts from the Email Server Profile record as well (all Alerts for mailboxes linked to that profile).

Finally don't forget that Server Side Sync for appointments will send out an Appointment invite or changes to the appointment to all attendees including those external to the organization. This was not acceptable in our business so we used a workflow to add text to the appointment that was captured by Exchange to prevent send as recommended in this Community Post.

So has anyone else used Server Side Sync successfully? What has been your experience? What other settings or tips to improve performance have you tried?

Appendix: Queries
There were several queries our consultant worked on with Microsoft to gather more details. I take no credit for their creation, just for seeing the results.

Backlog Status
Shows items currently waiting to be synced.


select sq.*
from (
select sub.FullName, eo.ObjectTypeCode, e.name as ObjectName, eo.SyncState, count(*) reccount
from SubscriptionSyncEntryOutlookBase eo with(nolock) inner join
(select distinct e.name, e.ObjectTypeCode from metadataschema.Entity e with(nolock)) e on eo.ObjectTypeCode=e.ObjectTypeCode inner join
subscription s with(nolock) on eo.SubscriptionId=s.SubscriptionId inner join
systemuserbase sub with(nolock) on s.SystemUserId=sub.SystemUserId
where eo.SyncState<>0
group by sub.FullName, eo.ObjectTypeCode, e.name, eo.SyncState
) sq
order by sq.reccount desc;

Mailbox Status
Show status and details for a particular mailbox. For part of our troubleshooting we actually set this up to run on a schedule and capture the details for specific mailboxes every 30 mins. Update the names list at the end to the users you want to capture for.

select sub.FullName
,m.ProcessingStateCode
,m.ACTDeliveryMethod
,m.UTCConversionTimeZoneCode
,m.PostponeMailboxProcessingUntil
,m.OutgoingEmailDeliveryMethod
,m.Username
,m.ProcessEmailReceivedAfter
,m.ModifiedOnBehalfBy
,m.NoEmailCount
,m.MailboxId
,m.IncomingEmailDeliveryMethod
,m.EnabledForIncomingEmail
,m.NoACTCount
,m.EmailAddress
,m.OutgoingEmailStatus
,m.EWSURL
,m.OrganizationId
,m.PostponeSendingUntil
,m.EmailServerProfile
,m.ModifiedOn
,m.OwnerId
,m.LastActiveOn
,m.TimeZoneRuleVersionNumber
,m.LastAutoDiscoveredOn
,m.TestEmailConfigurationScheduled
,m.IsEmailAddressApprovedByO365Admin
,m.ModifiedBy
,m.Password
,m.TestMailboxAccessCompletedOn
,m.ExchangeSyncStateXml
,m.ProcessAndDeleteEmails
,m.ReceivingPostponedUntil
,m.RegardingObjectId
,m.EntityImageId
,m.CreatedOn
,m.EmailRouterAccessApproval
,m.StatusCode
,m.ReceivingPostponedUntilForACT
,m.StateCode
,m.PostponeTestEmailConfigurationUntil
,m.UndeliverableFolder
,m.MailboxProcessingContext
,m.Name
,m.IsForwardMailbox
,m.TestEmailConfigurationRetryCount
,m.CreatedOnBehalfBy
,m.AllowEmailConnectorToUseCredentials
,m.HostId
,m.LastMessageId
,m.OwningBusinessUnit
,m.EnabledForACT
,m.IncomingEmailStatus
,m.CreatedBy
,m.TransientFailureCount
,m.IsACTSyncOrgFlagSet
,m.ACTStatus
,m.EnabledForOutgoingEmail
,m.RegardingObjectTypeCode
,m.RegardingObjectIdName
,m.OwnerIdType
,m.LastSuccessfulSyncCompletedOn
,m.ProcessedTimes
,m.AverageTotalDuration
,m.LastSyncErrorOccurredOn
,m.ItemsFailedForLastSync
,m.LastSyncErrorMachineName
,m.LastSyncErrorCount
,m.LastSyncErrorCode
,m.ItemsProcessedForLastSync
,m.LastSyncStartedOn
,m.LastSyncError
,m.OrgMarkedAsPrimaryForExchangeSync
,m.LastDuration
,m.ForcedUnlockCount
,m.ProcessingLastAttemptedOn
,m.LastMailboxForcedUnlockOccurredOn
,m.IsServiceAccount
,m.VerboseLoggingEnabled
,m.FolderHierarchy
,m.OfficeAppsDeploymentCompleteOn
,m.PostponeOfficeAppsDeploymentUntil
,m.OfficeAppsDeploymentScheduled
,m.MailboxStatus
,m.OfficeAppsDeploymentError
,m.OfficeAppsDeploymentStatus
from mailboxbase m with(nolock) inner join
SystemUserBase sub with(nolock) on m.OwnerId=sub.SystemUserId
where m.statecode=0 and sub.FullName in ('Last1, First1','Last2, First2','Last3, First3');

Microsoft Dynamics 365 Upgrade from 8.1 to 8.2.2

Recently my organization upgraded our 8.1 on-premise instance of Dynamics 365/CRM to version 8.2.2. This went very successfully and we had no major issues when testing or on go-live. The front-end changes are minimal so users were able to use the new version without needing re-training.

However, there were a few things that popped up and surprised us post upgrade:

• Call Direction: There is a slight change to the out-of-the-box logic when changing the "Direction" on a Phone Call. We have some custom scripting that sets the "Attendees" field. Since this field had 2 Attendees populated, when you change the Direction from Outgoing to Incoming it presents a warning message. If you proceed, then it will update both the Sender and Attendees field.
• Takeaway: be sure to test minor features when you upgrade such as changing the direction of a call.
• Resolution: We found that the message only occurs when clicking the OOB field (not when changing through workflows or business rules). So we created a custom "Direction" field then used a Business Rule to set and hide the OOB field as the Custom field changed.

• Advanced Find Column Focus: We love Advanced Find! So we have lots of views with lots of columns. On version 8.1, if you scrolled right in a view and then selected a line item, the view automatically scrolled to the far left (1st column). On version 8.2.2, we noticed that on some views it was scrolling to the right instead. Turns out it was scrolling to the Primary Field for that entity. 
• Resolution: The solution was to move the Primary Field to the left so users did not notice the change in behavior. This example was on the Opportunity entity where we had the "Topic" field further to the right so when a user viewed their opportunity and tried to select one it was "jumping". Problem solved with some simple System View updates!

• Workflow Ownership: When we first upgraded to 2016 (8.1 from 2011) we loved the new feature that you could import workflows without reassigning to yourself then back to your system user account. However, after upgrading to 8.2.2 this appears to not be working exactly as advertised. When importing an existing workflow assigned to another user, the import proceeds successfully. Workflow remains active and assigned to that user. However future system jobs are assigned to the importing user (as if they were the workflow owner). 
• Resolution: To fix you will need to reassign those workflows back to the owner you want.
• I have seen this community post for a similar issue but I do not see any Microsoft fixes yet.

• OOB Business Process Flows and Business Rules: There are some new Business Process Flows and Business Rules added. If you are not using these, make sure you disable. I did notice that the Business Process Flows were active but did not have an Active Process ID. I suspect (but have not confirmed) that they have dependencies to other OOB Process Flows that we had already deleted. So remember to check all the Active processes and see what you actually want to use.
• Resolution: Review and deactivate as necessary.

• Split Server Roles: The last gotcha in our upgrade is a bit past my technical depth but the synopsis is that we have multiple front and back end web servers. So this means that our upgrade is a two-step process. Once the upgrade is done to all of the servers you additionally need to upgrade the database.
• Identification: In CRM, go to Options > About to see the version. If the versions do not match then this could be the cause.
• Resolution: Open the Deployment Manager and see "Update Available". Perform this update.
• Takeaway: Ensure your server architecture is documented so you can identify where issues like this could exist (example your Development instance might all be on one server but production is broken out).

As I said, most of our upgrade went very smoothly. These issues were mainly invisible to the end users and we were able to resolve quickly.

Has anyone else done a similar upgrade? Have you seen the same issues? Or different ones I should be on the lookout for?

March 11 Blog Review

Let's jump right in to the blogs posted this week!

CRM Tip of the Day shared The "Silent Observer" Custom Activity. This talks about how you can set Custom Activities to not appear in Activity Menus. This allows you to hide them from users but still use them for tracking of certain communication or events.

Jukkan Niiranen posted on Everything as a Service - Not Just Your CRM. This expands upon Microsoft's announcement of the CRM 2016 Spring Wave just last week.

Sonoma Partners blogged on Surfacing the Right Business Process Flow. Here they walk through a bit of information on Business Process flows such as making them available for certain Security Roles, setting the order/priority and allowing users to switch between processes.

That's all for today. Have a great weekend!

February 19 Blog Review

The most important news you need to remember this week is that the CRMUGDC Q1 Chapter Meeting takes place on Wednesday, February 24th from 8:30-12:30 at the Microsoft Office in Chevy Chase, MD. We will be focused on reporting and talking about the importance of having good data. Make sure you check out the details and register here!

These events are great for everyone so you can network and get answers to your questions! If you can't make it on Feb 24, the next meeting is May 11th and will be focused on Service. Get a preview by reading Ledgeview Partner's blog Spotlighting Service in CRM 2016.

PowerObjects shared - Developers Rejoice! Solution Enhancements in CRM 2016. This walks through solution creation in CRM 2016 and shows how you can add specific components from an entity instead of the whole entity. This allows you to separate projects that have involve the same entity so they do not need to be released at the same time!

PowerObjects also posted Start Stretching! It's Dynamics CRM Training Time. Check this out to get more information on what you should be thinking about as you plan your CRM training. It talks about when users should get involved, how long the training should last, etc.

For those who are working to upgrade to 2016, take a look at Chris Cognetta's blog CRM 2016 Available but not Ready for Production. He talks about some items you need to be aware of as you look at 2016 and why you may want to consider waiting for Rollup 1.

That's all for today folks! Have a great day!

December 18 Blog Review

This will be the last blog review before my holiday time so let’s make sure it is interesting.

I must start by reminding you that the CRMUGDC 2016 Chapter Launch Party was just announced! This will be a fun networking event on Thursday January 21^st. It will feature eating, drinking and question-answering. Come out to learn more about all the user groups in DC, Maryland, and Virginia and talk with CRM users in your area. For more details, read the Blog or get Registered.

Within the last few weeks Microsoft announced the release of CRM 2016. This week they announced the acquisition of Metanautix, a data analytics company. No information is given about CRM integration but it will be interesting to see how this develops in the increasingly connected Microsoft product catalog. (Although Salesforce is mentioned in the announcement. Odd.)

With 2016 in mind, check out CRM Tip of the Day’s post on CRM 2016 Postponed Features. This talks about some items that were originally expected in CRM 2016 that have been pushed out to upcoming releases.

PowerObjects was very aware that most of the population was distracted with Star Wars this week and so their blogs reflect that. Be sure to read The Force is Strong with this Version: Top New Features in Microsoft Dynamics CRM 2016 Online and Light Side or Dark Side? Personalize Dynamics CRM 2015 with Themes.

Adam Vero posted Figuring Out Shares in the PrincipleObjectAccess POA table in CRM. Take a close look at this if you are experiencing any odd access issue and need to dig in to all the sharing going on in your organization.

That’s all for today folks. Happy Holidays!

December 11 Blog Review

This time of year is very busy with parties and gifts. So we will keep this quick so you can rush off to your cookie exchange.

Microsoft sent us the gift of CRM 2016 last week. This week came all of the documentation and more information about new features. The CRM Team shared the CRM 2016 Product Wave Documentation. This has links to many different documents and videos to help you prepare and transition to 2016.

The CRM Team also shared information about the new Task Flows feature. Check out, What are Task Flows and how do they compare to other tools in CRM? 

Then, when you are ready to build your own you should read, Creating and Customizing Task Flows.

Lastly for today, if you are looking to make a change in the new year you can review Sonoma Partner's Infographic on Resume Tips from the Recruiting Desk.

That's all for today folks. Have a lovely weekend!

December 4 Blog Review

Welcome to the first blog review of December! 2015 is quickly slipping away from us but don’t get too melancholy – CRM 2016 has been released!

Here are all the blogs to check out about the CRM 2016 announcement:

• Microsoft Dynamics CRM 2016 Now Available: ushering in a New Era of Intelligent Customer engagement (Microsoft)
• First Glance at Dynamics CRM 2016 (Jukka Niiranen)
• CRM 2016 is available (CRM Tip of the Day)
• Microsoft Dynamics CRM 2016 is here! (Ledgeview Partners)
• CRM 2016 General Availability Announced at Convergence EMEA! (Chris Cognetta)
• Microsoft Dynamics CRM 2016 (Carsten Groth)

Lots of new and enhanced features to look for. I’m sure there will be many more blogs about CRM 2016 in the coming weeks!

Let’s check out a couple other interesting blogs posted this week.

PowerObjects shared Take your Disk Space to the MAX: Automatically Delete Completed Workflow Jobs in CRM 2015. This walks you though how to set up your workflows to be removed when they complete. This is great to save space by removing the completed system jobs for workflows that have been fully tested and are running smoothly.

Leon Tribe posted an interesting article on Calculating the Day of the Week Without Code. This shows you how to use Business Rules to have a field show “Friday” when “12/4/2015” is entered in the trigger field.

That’s all for today folks. Enjoy your weekend!

November 20 Blog Review

Let’s jump right in to the CRM news of the past few weeks!

Update 0.2 for CRM 2015 On-Premise has been released. For all the information review CRM in the Field’s Overview Blog.

PowerObjects shared Manipulating the Value of Days Remaining between Today and the Expiration Date in Calculated Fields. This shows you how to modify the Days Remaining field so that it will not show a “-ve” value when the expiration date is in the past.

Adam Vero the CRMGuru posted an interesting post on Figuring Out Shares in the PrincipalObjectAccess POA Table in CRM. This provides more information about how shares are saved behind the scenes and how to view/interpret that data.

Chris Cognetta shared the Great Microsoft Cloud InfoGraphic! This shows the number of users that are on the different Microsoft cloud offerings. It shows why this is such a large area of emphasis and why this continues to be important for future growth.

Leon Tribe blogged about the Post MVP Summit Roundup. Unfortunately not too much information can be released yet but at least we can see everyone’s smiling faces and get a little glimpse into what it is like to be a CRM MVP. Also mentioned was that the CRM 2016 features that have already been announced were presented and provided a much better understanding of the capabilities. So be excited for more information at the end of the year!

Finally, one quick tip from CRM tip of the day is what to do with Disabled User’s Personal Views. Everyone runs into a situation where a user leaves and their views remain assigned to that now disabled user. Check out this post to see 3 possible solutions.

That’s all for today folks. Have a great day and Happy almost Thanksgiving! (In America, or happy almost Christmas, Thanksgiving was forever ago why are you still talking about it in Canada :) )

October 30 Blog Review

Happy Halloween!

Unfortunately (or maybe fortunately) I do not have any spooky CRM Stories for you other than users saying they didn’t do something the system says they did. Instead I have a quick blog review touching on the highlights of the last few weeks while I was busy with Summit and such. Let’s jump right in.

PowerObjects is working on a series about CRM Upgrades. Check out this installment – Upgrading CRM: Reimplementation vs. Upgrade – Choosing What’s Right for you. This talks about what the options of reimplementation and upgrade mean along with the reasons you might select one or the other.

EMEA Dynamics Support posted a helpful troubleshooting article, Dynamics CRM Bulk Delete – General Support Questions. This works through the common questions and issues that they deal with and how to respond.

A little bit ago we talked about the CRM 2016 Release Preview Guide. This is a very long document, so if you don’t have time to read it you should check out Leon Tribe’s slightly shorter summary – Review of the Latest Dynamics CRM Release Preview Guide. There are lots of great new features centered around Microsoft’s themes for Productivity, Intelligence, Mobility and Unified Service.

Sonoma Partners shared the Negative Effects of a Disjointed CRM Environment. This touches on the importance of ensuring your CRM supports and aligns with your business as it grows and integrates with all the necessary systems. Some of the impacts of a disjointed CRM environment include customer satisfaction, employee productivity and retention and company performance.

Lastly, I will put all of you in charge of reviewing the recent posts of CRM Tip of the Day. What was your favorite tip? Anything that can help your business?

Have a great weekend and a happy Halloween!  

P.S. Make sure you come out to the Washington, DC Chapter Meeting on November 4^th get more details and register here.