Microsoft Dynamics 365 Upgrading from 8.2.2 to 8.2.3

As part of troubleshooting Server Side Sync we upgraded our on-premise Dynamics 365 environment from version 8.2.2 to 8.2.3 (8.2.3 Details). There were several issues that occurred from this upgrade that I wanted to share to prevent you from falling into the same traps (or at least knowing how to recover when they arise).

Related Articles: Server Side Sync troubleshooting steps, Upgrade from 8.1.8.22 Issues

X-Anchor Header Change
The main reason we upgraded to this version was to ensure all integrations with Exchange used the X-Anchor Header. After the upgrade, this feature needs to be enabled by updating the OrgDB setting UseXAnchorMailboxInExchangeRequestHeader to True.

Navigation
After the upgrade, one of our Navigation menus had additional options. To resolve, I simply edited the navigation as part of the form editor and included this in a solution. Then imported this solution as part of the Live upgrade.

SQL Error
Post-upgrade several users reported getting a SQL Error when attempting to access Dynamics 365. This error would pop up anytime a list of records attempted to load without a wait/timeout. Direct links to Accounts could open but related records could not be viewed.

Teams are heavily used in our environment for access and sharing. We found that each of these users was a member of over 20 teams. When several teams were removed, access returned as normal.

Reopened Tasks
CRM for Outlook is also used in my organization. We have seen cases in the past where tasks or calls were created in Dynamics 365 from Outlook even though they already existed (lots of duplicates!). This issue is no longer occurring, but some tasks and calls are being reopened.

So post upgrade, its important to monitor modified Activities especially where they were created in the past and already have an Actual End populated.

Sharing Advanced Find
The last issue we have experienced (or at least we have found so far!) was related to sharing Advanced Finds. Users were able to share views with individual users but received an error when attempting to share with a team. Downloading the log file revealed "missing prvReadUserApplicationMetadata privilege". The User Application Metadata option is available on the Security Role Customization tab. I granted User level Read Access and this resolved the issue.

Has anyone else upgraded to 8.2.3? Any other issues I should be aware of?

User Application Metadata Security Update

Real life Tales of Server Side Sync

The idea of Server Side Synchronization for Microsoft Dynamics 365 is very exciting. It can enable the syncing you know and love to run even when Outlook (with CRM for Outlook) is not running. 

In my organization, we have teams who are scheduling appointments for other users. So it is important that Outlook reflects the most up-to-date information. If there are overlapping appointments that need to be rescheduled or if a user is not prepared for an appointment this is lost revenue.

So we enabled Server Side Sync to get past some of the limitations of CRM for Outlook (primarily that there needs to be a primary syncing computer and that syncing only occurs when Outlook is running). Unfortunately this did not run as smoothly as intended. Appointments were taking 12 hours to sync if they synced at all!

Things are now running smoothly (mostly) but it took time to get everything right and a lot of help from Microsoft support. Let's talk about the changes and items that helped us.

Background
We have been using Dynamics in my organization since CRM 4.0 and were on D365 8.2.2 on-premise when launching Server Side Sync. CRM for Outlook was in use for most users for the whole implementation. Users are syncing Calls, Tasks and Appointments. We are also using Exchange on-premise.

When enabling Server Side Sync we found that many (40,000+) duplicate calls and tasks were being created. We believe this was due to the history in Outlook and potential disconnects with the data (for example, the task knew it synced at one point but the Outlook profile was rebuilt and lost the connection so a new task was created).

Getting Started
When we launched Server Side Sync we learned several things we should have looked at prior to turning on for all users:

• We enabled SSS for all users and all options at once (incoming email, outgoing email, and Appointments/Contacts/Tasks). When we were having the performance issues we changed incoming email back to CRM for Outlook.
• The Email Server Profile was configured as part of our upgrade to 2016 so the "Process Email From" setting was set to October 2016. We updated this to the date SSS was turned on so older emails would not be processed.
• Outlook sync filters were also configured to sync items after October 2016. We also updated these to the same date SSS was enabled.
• Power: Throughout the troubleshooting we added more power to everything related to our asynchronous services servers. More CPU, more memory and additional processor cores.
• Also verify that the Organizer field is on the form for appointments and being set correctly. Verify this is set to a real user who will be syncing For example we had this set to a service account for some appointments (set via workflow) and these did not sync because that user did not have a calendar to sync to.
• Review your mailbox records and clean up any you do not need. If the user does not need to sync, do not turn it on to sync. If it is a service account or placeholder account of some kind, turn it off and disable it.

Nitty Gritty Settings
There are several settings that Microsoft recommended we update. I will explain the changes we made. When you look at these options you should look at your environment to see which values make sense.

Max Concurrent Connections
There is a suspicious setting in the Exchange Server Profile called "Max Concurrent Connections". This is related to the throttling policy in Exchange. We increased this setting to 100 initially (default is 10) and then increased to 1000. This means you also need to be working with your exchange team to determine what setting they are comfortable with.

AsyncMailboxesPerEvent
By default AsyncMailboxesPerEvent is set to 5. This means that 5 mailboxes will be attempting to process at once on the same thread. We were concerned that if there was an error in one mailbox it could delay the other 4 mailboxes in that group. So this setting was updated to 1. The default has been updated to 1 in some newer versions.

AsyncLongJobThreasholdInSeconds
The AsyncLongJobThreasholdInSeconds was updated to 86400. This value is typically lower for online environments where there are concerns about long-running async operations. The increase was recommended in our case to avoid any issues because of throttling (on the async server) because the sync was taking a long time.

X-Anchor Header and v8.2.3
In our fiddler tracing on the async servers we found that the X-Anchor mailbox header was not included. This is recommended as a best practice when working with exchange (source).

To add this header we needed to upgrade to the latest version of D365 on-premise 8.2.3. The X-Anchor change is not included in the release notes and is not enabled by default. To enable, after upgrading all servers you must update the Organization DB settings to set UseXAnchorMailboxInExchangeRequestHeader to true.

After upgrading to 8.2.3 and turning on the X-Anchor header we saw immediate improvements in the SSS behavior. Additionally some backlogs (SubscriptionSyncEntryOutlookBase table) of items to sync that we were monitoring began to improve after weeks of minimal change.

However we did have some issues with the upgrade. Check out my Upgrade from 8.2.2 to 8.2.3 blog for more information.

Related to SSS, the mailboxes processing during the upgrade needed to have "Test & Enable" repeated to keep functioning and the message sent from this process changed. Formerly the "Test & Enable" email message had a subject of "Test Message". Now it has a subject of "Your Mailbox is now connected to Dynamics 365". Post upgrade, we no longer saw issues with duplicate tasks being created but instead we did see cases of old (completed) activities being updated to a status of open. We monitored these cases via Advanced Find to clean up.

"Understanding" the Mailbox Details
When reviewing a specific Mailbox inside of Dynamics 365 there is a button on the ribbon to "Download Mailbox Details". This gives you a lovely file of information you do not know how to read. So here is what I learned about this file through the process.

1. Current Date and Time in UTC - All the times in the file are in UTC. If you don't feel like doing the conversion (me!) then you can just check the file name to compare current time to the time in the file.
2. Processing status - 1 means currently processing. 2 means not processing right now.
3. Time of last Processing attempt - also stored in CRM "Date Processing Last Attempted"
4. Next Processing for ACT - ACT = Appointments/Contacts/Tasks. This is the next time a sync will be attempted. This should be just a few minutes after the last attempted processing and should be in the future (i.e. if it's a past time then for some reason it didn't process when it was supposed to).
5. Last Successful Sync - Time of last successful, completed sync
6. Last Error on Sync - The last time a sync encountered an error
7. Processing details - number of items successfully processed on last sync and number of items that failed during the last sync
8. Error details - full details of all recent errors. These are also visible from the Alerts area of the mailbox. This will show you the full error details if it was not all visible in the interface.

Other Stuff 
There were a few things I skipped over. Throughout the process of working with Microsoft support we provided many logs, traces, mailbox statistics, mailbox details, memory dumps, perfmon details, etc. Some of these were from D365 but also some additionally from Exchange. We also used this tool to look at mailbox details but this did not lead to actionable steps.

After all of the steps above we are still monitoring progress and making sure things are processing as expected. To do this you can review the Mailboxes view and add the "Date of Last Processing Attempt" as well as the Status field. If it has not processed in a reasonable amount of time or is in a failed status, it is probably time to review the Alerts on that mailbox and to run "Test & Enable" again. Keep in mind you can also review all alerts from the Email Server Profile record as well (all Alerts for mailboxes linked to that profile).

Finally don't forget that Server Side Sync for appointments will send out an Appointment invite or changes to the appointment to all attendees including those external to the organization. This was not acceptable in our business so we used a workflow to add text to the appointment that was captured by Exchange to prevent send as recommended in this Community Post.

So has anyone else used Server Side Sync successfully? What has been your experience? What other settings or tips to improve performance have you tried?

Appendix: Queries
There were several queries our consultant worked on with Microsoft to gather more details. I take no credit for their creation, just for seeing the results.

Backlog Status
Shows items currently waiting to be synced.


select sq.*
from (
select sub.FullName, eo.ObjectTypeCode, e.name as ObjectName, eo.SyncState, count(*) reccount
from SubscriptionSyncEntryOutlookBase eo with(nolock) inner join
(select distinct e.name, e.ObjectTypeCode from metadataschema.Entity e with(nolock)) e on eo.ObjectTypeCode=e.ObjectTypeCode inner join
subscription s with(nolock) on eo.SubscriptionId=s.SubscriptionId inner join
systemuserbase sub with(nolock) on s.SystemUserId=sub.SystemUserId
where eo.SyncState<>0
group by sub.FullName, eo.ObjectTypeCode, e.name, eo.SyncState
) sq
order by sq.reccount desc;

Mailbox Status
Show status and details for a particular mailbox. For part of our troubleshooting we actually set this up to run on a schedule and capture the details for specific mailboxes every 30 mins. Update the names list at the end to the users you want to capture for.

select sub.FullName
,m.ProcessingStateCode
,m.ACTDeliveryMethod
,m.UTCConversionTimeZoneCode
,m.PostponeMailboxProcessingUntil
,m.OutgoingEmailDeliveryMethod
,m.Username
,m.ProcessEmailReceivedAfter
,m.ModifiedOnBehalfBy
,m.NoEmailCount
,m.MailboxId
,m.IncomingEmailDeliveryMethod
,m.EnabledForIncomingEmail
,m.NoACTCount
,m.EmailAddress
,m.OutgoingEmailStatus
,m.EWSURL
,m.OrganizationId
,m.PostponeSendingUntil
,m.EmailServerProfile
,m.ModifiedOn
,m.OwnerId
,m.LastActiveOn
,m.TimeZoneRuleVersionNumber
,m.LastAutoDiscoveredOn
,m.TestEmailConfigurationScheduled
,m.IsEmailAddressApprovedByO365Admin
,m.ModifiedBy
,m.Password
,m.TestMailboxAccessCompletedOn
,m.ExchangeSyncStateXml
,m.ProcessAndDeleteEmails
,m.ReceivingPostponedUntil
,m.RegardingObjectId
,m.EntityImageId
,m.CreatedOn
,m.EmailRouterAccessApproval
,m.StatusCode
,m.ReceivingPostponedUntilForACT
,m.StateCode
,m.PostponeTestEmailConfigurationUntil
,m.UndeliverableFolder
,m.MailboxProcessingContext
,m.Name
,m.IsForwardMailbox
,m.TestEmailConfigurationRetryCount
,m.CreatedOnBehalfBy
,m.AllowEmailConnectorToUseCredentials
,m.HostId
,m.LastMessageId
,m.OwningBusinessUnit
,m.EnabledForACT
,m.IncomingEmailStatus
,m.CreatedBy
,m.TransientFailureCount
,m.IsACTSyncOrgFlagSet
,m.ACTStatus
,m.EnabledForOutgoingEmail
,m.RegardingObjectTypeCode
,m.RegardingObjectIdName
,m.OwnerIdType
,m.LastSuccessfulSyncCompletedOn
,m.ProcessedTimes
,m.AverageTotalDuration
,m.LastSyncErrorOccurredOn
,m.ItemsFailedForLastSync
,m.LastSyncErrorMachineName
,m.LastSyncErrorCount
,m.LastSyncErrorCode
,m.ItemsProcessedForLastSync
,m.LastSyncStartedOn
,m.LastSyncError
,m.OrgMarkedAsPrimaryForExchangeSync
,m.LastDuration
,m.ForcedUnlockCount
,m.ProcessingLastAttemptedOn
,m.LastMailboxForcedUnlockOccurredOn
,m.IsServiceAccount
,m.VerboseLoggingEnabled
,m.FolderHierarchy
,m.OfficeAppsDeploymentCompleteOn
,m.PostponeOfficeAppsDeploymentUntil
,m.OfficeAppsDeploymentScheduled
,m.MailboxStatus
,m.OfficeAppsDeploymentError
,m.OfficeAppsDeploymentStatus
from mailboxbase m with(nolock) inner join
SystemUserBase sub with(nolock) on m.OwnerId=sub.SystemUserId
where m.statecode=0 and sub.FullName in ('Last1, First1','Last2, First2','Last3, First3');

September 2016 Blog Review

One of the big events of September was Microsoft Ignite. There are lots of highlight blogs available for you to check out or you can even watch the recordings of the key sessions. I did not watch all the sessions but I particularly enjoyed this Demo of HoloLens with Lowes. So many amazing possibilities in the future!

Let's jump into other blog news with some articles shared by PowerObjects:

• How We Get Marketing and Sales to Play Nice - with CRM as the parent in Charge - this reviews the common arguments from both sides and how CRM can help by providing transparent data and ways to better define the transition between marketing and sales as well as identify issues.
• Make a Field Required in a CRM Dialog - Use check conditions and child dialogs to effectively make a field required in this examle
• Color me CRM: Matching Theme and Logo Colors in CRM - This shows you how to quickly determine the right color for your navigation bar

Server-Side Sync is still relatively new so Microsoft produced a white paper on how it works. Check this out to gain a better understanding of the ins and outs of how your data is syncing.

If you are looking to migrate to CRM Online check out these Tips and Tricks from Dynamics CRM in the Field blog.

How to pick a partner is always a hot topic. There are so many factors to consider and things to keep in mind. Sonoma Partners offered some guidance in 5 questions to Ask When Evaluating CRM Consulting Firms.

Let's finish things off with a few notes from CRM Tip of the Day:

• Add Variables to you workflows - this is a great tip to remove explicit references from your workflows by using a custom action to set a variable
• You can now use timezone in your appointments - you can now build logic to mimic timezones in Outlook
• Add the number to your phone call records - great usability tip to display the phone number in your phone calls and then use a workflow to append this to the subject to make it even easier

That's all folks!

April 8 Blog Review

Let's jump right in to the CRM news that was shared this week.

As you probably know, Microsoft Envision was going on this past week. This replaced Convergence in some ways as a business focused Dynamics conference. Ledgeview Partners shares their quick recap and some helpful links in Microsoft Envision 2016 Sessions Available On-Demand.

CRM Tip of the Day shared two tips about Server Side Synchronization. First read User Experience Differences between Outlook and Server-Side Synchronization. This will give you some things to keep in mind as you decide if you want to take advantage of this feature. Then you can read Duplicate Activities in Outlook with Server Side Synchronization. If you have issues with Server Side Synchronization this gives you some ideas on where to begin your troubleshooting.

Finally, EMEA Dynamics CRM Support shared Dynamics CRM Document Generation. This shares the functionality available, quick details on how to set up and answers some common questions.

Also keep in mind that the CRMUG Summit early bird deadline is quickly approaching! You can also check out these tips for convincing your boss to let you go. I hope to see you there!

That's all for today folks!

July 10 Blog Review

It is a beautiful day! Really makes you want to go to a baseball game!

Let’s quickly jump in to the news of the day!

CRM Tip of the day posed a video: Tipsters Guide to Folder Level Tracking for End Users. This shows users how to set up folder level tracking when you are using Server Side Synchronization.

Hitachi Solutions shared a quick tip – Hey! Where are the Users in CRM 2015!? In CRM 2015, instead of having users directly under Administration you need to look under Security.

The CRM Support Blog posted Dynamics CRM Support with Windows 10, Edge browser, and Office 2016. As some have speculated only CRM 2013 and CRM 2015 will be supported with Windows 10.

Finally, Jukka Niiranen discusses the State of Dynamics in 2015. This is a look at many topics around the state of Microsoft and CRM.

That’s all I have folks. Have a great weekend!

January 30 Blog Review

It looks like there are lots of great posts to talk about this week. Before I do, a quick reminder, if you are in the Washington, DC area you should check out the DC CRMUG Chapter Meeting happening next week – Wednesday February 4^th.

Alight now on to the news of the week.

David Yack posted a great tip on the Risk of Adding Field Level Security Layer. This is a good reminder to consider all the implications before you turn on this feature.

PowerObjects posted about Testing Incoming Mailbox Configuration for Server Side Sync. This discusses what the Configuration Test Results mean when you are configuring Server Side Sync.

Hitachi Solutions posted a great walk through about calculating the Last Activity Date in CRM 2015. This build upon an old blog that calculated the last activity date with workflows. Now in CRM 2015 we can use rollup fields to greatly improve this process.

McGladrey walked through their favorite new features in CRM 2015: An Upgrade with more Conveniences. This touches on some features like the multi-entity search and managing synchronization filters within CRM.

The Microsoft Dynamics Team Blog released some videos for Microsoft Dynamics Marketing 2015 Update.

Mitch Miliam announced his 21 Squared CRM Administrator Course. This is a great course for CRM System Administrators. It covers the tools you need as an administrator as well as the basic tasks you will need to do within CRM and on the server.

In CRM 2015 you can now use JavaScript to interact with the process. Sonoma Partners discusses this in their post CRM 2015 – Scripting Options for Business Process Flows.

Also a few quick tips from CRM Tip of the Day:

• Change Labels on Composite Controls – Have you ever wanted to change the label of composite fields (address pops out to street 1, street 2, etc.; Name pops out to First, Middle, last)? Well here is your answer!
• CRM Server Licensing 101 – So perfect. There are no words.

So it has been a busy week! I hope I covered all the best bits. Have a great weekend!

November 21 Blog Review

It was a fairly quiet week in the CRM world. Everyone is busy working to be done before the holidays. Let’s take a look at some of the top posts.

PowerObjects taught us about Calculated Fields in Dynamics CRM 2015. This walks through an example to show you how to set up Calculated Fields.

Adam Vero posted a blog about his experience as a CRMUG member. Take a look if you are interested in what it means to be involved with CRMUG, lead a chapter meeting, and be an MVP at Summit.

CRM Tip of the Day also posted several helpful tips:

• Beware of the implications of “non searchable” fields – this discusses the implications and considerations to keep in mind when setting a field to searchable = no (also check out my blog on the topic comparing non searchable to field level security)
• Server Side Sync and View in CRM – there is a known issue that the View in CRM button does not work when using Server Side Sync
• Close Appointments from Outlook – It is easy to use Outlook to close CRM tasks but the same thing does not exist for appointments. Check out this tip to learn a potential workaround.

That’s all I have for today. Have a great weekend!