Sometimes the Problem is You

There has been an odd issue going on in my implementation for the last several months. With the help of Microsoft support, we solved the issue this week! However, the solution is very embarrassing because it was a self-made problem (I guess most of them are). But maybe my failures will help others so here goes!

The Problem

Towards the end of October, users started reporting that they were seeing the "Pending Email" warning when launching D365. We reviewed and saw there indeed there were email messages in pending status waiting to be sent.

All of these were workflow generated emails to a variety of different users. There didn't seem to be a common pattern in what these emails were, who they were to or the workflow that created them.

The Work

First to solve the issue we used a workflow to change the Email Message status back to Draft and re-send. Then we learned that everyone had already received the original message. So for some reason the Email in D365 said Pending Send even though it was actually sent and the Date Sent was even populated.

Next we created another new workflow that just updated the status from Pending Send to Sent. This was not solving the issue but it was preventing users from getting the Pending Send Warning message.

We believed the issue had something to do with our connection with Exchange. Maybe we weren't getting a response in a timely manner or something like that. So the "it's not my fault!" attitude was in full force. We basically started handling the symptoms (running the workflow to change status) and stalled on dealing with the problem.

Finally some bandwidth opened up so we decided it was time to engage Microsoft support on this issue. We worked with them for a little while to get traces, logs, etc. and provided all this data.

The Solution

Microsoft reviewed all the data we sent and said "hey, it looks like there is a workflow firing on all of these emails changing them to Pending Send". What? No way! They must be confused.

Sure enough, I found the workflow they were seeing with the step name they reported. It was doing exactly what they said was happening. I deactivated the workflow and the issue stopped occurring.

Then we thought about the timeline and figured out exactly what happened. (Side tip: we have turned Auditing on for the Processes entity. This makes it easy to check who created the workflow, who activated it, when, when was the owner changed, etc.)

• Earlier in October we had an issue with some emails failing (actually in a Failed state)
• One admin went in and created a workflow to change back to Pending Send so Server Side Sync would automatically pick up and re-send the email
• This was set to run on-Demand
• Scope = User
• Run on Create = Yes. This is fine when the owner was that admin. This is just for clean up, it won't be a problem, right?
• Automatically deleted completed workflow jobs = Yes
• Later in October there is an unrelated release involving other unrelated workflows
• Part of the release process is reassigning these workflows to the Service Account we use to run these workflows. I (I will take the blame!) reassigned all workflows recently modified to the Service Account not paying attention to the specific items modified for that release
• Now all emails being owned by/created by the Service Account, AKA all workflow generated emails, are being set to Pending Send after they are sent

As I said, not all emails were "stuck" in Pending Send status. I think this was still due to timing of the email actually being sent and email owner and those type of factors.

Lessons Learned

1. It can always be your/my fault
2. Review the Start When conditions even when just doing "quick"clean up items
3. When using a workflow for clean up or quick data manipulation, deactivate when done
4. Always communicate with others who are creating workflows and review so everyone is on the same page

So that is my embarrassing story for the week. I hope Microsoft does not put it on my permanent record. Please share your self-inflicted, system mishaps to help me feel better!

User Group Focus 2019 Recap

A few weeks ago I was at the User Group Focus event in Houston, TX. I attended Monday and Tuesday for the D365/CRM portion of the event. I presented several sessions, attended others and also went to the Rodeo! It was a great trip!

I participated in three sessions. Michael Ochs, Phyllis Eriksen and I presented an introduction to XrmToolBox which taught participants how to use it and we shared our favorite tools (be on the lookout for an encore webinar coming in June). I also presented on "Wrangling your Processes" (an introduction to Workflows in D365) and you can catch the encore webinar on May 22. Finally I participated in a panel about Upgrades. This was a good discussion on many tips for planning, upgrade methods, and preparing your users.

I also got to attend a few sessions and wanted to share some takeaways from those as well. For attendees, you can download all session materials from the User Group Focus Community.

Planning, Developing, and Surfacing Metrics for Sales and Service

Tad Thompson

The goal of this session was to discuss ways to maximize time for sales people by reducing clicks and effectively using screen real estate. Lots of great recommendations were discussed:

• Don't forget about Goals
• Utilize the Apps feature to minimize the number of unnecessary items users see
• Remove unused fields
• Use customization to drive user adoption by providing more information
• Calculated fields to give important stats, ex. time to qualification
• SLAs, shorten the SLA based on the customer's interest
• Business Process Flows to drive behavior (be careful not to overwhelm). Now you can have multiple BPFs per record so multiple users can be following separate flows at the same time.
• Editable grids can be helpful but ensure these are not used for continuous, mass data clean up (fix the actual problem!)
• Check out email feature to follow email and get insights and reminders related to it
• Reporting can be used to drive excitement and show the users what their data is powering.
• OOB charts still have value. These are real-time when PowerBI is not.
• Start with the PowerBI Templates and see what you can get value from quickly
• Embed your PowerBI dashboards in Dynamics
• For service, Flow allows you to listen to Twitter and create cases as needed

The Power Platform in 90 Mins

Michael Ochs

This session walked through an example of building a vacation request/approval tool using CDS, Flow, PowerApps, and PowerBI. D365 is an application that is only part of the Power Platform so we have so many other tools to explore! CDS (Common Data Service) was formerly thought of as the XRM Platform. Now everything is stripped away into apps and core entities such as Contacts, Accounts, Activities, etc. form the backbone of CDS.

PowerApps are broken down into Canvas and Model Driven. Canvas Apps are the type of app we are used to using on mobile devices. Model Driven Apps are part of Dynamics to only show what users need.

The objection of this session was to use all of these components to build line of business applications. This was a shortened version of the App in a Day session.

Better Activity Management
Geoff Ables

One of the most important features of a CRM system is activity tracking but you need to get the users to actually follow through. Start by working with the business to determine why they even want to track activities. We need to find the end motivation and then find ways to make it easy, fast and fun. The goal is to find ways to move from just reactive work to proactive work. This may also include watching users work through the current process to look for where efficiencies can be gained.

All of this boils down to the LUCK Principle. Listen Understand Connect Know. 

So, what are some practical ideas to explore?

• Compare the Outlook App with the Legacy Outlook Client to determine what is best. The app is recommended if there is nothing in the client that is needed. App is being recommended for new users. The App allows you to view dynamics data in Outlook without tracking the message.
• Relationship assistant provides reminders on the home screen to direct users. Activities are not automatically created but users are encouraged to take specific follow up actions. There are settings to control these reminders that are set by administrators. These are configured for the full organization.
• Neglected records monitoring. OOB neglected records reports may be beneficial. Additionally, calculated fields can be used to build fields to track Date of Last Contact so custom neglected rules can be developed.
• Workflows can also be used to automatically create follow up tasks. Create fields on Activities to indicate if a follow up is needed. Then automatically create when that activity is closed.

Real life Tales of Server Side Sync

The idea of Server Side Synchronization for Microsoft Dynamics 365 is very exciting. It can enable the syncing you know and love to run even when Outlook (with CRM for Outlook) is not running. 

In my organization, we have teams who are scheduling appointments for other users. So it is important that Outlook reflects the most up-to-date information. If there are overlapping appointments that need to be rescheduled or if a user is not prepared for an appointment this is lost revenue.

So we enabled Server Side Sync to get past some of the limitations of CRM for Outlook (primarily that there needs to be a primary syncing computer and that syncing only occurs when Outlook is running). Unfortunately this did not run as smoothly as intended. Appointments were taking 12 hours to sync if they synced at all!

Things are now running smoothly (mostly) but it took time to get everything right and a lot of help from Microsoft support. Let's talk about the changes and items that helped us.

Background
We have been using Dynamics in my organization since CRM 4.0 and were on D365 8.2.2 on-premise when launching Server Side Sync. CRM for Outlook was in use for most users for the whole implementation. Users are syncing Calls, Tasks and Appointments. We are also using Exchange on-premise.

When enabling Server Side Sync we found that many (40,000+) duplicate calls and tasks were being created. We believe this was due to the history in Outlook and potential disconnects with the data (for example, the task knew it synced at one point but the Outlook profile was rebuilt and lost the connection so a new task was created).

Getting Started
When we launched Server Side Sync we learned several things we should have looked at prior to turning on for all users:

• We enabled SSS for all users and all options at once (incoming email, outgoing email, and Appointments/Contacts/Tasks). When we were having the performance issues we changed incoming email back to CRM for Outlook.
• The Email Server Profile was configured as part of our upgrade to 2016 so the "Process Email From" setting was set to October 2016. We updated this to the date SSS was turned on so older emails would not be processed.
• Outlook sync filters were also configured to sync items after October 2016. We also updated these to the same date SSS was enabled.
• Power: Throughout the troubleshooting we added more power to everything related to our asynchronous services servers. More CPU, more memory and additional processor cores.
• Also verify that the Organizer field is on the form for appointments and being set correctly. Verify this is set to a real user who will be syncing For example we had this set to a service account for some appointments (set via workflow) and these did not sync because that user did not have a calendar to sync to.
• Review your mailbox records and clean up any you do not need. If the user does not need to sync, do not turn it on to sync. If it is a service account or placeholder account of some kind, turn it off and disable it.

Nitty Gritty Settings
There are several settings that Microsoft recommended we update. I will explain the changes we made. When you look at these options you should look at your environment to see which values make sense.

Max Concurrent Connections
There is a suspicious setting in the Exchange Server Profile called "Max Concurrent Connections". This is related to the throttling policy in Exchange. We increased this setting to 100 initially (default is 10) and then increased to 1000. This means you also need to be working with your exchange team to determine what setting they are comfortable with.

AsyncMailboxesPerEvent
By default AsyncMailboxesPerEvent is set to 5. This means that 5 mailboxes will be attempting to process at once on the same thread. We were concerned that if there was an error in one mailbox it could delay the other 4 mailboxes in that group. So this setting was updated to 1. The default has been updated to 1 in some newer versions.

AsyncLongJobThreasholdInSeconds
The AsyncLongJobThreasholdInSeconds was updated to 86400. This value is typically lower for online environments where there are concerns about long-running async operations. The increase was recommended in our case to avoid any issues because of throttling (on the async server) because the sync was taking a long time.

X-Anchor Header and v8.2.3
In our fiddler tracing on the async servers we found that the X-Anchor mailbox header was not included. This is recommended as a best practice when working with exchange (source).

To add this header we needed to upgrade to the latest version of D365 on-premise 8.2.3. The X-Anchor change is not included in the release notes and is not enabled by default. To enable, after upgrading all servers you must update the Organization DB settings to set UseXAnchorMailboxInExchangeRequestHeader to true.

After upgrading to 8.2.3 and turning on the X-Anchor header we saw immediate improvements in the SSS behavior. Additionally some backlogs (SubscriptionSyncEntryOutlookBase table) of items to sync that we were monitoring began to improve after weeks of minimal change.

However we did have some issues with the upgrade. Check out my Upgrade from 8.2.2 to 8.2.3 blog for more information.

Related to SSS, the mailboxes processing during the upgrade needed to have "Test & Enable" repeated to keep functioning and the message sent from this process changed. Formerly the "Test & Enable" email message had a subject of "Test Message". Now it has a subject of "Your Mailbox is now connected to Dynamics 365". Post upgrade, we no longer saw issues with duplicate tasks being created but instead we did see cases of old (completed) activities being updated to a status of open. We monitored these cases via Advanced Find to clean up.

"Understanding" the Mailbox Details
When reviewing a specific Mailbox inside of Dynamics 365 there is a button on the ribbon to "Download Mailbox Details". This gives you a lovely file of information you do not know how to read. So here is what I learned about this file through the process.

1. Current Date and Time in UTC - All the times in the file are in UTC. If you don't feel like doing the conversion (me!) then you can just check the file name to compare current time to the time in the file.
2. Processing status - 1 means currently processing. 2 means not processing right now.
3. Time of last Processing attempt - also stored in CRM "Date Processing Last Attempted"
4. Next Processing for ACT - ACT = Appointments/Contacts/Tasks. This is the next time a sync will be attempted. This should be just a few minutes after the last attempted processing and should be in the future (i.e. if it's a past time then for some reason it didn't process when it was supposed to).
5. Last Successful Sync - Time of last successful, completed sync
6. Last Error on Sync - The last time a sync encountered an error
7. Processing details - number of items successfully processed on last sync and number of items that failed during the last sync
8. Error details - full details of all recent errors. These are also visible from the Alerts area of the mailbox. This will show you the full error details if it was not all visible in the interface.

Other Stuff 
There were a few things I skipped over. Throughout the process of working with Microsoft support we provided many logs, traces, mailbox statistics, mailbox details, memory dumps, perfmon details, etc. Some of these were from D365 but also some additionally from Exchange. We also used this tool to look at mailbox details but this did not lead to actionable steps.

After all of the steps above we are still monitoring progress and making sure things are processing as expected. To do this you can review the Mailboxes view and add the "Date of Last Processing Attempt" as well as the Status field. If it has not processed in a reasonable amount of time or is in a failed status, it is probably time to review the Alerts on that mailbox and to run "Test & Enable" again. Keep in mind you can also review all alerts from the Email Server Profile record as well (all Alerts for mailboxes linked to that profile).

Finally don't forget that Server Side Sync for appointments will send out an Appointment invite or changes to the appointment to all attendees including those external to the organization. This was not acceptable in our business so we used a workflow to add text to the appointment that was captured by Exchange to prevent send as recommended in this Community Post.

So has anyone else used Server Side Sync successfully? What has been your experience? What other settings or tips to improve performance have you tried?

Appendix: Queries
There were several queries our consultant worked on with Microsoft to gather more details. I take no credit for their creation, just for seeing the results.

Backlog Status
Shows items currently waiting to be synced.


select sq.*
from (
select sub.FullName, eo.ObjectTypeCode, e.name as ObjectName, eo.SyncState, count(*) reccount
from SubscriptionSyncEntryOutlookBase eo with(nolock) inner join
(select distinct e.name, e.ObjectTypeCode from metadataschema.Entity e with(nolock)) e on eo.ObjectTypeCode=e.ObjectTypeCode inner join
subscription s with(nolock) on eo.SubscriptionId=s.SubscriptionId inner join
systemuserbase sub with(nolock) on s.SystemUserId=sub.SystemUserId
where eo.SyncState<>0
group by sub.FullName, eo.ObjectTypeCode, e.name, eo.SyncState
) sq
order by sq.reccount desc;

Mailbox Status
Show status and details for a particular mailbox. For part of our troubleshooting we actually set this up to run on a schedule and capture the details for specific mailboxes every 30 mins. Update the names list at the end to the users you want to capture for.

select sub.FullName
,m.ProcessingStateCode
,m.ACTDeliveryMethod
,m.UTCConversionTimeZoneCode
,m.PostponeMailboxProcessingUntil
,m.OutgoingEmailDeliveryMethod
,m.Username
,m.ProcessEmailReceivedAfter
,m.ModifiedOnBehalfBy
,m.NoEmailCount
,m.MailboxId
,m.IncomingEmailDeliveryMethod
,m.EnabledForIncomingEmail
,m.NoACTCount
,m.EmailAddress
,m.OutgoingEmailStatus
,m.EWSURL
,m.OrganizationId
,m.PostponeSendingUntil
,m.EmailServerProfile
,m.ModifiedOn
,m.OwnerId
,m.LastActiveOn
,m.TimeZoneRuleVersionNumber
,m.LastAutoDiscoveredOn
,m.TestEmailConfigurationScheduled
,m.IsEmailAddressApprovedByO365Admin
,m.ModifiedBy
,m.Password
,m.TestMailboxAccessCompletedOn
,m.ExchangeSyncStateXml
,m.ProcessAndDeleteEmails
,m.ReceivingPostponedUntil
,m.RegardingObjectId
,m.EntityImageId
,m.CreatedOn
,m.EmailRouterAccessApproval
,m.StatusCode
,m.ReceivingPostponedUntilForACT
,m.StateCode
,m.PostponeTestEmailConfigurationUntil
,m.UndeliverableFolder
,m.MailboxProcessingContext
,m.Name
,m.IsForwardMailbox
,m.TestEmailConfigurationRetryCount
,m.CreatedOnBehalfBy
,m.AllowEmailConnectorToUseCredentials
,m.HostId
,m.LastMessageId
,m.OwningBusinessUnit
,m.EnabledForACT
,m.IncomingEmailStatus
,m.CreatedBy
,m.TransientFailureCount
,m.IsACTSyncOrgFlagSet
,m.ACTStatus
,m.EnabledForOutgoingEmail
,m.RegardingObjectTypeCode
,m.RegardingObjectIdName
,m.OwnerIdType
,m.LastSuccessfulSyncCompletedOn
,m.ProcessedTimes
,m.AverageTotalDuration
,m.LastSyncErrorOccurredOn
,m.ItemsFailedForLastSync
,m.LastSyncErrorMachineName
,m.LastSyncErrorCount
,m.LastSyncErrorCode
,m.ItemsProcessedForLastSync
,m.LastSyncStartedOn
,m.LastSyncError
,m.OrgMarkedAsPrimaryForExchangeSync
,m.LastDuration
,m.ForcedUnlockCount
,m.ProcessingLastAttemptedOn
,m.LastMailboxForcedUnlockOccurredOn
,m.IsServiceAccount
,m.VerboseLoggingEnabled
,m.FolderHierarchy
,m.OfficeAppsDeploymentCompleteOn
,m.PostponeOfficeAppsDeploymentUntil
,m.OfficeAppsDeploymentScheduled
,m.MailboxStatus
,m.OfficeAppsDeploymentError
,m.OfficeAppsDeploymentStatus
from mailboxbase m with(nolock) inner join
SystemUserBase sub with(nolock) on m.OwnerId=sub.SystemUserId
where m.statecode=0 and sub.FullName in ('Last1, First1','Last2, First2','Last3, First3');

D365UG/CRMUG Houston 2019

It's less than a month until D365UG/CRMUG Focus conference in Houston. This event features 90-minute deep dive sessions in a smaller environment. This will be a great way to learn more about the product and discuss your questions with speakers.

I will be participating in 3 sessions. Make sure you add them to your schedule!

• Monday March 11th 1pm: XrmToolBox and Other Helpful Tools for Administrators. This panel will feature Michael Ochs and Phyllis Eriksen sharing their favorite tools in XrmToolBox as well as other tools that make your D365 life easier.
• Tuesday March 12th 8am: Wrangling Your Business Processes Using Workflows. Learn the how to get started writing workflows. We will explain the basics of scope, start when and steps to build.
• Tuesday March 12th 1:15pm: It’s Time to Automate That! Dive deeper into workflows Phyllis Eriksen and look at solving real world problems.

What are you hoping to learn in Houston?

Copying Records with GUID in Microsoft Dynamics 365

If you are using workflows it is likely that you need to link to specific records in the environment. One issue with this occurs if the record is manually created in each environment so when migrating workflows via solutions you still need to manually edit the workflow and re-link to the correct record. This is because the record GUIDs are not consistent across the environments.

But there is a solution!

I was working on this very issue today and had to re-learn how to handle so I want to document it for myself and hopefully it will be beneficial for you too. 

In my case we are using a Queue to send an email from a shared inbox via a workflow. So what I can do is create the Queue in our Development environment and then use an Import to create in other environments with the same GUID.

Steps: 

1. Create Queue in Development environment.. Populate with all necessary details such as name, email address, etc.

   

   Create Queue record in Development environment

2. Create an Advanced Find of these Queue Details. You will want to include any fields you populated such as the email address and description.

   

   Advanced Find to isolate new Queue details
   

   

   Export fields from Advanced Find to get GUID and ensure data matches

3. Export the details from this Advanced Find. This will include the GUID of the record.
4. Open this file and expose columns A, B, C.
5. You can delete columns B and C. Then rename A to "GUID" for clarity. Save this file as a .csv
6. In your next environment, click on Import Data and select the file you just saved.
7. For Mapping, select the entity you want to import to (in this case, Queue) and then map any fields that are needed.
• Map your "GUID" field to "Queue (Primary Key)"
• Keep in mind some fields are set by the system so they may not let you map them. These can be ignored.



When importing map the "GUID" column to the Primary Key for the entity

8. You can choose to save this map if you like for future environments. Then click "Submit"
9. When the import is done, verify the new record was created and you can check the GUID. All should match.

If you entered a name for your Data Map (last step of import process), then you should follow these steps before moving to the next environment:

• Settings -> Data Management -> Data Maps
• Select the one you created and click "Export"
• In your next environment, navigate to the same area and click "Import"

In your next environment you will use the existing file (from your Development environment) from step 5, no need to export again. For step 7, you can either remap or select the imported data map name.

I hope this is helpful for you! Have you had this issue in your organization? How do you handle?

Auto Sending Emails as Another User in Dynamics 365

Workflows in Dynamics 365 can be used to send emails and even send on behalf of another user. This is used in our organization to notify teams about actions needed on their records via automatic or on-demand workflows. When you are adding this functionality, there are a few settings to keep in mind.

Setting up Workflow to Send Email

Create your workflow based on your business logic. Inside this process you can include a "Send Email" step with the details you want to be communicated. By default this would send from the workflow owner (or trigger user if run on-demand) but sometimes you want this to be from a specific user. So populate the from information with a user or dynamic reference to the user you want it sent from.

Workflow Owner Permissions

If sending from another user, the workflow owner needs to have access to Send on Behalf of another user. 

On a Security Role this is on the Business Management Tab under Miscellaneous Privileges as "Send Email as Another User". This is included in the System Administrator and System Customizer Roles by default.

Miscellaneous Privileges on Business Management Tab

Outgoing Mail Options

Now you have to make sure that these emails actually go out! If you are using CRM for Outlook, these emails will only send when Outlook is open and running. So this is a good opportunity to use Server Side Sync. You can turn this on just for Outgoing email if you are not ready to use for everything.

Personal User Settings

Last step is to ensure you have permissions to send as that user. So the user's personal settings need to be updated to allow others to send email on their behalf. This can be done with XrmToolBox using the User Settings Utility. 

1. Launch XrmToolBox and connect to your organization
2. Open the User Settings Utility
3. Click "Load Users and Settings" to pull in users from your organization
1. You can also select a view for the users to limit the list you are looking at. This can be helpful if all users for a team need to start having emails sent.
4. Select the users you want to edit
5. Under "Email Settings" change "Allow emails on my behalf" to "Yes"
6. Click "Update User(s) Settings" in the tool bar

XrmToolBox User Settings Utility Settings

I hope this helps automate automate your processes!