Sometimes the Problem is You

There has been an odd issue going on in my implementation for the last several months. With the help of Microsoft support, we solved the issue this week! However, the solution is very embarrassing because it was a self-made problem (I guess most of them are). But maybe my failures will help others so here goes!

The Problem

Towards the end of October, users started reporting that they were seeing the "Pending Email" warning when launching D365. We reviewed and saw there indeed there were email messages in pending status waiting to be sent.

All of these were workflow generated emails to a variety of different users. There didn't seem to be a common pattern in what these emails were, who they were to or the workflow that created them.

The Work

First to solve the issue we used a workflow to change the Email Message status back to Draft and re-send. Then we learned that everyone had already received the original message. So for some reason the Email in D365 said Pending Send even though it was actually sent and the Date Sent was even populated.

Next we created another new workflow that just updated the status from Pending Send to Sent. This was not solving the issue but it was preventing users from getting the Pending Send Warning message.

We believed the issue had something to do with our connection with Exchange. Maybe we weren't getting a response in a timely manner or something like that. So the "it's not my fault!" attitude was in full force. We basically started handling the symptoms (running the workflow to change status) and stalled on dealing with the problem.

Finally some bandwidth opened up so we decided it was time to engage Microsoft support on this issue. We worked with them for a little while to get traces, logs, etc. and provided all this data.

The Solution

Microsoft reviewed all the data we sent and said "hey, it looks like there is a workflow firing on all of these emails changing them to Pending Send". What? No way! They must be confused.

Sure enough, I found the workflow they were seeing with the step name they reported. It was doing exactly what they said was happening. I deactivated the workflow and the issue stopped occurring.

Then we thought about the timeline and figured out exactly what happened. (Side tip: we have turned Auditing on for the Processes entity. This makes it easy to check who created the workflow, who activated it, when, when was the owner changed, etc.)

• Earlier in October we had an issue with some emails failing (actually in a Failed state)
• One admin went in and created a workflow to change back to Pending Send so Server Side Sync would automatically pick up and re-send the email
• This was set to run on-Demand
• Scope = User
• Run on Create = Yes. This is fine when the owner was that admin. This is just for clean up, it won't be a problem, right?
• Automatically deleted completed workflow jobs = Yes
• Later in October there is an unrelated release involving other unrelated workflows
• Part of the release process is reassigning these workflows to the Service Account we use to run these workflows. I (I will take the blame!) reassigned all workflows recently modified to the Service Account not paying attention to the specific items modified for that release
• Now all emails being owned by/created by the Service Account, AKA all workflow generated emails, are being set to Pending Send after they are sent

As I said, not all emails were "stuck" in Pending Send status. I think this was still due to timing of the email actually being sent and email owner and those type of factors.

Lessons Learned

1. It can always be your/my fault
2. Review the Start When conditions even when just doing "quick"clean up items
3. When using a workflow for clean up or quick data manipulation, deactivate when done
4. Always communicate with others who are creating workflows and review so everyone is on the same page

So that is my embarrassing story for the week. I hope Microsoft does not put it on my permanent record. Please share your self-inflicted, system mishaps to help me feel better!