Gotcha: DateTimeKind and FetchXml

I was going through the FetchXML schema and found these two interesting attributes. The first one is “utc-offset” on the “fetch” node.

utcoffset

The other one is “usertimezone” on the “attribute” node.

usertimezone

Both these attributes give an impression, that it is possible to return datetime attributes in a different timezone instead of UTC, but in reality they don’t seem to do anything.

The only way to return datetime in the user’s local timezone, is by using the deprecated ExecuteFetchRequest. I will demonstrate this with XrmToolBox. This is my simple fetch query.

fetchxml-with-datetime

Below is the raw fetch result

fetch-xml-response-raw

Below is the table grid result

fetch-xml-response-retrievemultiple

I execute the FetchXML using the FetchXML Builder tool. As you can see, the datetime values are different. This is because when you use “RetrieveMultiple” to execute the fetch query, the datetime returned is always UTC and has to be converted to the local timezone. ExecuteFetchRequest is now deprecated and RetrieveMultiple is the recommended way to execute fetchxml. These two points really puzzle me:

  1. Why user-timezone and utc-offset don’t seem to do anything. Are they internally used?
  2. Why MS decided to change the datetime behaviour

So, this is a good point to remember when you are executing fetchxml using “FetchXML Tester” or have set the Result view to Raw fetch result in “FetchXML Builder” or using a mixture of ExecuteFetchRequest and RetrieveMultiple in your code.

References:

  1. ExecuteFetchRequest class
  2. XrmToolBox Issue #326

 

Bookmarklet: Copy and Paste Lookup

Lookups (including partylist and customer) cannot be copy pasted. I have encountered couple of scenarios where I needed this capability. I have developed these bookmarklets to solve this issue.

Copy Lookup

Select and the drag the code below to the bookmark bar.

javascript:(function(){let contentPanels=Array.from(document.querySelectorAll('iframe')).filter(function(d){return d.style.visibility!=='hidden'});if(contentPanels&&contentPanels.length>0){let Xrm=contentPanels[0].contentWindow.Xrm;let currentControl=Xrm.Page.ui.getCurrentControl();if(currentControl&¤tControl.getControlType()==='lookup'){let currentLookup=currentControl.getAttribute().getValue();if(currentLookup){let serialisedLookupValue=JSON.stringify(currentLookup.map(x=>{let c={};({id:c.id,name:c.name,type:c.type,typename:c.typename,entityType:c.entityType}=x);return c;}));sessionStorage.setItem('ryr_serialisedLookup',serialisedLookupValue);alert('Lookup copied. Ready to paste');}}else{alert('No field has been selected or the currently selected field is not a lookup');}}else{alert('Entity form not detected');}})();void 0;

Source:

(function () {
	let contentPanels = Array.from(document.querySelectorAll('iframe')).filter(function (d) {
			return d.style.visibility !== 'hidden'
		});
	if (contentPanels && contentPanels.length > 0) {
		let Xrm = contentPanels[0].contentWindow.Xrm;
		let currentControl = Xrm.Page.ui.getCurrentControl();
		if (currentControl && currentControl.getControlType() === 'lookup') {
			let currentLookup = currentControl.getAttribute().getValue();
			if (currentLookup) {
				let serialisedLookupValue = JSON.stringify(
						currentLookup.map(x =  > {
								let c = {};
								({
									id : c.id,
									name : c.name,
									type : c.type,
									typename : c.typename,
									entityType : c.entityType
								}
										 = x);
								return c;
							}));
				sessionStorage.setItem('ryr_serialisedLookup', serialisedLookupValue);
				alert('Lookup copied. Ready to paste');
			}
		} else {
			alert('No field has been selected or the currently selected field is not a lookup');
		}
	} else {
		alert('Entity form not detected');
	}
})();

Paste Lookup

Select and the drag the code below to the bookmark bar.

javascript:(function(){let contentPanels=Array.from(document.querySelectorAll('iframe')).filter(function(d){return d.style.visibility!=='hidden'});if(contentPanels&&contentPanels.length>0){let Xrm=contentPanels[0].contentWindow.Xrm;let currentControl=Xrm.Page.ui.getCurrentControl();if(currentControl&¤tControl.getControlType()==='lookup'){let currentLookup=currentControl.getAttribute();let copiedLookupValue=sessionStorage.getItem('ryr_serialisedLookup');if(copiedLookupValue){currentLookup.setValue(JSON.parse(copiedLookupValue));} else{alert('Please select a lookup to copy first before pasting');}}else{alert('No field has been selected or the currently selected field is not a lookup');}}else{alert('Entity form not detected');}})();void 0;

Source:

(function () {
	let contentPanels = Array.from(document.querySelectorAll('iframe')).filter(function (d) {
			return d.style.visibility !== 'hidden'
		});
	if (contentPanels && contentPanels.length > 0) {
		let Xrm = contentPanels[0].contentWindow.Xrm;
		let currentControl = Xrm.Page.ui.getCurrentControl();
		if (currentControl && currentControl.getControlType() === 'lookup') {
			let currentLookup = currentControl.getAttribute();
			let copiedLookupValue = sessionStorage.getItem('ryr_serialisedLookup');
			if(copiedLookupValue){
				currentLookup.setValue(JSON.parse(copiedLookupValue));
			}
			else{
				alert('Please select a lookup to copy first before pasting');
			}
		} else {
			alert('No field has been selected or the currently selected field is not a lookup');
		}
	} else {
		alert('Entity form not detected');
	}
})();

Instructions

  1. Select the lookup field to be copied. The lookup will usually be highlighted with a blue background when it is selectedlookup-selected
  2. Run the Copy Lookup bookmarklet. You will get a alert message confirming that the lookup has been copied
  3. Select the lookup field that is the target of the paste. The lookup has to be of the same type i.e. if the copied lookup was a contact lookup, the target lookup also has to be a contact lookup
  4. Run the Paste Lookup bookmarklet
  5. Save the record

I have tested these bookmarklets only in Chrome 53 and Firefox 50.

Convert Personal View to System View

When you design a query from the “Advanced Find” window, you can save the query for future use. I call this “Personal View”, but the official name for this is “Saved View”. This view is only visible to the person who saved the view (unless it is shared/reassigned).

As more personal views get created, it becomes an issue during migration, as these are not transported in the solution xml. You can of course use Solution Extender to copy this across, but I would like to do this right from the Advanced Find window. Apart from data migration, another scenario where you might find the need to create a system view from a personal view, is when you are constantly sharing views to a large number of people of group. When you are doing this, it is good time to actually make this a system view.

With these scenarios in mind, I have developed a solution that simplifies the process of creating a system view from a personal view. After installing the managed solution, you will see a new button called “Promote to System View” in the advanced find, when you switch to the “Saved Views” tab.

screenshot

To create a system view from the personal view, simply choose the views that need to be converted, and click the “Promote to System View” button. A message will be displayed after the conversion is complete. Once you refresh the window, you should be able to see the newly created system view(s).

Please log any issues/feedback/feature request in the github repo -> https://github.com/rajyraman/Personal-View-to-System-View/issues

You can download the managed solution from https://github.com/rajyraman/Personal-View-to-System-View/releases

Plugin Integration Tests using FakeXrmEasy

XrmUnitTest and FakeXrmEasy are two testing frameworks that are specifically targeted towards Dynamics CRM/Dynamics 365. It is possible to do both unit test as well as integration tests using both these frameworks. In this first post, I would like to cover FakeXrmEasy. Here is a sample plugin code that I would like to test:

using Microsoft.Xrm.Sdk;
using System;
using Microsoft.Crm.Sdk.Messages;

namespace SamplePlugin
{
    public class LastNameUpperCasePlugin : IPlugin
    {
        public void Execute(IServiceProvider serviceProvider)
        {
            ITracingService tracer = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
            IPluginExecutionContext context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
            IOrganizationServiceFactory factory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
            IOrganizationService service = factory.CreateOrganizationService(context.UserId);

            try
            {
                Entity entity = (Entity)context.InputParameters["Target"];
                //not required. Just added to demonstrate that we are connecting to a real crm org service
                var response = service.Execute(new WhoAmIRequest());

                var lastName = entity.GetAttributeValue<string>("lastname");
                if (!string.IsNullOrEmpty(lastName))
                {
                    entity["lastname"] = lastName.ToUpper();
                }
            }
            catch (Exception e)
            {
                throw new InvalidPluginExecutionException(e.Message);
            }
        }
    }
}

In order to test this plugin, we have to create a unit test project. Here are the steps for it:

  1. Create a unit test  project from the existing plugin solution by right clicking and choose Add->New Project->Visual C#->Test->Unit Test Project
  2. Use nuget to add the correct FakeXrmEasy reference to the test project. In this example I am using “FakeXrmEasy.365” as I am connecting to a Dynamics 365 instance online. Refer https://www.nuget.org/profiles/jmontana to find out all the different versions that are relevant to your CRM version.
  3. Create a App.config file, if it doesn’t already exist. Here is my connection string. Refer https://msdn.microsoft.com/en-us/library/mt608573.aspx to find the correct format of the connection string for you crm instance.connection-string

Below is my test for the plugin

using System;
using System.Collections.Generic;
using System.Web.Configuration;
using FakeXrmEasy;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.Xrm.Sdk;

namespace SamplePlugin.Test
{
	[TestClass]
	public class LastNamePluginUnitTests
	{
		[TestMethod]
		public void Should_Set_LastName_In_UpperCase()
		{
			var context = new XrmRealContext
			{
				ProxyTypesAssembly = typeof(LastNameUpperCasePlugin).Assembly,
				ConnectionStringName = "CRMOnline"
			};
			var executionContext = context.GetDefaultPluginContext();
			var target = new Entity("contact")
			{
				["lastname"] = "Power",
				Id = Guid.NewGuid()
			};
			executionContext.MessageName = "Create";
			executionContext.Stage = 20;
			executionContext.PrimaryEntityId = target.Id;
			executionContext.PrimaryEntityName = target.LogicalName;
			executionContext.InputParameters = new ParameterCollection
			{
				new KeyValuePair<string, object>("Target", target)
			};
			context.ExecutePluginWith<LastNameUpperCasePlugin>(executionContext);
			Assert.AreEqual("POWER",
				((Entity) executionContext.InputParameters["Target"]).GetAttributeValue<string>("lastname"));
		}
	}
}

In this test I am creating a scenario where the plugin is running pre-create. You can change to add pre/post entity images or target a different message and stage. If we debug our integration test, we can see that we are connected to the real org service.

Debugger.png

I have used latebound entity in this example, but you can use the same code with early bound entities generated using crmsvcutil or Early Bound Generator.

You can use a similar approach for testing custom workflow assemblies as well. In that case you would be using “GetDefaultWorkflowContext” instead of “GetDefaultPluginContext” to get the execution context.

Footnote: Technically this test could have been better written as an unit test instead of a integration test. But the purpose of this post is to demonstrate the points below:

  1. How to build a fake plugin execution context
  2. How to use connection strings to create a real OrganizationService instance
  3. How to use the real OrganizationService with the fake plugin execution context

EDIT (22/11/2016): Thank you Jonas (@rappen) for pointing out the plugin stage should be pre-create for this to work. Updated context stage to 20.

Server Side Sync – Polling interval

I have been battling with getting server side sync and forward mailbox working the way I want it to over the last couple of weeks. Googling hasn’t helped me a lot on this regard, and so in this post I will explain my understanding of the server side sync process.

The biggest hurdle that I faced in understanding the server side sync process is how little detail there is in the mailbox sync errors. I am not talking about the CRM Tracing, just the standard OOB logging in the application (like workflow error). Lot of times, I have seen unknown errors and mailboxes getting disabled randomly, but no detailed log in the application to indicate what the issue could be.

I used the following query in LINQPad with the CRM driver installed.

from m in MailboxSet
where m.EmailAddress == "email@email.com"
select new {
m.EmailAddress,
PostponeMailboxProcessingUntil = m.PostponeMailboxProcessingUntil.Value.ToLocalTime(),
m.ProcessedTimes,
ProcessingLastAttemptedOn = m.ProcessingLastAttemptedOn.Value.ToLocalTime(),
ReceivingPostponedUntil = m.ReceivingPostponedUntil.Value.ToLocalTime(),
PostponeSendingUntil = m.PostponeSendingUntil.Value.ToLocalTime(),
m.NoEmailCount,
m.TransientFailureCount,
m.LastSuccessfulSyncCompletedOn,
m.LastSyncError,
m.LastSyncErrorCode,
m.LastSyncErrorCount,
m.LastSyncErrorMachineName,
m.LastSyncErrorOccurredOn,
m.LastSyncStartedOn,
m.ItemsFailedForLastSync,
m.ItemsProcessedForLastSync
}

Here is a sample result.

sss

Based my tests, here is how I think it works for incoming email. The same is true for outbound emails. I am assuming that the mailboxes are not disabled and email address is approved:

  1. Async service starts to process the email at time specified in the PostponeMailboxProcessingUntil field
  2. Async service starts processing the inbound emails and creates email records in CRM
  3. In order to set the PostponeMailboxProcessingUntil and the ReceivingPostponedUntil , for the next run async service looks into the NoEmailCount field field. The NoEmailCount field is incremented every time the async service attempts to process email. If it did create some emails in CRM, the counter is reset to zero.

Based on my tests, I believe that if no email is sent/received in 15 minutes, the polling period is fixed at around 5 minutes. Here are my test results.

sss-tests

As you can see based on my tests, the polling interval gradually starts climbing when there are no emails to process and hovers around 5 minutes, after 15 minutes of inactivity i.e. no email records were created on CRM and/or no emails were sent from CRM.

If you have a queue setup in CRM, and forward an email to the queue, 5 minutes is a very long time to wait for the async service to pickup the email from Exchange. I saw couple of posts about how this polling interval can be changed by modifying the “Minimum Polling Interval” field.

polling-interval

The recommended value for this field is “0” and even when I changed this to 1 minute, I haven’t seen it influence the “ProcessingLastAttemptedOn“. I tried to change “PostponeMailboxProcessingUntil” and “ReceivingPostponedUntil” for the mailbox though the SDK, but since they are readonly fields, they cannot be changed. All the tests were carried out in OnPrem CRM2015, and so I cannot confirm whether this is the same behaviour in CRM2016 or Dynamics 365.

EDIT (18/11/2016): This 5 minute polling interval seems to be a “by-design” behavior. Thank you Andre Margono for sending this info. Here is what the MSDN article says:

When synchronization by using server-side synchronization occurs, the process is dynamic and unique for each user’s mailbox. The synchronization algorithm ensures that mailboxes are synced according to dynamic parameters such as the number of email messages and the activity within the mailbox. Normally, email synchronization occurs every 5 minutes. When a mailbox has many email messages, the interval can be reduced dynamically to 2 minutes. If the mailbox is less active, the interval can be increased up to 12 minutes. Generally speaking, you can assume that a mailbox will be synced at least once every 12 minutes. Note that you can’t manually synchronize records through server-side synchronization and when you track email (Track button), this occurs immediately.

tl;dr; If your CRM application is not processing any email (sending/receiving) in 15 minutes, the polling interval will eventually be fixed at around 5 minutes. When an email is received, async service will start processing more frequently, but the polling period will slowly climb with no-activity until it gets to 5 minutes.

Export to Excel using Dynamics 365 SDK

With the new Dynamics 365 release, a new message has been added that making exporting FetchXML results really simple. This message is not documented, hence is technically unsupported. With that word of warning, I will show you how to utilise this new message to export data from Dynamics 365 to an Excel file.

ExportToExcel message definition

Parameter Type
View EntityReference
FetchXml string
LayoutXml string
QueryApi string
QueryParameters InputArgumentCollection

Code

using System;
using System.Collections.Generic;
using System.Configuration;
using System.IO;
using System.Linq;
using System.ServiceModel;
using Microsoft.Crm.Sdk.Messages;
using Microsoft.Xrm.Client;
using Microsoft.Xrm.Client.Services;
using Microsoft.Xrm.Sdk;

namespace Experiments
{
    class Program
    {
        private static OrganizationService _orgService;
        static void Main(string[] args)
        {
            try
            {
                CrmConnection connection = CrmConnection.Parse(
                    ConfigurationManager.ConnectionStrings["CRMConnectionString"].ConnectionString);

                using (_orgService = new OrganizationService(connection))
                {
                    var exportToExcelRequest = new OrganizationRequest("ExportToExcel");
                    exportToExcelRequest.Parameters = new ParameterCollection();
                    //Has to be a savedquery aka "System View" or userquery aka "Saved View"
                    //The view has to exist, otherwise will error out
                    //Guid of the view has to be passed
                    exportToExcelRequest.Parameters.Add(new KeyValuePair<string, object> ("View",
                        new EntityReference("userquery", new Guid("{0B915102-24A7-E611-8101-1458D05B1178}"))));
                    exportToExcelRequest.Parameters.Add(new KeyValuePair<string, object>("FetchXml", @"
                    <fetch distinct='false' no-lock='false' mapping='logical' returntotalrecordcount='true'>
	                    <entity name='contact'>
		                    <attribute name='fullname' />
	                    </entity>
                    </fetch>"));
                    exportToExcelRequest.Parameters.Add(new KeyValuePair<string, object>("LayoutXml", @"
                    <grid name='resultset' object='2' jump='fullname' select='1' icon='1' preview='1'>
	                    <row name='result' id='contactid'>
		                    <cell name='fullname' width='300' />
	                    </row>
                    </grid>"));
                    //need these params to keep org service happy
                    exportToExcelRequest.Parameters.Add(new KeyValuePair<string, object>("QueryApi", ""));
                    exportToExcelRequest.Parameters.Add(new KeyValuePair<string, object>("QueryParameters",
                        new InputArgumentCollection()));
                    var exportToExcelResponse = _orgService.Execute(exportToExcelRequest);
                    if (exportToExcelResponse.Results.Any())
                    {
                        File.WriteAllBytes("Active Contacts.xlsx", exportToExcelResponse.Results["ExcelFile"] as byte[]);
                    }
                }
            }
            catch (FaultException<OrganizationServiceFault> ex)
            {
                string message = ex.Message;
                throw;
            }
        }
    }
}

Closing Notes:

  1. “View” parameter can accept “userquery” or “savedquery”, but they have to exist i.e. you can’t pass empty Guid.
  2. The fetchxml and layoutxml can be different from what is in the “savedquery” or “userquery”. Hence, you can create a “Personal View” just so that you can use it in this message, but modify the fetchxml and layoutxml to whatever you want.
  3. The name of the tab in the Excel output file will be the name of the view specified in the “View” parameter
  4. This message can be executed from Javascript as well, but you will get a base64 string instead of a byte array in the response.

Please vote up my request on Connect (logged Feb 2015) -> https://connect.microsoft.com/site687/feedback/details/1127874/export-to-excel-sdk-message so that this message can be made available as an unbound WebAPI action.

Quick summary of Dynamics 365 Licensing

I was reading through the new Dynamics 365 Licensing details, and I thought I will share what I learnt about the changes in Dynamics 365 licensing model compared to CRMOnline. Below is a the summary of my understanding.

  • These are four licence types: Enterprise Edition Plan 2, Enterprise Edition Plan 1, Plan 1 Applications, Enterprise Team Members and Operations Application.
  • Enterprise Edition Plan 2 is the most expensive (and with most number of features) and Enterprise Team members is the least expensive (and least number of features).
  • Enterprise Edition Plan 2 includes everything. There is no tiered pricing, but Plan 2 users are counted towards Plan 1 tiered pricing.
  • Enterprise Edition licensing includes dual use rights i.e. you are licensed to use both online and on-premise edition of Dynamics 365. However, you need be on CRM2013 or higher to use the on-premise license.
  • There is no minimum seat requirement for Enterprise Edition Plan 1 and Plan 1 Applications.
  • There is a 20 user minimum requirement for Enterprise Edition Plan 2 and Operations Application.
  • Enterprise Edition Plan 1 includes a non-production instance. No minimum user condition (unlike the current CRMOnline, which has 25 user minimum).
  • Operations Application includes 2 sandbox instances.
  • Enterprise Edition Plan 1, Plan 1 Applications and Enterprise Team Members includes Portal (AdxStudio).

This is the new pricing model

image

I have developed an Excel sheet to calculate the pricing for various license types. You view/download the Excel file from https://1drv.ms/x/s!APzjERKFC6gOtzw

Please let me know if there are any errors.

References:

  1. Introduction to Dynamics 365 Licensing
  2. Pricing and Licensing updated (PPTX) -> This file is no longer available for download
  3. Pricing and Licensing FAQ (PDF) -> This file is no longer available for download