Pogo69's Blog

December 2, 2012

Adventures in Multi-Threaded CRM 2011 Client Development

Filed under: .NET, C#, CRM — pogo69 [Pat Janes] @ 12:52

Overview

I’ve recently completed development of a process to auto-magically batch import data on a weekly basis for a client.

I have no control over the format in which the data will be received; the client (or their technical representatives) have decided that I will receive the entire set of Active Associations (data representing a link between a Contact and Account) each import.  I will not, however, receive any indication of Inactive (that is, deactivated) associations; as such, the import process will “clean up” prior to each import, deleting all active associations and updating each affected Contact (setting a custom Status attribute to Inactive).

I will discuss the import process in another blog posting; the focus of this post is the “clean up”; specifically the development process of deleting and updating approximately 100,000 records (each) as efficiently as possible.

Early Bound Classes, Large Record Sets and Memory Consumption

The initial iteration of my clean up program, simply queried for a set of all Contacts that were currently set to CustomStatus == Active.  It then looped through each record and set them, one by one, to Inactive.  In pseudo-code:

var contactIds =
  (
    from c in xrm.ContactSet
    where c.new_status == 1
    select c.Id
  ).ToList();

foreach (Guid contactId in contactIds)
{
  Contact c = new Contact
  {
    Id = contactId,
    new_status = 2
  };

  xrm.Update(c);
}

This caused the console program to consume huge amounts of memory to the point that the program stopped working.  I thought it may have to do with the ServiceContext caching objects, so I turned caching off completely:

xrm.MergeOption = MergeOption.NoTracking;

It had no effect (on memory); so I started “batching” my updates.

Enter the ThreadPool

I decided to use System.Threading.ThreadPool to operate simultaneously on batches of records.  Initially, my code selected 1,000 records at a time:

var contactIds =
  (
    from c in xrm.ContactSet
    where c.new_status == 1
    select c.Id
  ).Take(1000).ToList();

foreach (Guid contactId in contactIds)
{
  ThreadPool.QueueUserWorkItem(id =>
  {
    Contact c = new Contact
    {
      Id = id,
      new_status = 2
    };

    xrm.Update(c);
  }, contactId);
}

This would run for a little while, before erroring.  Trawling through the CRM Trace Log files indicated that I was encountering Timeout errors.  WCF Services have a throttling mechanism that allows only a certain number of simultaneous connections.  So I throttled my client code:

ThreadPool.SetMaxThreads(10, 10);

The first parameter allows only 10 concurrently running threads in the ThreadPool.  All other requests must wait for a spare thread to become available, effectively throttling my multi-threaded client code.

But it still errored; this time, I started generating SQL Deadlock errors.  It was intermittent and random.  It appears that I still had threads doing multiple updates when the next batch of candidate Contacts was being selected.  So I had to separate the select and update operations by “waiting” for each batch of updates to complete before going back and selecting the next batch.

In order to wait for a batch of threads, I needed to set up an array of WaitObjects.  The maximum number of objects that can be waited on simultaneously is 64, so that became my batch size:

class Program
{
  #region Local Members
  private static int BATCH_SIZE = 64;
  #endregion

  #region Thread Worker Classes
  private class ContactUpdater
  {
    XrmServiceContext _xrm;
    Guid _contactId;
    ManualResetEvent _doneEvent;

    public ContactUpdater(XrmServiceContext xrm, Guid contactId, ManualResetEvent doneEvent)
    {
      this._contactId = contactId;
      this._doneEvent = doneEvent;
      this._xrm = xrm;
    }
    public void UpdateContact(Object o)
    {
      // de-activate contact
      Contact contact = new Contact
      {
        Id = this._contactId,
        new_Status = (int)Contactnew_Status.Inactive
      };

      // update
      this._xrm.Update(contact);

      // signal that we're done
      this._doneEvent.Set();
    }
  }
  #endregion

  static void Main(string[] args)
  {
    ThreadPool.SetMaxThreads(10, 10);

    using (XrmServiceContext xrm = new XrmServiceContext("Xrm"))
    {
      xrm.MergeOption = Microsoft.Xrm.Sdk.Client.MergeOption.NoTracking;

      // update ALL current active contacts
      while (true)
      {
        // retrieve batch
        List<Guid> contactIds =
          (
            from c in xrm.ContactSet
            where c.new_Status.GetValueOrDefault() == (int)Contactnew_Status.Active
            select c.Id
          ).Take(BATCH_SIZE).ToList();

        if (contactIds.Count == 0) { break; }

        ManualResetEvent[] doneEvents = new ManualResetEvent[contactIds.Count];

        // spawn processing threads
        for (int i = 0; i < contactIds.Count; i++)
        {
          ContactUpdater contactUpdater = new ContactUpdater(xrm, contactIds[i], (doneEvents[i] = new ManualResetEvent(false)));

          ThreadPool.QueueUserWorkItem(contactUpdater.UpdateContact);
        }

        // wait for all threads to complete before processing next batch
        WaitHandle.WaitAll(doneEvents);
      }
    }
  }
}

Conclusion

This provides a framework for the multi-threaded updating of a large number of CRM Entity Instances.  With a little bit of tinkering you could get it to fit most batch update situations.

You could probably also play with the ThreadPool maximum thread count; I haven’t, but 10 seemed like a conservatively good number.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: