User Profile Sync Issues

I just came across an issue in one of our development environments with User Profile Sync. UPS can be the bane of a SharePoint guy's existence if you run into issue with it.  I have spent quite a bit of time with Microsoft Premier Support working on UPS issues they are very well versed in resolving the issues with UPS.  I highly recommend reading this article on the UPS infrastructure it has completely changed since the MOSS 2007 days http://technet.microsoft.com/en-us/library/gg188041.aspx.  Here are my top tips for resolving UPS issues:

Patch your SharePoint Farm
UPS RTM had some "interesting features" which have most likely caused the issue your are currently having the June 2011 Cumulative Update resolved a substantial amount of the issues.  Patch to a minimum of June 2011!

Re-Provision the FIM configuration
This will fix 95% of User Profile Sync issues something will get corrupt with the connection between SharePoint and FIM.  Stopping the User Profile Synchronization Service de-provisions the FIM configuration and starting it re-provisions it.  It does a substantial amount of configuration that is why it takes so long.

Generally the following procedure will resolve the issue:

1. Stop the User Profile Synchronization Service (Central Administration --> Manage Services on server)
2. Wait until the service has completely stopped!!
3. iisreset
4. Start the User Profile Synchronization Service (Central Administration --> Manage Services on server)
5. iisreset

The iisresets appear to be needed due to the ability to reference the Microsoft.ResourceManagement dlls in the GAC.

If this does not resolve your issue you may need an extended outage:

1. Stop the User Profile Synchronization Service (Central Administration --> Manage Services on server)
2. Stop the User Profile Service 
3. Wait for an hour (I have seen configuration changes occur after the services stop)
4. Restart the server 
5. Start the User Profile Service 
6. Start the User Profile Synchronization Service and the User Profile Service
7. Wait for an hour (I have seen configuration changes occur after the services start as well)
8. Restart the server

Some of the error messages I have seen that this will resolve are:
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.ResourceManagement, Version=4.0.2450.34, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.  

File Not Found