User Profile Sync Issues

I just came across an issue in one of our development environments with User Profile Sync. UPS can be the bane of a SharePoint guy's existence if you run into issue with it.  I have spent quite a bit of time with Microsoft Premier Support working on UPS issues they are very well versed in resolving the issues with UPS.  I highly recommend reading this article on the UPS infrastructure it has completely changed since the MOSS 2007 days http://technet.microsoft.com/en-us/library/gg188041.aspx.  Here are my top tips for resolving UPS issues:

Patch your SharePoint Farm
UPS RTM had some "interesting features" which have most likely caused the issue your are currently having the June 2011 Cumulative Update resolved a substantial amount of the issues.  Patch to a minimum of June 2011!

Re-Provision the FIM configuration
This will fix 95% of User Profile Sync issues something will get corrupt with the connection between SharePoint and FIM.  Stopping the User Profile Synchronization Service de-provisions the FIM configuration and starting it re-provisions it.  It does a substantial amount of configuration that is why it takes so long.

Generally the following procedure will resolve the issue:

1. Stop the User Profile Synchronization Service (Central Administration --> Manage Services on server)
2. Wait until the service has completely stopped!!
3. iisreset
4. Start the User Profile Synchronization Service (Central Administration --> Manage Services on server)
5. iisreset

The iisresets appear to be needed due to the ability to reference the Microsoft.ResourceManagement dlls in the GAC.

If this does not resolve your issue you may need an extended outage:

1. Stop the User Profile Synchronization Service (Central Administration --> Manage Services on server)
2. Stop the User Profile Service 
3. Wait for an hour (I have seen configuration changes occur after the services stop)
4. Restart the server 
5. Start the User Profile Service 
6. Start the User Profile Synchronization Service and the User Profile Service
7. Wait for an hour (I have seen configuration changes occur after the services start as well)
8. Restart the server

Some of the error messages I have seen that this will resolve are:
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.ResourceManagement, Version=4.0.2450.34, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.  

File Not Found

Disable All Content Deployment Jobs

Just a quick script drop we have around 20 content deployment jobs that require disabling and enabling when we are service packing / upgrading.  This number is increasing constantly and will eventually be quite large.  Here is a one liner to disable and re-enable your content deployment jobs, I have two ps1 files on the server (Turn Off Content Deployment Jobs and Turn On Content Deployment Jobs):

Enable
Get-SPContentDeploymentJob | ForEach-Object {$_.IsEnabled = $true; $_.Update()}


Disable
Get-SPContentDeploymentJob | ForEach-Object {$_.IsEnabled = $false; $_.Update()}

Automated SSL Certificate Import (Certutil)

Just a quick script drop.  You can use this script to import SSL certificates.  It will import all the the PFX files in the directory to your COMPUTER\Personal\ Store.  Which is the required store for your SSL Certificates.  It does not import intermediate or CA certificates.  I might add that functionality one day.  Any questions post a comment below:

1. Create a Directory called "SSL" (or something like that)
2. Drop all your PFX files into the SSL folder
3. Create a Batch File "InstallCert.bat" 
4. Run a command prompt as administrator
5. Change Directory to your batch file and run it

Batch File Contents


set CURDIR=%CD%
for /f "usebackq delims=|" %%f in (`dir /b "%CURDIR%" ^| findstr /i pfx`) do certutil.exe -f -p <PFX File Password> -importpfx "%CURDIR%\%%f"
pause

SharePoint 2010 Automated Patcher (Alpha Release)

Here is an alpha version of the source code.  There is still a lot of work to do one this one but it is a starting point.  I used AutoSPInstaller as the foundation for the launching http://autospinstaller.codeplex.com/.

USAGE:

1. Download SharePoint Foundation SP1 / SharePoint Server SP1 / Office Web Apps SP1 and put them in the SP directory
2. Download the latest cumulative update and put it in the CU directory
3. Copy files to all SharePoint Servers
4. Open Command Prompt as an administrator
5. Run Launcher.bat on each SharePoint Server
6. Run Products and Configuration Wizard on each server
7. Done

This is the end to end procedure that we follow here:

SharePoint Service Pack / Cumulative Update Procedure

The following procedure outlines the SharePoint patching process. This is appropriate for both Service Packs and Cumulative Updates. This does not include Windows Service Packs or Hot Fixes.

The procedure has been developed for a farm that has two Web Front End Servers and one Application Server. This procedure can be scaled up to suit a larger farm.

This procedure is to be used for the following SharePoint and Related Services Patches:

• SharePoint Foundation
• SharePoint Server
• Office Web Applications

SharePoint Patches will be packaged and installed via the command line. 

Upgrading a SharePoint Farm is a two phase process which consists of:

Phase	Description
Update Phase	During the update phase the farm can continue to be in production with no downtime. The update process consists of installing the binaries for the next version of SharePoint on each SharePoint server
Upgrade Phase	During the upgrade phase the SharePoint farm must be taken off line as concurrent connections during an upgrade can cause locks thus causing upgrade failures. During the Upgrade Phase the SharePoint Configuration database and Content databases are upgraded to the latest version.

Before proceeding with this procedure verify that the following conditions are true:

• All the Web Front End Servers are load balanced and are in the rotation of the TMG Load Balancer
• All the farm servers are operating correctly
• All the databases are active and operating correctly
• SQL Database backups are present from the previous evening

Do not proceed with the upgrade if any of the preceding conditions are not true. Resolve all issues before you continue.

Update Phase

The following illustration shows the sequence of steps that are required to install the update on the farm.

1. Copy installation package to all SharePoint Servers (this will be done ahead of time)
2. Drain Stop PNWB1 from rotation in the TMG Load Balancer
          a. Check to validate that all users have been relocated to PNWB2 by checking the Web Service Current Connections performance monitor variable
          b. Shutdown PNWB1
          c. Snapshot PNWB1
          d. Start PNWB1
3. Run Command Prompt as an administrator, Browse to SharePoint Patcher Location and Run Launcher.bat on PNWB1
          a. Monitor the Upgrade Status until it completes successfully
          b. Verify Upgrade
          c. Test to verify the server is operational
4. Bring PNWB1 back into rotation in the TMG Load Balancer
5. Drain stop PNWB2 from rotation in the TMG Load Balancer
          a. Check to validate that all users have been relocated to PNWB2 by checking the Web Service Current Connections performance monitor variable
          b. Shutdown PNWB2
          b. Snapshot PNWB2
          c. Start PNWB2
6. Run Command Prompt as an administrator, Browse to SharePoint Patcher Location and Run Launcher.bat on PNWB1PNWB2
          a. Monitor the Upgrade Status until it completes successfully
          b. Verify Upgrade
          c. Test to verify the server is operational
7. Bring PNWB2 back into rotation in the TMG Load Balancer
          a. Snapshot PNAP1
8. Run upgrade.bat on PNAP1
          a. Monitor the Upgrade Status until it completes successfully
b. Verify Upgrade
          c. Test to verify the server is operational
9. At this point all SharePoint Servers have been upgraded to the latest binaries

Upgrade Phase

The following illustration shows the sequence of steps that are required to finish the patching process by upgrading the farm servers. During this process, the sites that are being upgraded will not be available to users.

1. Remove both PNWB1 and PNWB2 from rotation in the TMG Load Balancer

2. Run SharePoint Products Configuration Wizard on the Central Administration Server PNAP1

3. Run SharePoint Products Configuration Wizard on PNWB1

4. Run SharePoint Products Configuration Wizard on PNWB2

5. Bring PNWB2 and PNWB2 back into rotation in the TMG Load Balancer

Upon completion of User Acceptance Testing remove the Snapshots created in the Update Phase

SharePoint 2010 December 2011 Cumulative Update Install Fails

I have found an issue with the installation of SharePoint 2010 December 2011 Cumulative Update.  If you are seeing the following error in you Application Log:

Product: Microsoft Shared Components - Update 'Hotfix for Microsoft Office Server (KB2597014)' could not be installed. Error code 1603. Additional information is available in the log file C:\Users\??\AppData\Local\Temp\osrv-x-none_MSPLOG.LOG.

Open the file and if you search for return value 3 you will find the actual error message just above the return code.  In my situation the error was:


MSI (s) (04:18) [12:33:42:190]: Executing op: End(Checksum=0,ProgressTotalHDWord=0,ProgressTotalLDWord=9722923)
MSI (s) (04:18) [12:33:42:425]: Assembly Error:The process cannot access the file because it is being used by another process.

MSI (s) (04:18) [12:33:42:425]: Note: 1: 1935 2: {E3DD2806-A5AB-43D8-AE84-DFAF878F579C} 3: 0x80070020 4: IAssemblyCacheItem 5: Commit 6: Microsoft.Office.Server.FilterControls,fileVersion="14.0.6108.5000",version="14.0.0.0000000",culture="neutral",publicKeyToken="71E9BCE111E9429C",processorArchitecture="MSIL"
MSI (s) (04:18) [12:33:42:425]: Note: 1: 2205 2:  3: Error
MSI (s) (04:18) [12:33:42:425]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1935
MSI (s) (04:18) [12:33:42:428]: Note: 1: 2205 2:  3: Error
MSI (s) (04:18) [12:33:42:428]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709
MSI (s) (04:18) [12:33:42:428]: Product: Microsoft Shared Components -- Error 1935. An error occurred during the installation of assembly 'Microsoft.Office.Server.FilterControls,fileVersion="14.0.6108.5000",version="14.0.0.0000000",culture="neutral",publicKeyToken="71E9BCE111E9429C",processorArchitecture="MSIL"'. Please refer to Help and Support for more information. HRESULT: 0x80070020. assembly interface: IAssemblyCacheItem, function: Commit, component: {E3DD2806-A5AB-43D8-AE84-DFAF878F579C}

A simple reboot and re-run of the install resolved the issue.

I am going to do some additional testing to see if I can establish why this requires a restart / re-run.

SharePoint 2010 Automated Patcher

SharePoint Patching is a pain especially when you have 29 production servers and 60 non production servers like we do.  I have written an automated patcher to cut the operational overhead of SharePoint patching.  I am running the first test as I write this and it is looking really positive.  I should be able to get a couple up here by Wednesday I would say.

How it works, you put your Cumulative Update in the CU directory and run install.bat.  This is what it does.

• Checks to make sure you are past SP1 if you are not it installs SharePoint Foundation / Server SP1
• Checks to see if Office Web Apps are installed it will install SP1 for Office Web Apps if you have it installed
• Checks to see what CU's you have put in the directory and installs them one after the other

What it does not do:

• PSCONFIG Upgrade that will be a separate script
• Only supports the two latest CU's, if people would like previous CU's post a comment below and I will write the code
• Drain Stop the node from the TMG cluster (I am going to add this in next release)
• Check to see if you are a Farm admin  (I am going to add this in next release)

Post a comment if you want additional functionality

<UPDATE>  Got a bit held up with this due to other priorities.  Doing the first release into production this afternoon if that is successful.  I will hopefully have something up here tomorrow</UPDATE>

Scripted Decommission SharePoint 2007 Web Application

We have approval to decommission one of our last SharePoint 2007 Web Applications! Only two more to go then we are a full 2010 shop!!  I was going to write a script to perform the whole process with one click but can not justify it as we only have two web applications left to decommission.  I am happy with this THREE step process.  Feel free to use it or post comments.

Get the list of Site Collections

1. Run the following command to get all of the site collection URL's and
2. stsadm -o enumsites -url https://webapplication.url > c:\temp\SiteCollections.xml

This will give you all of the details in an XML file that you need to work with

Generate the STSADM / SQL Delete statements

1. Open the XML file in Excel 2010 (it will put everything into columns for you
2. Create a new column with the following formula (this is your "Site Collection Delete Script")
3. ="stsadm -o deletesite -url " &[@Url] & " -gradualdelete"
4. Create a new column with the following formula (this is your "Content DB Delete Script")
5. ="stsadm -o deletecontentdb -url https://webapplication.url  -databasename " & [@ContentDatabase]
6. In this web application we have one Site Collection for each Content Database if you do not you will have to remove non Unique values for this script (or just let it fail)
7. Create a new column with the following formula (this is your "DROP Database script")
8. ="DROP DATABASE [" & [@ContentDatabase] &"]"
9. Your scripts will look like this:

• stsadm -o deletesite -url https://webapplication.url/stuff/eight -gradualdelete
• stsadm -o deletecontentdb -url https://webapplication.url -databasename eight -data
• DROP DATABASE [eight]

I always thought that if you have a multi server farm you had to specify the database server to delete the content database.  Turns out you don't.

Run the scripts

1. Run the "Delete Site Collection Script"
2. Check to make sure you got all of the Site Collections by going to Central Admin --> Application Management --> Content Databases
3. Every Content Database should by in a stopped State and have 0 Sites
4. Run the "Content Database Delete Script"
5. Check to make all the Content Databases have been deleted by going to Central Admin --> Application Management --> Content Databases
6. This only detaches the databases from SharePoint!  You still have Drop the Databases
7. Run the "DROP Database script" (on the SQL Server)

The only thing left to do is delete the Web Application I did that in Central Admin though not using stsadm.

If some one writes a one click decom prodecdure let me know.