This has been nagging me for about a month.  On my personal SBS 2008 box, since mid-October, I have been getting this error repeatedly.  When logged on to the server console, a Windows Error Reporting dialog pops up every minute reporting that Taskeng.exe has stopped working.

Taskeng is the main process of Windows Task Manager, which has a console under Administrative Tools. 

What was causing Task Scheduler to crash?  I very quickly found it—Certificate Services Client was causing it, as you may have seen in the Event Viewer screenshot;  it throws a error message right before Task Scheduler reports the crash.  It has three tasks:  SystemTask, UserTask and UserTask-Roam.  On my machine, SystemTask was crashing and trying to restart.  That accounted for the repeating WER dialogs.

But what is the Certificate Services Client?  What do its three tasks do?

The Certificate Services Client (CSC, not to be confused with Client-Side Caching) is a component of Windows that allows users to use their site certificates, which are often used for VPN, email authentication and many other services on a corporate network.  It’s been a part of Windows for awhile, and it’s on Windows Server 2008 and by extension SBS 2008.

A very important part of the CSC is autoenrollment:  Autoenrollment is the process of keeping track of user certificates, expiring them and renewing them as needed.  PAUTOENR.DLL is the DLL that manages this process, running with DIMSJOB.DLL and DIMSROAM.DLL under Task Scheduler.

After a lot of tracing with Process Monitor and a lot of research online, I have a conclusion:  The CSC, and the Autoenrollment feature, each keep a cache of certificate information.  It was corrupted.   I suspect a certificate expired sometime in October or a bit before, and the Autoenrollment choked on it.

I must note at this point that the CSC is a client.  On an SBS machine, it runs alongside the Certificate Services role, but it is totally separate so it does no good to troubleshoot Cert Services if it is otherwise working.  Don’t touch Cert Services.

This is an involved process with several reboots so you will need patience.

1. First, pull up Task Scheduler from Administrative Tools.  Select “Task Scheduler Library” and open Microsoft\Windows\CertificateServicesClient.   We are going to disable each of the three tasks before we delete the cache.  On each task, SystemTask, UserTask and UserTask-Roam, right-click and select End.  Repeat this for each task and select Disabled.
2. Follow these steps to delete the certificate cache on the server:  Go to the LOCAL SYSTEM profile at C:\Windows\System32\config\systemprofile\AppData\LocalLow\Microsoft\CryptnetUrlCache.  Copy the CryptNetURLCache folder to a safe place.  Two folders are there—Content and Metadata.   Go to each folder and delete the files therein—don’t delete the Content and Metadata folders themselves.
3. Run Regedit as an administrator and go to this key, HKEY_LOCAL_MACHINE\Software\Microsoft\SystemCertificates\AuthRoot\Certificates.  Delete all the keys under this key.   That takes care of the certificate cache.
4. Now, delete the Autoenrollment cache.  Go to Regedit and to HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\AutoEnrollment\AEDirectoryCache.  Delete all the keys under that key.
5. Go back to Task Scheduler.  Open the CertificateServicesClient folder that we had opened before (Task Scheduler Library\Microsoft\Windows\CertificateServicesClient).  Right-click on each task and select “Enable”.
6. We aren’t done yet.  The Certificate caches have been empty, but they need to be repopulated.  To do this, we need to change the local security policy on the machine to turn on autoenrollment temporarily until the caches are repopulated.  By default (on my SBS box at least), autoenrollment is not turned on.  This is often done on Windows networks via the Default Domain policy but I am really uneasy about making a global change to fix a local problem so the local security policy is the best way.
7. Open secpol.msc as an administrator to access the local security policy.  Click on Public Key Policies.  On the right pane, double-click Certificate Services Client—Auto-Enrollment Policies.  In the dialog, click on the Configuration Model dropdown box and select Enabled.  There are two checkboxes:  Check both boxes, “Renew expired certificates, update pending certificates and remove revoked certificates” and “Update certificates that use certificate templates”.  Click OK.
8. From an elevated command prompt, enter the command gpupdate /force to make the change apply to the machine.
9. Restart the server.
10. Log back on and reopen the Task Scheduler and the CertificateServicesClient task.  SystemTask and UserTask will show as running.  Click on SystemTask, and click the History tab.  The most recent events in History should show that the task started normally and is currently running.
11. While still on the SystemTask window, right-click and select End.  This task normally only runs when the system is restarted, and every eight hours thereafter.
12. Next, we will undo our changes to the local security policy.  Open secpol.msc as an administrator and select Public Key Policies.  Click on Certificate Services Client—Auto-Enrollment Policies.  On the Configuration Model dropdown box, select “Not Configured” and click OK.
13. From an elevated command prompt, reenter gpupdate /force to undo the change.
14. Restart the server again.
15. Log back on once again and reopen the CertificateServicesClient task.  SystemTask should be listed as Ready or Running.  UserTask should be Running.  UserTask-Roam should be Ready.  Look at the event history for each task.  If there are no errors since the last reboot, the problem is resolved.  You may want to try running the SystemTask as a test, if it is not already running.

Microsoft has recently released a patch to address a well-publicized vulnerability in Windows where programs can load “bad” DLL’s from a WebDAV or SMB share.   However, you need to create a new registry entry to modify Window’s behavior on DLL loading and close the hole.

The new registry entry is CWDIllegalinDllSearch, and there is more information, and a Fix It tool, on Microsoft’s Security Research and Defense blog.  Susan Bradley rightly points out that their advice is not really suitable when there are numerous PC’s in the network, especially when they are all managed by Group Policy.  She mentioned a solution involving Group Policy Preferences.

It’s not often I can keep up with Susan, but I have figured out how she’d do it!  We’ll make a new Group Policy Preference with the required registry settings.  This preference could be applied to the default domain, but I have chosen to limit it to client machines, so it will be created for the SBSComputers OU. 

1. Preconditions:  All your PC’s must have Group Policy Preferences installed;  this is standard for Vista and higher, but available as an update for XP SP3.
2. They must also have KB2264107 installed first.  Do it now if you haven’t already.
3. Open the Group Policy Management Console, select your domain, and go through the OU tree until you reach SBSComputers.  
4. Right-click, select “Create a GPO and link it here”.
5. Pick a name, “DLL Preloading Vulnerability Prevention”.  Click OK.
6. Find the GPO you just created and right-click it.  Select Edit.
7. In the GPO editor, select Computer Configuration/Preferences/Windows Settings/Registry.
8. Right click the Registry item and select New/Registry Item.
9. In the window shown below, to the right of the Key Path item box, click the […] button.

10.  Select SYSTEM/CurrentControlSet/Control/Session Manager.  Do not open any subkeys under Session Manager.  Click Select.

11.  Enter the new value name, CWDIllegalinDllSearch.

12.  Enter the value type, REG_DWORD.

13.  For value data, enter 2.

14.  Click OK.

15.  Close the GPO Editor, and the GPMC console.

If you wish, you can refresh the GPO settings at each client by doing a gpupdate /force. 

The “default” setting of 2 for CWDIllegalinDllSearch blocks DLL loading from Webdav and SMB’s.  There is a more restrictive setting that blocks DLL loading from USB devices.  For that, you would select the Hexadecimal radio button in the dialog above and enter FFFFFFFF.  Until I know how this affects the system I am sticking with the defaults.  It occurs to me that if you have techs who troubleshoot with thumb-drive utilities, this may break them.  Just so you know.

There are other ways to implement this in Group Policy;  one method I have seen involves creating a new GP template with the registry settings.  I think  GP Preferences are the better way to go as, with a custom template, the registry settings are "tattooed" (this is the official GP nomenclature) onto the machine’s registry and remain there even if the policy is removed.  I like to have a clean machine with transparent settings, even though this setting will be permanent through the life of Windows.

Thanks for the hints, Susan!

Exchange 2007 SP3 is now available.  Unlike SP2, there is no wrapper needed to install it under SBS 2008.

However, you may see a few wrinkles in the update process.  You will need to stop the Windows SBS Manager (datacollectorsvc) and, if you have File Server Resource Manager installed, srmsvc.  In Powershell do:

stop-service datacollectorsvc

stop-service srmsvc

The service pack setup should then proceed normally.  It takes about a half-hour, and to be safe you should reboot the server after the update completes.

SBS shops are awaiting Windows 7—and several other components needed by SBS 2008 before it can accept Windows 7 clients.

First of these is Windows Server Update Services 3.0 Service Pack 2.  It will be out on Automatic Update this Tuesday the 25th.  I’ve been running this in the beta for six months with no incidents.  This update is needed for Windows 7 clients to see the WSUS server.

The second update is Small Business Server 2008 Update Rollup 3.  This updates the Connect Computer wizard so that Windows 7 is recognized as a valid client.  (It also changes the WMI filters in Group Policy.)

The latter update is not listed to be out on the 25th.  As many partners have SBS shops, a number of them will try to join their new Windows 7 box to their SBS network and fail.

The Official SBS Blog has a workaround for the Windows 7 beta that should still work with the release version.  It’s the manual version of the rollup patch minus the other fixes.

UPDATE:  Update Rollup 3 has been released after all:

For me, Windows 7 is five three days and counting…

In our migration to SBS 2008 at SATV, I got bitten by a network configuration that may happen to you as well, so take note.

SBS 2008, like past versions, has a wizard to configure your network connection;  in the old days it was called the CEICW;  now it is the Connect to the Internet Wizard (CTIW).  It’s a great tool.  It sets the defaults and lines things up just the way they should be on a small network.  I love it since it sets a known baseline for me.  Then, when I run the wizard, I’m free to do all the customizing I need.

That’s also the bad part:  It sets, or resets the defaults and sets the baseline.  If your baseline is different from the one SBS has out of the box, you need to know that before you run the wizard so that you can set things right.

The above screenshot is our DHCP configuration.  We have a number of static IP devices.  To give us space and consistent allocations, I reserved .1 through .50 and .241 through .254 in our subnet for static devices.

Normally, SBS will reserve .1 through .10 and optionally .254 if the router is at that address, as it is in our installation.  You’re on the hook for any other exclusions you need to make.  If you already have an exclusion for .1 to .20, for example, the CTIW will delete it, and make its new default exclusion for .1 to .10.

What if you have static IP’s in the .11 through .20 area?

That has some interesting effects on a Windows network.  Some devices may not respond.  DNS entries might mysteriously disappear (since Windows DNS is dynamically updated.)  You might wonder if your AD is corrupted.

And your VPN clients may not be able to access internal network resources.  Or their connections may work the next time but not the next time after that.

I’m fortunate not to find out what would have happened if I had a DC in that IP range!

Moral:  Pay close attention to your existing configuration.  Perform the migration.  As soon as it’s complete and you’ve run the CTIW, open the DHCP console, open your server, the IP4 tab, your scope and your address range, create an exclusion that covers your devices.

Since the CTIW already makes an exclusion for .1 through .10, I prefer to make another exclusion from .11 to whereever my highest reserved number should be, usually .20, .30 or .50.  Then I may make another exclusion at the high end of my subnet, say, .241 to .254.  (If your router is at .254, it may already have an exclusion.)

UPDATE:  I’ve revised this to cover the “This security ID may not be assigned as the owner of this object” error that is commonly seen when configuring folder redirection.

One challenge you may run into in an SBS 2008 migration is Folder Redirection.  The migration document makes it very clear that if your client PCs have been configured for folder redirection, you must make sure your users log in while both servers are connected to the network.

Sometimes, this may not be possible.  The migration period is for 21 days at most.  SATV, like other businesses, has a number of part-time or contract workers that may have access to our IT, like a webmaster, an accountant or an engineer.  They may come in just once a month, or for just a few weeks around the end of the fiscal year.

Any way it’s expressed, you probably do have users that aren’t logged on during the migration period, and won’t be aware of anything different until you tell them the next time they’re there.

Or they get errors like the one above.

There have been problems with folder redirection in Windows XP SP3, as explained in several threads on the TechNet forums,  "GP error with folder redirections (ids 107, 1504, 1509, 1085)", and "Folder Redirection on Windows XP SP3".

Distilling the advice, there are two things you should check.

1. Check the destination folder, which will be C:\Users\FolderRedirections, on a stock SBS 2008 server.  Check the permissions on the folders for each affected user and make sure the user has Full Control and ownership.
2. If the source server isn’t online anymore, copy the files manually from the old SBS machine, which should be under the share Computer Folders$.  You should have had a backup or have kept the old server around just in case.  Then open Group Policy Manager, select the Small Business Server Folder Redirection policy, make a backup of the GPO and edit it like this:

Uncheck “Move the contents of Documents to the new location”.  Also uncheck “Grant the user exclusive rights to Documents”.  Click OK.

On the machine of the affected user, log on as administrator.  Open a command prompt.  Refresh and force the group policy by typing:

gpupdate /force

Click OK to log off.  Log back on as admin one more time and then log off.

This explanation of XP SP3’s folder redirection comes from the thread on TechNet Forums:

There is a difference in the way Fdeploy.dll works in SP3. I am not too sure if this is a deliberate change to the functionality, however.

If you have a client that is running Windows XP SP3 and their redirect is updated to a new location, you need to ensure that the source is reachable as well as the destination. If the source is not reachable for fdeploy.dll to verify that its moving content, the function will fail and the redirection path will not be updated to the new destination. This differs from SP2 in that if the source isn’t reachable, the process ‘moves on’ and updates the folder redirect location to the new destination.

If you have replicated the content to the new area using Robocopy or similar utilities and intend to do so as a procedure, I recommend that you update your GPO and uncheck the "Move the contents of My Documents to the new location" option in the GPO (which is enabled by default on a new GPO). Once the GPO change has replicated to all sites, have the client logout and log back in and verify the results.

There is another option cited in the threads, replacing a DLL with one from XP SP2, which I do not recommend;  our machines have had XP SP3 for over a year and I really don’t want to make that kind of change.  System File Protection may not let the DLL be replaced anyway.

If it still doesn’t work, a few other links: 

• Turn on Folder Redirection logging on the client.
• Security Recommendations for Folder Redirection
• Event ID 101 and Event ID 1000 Messages May Be Displaye…

Good luck with your migration!

UPDATE:  Phil has some kind words for me!

Today, I thought of the old saying about being nibbled to death by ducks.

I had all sorts of little problems crop up today.  We have a PBX with a built-in router that I had hoped we would be able to use.  No such luck.  It locked up after hours for no good reason.  Not good.  We have a cable modem/router that will be familiar to many, since it is the one provided to business customers of this large cable operator;  we are now using its firewall capabilities.

It works better.  At least it hasn’t locked up.

But the VPN works.  Our PBX vendor will gladly license our appliance to be a VPN server, but with SBS it’s a waste;  we already have a license for as many people as we expect to use it (not many, mainly staff.)

(Our staff uses Macs at home, so Remote Web Workplace is not a great fit.  I’m aware of the disadvantages of a VPN, but the staff is most familiar with it so that’s what I’m going with.)

I also had all the usual problems one has with 6-year old workstations;  I don’t believe in “Winrot”, but these machines have been used hard.  (To their credit, Dell Optiplexes are very good machines.)

Two larger problems I had to deal with:

1) When the users are migrated from SBS 2003, the users themselves aren’t migrated as such—they’re already there in Active Directory.  But the user roles and permissions have to be assigned at the new server, otherwise they’ll not show up in the console.

Straightforward, mostly.

But I had gotten ahead of myself and copied User’s Shared Folders from the old SBS 2003 server to the 2008 machine.

The User Role Wizard doesn’t like that.  It will report it can’t set quota because the folders already exist.

OK, I move the folders to somewhere else and try again.

Nope.  This time it says it can’t set quota on folder “xxxxxx”.  Turns out folder “xxxxxx” doesn’t exist.  Fix:  Run the User Role wizard and note what folders it complains about.

In the folder assigned to users’ shared folders, make a folder or folders with the name the wizard lists.  If if complains about a folder named “davidmoisan”, then create that new folder named “davidmoisan”.

Rerun the User Role Wizard and tell it to add a role (not replace).  Do this until you get green checkboxes for every user going forward to the new server.

This is known, in sysadmin practice, as “Giving the bleeping computer a cookie until it gets what it wants and is happy with it!”

Of course, you’re using the SBS 2008 Group Migration Tool, right?

2)  I had problems getting the SBS 2008 Backup wizard to work;  I got an error “Unable to configure the backup schedule”.  The log had an ugly error about something being missing in Task Scheduler, so I assumed the worst.

There are two possible causes:  The old EMEA SBS blog suggests it’s due to having  a backup device that’s too small for the data being backed up.

Our backup disk is a pair of LaCie Big Disk Quadra, 2 TB each.  Not likely to be too small!

The other reason was uncovered by Nick Whittome:  His machine, and ours, is a Dell PowerEdge that was configured at the factory.  It has a 2 gig FAT partition named “OS”.  The Backup Wizard does not like this.  His fix, and mine, is to convert the 2 gig FAT partition to NTFS.

I’m wondering if it’ll break the diagnostic partition that Dell installs on its machines;  it usually puts that in a tiny EISA partition (that is on the server.)  I haven’t gone into the partition to see what’s actually in it.

Fix is to type in a command prompt:

convert <drive> /fs:ntfs 

where <drive> is the Dell OS partition (usually D:.)

I’ve heard nothing about this.  The SBS team was supposed to post about this but that was in February and it’s now June.  There had to have been hundreds of Dell boxes with SBS.  Why I never knew this…?

But the fix worked and I should get a good backup report in my morning email.

With that, nearly everything is done.  I only have to verify that all the workstations are applying folder redirection to the new server, redirect some file shares, and decommission the old server.

One last post in this series and I’ll be done.

Our migration to SBS 2008 at SATV is moving along.  Last week, Exchange public folders and mailboxes were moved to the new server.  I’ve been warned that this is the longest step of the migration.  Depending on replication settings, it could take hours or days to move the public folders over to Exchange 2007.

It took about 28 hours.   The public folders were migrated starting at 10 AM Thursday, and finally finished on Friday afternoon.  The migration instructions have you look at the output of a Exchange command shell to determine if the migration is complete.  Unfortunately, the instructions are vague and the command output is very verbose.  There should perhaps be a filter and a format-table.  Perhaps next time.

Instead, follow the instructions in You Had Me At EHLO…  How to Remove a Public Folder Database in Exchange Server 2007 RTM.  When you are on the source Exchange server, look at Public Folder Instances.  It will be populated with objects while replication takes place.

The screenshot above was taken after replication was complete and I was ready to go on to the next step of the migration.

If you are administering SBS 2008, you might have found Sharepoint broken recently.  There have been no less than three problems and fixes out there:  SBS 2008 Update Rollup 2 (KB 960911) Installation Failure, Companyweb Inaccessible After Sharepoint 3.0 Service Pack 2, Files in Companyweb are Opening Read-Only After SBS 2008 UR2, and Event 2436 for SharePoint Services 3 Search.

Sharepoint broke on my personal SBS box.  In my case the Update Rollup was applied properly so far as I knew, but Companyweb gave 404 errors when I tried opening it.  Not only that, I was getting multiple event 2436 errors in my application log every five minutes.

To fix the 404 errors, I manually reinstalled the Sharepoint 3.0 Service Pack 2 file.  I got the idea from a posting from someone who tried to install Sharepoint 3.0 SP1 and broke it.  My procedure is the same though the files involved are different:

1. Download the x64 version of the Sharepoint service pack.
2. On the server, run the service pack with the /extract option and extract the files to a directory of your choice: 

wssv3sp2-kb953338-x64-fullfile-en-us.exe /extract

3. In the directory you just made, there’ll be three MSP files:

   stswwsp2.msp                                                              
   wssmuisp2-en-us.msp                                                       
   wsswwsp2.msp                                                              
   

4. In an elevated PowerShell prompt, cd to the directory and invoke each of the files one at a time:

   invoke-item stswwsp2.msp                                                              
   invoke-item wssmuisp2-en-us.msp                                                       
   invoke-item wsswwsp2.msp

5. Note that it might take a long while to patch and the process may be stuck at times.  This is normal. 

Check Companyweb from the server.  If it works, continue applying the steps listed in each of the linked articles, if you haven’t done so already.  In my case I applied Event 2436 for SharePoint Services 3 Search and Files in Companyweb are Opening Read-Only After SBS 2008 UR2 and my Companyweb is now working.

I’m not sure what caused the 2436 errors but if you are troubleshooting Sharepoint on SBS 2008 you should follow all the steps in all four articles to determine your problem.  Don’t follow any diagnostics you may see for older versions of Sharepoint.