It’s been a year since my last post, Hac-Man Won the Retrochallenge! I’ve been very busy at SATV and with the Salem Commission on Disabilities. SATV is coming up on our franchise renewal year; the current contract runs out 11 months from now, in September 2014, and we started the planning process a year and a half ago.
My part of the process is to determine, for a 10 year contract cycle, what IT we need. Keep in mind that it’s practically impossible to determine technology trends more than a few years out. Rather than even try doing that, I just made estimations based on our current technology, our equipment refresh schedule (which varies by the technological area) and our unmet needs, pain points and so forth.
It was a very exhausting process that I can’t fully get into here, but let’s just say from September of last year until September this year, I and our executive director, had our heads down in this. I also had to refresh our network server sooner than we had anticipated as our backup Dell 1800 server was 8 years old and showing ECC memory errors. It was also the last 32-bit Windows server in the building long after Microsoft moved to 64-bit only server OS’s. That machine had a good life.
Our newer Dell 1900 was running good, but it was also running SBS 2008. As many readers know, there was a major disturbance in the SBS world last year. Microsoft discontinued SBS! The operating system I had been certified on twice (SBS 2008 and SBS 2011) was dead. SATV had been an SBS shop since the old BackOffice 4.0 days and I knew we would have to find a replacement head on.
I can’t get into any more detail about this either, but I like to quote an old phrase all the time, “The cobbler’s children go barefoot”. Between SATV’s franchise preparations, my own research into a life without Small Business Server, and numerous other personal circumstances, I had no energy for blogging or much of anything else.
I even missed the screaming about Windows 8. I had tried the developer preview, so I knew it was going to be a very different experience, but I had no time whatsoever to even try it out thereafterwards I finally got to use it after all when I convinced SATV to get me a staff laptop with Win 8 on it. (After using Win 8, oddly enough, I dislike Win 7 now.)
I also spent the past three months of summer and early fall migrating from SBS 2008, and Windows Server 2008, to a new server that is running Windows Server Essentials, Exchange and Sharepoint in a virtualized box hosted by our new Dell T320 server in the picture.
That was work.
The absolute low point was spending my birthday (July 24th and do-not-ask-please) with a very sick new Windows Hyper-V server that would not stay running (due to corrupted everything, as it turned out. Plus a Windows servicing stack that lied to me about what was and what was not installed, another story.) That afternoon I only remember staggering to my favorite takeout joint in Salem, going home in a daze, nomming down steak tips, and then falling into bed. The topper was me remoting into that same server four hours later because I could not sleep! (!!)
I am extremely stubborn and hyperfocused with computer problems. Now imagine 12 months of this. I’m surprised I’m even still here.
I’m just in time for Windows 8.1, though, and I did get new server hardware at home to write about. I’m catching up on my gaming, too, and I have no shortage of topics now.
Stay on the channel.
This has been nagging me for about a month. On my personal SBS 2008 box, since mid-October, I have been getting this error repeatedly. When logged on to the server console, a Windows Error Reporting dialog pops up every minute reporting that Taskeng.exe has stopped working.
Taskeng is the main process of Windows Task Manager, which has a console under Administrative Tools.
What was causing Task Scheduler to crash? I very quickly found it—Certificate Services Client was causing it, as you may have seen in the Event Viewer screenshot; it throws a error message right before Task Scheduler reports the crash. It has three tasks: SystemTask, UserTask and UserTask-Roam. On my machine, SystemTask was crashing and trying to restart. That accounted for the repeating WER dialogs.
But what is the Certificate Services Client? What do its three tasks do?
The Certificate Services Client (CSC, not to be confused with Client-Side Caching) is a component of Windows that allows users to use their site certificates, which are often used for VPN, email authentication and many other services on a corporate network. It’s been a part of Windows for awhile, and it’s on Windows Server 2008 and by extension SBS 2008.
A very important part of the CSC is autoenrollment: Autoenrollment is the process of keeping track of user certificates, expiring them and renewing them as needed. PAUTOENR.DLL is the DLL that manages this process, running with DIMSJOB.DLL and DIMSROAM.DLL under Task Scheduler.
After a lot of tracing with Process Monitor and a lot of research online, I have a conclusion: The CSC, and the Autoenrollment feature, each keep a cache of certificate information. It was corrupted. I suspect a certificate expired sometime in October or a bit before, and the Autoenrollment choked on it.
I must note at this point that the CSC is a client. On an SBS machine, it runs alongside the Certificate Services role, but it is totally separate so it does no good to troubleshoot Cert Services if it is otherwise working. Don’t touch Cert Services.
This is an involved process with several reboots so you will need patience.
- First, pull up Task Scheduler from Administrative Tools. Select “Task Scheduler Library” and open Microsoft\Windows\CertificateServicesClient. We are going to disable each of the three tasks before we delete the cache. On each task, SystemTask, UserTask and UserTask-Roam, right-click and select End. Repeat this for each task and select Disabled.
- Follow these steps to delete the certificate cache on the server: Go to the LOCAL SYSTEM profile at C:\Windows\System32\config\systemprofile\AppData\LocalLow\Microsoft\CryptnetUrlCache. Copy the CryptNetURLCache folder to a safe place. Two folders are there—Content and Metadata. Go to each folder and delete the files therein—don’t delete the Content and Metadata folders themselves.
- Run Regedit as an administrator and go to this key, HKEY_LOCAL_MACHINE\Software\Microsoft\SystemCertificates\AuthRoot\Certificates. Delete all the keys under this key. That takes care of the certificate cache.
- Now, delete the Autoenrollment cache. Go to Regedit and to HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\AutoEnrollment\AEDirectoryCache. Delete all the keys under that key.
- Go back to Task Scheduler. Open the CertificateServicesClient folder that we had opened before (Task Scheduler Library\Microsoft\Windows\CertificateServicesClient). Right-click on each task and select “Enable”.
- We aren’t done yet. The Certificate caches have been empty, but they need to be repopulated. To do this, we need to change the local security policy on the machine to turn on autoenrollment temporarily until the caches are repopulated. By default (on my SBS box at least), autoenrollment is not turned on. This is often done on Windows networks via the Default Domain policy but I am really uneasy about making a global change to fix a local problem so the local security policy is the best way.
- Open secpol.msc as an administrator to access the local security policy. Click on Public Key Policies. On the right pane, double-click Certificate Services Client—Auto-Enrollment Policies. In the dialog, click on the Configuration Model dropdown box and select Enabled. There are two checkboxes: Check both boxes, “Renew expired certificates, update pending certificates and remove revoked certificates” and “Update certificates that use certificate templates”. Click OK.
- From an elevated command prompt, enter the command gpupdate /force to make the change apply to the machine.
- Restart the server.
- Log back on and reopen the Task Scheduler and the CertificateServicesClient task. SystemTask and UserTask will show as running. Click on SystemTask, and click the History tab. The most recent events in History should show that the task started normally and is currently running.
- While still on the SystemTask window, right-click and select End. This task normally only runs when the system is restarted, and every eight hours thereafter.
- Next, we will undo our changes to the local security policy. Open secpol.msc as an administrator and select Public Key Policies. Click on Certificate Services Client—Auto-Enrollment Policies. On the Configuration Model dropdown box, select “Not Configured” and click OK.
- From an elevated command prompt, reenter gpupdate /force to undo the change.
- Restart the server again.
- Log back on once again and reopen the CertificateServicesClient task. SystemTask should be listed as Ready or Running. UserTask should be Running. UserTask-Roam should be Ready. Look at the event history for each task. If there are no errors since the last reboot, the problem is resolved. You may want to try running the SystemTask as a test, if it is not already running.
Microsoft has recently released a patch to address a well-publicized vulnerability in Windows where programs can load “bad” DLL’s from a WebDAV or SMB share. However, you need to create a new registry entry to modify Window’s behavior on DLL loading and close the hole.
The new registry entry is CWDIllegalinDllSearch, and there is more information, and a Fix It tool, on Microsoft’s Security Research and Defense blog. Susan Bradley rightly points out that their advice is not really suitable when there are numerous PC’s in the network, especially when they are all managed by Group Policy. She mentioned a solution involving Group Policy Preferences.
It’s not often I can keep up with Susan, but I have figured out how she’d do it! We’ll make a new Group Policy Preference with the required registry settings. This preference could be applied to the default domain, but I have chosen to limit it to client machines, so it will be created for the SBSComputers OU.
- Preconditions: All your PC’s must have Group Policy Preferences installed; this is standard for Vista and higher, but available as an update for XP SP3.
- They must also have KB2264107 installed first. Do it now if you haven’t already.
- Open the Group Policy Management Console, select your domain, and go through the OU tree until you reach SBSComputers.
- Right-click, select “Create a GPO and link it here”.
- Pick a name, “DLL Preloading Vulnerability Prevention”. Click OK.
- Find the GPO you just created and right-click it. Select Edit.
- In the GPO editor, select Computer Configuration/Preferences/Windows Settings/Registry.
- Right click the Registry item and select New/Registry Item.
- In the window shown below, to the right of the Key Path item box, click the […] button.
10. Select SYSTEM/CurrentControlSet/Control/Session Manager. Do not open any subkeys under Session Manager. Click Select.
11. Enter the new value name, CWDIllegalinDllSearch.
12. Enter the value type, REG_DWORD.
13. For value data, enter 2.
14. Click OK.
15. Close the GPO Editor, and the GPMC console.
If you wish, you can refresh the GPO settings at each client by doing a gpupdate /force.
The “default” setting of 2 for CWDIllegalinDllSearch blocks DLL loading from Webdav and SMB’s. There is a more restrictive setting that blocks DLL loading from USB devices. For that, you would select the Hexadecimal radio button in the dialog above and enter FFFFFFFF. Until I know how this affects the system I am sticking with the defaults. It occurs to me that if you have techs who troubleshoot with thumb-drive utilities, this may break them. Just so you know.
There are other ways to implement this in Group Policy; one method I have seen involves creating a new GP template with the registry settings. I think GP Preferences are the better way to go as, with a custom template, the registry settings are "tattooed" (this is the official GP nomenclature) onto the machine’s registry and remain there even if the policy is removed. I like to have a clean machine with transparent settings, even though this setting will be permanent through the life of Windows.
Thanks for the hints, Susan!
Exchange 2007 SP3 is now available. Unlike SP2, there is no wrapper needed to install it under SBS 2008.
However, you may see a few wrinkles in the update process. You will need to stop the Windows SBS Manager (datacollectorsvc) and, if you have File Server Resource Manager installed, srmsvc. In Powershell do:
stop-service datacollectorsvc stop-service srmsvc
The service pack setup should then proceed normally. It takes about a half-hour, and to be safe you should reboot the server after the update completes.
SBS shops are awaiting Windows 7—and several other components needed by SBS 2008 before it can accept Windows 7 clients.
First of these is Windows Server Update Services 3.0 Service Pack 2. It will be out on Automatic Update this Tuesday the 25th. I’ve been running this in the beta for six months with no incidents. This update is needed for Windows 7 clients to see the WSUS server.
The second update is Small Business Server 2008 Update Rollup 3. This updates the Connect Computer wizard so that Windows 7 is recognized as a valid client. (It also changes the WMI filters in Group Policy.)
The latter update is not listed to be out on the 25th. As many partners have SBS shops, a number of them will try to join their new Windows 7 box to their SBS network and fail.
UPDATE: Update Rollup 3 has been released after all:
For me, Windows 7 is
five three days and counting…
In our migration to SBS 2008 at SATV, I got bitten by a network configuration that may happen to you as well, so take note.
SBS 2008, like past versions, has a wizard to configure your network connection; in the old days it was called the CEICW; now it is the Connect to the Internet Wizard (CTIW). It’s a great tool. It sets the defaults and lines things up just the way they should be on a small network. I love it since it sets a known baseline for me. Then, when I run the wizard, I’m free to do all the customizing I need.
That’s also the bad part: It sets, or resets the defaults and sets the baseline. If your baseline is different from the one SBS has out of the box, you need to know that before you run the wizard so that you can set things right.
The above screenshot is our DHCP configuration. We have a number of static IP devices. To give us space and consistent allocations, I reserved .1 through .50 and .241 through .254 in our subnet for static devices.
Normally, SBS will reserve .1 through .10 and optionally .254 if the router is at that address, as it is in our installation. You’re on the hook for any other exclusions you need to make. If you already have an exclusion for .1 to .20, for example, the CTIW will delete it, and make its new default exclusion for .1 to .10.
What if you have static IP’s in the .11 through .20 area?
That has some interesting effects on a Windows network. Some devices may not respond. DNS entries might mysteriously disappear (since Windows DNS is dynamically updated.) You might wonder if your AD is corrupted.
And your VPN clients may not be able to access internal network resources. Or their connections may work the next time but not the next time after that.
I’m fortunate not to find out what would have happened if I had a DC in that IP range!
Moral: Pay close attention to your existing configuration. Perform the migration. As soon as it’s complete and you’ve run the CTIW, open the DHCP console, open your server, the IP4 tab, your scope and your address range, create an exclusion that covers your devices.
Since the CTIW already makes an exclusion for .1 through .10, I prefer to make another exclusion from .11 to whereever my highest reserved number should be, usually .20, .30 or .50. Then I may make another exclusion at the high end of my subnet, say, .241 to .254. (If your router is at .254, it may already have an exclusion.)