I recently visited a customer site who has student email hosted by Microsoft’s Live@EDU program.
While on-site, they reported the ILM/OLSync/PCNS (Password Sync) had stopped working. Users were able to change their LiveID password with Microsoft, but the one-way password-reset sync from Active Directory no longer worked.
For those unfamiliar with any of those above phrases, this article is not for you. But you can read about them here:
A quick overview of PCNS requirements and configuration:
- PCNS schema extension is in place
- PCNS is installed on all domain controllers in the user’s domain
- Inclusion Group(s) Defined
- Password Synchronization is configured within the ILM software
For my customer, this was all configured. And PCNS worked when I left them a few months back. What changed?
I stopped by the ILM server to have a look at things. PCNS is not so great about reporting errors by default, so I wanted to enable more logging. To do this you need to create a registry key called FeaturePwdSyncLogLevel.
FeaturePwdSyncLogLevel isn’t documented much on TechNet as far as I’m aware, but it is explained on this public forum here:
PCNS
For PCNS, four logging levels are controlled by adding the EventLogLevel (REG_DWORD) entry to the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PCNSSVC\Parameters
· 0 = Minimal logging
· 1 = Normal logging (default)
· 2 = High logging
· 3 = Verbose logging
MIIS 2003
For MIIS 2003, four logging levels are controlled by adding the FeaturePwdSyncLogLevel (REG_DWORD) entry to the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\miiserver\Logging
· 0 = Minimal logging
· 1 = Normal logging (default)
· 2 = High logging
· 3 = Verbose logging
So after configuring my ILM 2007 FP1 server (despite the above quote saying “MIIS 2003”) with this value (and restarting) I once again attempted a password reset.
Now I am able to see the following error in the application log:
| Log Name: Application Source: MIIServer Event ID: 6905 Task Category: (8) Level: Information Keywords: Classic Computer: gcilm.domain.com Description: A password notification was received but could not be processed because the corresponding object is a disconnector. Additional information: Reference ID: {6BC69D88-9392-4FEE-B050-940537F4063F} Source Object GUID: {75562CF8-BF46-4CE2-94FA-89B3EABA60D8} Source DN: CN=10187,OU=Y2013,OU=Students,OU=User Accounts & Groups,DC=domain,DC=com Source MA Name: OnPremise |
What we see in this error is that ILM cannot sync the password to the Live@EDU mailbox because ILM thinks the account is a “disconnector”.
Essentially this means ILM doesn’t have a matching object for this user. This does not mean the mailboxes in Live@EDU are gone or damaged – just that ILM doesn’t have a relationship between them and an on-premise AD account anymore.
For more about disconnectors, see the article: Don’t call me disconnector !!!
So how do we fix this?
Objects become disconnected in OLSync when the rules defined by the OLSync code are violated. So let’s look at the rules. Our best public source for these rules is in this article, titled: How Outlook Live Directory Sync Works. Within you’ll find a section called OLSync filtering logic:
When OLSync runs, ILM filters out objects in the following order. After an object is filtered out, ILM won’t evaluate it again, nor will the object be copied to the ILM metaverse for synchronization.
1. Recipient objects that don’t have required attributes ILM reads the following recipient objects. If any of the required attributes are empty (null), the recipient object is filtered out.
Recipient object type Required attributes Mailbox-enabled user mail, legacyExchangeDN, proxyAddresses Mail-enabled user mail, targetAddress User (AD DS or Active Directory only; no Microsoft Exchange installed) Mail-enabled contact mail, targetAddress Distribution group, dynamic distribution group, or security group mail, proxyAddresses, mailNickName 2. Recipient objects where the adminCount attribute is set to 1 The adminCount attribute is used to identify users in protected administrator groups, such as the Domain Admins and Administrators. If the adminCount attribute is set to 1 on any recipient object, it is filtered out.
3. Mailbox-enabled user objects that are specified as mailbox plans, discovery mailboxes, or arbitration mailboxes The msExchRecipientTypeDetails attribute is used to identify mailboxes that are specified as mailbox plans, discovery mailboxes, or arbitration mailboxes. These mailbox-enabled users are filtered out.
4. The mail attribute on an AD DS or Active Directory-only user that doesn’t match the provisioning domain In an on-premises environment where Microsoft Exchange hasn’t been installed, OLSync filters out all user objects where the mail attribute doesn’t contain an SMTP address that matches the provisioning domain.
5. The attribute used to generate the Windows Live ID doesn’t match any of the accepted domains The final pass filters out recipient objects that are configured for auto-provisioning but don’t have an accepted domain match in the attribute that is used to generate the Windows Live ID.
The attribute used to generate the Windows Live ID must contain a domain name that matches one of the accepted domains that you have configured in Outlook Live. As described in step 4, by default, OLSync looks to the user principal name (UPN) for a match unless you have set the MVWindowsLiveIdAttributeName parameter to use a different attribute. In this case, OLSync matches the SMTP address that is stored in the attribute that you have specified in the MVWindowsLiveIdAttributeName parameter. In any case, if OLSync can’t find a match to an accepted domain, the recipient object is filtered out.
After ruling out many of the above conditions, I thought more on item 2. I checked and realized the adminCount attribute on ALL of my users now is populated with a “1”!
This attribute is automatically populated by the PDC emulator for all users in “restricted” groups. Sadly, if you remove users from these groups, the same process doesn’t bother to clear that attribute! You have to do it manually.
More on this topic here: AdminSDHolder – or where did my permissions go?
When clearing this attribute make sure you set it to null. ADSIedit has a “clear” button to nullify an attribute:
Of course I have a few remaining questions:
- Will this “1” return?
- How did this occur?
- How do I script this fix?
- The answer to #1 is yes – if you don’t fix the root cause. Every hour the protected groups are evaluated and 1’s re-stamped if appropriate. So we need to fix the
cause
- before we fix the symptom.
- How did this occur?
- After looking at the group memberships of each user I verified users were NOT in a protected group; or at least until I checked
nested
- group memberships – Bingo!
- Someone had added the Domain Users group to Print Operators; a protected group! Why was this done – who knows. But I removed this membership.
- Now time for cleanup.
Using the free Quest PowerShell tools, I wrote the following script to identify the scope of the problem:
| Get-QADUser –SizeLimit 0 -IncludeAllProperties -SearchRoot “domain.com/User Accounts & Groups/Students” | where {$_.admincount -eq “1”} |
And followed up with this one to clear the attribute:
| Get-QADUser –SizeLimit 0 -IncludeAllProperties -SearchRoot “domain.com/User Accounts & Groups/Students” | where {$_.admincount -eq “1”} | Set-QADUser -ObjectAttributes @{adminCount= $null} |
Note, in the scripts I am only looking in a specific OU. You’ll need to adjust this for your environment. In fact, you can remove the SearchRoot switch all together if you want, as the adminCount will be re-populated for users in protected groups in the next hour anyway.
After running this script you can re-run the “startsync.ps1 –firstrun” to repair the relationships between the objects.
That’s it – bye for now!
Mike Crowley's Whiteboard 