Category: Active Directory

Win RM fails on DC with event ID 142

Posted on by Hayes Jupe

For a while i have had a niggling issue where on a DC that is used by a number of in-house coded applications, WinRM would fail intermittently with the following:

Log : Microsoft-Windows-WinRM/Operational

EventID : 142

Event Message: WSMan operation Enumeration failed, error code 2150859046

There isn’t much to go on for this error when googling – and MS support – well… no point in trying that.

After verifying permissions and configuration, checking server resources etc… i was at a point where i didnt know how to “fix” it or even have any leads.

I initially put in a simply script to restart the service nightly… but every now and again, the stop of the service would hang…. so i’d have to kill the process.

I’ve ended up going down a path of:

  • Attaching a scheduled task to eventID 142
  • To get around powershell restrictions – have it launch a batch file containing

reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\PowerShell /v ExecutionPolicy /t REG_SZ /d unrestricted /f
powershell.exe -NoProfile -NoLogo -NonInteractive -ExecutionPolicy Unrestricted -File C:\data\TerminateAndRestartWinRM.ps1
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\PowerShell /v ExecutionPolicy /t REG_SZ /d AllSigned /f

TerminateAndRestartWinRM.ps1 contains

Start-Transcript C:\Data\WinRMTerminate.log

write-host “Getting the WinRM ProcessID”
$winRMService = Get-WmiObject -Class Win32_Service -Filter “Name=’WinRM'”
$processId = $winRMService.ProcessId

write-host “Terminating processID: $ProcessId”
Stop-Process -Id $processId -Force

write-host “Sleeping for 10 seconds to wait for process to terminate”
Start-Sleep -seconds 10

write-host “Starting WinRM”
# Start the WinRM service
Start-Service -Name WinRM

Stop-Transcript

 

Not the best thing ever – and i generally don’t like these types of “hacky” solutions…. but given that MS has moved from “mostly unsupported” to “completely unsupported” for everything that isn’t in Azure…. (which even then is mostly unsupported)… we don’t have much choice anymore.

Documenting AD ACL’s

Posted on by Hayes Jupe

A while ago i joined an organisation whose MS estate was in need of a significant amount of love, time and effort. Getting them off of 2012 R2 DC’s and onto 2022 DC’s, upgrading the forest/domain functional levels and getting replication times down were the obvious first jobs… but once they were done – there was so many other things to do – it was hard to know what to go with first. So… i made a start on all of it at once – knowing that it would probably take all year to get the AD into a semblance of decent condition.

The more i looked, the more i found… one thing that was/is particularly disturbing is that the DS ACL’s have been fucked with at the top level – and flowed down to all descendant objects for some admin accounts, service accounts etc…. stuff that clearly doesnt need, or has never needed that level of access….

Before changing anything, the goal is to document the permissions – as is a spaghetti of inherited, non-inherited and multi-nested groups applied at many different levels…. resulting in one severe head-fuck for anyone trying to do anything effective with permissions delegation.

First of all i tried

https://powershellisfun.com/2022/08/22/report-on-active-directory-container-permissions-using-powershell/

A decent solution – which works perfectly in my test environment, but in the prod environment with thousands of OU’s and a stupid level of excessive custom permissions, uses approx 4GB of memory before dying consistently. So while this is definitely a good script – it just doesn’t work in this prod environment…. and that’s because of how fucked the environment is, not because the script is bad.

I moved on and found

https://github.com/canix1/ADACLScanner

Which seems to be an exceedingly nice (powershell based) AD ACL solution…. an optional GUI, plenty of configuration options and great output options – a really good solution.

For me – i needed to tick “inherited permissions”… as it is important for me to demonstrate how incredibly stupid (in case you haven’t noticed, I’m still flabbergasted that someone would do this….) it is to allocate permissions at the top level of a domain – along with having complete documentation.

 

Well done & thanks to the author – Robin Granberg – for creating a genuinely awesome tool.

 

Now comes the hard bit – removing the permissions without breaking anything.

Otherwise healthy DC failing DFS-R, pointing to DC that no longer exists

Posted on by Hayes Jupe

Today i had a DC that was otherwise healthy, but reporting error 4612 and 5012 in the DFS Replication log, specifically:

The DFS Replication service failed to communicate with partner <decommissioned DC name> for replication group Domain System Volume. The partner did not recognize the connection or the replication group configuration.

My first port of call was to open ADSIEdit.msc and check

CN=Topology,CN=Domain System Volume,CN=DFSR-GlobalSettings,CN=System,DC=domain,DC=com,DC=au

but the dead server was not in there.

After some googling, i found a reference to

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DFSR\Parameters\SysVols\Seeding SysVols\DomainName\Parent Computer

and sure enough – this was referencing the now decommissioned DC – no idea how it happened. The new/old DC’s were both online together for over a month – it should not have been still seeding…. but obviously something went wrong.

Updated the name to a DC that existed, restarted the DFS-R service, waited about 15 seconds – all is now good.

Issues with mailbox migration, multiple identities where tenant wide retention policies in use

Posted on by Hayes Jupe

This is a somewhat niche issue – which is why im documententing it here.

Scenario

  • Exchange migration to exchange online which i came into approx 75% of the way through – so i don’t have any history on why some things have happened (and there is no useful doco)
  • Tenant wide retention policies are in place for all data (legislative requirement im led to believe for this client)
  • Identity sync via AADConnect
  • Some mailboxes cannot be moved. Powershell error message from new-moverequest indicates that the identity is not unique

Investigation

  • Start off by looking at the AAD Object sync with
    • Connect-MSOLService
    • (Get-MsolUser -UserPrincipalName identity@goes.here.com).errors.errordetail.objecterrors.errorrecord| fl ErrorCode
    • The output, will likely look something like this:
      • The value “<guid-value>” of property “ArchiveGuid” is used by another recipient object. Please specify a unique value.
  • Next up, we want to have a look at the potential duplicate objects
    • Connect-ExchangeOnline
    • Get-recipient -identity <identity> -includesoftdeletedrecipients
      • This will likely show you 2 (or more) mail user objects
    • To confirm the soft-deleted mailuser object you can use
      • Get-MailUser -resultsize unlimited -SoftDeletedMailUser -Identity <identity> | fl *guid*
      • Notice the ArchiveGUID returned is the same as the ArchiveGUID from the Get-MSOLuser error retrieved earlier in the investigation
    • If you then try and run the obvious next step
      • Get-MailUser -Identity <identity> -SoftDeletedMailUser | Remove-MailUser
      • You will get an error similar to
        • Remove-MailUser: The operation couldn’t be performed because object ‘Soft Delete d Objects\<identity>’ couldn’t be found on ‘SYBPR01A004DC01.AUSPR01A004 .PROD.OUTLOOK.COM’

Now, i know what your thinking “just exclude the mailbox from the retention policy” – and there within lies the issue…. there is no mailbox, only a mail user object, but with an archive mailbox that has been retained by the retention policy after the primary mailbox has been removed. It is then, to my knowledge, impossible to exclude that archive mailbox from retention – as its associated with a mailuser – not a mailbox.

As to how these identities got into this state…. absolutely no idea. I wasn’t around for the earlier parts of the project – but given some other things i’ve seen at the client, standardisation and documentation appear to be frowned upon (which is why i’m getting out ASAP)

 

Solution

The unfortunate solution is to log a call with O365 support.

I included all of the above information in my original support request and was still asked to run a “get-mailbox”… i included all the info again (and again, and again over a teams call showed them the exact same errors and data that i sent them) – and eventually they got the point (took approx 15 business days) and sent it to an internal team, who deleted the objects

Unfortunately i cant post the case number for reference (as it would potentially identify the client) – but maybe pointing MS support to this article might speed the process for others (?). Ideally, there would be a way around this, without engaging support – but there is not as far as I’m aware as of June 2023.

Cleaning up DNS after DC demotion

Posted on by Hayes Jupe

For many of our clients, this is not a big deal…. however recently i was involved in an AD upgrade for an environment with 100’s of sites… and this environment being…. not well kept… wasn’t surprising that many “old” DC entries did not clean up nicely.

With that in mind, it was time to pull out my terrible powershell skills (and ask one of my guys for help when i got stuck)

This client has a couple of forward of lookup zones, but 100’s of reverse lookup zones…. so in order to ensure the name server was gone from all of these zones i used

Get-DnsServerZone -ComputerName <Name of DNS Server> | where {$_.IsReverseLookupZone -eq “True”} | ForEach-Object {Try {Remove-DnsServerResourceRecord -ZoneName $_.ZoneName -RRType “NS” -RecordData “<name of the old server i wasnt to remove witha . at the end” -Name “@” -force} catch {“$_”}}

For cleaning out the sites i then used

Get-DnsServerResourceRecord -ComputerName <Name of DNS Server> -RRType “SRV” -ZoneName <name of zone> | where {$_.RecordData.Domainname -like ‘*servername*’} | Remove-DnsServerResourceRecord -ZoneName <name of zone> -force

if you want to check (without removing) – or simply verify… run

Get-DnsServerResourceRecord -ComputerName <Name of DNS Server> -RRType “SRV” -ZoneName <name of zone> | where {$_.RecordData.Domainname -like ‘*servername*’}

 

Exchange migration and AdminSDProp

Posted on by Hayes Jupe

I recently did a piece of work for a client – moving from Exchange 2010 to 2016. Nothing too exciting…. but they did have an interesting issue.

Once migrating some of test mailboxes, inheritance in AD had to be enabled for a few admin accounts before they could connect via outlook and activesync – to be expected (yes yes, i know admin accounts shouldn’t have mailboxes, but we all know that some clients still do this – and thats not the focus of this post)

What was interesting, was that on further investigation – every account has AdminCount set to “1” and had inheritance disabled – not something to handle manually..

On further investigation, it was found that via some group nesting, all users were members of print operators.

Groups with AdminCount=1 can be located utilising the powershell

Get-ADGroup -LDAPFilter “(admincount=1)”

The client did not want to immediately reverse this due to potential client impacts – and while i disagreed – excluding a group from AdminSDHolder was not something i had looked into before – so i was interested.

A short amount of googling later – and reading a long list of articles, we decided to exclude “print operators” from AdminSDHolder. Two of the better articles (for reference) around this were:

https://social.technet.microsoft.com/wiki/contents/articles/22331.adminsdholder-protected-groups-and-security-descriptor-propagator.aspx

https://social.technet.microsoft.com/Forums/windows/en-US/ddd8d964-6c8b-42b0-b170-2cacaa283d1c/adminsdholder-remove-groups-server-operators-print-operators-backup-operators?forum=winserverDS

The condensed version of the overall solution is:

  • In order to exclude a group from AdminSDHolder, you can utilise ADSIEdit to modify  the property dsHeuristic under “CN=Directory Services,CN=Windows NT,CN=Services,CN=Configuration,DC=YourDomain,DC=com”
  • The value can be calculated depending what groups you wish to exclude, the 2nd linked technet social post above has a really nice explanation
  • in my case, i needed to it to “0000000001000004” (without the quotes)
  • Once this is done, clear the AdminCount property from the appropriate group (in my case, this was “print operators” + another group within the long-line of nesting this client had)
  • Re-run the powershell – “Get-ADGroup -LDAPFilter “(admincount=1)” to verify the groups no longer show up
  • Once this is done, we need to remove the “adminCount” from each of the affected user accounts and enable inheritance – to do that, you can run the below script

 

$users = Get-ADUser -ldapfilter “(objectclass=user)” -searchbase “<DN of path you wish to use for your search base>”
#$users = Get-ADUser -Identity <username> ‘ Use this for testing on a single user first

#Get domain values
$domain = Get-ADDomain
$domainPdc = $domain.PDCEmulator
$domainDn = $domain.DistinguishedName

#HashTable to be used for the reset
$replaceAttributeHashTable = New-Object HashTable
$replaceAttributeHashTable.Add(“AdminCount”,0)

$isProtected = $false ## allows inheritance
$preserveInheritance = $true ## preserve inheritance rules

ForEach($user in $users)
{
# Binding the users to DS
$ou = [ADSI](“LDAP://” + $user)
$sec = $ou.psbase.objectSecurity

Set-ADUser -identity $user -clear adminCount

if ($sec.get_AreAccessRulesProtected())
{
#Changes AdminCount back to &lt;not set&gt;
Get-ADuser $user.DistinguishedName -Properties “admincount” | Set-ADUser -Remove $replaceAttributeHashTable -Server $domainPdc
#Change security and commit
$sec.SetAccessRuleProtection($isProtected, $preserveInheritance)
$ou.psbase.commitchanges()
}
}

 

References:

https://social.technet.microsoft.com/wiki/contents/articles/22331.adminsdholder-protected-groups-and-security-descriptor-propagator.aspx

https://sdbrett.com/BrettsITBlog/2016/12/discover-clear-admincount-powershell/

https://docs.microsoft.com/en-us/previous-versions/technet-magazine/ee361593(v=msdn.10)?redirectedfrom=MSDN

https://blogs.msdn.microsoft.com/muaddib/2013/12/30/how-to-modify-security-inheritance-on-active-directory-objects-using-powershell/

https://blogs.technet.microsoft.com/chadcox/2018/01/08/adposh-find-and-fix-adminsdholder-orphans-admincount/

http://www.selfadsi.org/extended-ad/ad-permissions-adminsdholder.htm

https://social.technet.microsoft.com/Forums/windows/en-US/ddd8d964-6c8b-42b0-b170-2cacaa283d1c/adminsdholder-remove-groups-server-operators-print-operators-backup-operators?forum=winserverDS

 

Active Directory 2019 and Exchange 2019 – what’s new

Posted on by Hayes Jupe

The short answer is – not much.

Exchange 2019 was released a few weeks back, but was effectively un-usable, as Exchange 2019 requires Windows Server 2019…. and Windows server 2019 got pulled from release (like Windows 10 1809) due to some issues.

Windows Server 2019 was re-released a few days ago, which allowed nerds everywhere (including me) to put Server 2019 and Exchange 2019 into a test environment.

The most striking thing that is immediately noticeable is that everything looks the same…. The install process, the GUI, the management, all looks the same as it did in 2016. To me, this is a good thing – while Microsoft of the past seemed to believe that moving functions between areas was good – some consistency is nice to have too.

 

Active Directory

First appearances indicate there is nothing new in AD 2019, the installation process and management is exactly the same as 2016.

While installing, there is not even an option to set the forest and domain functional level to “2019” – only 2016.

A quick look at the schema version indicates it has increased and quick google finds this article

https://blogs.technet.microsoft.com/389thoughts/2018/08/21/whats-new-in-active-directory-2019-nothing/

So, while there is something new in the schema, its an incredibly small update….. and there are no new features or functionality of any type to focus on.

 

Exchange 2019

Exchange 2019 is a bit the same as AD, everything appears to be the same as Exchange 2016, from the install process to the management interface.

A google comes up with this

https://practical365.com/exchange-server/should-you-upgrade-to-exchange-server-2019/

So there are some changes and feature updates – but these updates may not have an impact/matter to your organization.

 

I found these two releases interesting overall as

  • AD is the core of many enterprise networks
  • Exchange is a core business application

To see a new release of both of these products with very minimal improvements I think demonstrates where all Microsoft’s development effort is going (which, to be fair, we already knew)

 

Importing AD powershell module into Windows PE and then using encrypted creds

Posted on by Hayes Jupe

Powershell makes life much easier than vbscript…. however it does have its downsides…  signing policy can sometimes be a bit of pain and the modules you need have to be available…. which is an issue in particular for Windows PE.

Mick (good aussie name there) was nice enough to write a blog on how to import powershell into PE – without having to add it statically to the boot wim – http://mickitblog.blogspot.com.au/2016/04/import-active-directory-module-into.html

I was a little lazy here and copied both x86 and x64 required directories via robocopy rather than determining the version via powershell like Mick did.

The next step however is the more important one…. a task sequence doesn’t allow us to run a powershell command in PE with credentials, we need a secure way of running the command. In my case, I want to delete a computer object….

Step 1 – Generate a key file (perform on any full OS)

$KeyFile = “\\sccm\PSource$\OSD.DeleteComputer\DeleteComputer.key“

$Key = New-Object Byte[] 16

[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($Key)

$Key | out-file $KeyFile

 

Step 2 – Encrypt a password using the key

$PasswordFile = “\\sccm\PSource$\OSD.DeleteComputer\DeleteComputer.txt“

$KeyFile = “\\sccm\PSource$\OSD.DeleteComputer\\DeleteComputer.key“

$Key = Get-Content $KeyFile

$Password = “Your password here” | ConvertTo-SecureString -AsPlainText -Force

$Password | ConvertFrom-SecureString -key $Key | Out-File $PasswordFile

 

Step 3 – Create your script utilising the creds – (Below is the one I use to delete a computer object)

Import-module ActiveDirectory

#SCCM TS Object
$tsenv = New-Object -COMObject Microsoft.SMS.TSEnvironment

#SCCM Variables
$CompName = $tsenv.Value(“_SMSTSMachineName”)

# Get current path in order to get encrypted password
$MyDir = [System.IO.Path]::GetDirectoryName($myInvocation.MyCommand.Definition)
$User = “Domain\Account”
$PasswordFile = “$MyDir\DeleteComputer.txt”
$KeyFile = “$MyDir\DeleteComputer.key”
$key = Get-Content $KeyFile
$MyCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $User, (Get-Content $PasswordFile | ConvertTo-SecureString -Key $key)

# Remove the computer from AD
Remove-ADComputer -Identity $CompName -server <DC name required> -Credential $MyCredential -confirm:$false

 

Now before you say it…. yes, this is not very secure. It will stop a random snooper type person from seeing a plain text password…. but it will not stop someone who has 1/2 an idea about pressing F8 to get into the running TS (if you have it enabled) and then grabbing the key and txt and being able to use them…. so use (or don’t use) appropriately for your environment.

Making your software product too hard to obtain….

Posted on by Hayes Jupe

A client has recently merged and are looking at doing a merging into a single forest/domain structure – but up until that happens, they are keen to get the basics, immediately – such as file, calendar and address book sharing – all of which is fair enough.

It had been a while since I looked at GALsync-type solutions – so I did a bit of a search to find what was current.

One of the solutions I came across was Quest Quick Connect Express for Active Directory via http://www.open-a-socket.com/index.php/2010/12/23/gal-sync-with-quest-quick-connect-express-for-active-directory/

I’ve never been a huge fan of the quest products – they seem to get sold pretty hard by Dell consultants who don’t care if the product actually meets the needs of the client – and they are fucking expensive, however, when used for the right job – they actually do work pretty well.

In this case, I found the dell download link (https://support.software.dell.com/download-install-detail/5556386) and proceeded to try and obtain this free product. Two account sign ups later – I’m not entitled to the software…. so does that mean its now paid software when it previously free? Fair enough – might be worth mentioning that by having a licensing link on the page, or pricing etc… something to indicate it isn’t free anymore.

Is there a trial available ? Doesn’t look like it.

In short Dell – that fucking sucks. I don’t care that its not free anymore – I care that you made me jump through hoops to find out that it wasn’t free anymore – and also gave me no indication as to what the price is, where I can find out what the price is, what (if any) trial/demo’s are available etc. Absolutely zero useful information.