Cleaning up DNS after DC demotion

For many of our clients, this is not a big deal…. however recently i was involved in an AD upgrade for an environment with 100’s of sites… and this environment being…. not well kept… wasn’t surprising that many “old” DC entries did not clean up nicely.

With that in mind, it was time to pull out my terrible powershell skills (and ask one of my guys for help when i got stuck)

This client has a couple of forward of lookup zones, but 100’s of reverse lookup zones…. so in order to ensure the name server was gone from all of these zones i used

Get-DnsServerZone -ComputerName <Name of DNS Server> | where {$_.IsReverseLookupZone -eq “True”} | ForEach-Object {Try {Remove-DnsServerResourceRecord -ZoneName $_.ZoneName -RRType “NS” -RecordData “<name of the old server i wasnt to remove witha . at the end” -Name “@” -force} catch {“$_”}}

For cleaning out the sites i then used

Get-DnsServerResourceRecord -ComputerName <Name of DNS Server> -RRType “SRV” -ZoneName <name of zone> | where {$_.RecordData.Domainname -like ‘*servername*’} | Remove-DnsServerResourceRecord -ZoneName <name of zone> -force

if you want to check (without removing) – or simply verify… run

Get-DnsServerResourceRecord -ComputerName <Name of DNS Server> -RRType “SRV” -ZoneName <name of zone> | where {$_.RecordData.Domainname -like ‘*servername*’}

 

Powershell – Test Network Connection

I’m a little embarrassed that i have only just found out about this.

I, like many old timers, have been using telnet to verify network connectivity over certain ports for many years, with commands such as “telnet www.adexis.com.au 443” in order to verify that specific ports are accessible from the machine im working on. This is very handy when an issue arises with certain machines, especially those in more secured parts of the network, are not working for certain things. The network team always says its not them…. and a quick telnet can help with proving or disproving that.

Today, while compiling some information for a MS support case, i needed to demonstrate that the ports were open…. however, one of the issues with telnet is that you either get an error (port is not accessible) or a blank screen (indicating that it is accessible) which isn’t great for relaying information to a 3rd party….

Enter the powershell command “TNC” or test-networkconnection

tnc www.adexis.com.au -port 443

ComputerName : www.adexis.com.au
RemoteAddress : 10.x.x.x
RemotePort : 443
InterfaceAlias : Ethernet
SourceAddress : 10.x.x.x
TcpTestSucceeded : True

 

In order to get a little bit more information, you can add “-informationlevel Detailed”

tnc www.adexis.com.au -port 443 -InformationLevel detailed

ComputerName : www.adexis.com.au
RemoteAddress : 10.x.x.x
RemotePort : 443
NameResolutionResults : 10.x.x.x
MatchingIPsecRules :
NetworkIsolationContext : Private Network
InterfaceAlias : Ethernet
SourceAddress : 10.x.x.x
NetRoute (NextHop) : 0.0.0.0
TcpTestSucceeded : True

 

much more friendly when having to relay the information onto a 3rd party – and something ill be using in place of telnet from here on in.

Office 365 installing bing as default search engine from Feb 2020

This is one of the entries that kinda needs a “so bad its kinda funny” category.

Microsoft, due to their unique understanding of customer needs, will be setting the default search engine in chrome to bing when you update to Office 365 version 2002.

https://techcommunity.microsoft.com/t5/office-365-blog/introducing-and-managing-microsoft-search-in-bing-through-office/ba-p/1110974

https://docs.microsoft.com/en-us/deployoffice/microsoft-search-bing

https://docs.microsoft.com/en-us/deployoffice/microsoft-search-bing#how-to-exclude-the-extension-for-microsoft-search-in-bing-from-being-installed

https://en.wikipedia.org/wiki/Browser_hijacking#Microsoft_Office_365_ProPlus

The best bit of the 1st article has to be the comments section, with universal love and praise of Microsoft for making this wise decision.

fortunately for the users in enterprise environments, it just a simple on/off toggle – the fact that many users in enterprise environments struggle with absolutely any dialogue box doesn’t seem to phase the decision makers.

 

Anyway, if i can delve back into reality for a second…. for enterprise admins out there that don’t want to generate a bucketload of support calls due to this bafflingly bad decision, you have a couple of options

  • Grab the Office admx templates from https://www.microsoft.com/en-us/download/details.aspx?id=49030 (must be version 4996.1000 or later)
  • Set Computer Configuration\Policies\Administrative Templates\Microsoft Office 2016 (Machine)\Updates\ Don’t install extension for Microsoft Search in Bing that makes Bing the default the search engine = Enabled
  • Utilise the config.xml at deployment time, with the line
    • <ExcludeApp ID=“Bing” />
  • If the extension has already been installed, utilise “C:\Program Files (x86)\Microsoft\DefaultPackMSI\MainBootStrap.exe” uninstallAll 

 

As at time of writing, i have not tested any of these – but when the update is rolled out – scheduled for middle of February 2020 – i will update this article with any additional info at that time.

 

Seems to be in the same vein as the  “SCCM will effectively stop working if it detects 3rd party MDM on a device“…. an equally bad decision, but one that effects far fewer people than this moronic BS.

Deploying the new chromium based edge via SCCM

New Edge, Credge, Edgium or Credgium – no matter what you call it, the Edge browser based on chromium was released on Jan 15th 2020.

To deploy the new versions using SCCM 1910 (or above)

  • Add “Microsoft Edge” a product that is sync’ed in your software update point component properties
    • Somewhat confusingly, the product is under the sub-category of “Windows”

  • Then navigate to Software Library > Microsoft Edge Management | All Microsoft edge update

 

  • Right click and select “:synchronise software updates”
  • You can follow the progress of this process by checking wsyncmgr.log – the same log as you use to follow all software update sync’s
  • At this stage – you can wait for anywhere from 5 minutes (for an environment that has sync’ed relatively recently) to hours (if this is your first sync)
  • Once completed, you can refresh by hitting F5 – and you should see some content update for Edge in the right-hand pane
  • Now right-click on the top node “Microsoft Edge Management” (yes, this is not intuitive) and select “Create Microsoft Edge application”
    • Select and name and a source directory to be used
    • Select a channel, or specific version that you wish to deploy. If you’re not sure here – use “stable” and “latest”
    • Deploy to a collection, select your DP’s etc – all pretty standard stuff here
  • You’ll notice that an application is created for you under the “applications” node – this is different to other software updates which are created as Software Update groups (SUG’s)
  • Personally, i’d prefer to see it all managed under the one node – however, its still a good feature overall – as simplifies deployment of edge greatly – but still allows the admin to go and edit properties of the deployment if… for the invetiable situation of where the Microsoft pre-defined install doesn’t meet your orgs needs

 

Once completed, as per any deployment – monitor your deployment via the “deployments” area in the monitoring tab.

 

*Update 28/02/2020 *

When installing, you may get the following error (or similar)

App install failed.
Install application action failed: ‘APP.Edge.Latest’. Error Code0x80004005
Sending StatusMessage
Setting the authenticator.
CLibSMSMessageWinHttpTransport::Send: WinHttpOpenRequest – URL: SCCM01.company.com.au:80 CCM_POST /ccm_system/request
Not in SSL.
Request was successful.
hrInstallation, HRESULT=80004005 (installapplication.cpp,989)
pInstall->InstallApplications(saAppNames, sContinueOnError), HRESULT=80004005 (main.cpp,284)
Install application action cannot continue. ContinueOnErrorFlag is set to false.
Install Static Applications failed, hr=0x80004005

This is due to the powershell policy…. to get around this, modify the command line from this

powershell -File “.\Install-Edge.ps1” -MSIName “MicrosoftEdgeEnterpriseX64.msi” -ChannelID “{56eb18f8-b008-4cbd-b6d2-8c97fe7e9062}”

to this

powershell -executionpolicy bypass -File “.\Install-Edge.ps1” -MSIName “MicrosoftEdgeEnterpriseX64.msi” -ChannelID “{56eb18f8-b008-4cbd-b6d2-8c97fe7e9062}”

SCEP updates fail on Windows 7 and 2008R2 clients where they are not patched

Ok, so, before the flood of hate comes my way

  1. Yes, its 2020, you should not be running Server 2008 R2 or Windows 7…… but the (unfortunate) reality is that many places still are
  2. Even if you are running them, they should be fully patched! “Patches break applications”…. mostly bullshit…. it does happen… but happens much much much less than what some admins claim.

Anyway, for those of you out there that have Windows 7 or 2008R2 SCCM clients that are, for whatever reason, unpatched, you may have also noticed that SCEP updates stopped working on them around October 21st 2019.

That’s because of this – https://www.microsoft.com/en-us/wdsi/defenderupdates

specifically the lines

Note: Starting on Monday October 21, 2019, the Security intelligence update packages will be SHA2 signed.
Please make sure you have the necessary update installed to support SHA2 signing, see 2019 SHA-2 Code Signing Support requirement for Windows and WSUS.

IF you try and install SCEP updates you will get an entry in the system event log similar to this:

Microsoft Antimalware has encountered an error trying to update signatures.
New Signature Version: 1.307.1945.0
Previous Signature Version:
Update Source: User
Update Stage: Install
Source Path:
Signature Type: AntiSpyware
Update Type: Full
User: Domain\User
Current Engine Version: 1.1.16600.7
Previous Engine Version:
Error code: 0x800b0109
Error description: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.

 

As this link – https://support.microsoft.com/en-us/help/4472027/2019-sha-2-code-signing-support-requirement-for-windows-and-wsus  specifies, you must install

  1. KB4474419
  2. KB4490628

once these updates are installed (and reboot of course), SCEP updates will then install.

Finding the right SQL view for your SCCM report

One of the banes of many SCCM admins existence is reporting. Some SCCM people are SQL guns, others, like me, know what they need to to get by, but a strong SQL understanding – that just isn’t me (and i know i’m not alone!).

One of the key things with SCCM reports, and something that some consultancies we deal with ignore is the requirement to use views for your reports, not directly talk to the tables.

The official SCCM doco dances around this and doesn’t explicitly state it (even though it does reference views constantly, it does not specifically state that using tables is unsupported – and it should)

https://docs.microsoft.com/en-us/sccm/core/servers/manage/creating-custom-report-models-in-sql-server-reporting-services

https://docs.microsoft.com/en-us/sccm/develop/core/understand/sqlviews/sql-server-views-configuration-manager

 

These posts are far better at laying it out as it is

https://www.enhansoft.com/what-are-the-supported-sql-server-views-to-use-with-sccm-reporting/

https://www.enhansoft.com/why-is-it-important-to-use-supported-sql-server-views-with-sccm-reporting/

 

The TL;DR version of the ehansoft articles is:

  • The SCCM reporting services account is specified during setup and is granted access to all the views – but not the tables etc
  • The 2nd enhansoft article lays out why views are utilised instead of tables – and the associated benefits
  • You can grant datareader access to accounts over the SCCM database to get around this – but its officially (MS) unsupported, and can cause a range of issues (again, laid out in the 2nd article)
    • There are people out on the web that recommend just allocating permissions – even some MVPs – which doesn’t help
  • My recommendation is to remain in “supported” territory and update the report to use views rather than tables directly (knowing full well that we have all referenced tables directly at some point in our careers!)

 

A couple of things i have found handy with this are:

 

When you know the table you wish to reference, but aren’t sure which views you can use (and you want to do the right thing and use a view)

SELECT * FROM INFORMATION_SCHEMA.VIEWS WHERE VIEW_DEFINITION like ‘%tablename%’ OR VIEW_DEFINITION like ‘%tablename%’ OR VIEW_DEFINITION like ‘%tablename%’

 

When you can find what your looking for in the database

https://www.apexsql.com/sql-tools-search.aspx

 

 

XBOX game pass for PC

I recently picked up XBox game pass for PC. It was (is if your reading this soon after posting!) $1 for the first month and $5/month after that currently…. although i fully expect that the price will substantially increase if it manages to take off.

I subscribed to Xbox game pass purely to play “the outer worlds” without having to wait for 6 months for it to turn up on steam. While it is available via the epic games store – full price simply doesn’t compare to $1 (even though the outer worlds is worth every cent of its full price tag)

So – keep in mind that Xbox game pass is in Beta (which is watered down somewhat by the fact that “beta” is increasingly used as a long term excuse for a shit product)

  • The library of games is OK – but not great. For $1 or the reverting price of $5 month it is pretty good. I played “dead cells” for example because it was available via the pass – and discovered how absolutely fucking brilliant that game is. Overall though, there would maybe only be 5-6 games that im truly interested in on the current list… clearly very much based on personal preferences.
  • The speed and responsiveness of the client is just terrible. Clicking a game results in a 10-15 second wait for the main information screen to show, downloads of games are excruciatingly slow – be ready to leave your PC on overnight in order to download a couple of your games. Once your in game, all is good….
  • The app regularly “black screens” (which appears to be a common issue). This is sometimes resolved by restarting the client and sometimes by restarting the PC.
  • The app regularly doesn’t automatically log in
  • The app, if shutdown 1/2 way through a download will commonly “lose” all knowledge about that app, and you’ll need to go back into the store and re-select “install”. So don’t bother queuing up 4 or 5 installs unless you plan on leaving your PC on overnight.
  • Because it downloads everything as “WindowsApps”, most people will have to change their default windows app install location to their larger drive. This ofcourse sets the default for all windows apps – not just the games – this is just poor design. The config/save files etc can also be difficult to find due to the whacky folder structure – but to be fair, steam suffers from this a little as well – as there are no industry standards for this.
  • And perhaps the biggest downside, the app has been shoe-horned in as a way to get people onto the latest versions of Windows 10. While this isn’t an issue for me, i imagine it will be for some people – but more to the point, its that the thinking around the app/storefront seems to be “let leverage this to force Win 10 upgrades and windows apps usage” as opposed to “lets make a genuinely good game storefront/platform to challenge steams dominance and encourage crossplay between Xbox and PC”

 

To sum up – ill be keeping my Xbox game pass (for PC) for at least a few months – primarily because it represents good value for money right now. Given where Microsoft seem to have their focus, i wouldn’t be surprised if this goes the same way as the ultra-successful “games for Windows live” – but i would also be happy to be wrong!

Exchange migration and AdminSDProp

I recently did a piece of work for a client – moving from Exchange 2010 to 2016. Nothing too exciting…. but they did have an interesting issue.

Once migrating some of test mailboxes, inheritance in AD had to be enabled for a few admin accounts before they could connect via outlook and activesync – to be expected (yes yes, i know admin accounts shouldn’t have mailboxes, but we all know that some clients still do this – and thats not the focus of this post)

What was interesting, was that on further investigation – every account has AdminCount set to “1” and had inheritance disabled – not something to handle manually..

On further investigation, it was found that via some group nesting, all users were members of print operators.

Groups with AdminCount=1 can be located utilising the powershell

Get-ADGroup -LDAPFilter “(admincount=1)”

The client did not want to immediately reverse this due to potential client impacts – and while i disagreed – excluding a group from AdminSDHolder was not something i had looked into before – so i was interested.

A short amount of googling later – and reading a long list of articles, we decided to exclude “print operators” from AdminSDHolder. Two of the better articles (for reference) around this were:

https://social.technet.microsoft.com/wiki/contents/articles/22331.adminsdholder-protected-groups-and-security-descriptor-propagator.aspx

https://social.technet.microsoft.com/Forums/windows/en-US/ddd8d964-6c8b-42b0-b170-2cacaa283d1c/adminsdholder-remove-groups-server-operators-print-operators-backup-operators?forum=winserverDS

The condensed version of the overall solution is:

  • In order to exclude a group from AdminSDHolder, you can utilise ADSIEdit to modify  the property dsHeuristic under “CN=Directory Services,CN=Windows NT,CN=Services,CN=Configuration,DC=YourDomain,DC=com”
  • The value can be calculated depending what groups you wish to exclude, the 2nd linked technet social post above has a really nice explanation
  • in my case, i needed to it to “0000000001000004” (without the quotes)
  • Once this is done, clear the AdminCount property from the appropriate group (in my case, this was “print operators” + another group within the long-line of nesting this client had)
  • Re-run the powershell – “Get-ADGroup -LDAPFilter “(admincount=1)” to verify the groups no longer show up
  • Once this is done, we need to remove the “adminCount” from each of the affected user accounts and enable inheritance – to do that, you can run the below script

 

$users = Get-ADUser -ldapfilter “(objectclass=user)” -searchbase “<DN of path you wish to use for your search base>”
#$users = Get-ADUser -Identity <username> ‘ Use this for testing on a single user first

#Get domain values
$domain = Get-ADDomain
$domainPdc = $domain.PDCEmulator
$domainDn = $domain.DistinguishedName

#HashTable to be used for the reset
$replaceAttributeHashTable = New-Object HashTable
$replaceAttributeHashTable.Add(“AdminCount”,0)

$isProtected = $false ## allows inheritance
$preserveInheritance = $true ## preserve inheritance rules

ForEach($user in $users)
{
# Binding the users to DS
$ou = [ADSI](“LDAP://” + $user)
$sec = $ou.psbase.objectSecurity

Set-ADUser -identity $user -clear adminCount

if ($sec.get_AreAccessRulesProtected())
{
#Changes AdminCount back to &lt;not set&gt;
Get-ADuser $user.DistinguishedName -Properties “admincount” | Set-ADUser -Remove $replaceAttributeHashTable -Server $domainPdc
#Change security and commit
$sec.SetAccessRuleProtection($isProtected, $preserveInheritance)
$ou.psbase.commitchanges()
}
}

 

References:

https://social.technet.microsoft.com/wiki/contents/articles/22331.adminsdholder-protected-groups-and-security-descriptor-propagator.aspx

https://sdbrett.com/BrettsITBlog/2016/12/discover-clear-admincount-powershell/

https://docs.microsoft.com/en-us/previous-versions/technet-magazine/ee361593(v=msdn.10)?redirectedfrom=MSDN

https://blogs.msdn.microsoft.com/muaddib/2013/12/30/how-to-modify-security-inheritance-on-active-directory-objects-using-powershell/

https://blogs.technet.microsoft.com/chadcox/2018/01/08/adposh-find-and-fix-adminsdholder-orphans-admincount/

http://www.selfadsi.org/extended-ad/ad-permissions-adminsdholder.htm

https://social.technet.microsoft.com/Forums/windows/en-US/ddd8d964-6c8b-42b0-b170-2cacaa283d1c/adminsdholder-remove-groups-server-operators-print-operators-backup-operators?forum=winserverDS

 

VMWare guest server CPU and memory issues

Got a call from a client who was having issues with the SQL instance on their SCCM server – and investigation showed that the SQL service was crashing due to various memory errors (event log and SQL logs) – but the descriptions weren’t overly helpful.

The SQL exception.log shows errors such as

09/12/19 12:23:58 spid 125 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION writing address 000001E1F29E3390 at 0x000001E1F29E3390

 

After a bit of investigation, i noticed that the “system” task in task manager was constantly utilising between 20-40% CPU. The “system” task has no associated command line in task manager, so tracking it down required the use of the ever-helpful sysinternal tools – in this case, process explorer.

Once opening process explorer, you can go to the properties of the “system” process and view all its threads – and most importantly, sort by CPU usage.

In this case, i could see that Vmmemctl.sys was using the vast majority of the CPU time within this process.

A quick google lead me to this https://kb.vmware.com/s/article/2138677

While i wasn’t getting blue screens, i was definitely getting memory errors – so this lined up.

Checking the installed programs, i could then see that VMWare tools 10.2.5 was installed, but so was 9.1.

Removed VMWare tools 9.1 from the server and the CPU use immediately dropped – and the memory issues, at least so far, are not longer occurring.

Surprisingly, this didn’t seem to require a reboot after the VMWare tools 9.1 uninstall.

I guess the moral of this story (post) is – keeping your VMWare tools version up to date is wise….. but don’t forget to uninstall old versions as well.

SCCM 1906 release

SCCM 1906 released! – https://docs.microsoft.com/en-us/sccm/core/plan-design/changes/whats-new-in-version-1906

If you want it right now – you can opt-in via the fast ring script – https://go.microsoft.com/fwlink/?linkid=2099733

Another belting release from the SCCM team – while some releases have different focuses over time – generally releases have tended to have something which makes day-to-day admin life a bit easier for someone…. and sometimes, big things, such as passive site servers, that just structurally make the product substantially better.

Anyway – out of this release comes a couple of items that i think are particularly of note to me (other may be interested in different parts – depending on your setup):

  • Site maintenance UI is friendlier
  • Use your distribution point as an in-network cache server for Delivery Optimization
  • Support for Windows Virtual Desktop
  • OneTrace – a preview of a tool which could be dubbed the next-gen of CMTrace…
  • Improvements to co-management auto-enrollment
  • Retry the install of pre-approved applications
  • Task sequence debugger – not there yet – but great that its being thought about – can see this being exceedingly handy as this matures
  • The Disable BitLocker task sequence step has a new restart counter
  • Additional options for WSUS maintenance – these are brilliant. Hopefully this release will also fix bug id 4808740 – i haven’t been able to test this as yet.
  • New Windows 10, version 1903 and later product category – this enables admins everywhere to further reduce the number of updates stored by WSUS – which leads to improvements with server and client performance
  • Role-based access for folders – finally!
  • Administration service support for security nodes – this is potentially a big change – keen to test this out in a bigger environment
  • Collections tab in devices node – gradually chipping away at right click tools functionality
  • Multiselect and delete packages – finally!