The challenges of click-once and other per-user install methods

Posted on by Hayes Jupe

A while back, one of the main vendors for the organisation im now working for had a bit of an indignant chuckle when i asked for their supposed enterprise-grade software to have a silent install…. “Just use the click-once version – its fine for everyone else”

Well – needless to say, i saw red, what an incredibly fucking stupid thing to say… and im finally getting around to writing blog article as to why thats the case.

So, with that said – lets have a look at a few of the challenges, and positives, of user-based installs – be it click-once or other.

 

Administrator rights

Any per-user install can generally be installed without requiring administrator rights or elevation.

Now, in some instances, this can be a good thing…. think about at the start of Covid when many people moved to working form home – and some (not all) IT departments did not have the capbility of deploying software to remote machines. The ability to get users to download and install teams (for example) would have reduced the load of those IT departments at the time.

The other side of this is, it also creates an opportunity for users to install software which is not on the companies approved list. Which creates potential for support hassle, security vulnerabilities and inconsistent business processes. Application whitelisting controls will address this – but again, not all enterprise IT departments are at that maturity level (and yes, i agree, they should be – but thats just not the reality sometimes)

 

Click Once installation performance and size

Click once has various optimisations to decrease the initial install time…. but the reality is, if an application is 600MB, its not going to perform well over a slow network connection…. and even with a LAN connection, it lacks the options you get when deploying an application with something designed for mass-deployment, including slow network links, like SCCM (.e.g peer caching, pre-staging etc)

For organisations that have a large number of users cycling through a group of machines (im thinking healthcare – but you may have your own example) – 100 people all installing a 600MB application into their user profile starts to become an issue.

 

Management

Enterprise management and deployment apps (im thinking SCCM and group policy, but insert your fave here) have plenty of built-in capabilities to install, inventory and report on applications. Want to find all the versions of Acrobat Reader that aren’t the current version? – sure, give me a few minutes – and then a couple more to get them all updated to the current version.

Per-user installs break this management model by storing the program files in the user profile… now obviously we can start inventoring those paths as well…. but we have gone from looking at WMI, C:\Program files and Program files (x86), to now looking at all the user profiles on a machine…. and as anyone will tell you that does that stuff for a living, per-user installs seem to have a fanatical hatred of using a standard directory structure and prefer putting in random GUID’s here and there along the way – making it more difficult (but generally not impossible) to optimise that process.

In addition, when click-once does break (and it does) – you are dealing with a different type of support – and potentially much more of it (due to the multiple installs per machine).

We now are potentially managing multiple copies per machine, in locations that generally dont match a pattern, rather than 1 per-machine in a standard location. This then has flow on effects for application whitelisting, support – and the big one, security.

 

Security

For the purposes of this section, we are going to assume that application whitelisting is in place and that the application in question is a part of your companies approved software list – as it becomes more of a process and governance dicussion if this isnt the case. (which iun turn becomes a security issue if these aren’t in place)

Lets say you have an application called “LostSouls” (named after the band i cannot stop listening to at the moment).

LostSouls deploys via ClickOnce….. and 1800 people have installed it across the org.

LostSouls v1.1 turns out to have a major security vulnerability in one of its supporting dll’s…. fair enough, that happens… and LostSouls v1.2 comes out, which addresses that vulnerability….

“Dont worry – we have fixed it” the vendor says… “it will update the next time they open the app” the vendor (correctly) says…. the technically clueless management and project manager nod in agreement – because… why should they care about reality now? Ignorance is bliss after all.

So out of those 1800 installs of 1.1…. for various reasons (staff leaving, people using different machines etc), only 1200 people start it up again on the same machine – so those 1200 installs get updated… but we now have 600 installs sitting in user profiles that are vulnerable. 600 installs that i (as a deployment admin) need to find and address – which is achievable – but ofcourse uses time and effort that would better be spent elsewhere… and more to the point, just making life much harder than it needs to be.

 

To sum up

There is nothing wrong with applications that present the option to be installed as a per-machine or a per-user install…. and while i might not be a fan of per-user installs – there clearly are specific use cases where that model fits.

User-based installs become an issue where:

  • User-based install is the only option
  • The install size is large (lets say anything over 500MB) and
    • Is intended to be used across all sites – even those with slow network links – this can lead to significant delays before the user can use the app after an update
    • You have user-base that roams between machines commonly – and the mutliple installs across multiple machines ends up using significant disk space
  • Support. There’s no nice way of putting it – its a extremely poor deployment model which is far more difficult to manage when compared to your traditional per-machine install.
  • When there is a security issue in a release, where that issue can be exploited simply by the existance of the vulnerable dll (for example – using dll preloading/hijacking) – the difficulty of finding and removing or updating all of these for a per-user application is significant
  • Dealing with smug fuckwit salespeople that have absolutely no idea why click once is a bad idea and say idiotic shit like “our other clients use it and dont have a problem”… yes, your other clients with 100 PCs in their entire org across 1 site who have no dedicated IT function and dont know any better.

In short, if you turn up to any reasonable size client and say “we deploy our 700Mb memory leak simulator (yes, im thinking of partrcular piece of software here) via click once:” – any comptent IT people should laugh you out the door.

While i realise there is zero chance of salespeople caring about their product’s actual capabilities and likewise, zero chance of managers starting to care about buying products that are complete fucking shit and expecting internal techs to somehow make them work – i was disappointed when i couldn’t find any existing articles on the shittiness of per-user applications…. so… now there is one.

 

Other articles that touch on the issues of ClickOnce

https://www.stevestreeting.com/2013/05/12/friends-dont-let-friends-use-clickonce/#:~:text=ClickOnce%20forces%20you%20to%20put,your%20customer’s%20time%20on.

https://www.codemag.com/article/0902031/Eight-Evil-Things-Microsoft-Never-Showed-You-in-the-ClickOnce-Demos-and-What-You-Can-Do-About-Some-of-Them

All You Need Is One – A ClickOnce Love Story

 

Documenting GPPref drive and printer mappings

Posted on by Hayes Jupe

GPPrefs are great – but in true MS fashion – they are very much a bolt-on from when they were acquired many years ago – and the ability to script reading values (let alone creation) is somewhere limited due to this.

This dude made a fantastic script to document GPPref drive mappings here

Getting all GPP Drive maps in a Domain with PowerShell

You should be able to go to the link above for the original script, i have a slightly updated version below, which outputs the content to a CSV instead of to screen.

<#
.SYNOPSIS
The script finds the GPP Drive Maps in your domain.
.NOTES
File Name: Get-GPPDriveMaps.ps1
Author : Johan Dahlbom, johan[at]dahlbom.eu
The script are provided “AS IS” with no guarantees, no warranties, and it confer no rights.
#>
#Import the required module GroupPolicy
try
{
Import-Module GroupPolicy
}
catch
{
throw “Module GroupPolicy not Installed”
}
$GPO = Get-GPO -All

foreach ($Policy in $GPO){

$GPOID = $Policy.Id
$GPODom = $Policy.DomainName
$GPODisp = $Policy.DisplayName

if (Test-Path “\\$($GPODom)\SYSVOL\$($GPODom)\Policies\{$($GPOID)}\User\Preferences\Drives\Drives.xml”)
{
$DriveXML = Get-Content “\\$($GPODom)\SYSVOL\$($GPODom)\Policies\{$($GPOID)}\User\Preferences\Drives\Drives.xml”

foreach ( $drivemap in $DriveXML.Drives.Drive ) {

$GPOName = $GPODisp
$DriveLetter = $drivemap.Properties.Letter + “:”
$DrivePath = $drivemap.Properties.Path
$DriveAction = $drivemap.Properties.action.Replace(“U”,”Update”).Replace(“C”,”Create”).Replace(“D”,”Delete”).Replace(“R”,”Replace”)
$DriveLabel = $drivemap.Properties.label
$DrivePersistent = $drivemap.Properties.persistent.Replace(“0″,”False”).Replace(“1″,”True”)
$DriveFilterGroup = $drivemap.Filters.FilterGroup.Name

Add-Content -Value “$GPOName,$DriveLetter,$DrivePath,$DriveAction,$DriveLabel,$DrivePersistent,$DriveFilterGroup” -Path C:\Temp\GPODriveMaps.csv
}
}
}

 

In addition, i used this script as a base to make one that will document printer mappings from GPPrefs

<#
.SYNOPSIS
The script finds the GPP Printer Maps in your domain.
.NOTES
File Name: Get-GPPPrinterMaps.ps1
The script are provided “AS IS” with no guarantees, no warranties, and it confer no rights.
#>
#Import the required module GroupPolicy
try
{
Import-Module GroupPolicy -ErrorAction Stop
}
catch
{
throw “Module GroupPolicy not Installed”
}
$GPO = Get-GPO -All

foreach ($Policy in $GPO){

$GPOID = $Policy.Id
$GPODom = $Policy.DomainName
$GPODisp = $Policy.DisplayName

if (Test-Path “\\$($GPODom)\SYSVOL\$($GPODom)\Policies\{$($GPOID)}\User\Preferences\Printers\Printers.xml”)
{

$PrinterXML = Get-Content “\\$($GPODom)\SYSVOL\$($GPODom)\Policies\{$($GPOID)}\User\Preferences\Printers\Printers.xml”

foreach ( $PrinterMap in $PrinterXML.Printers.SharedPrinter) {

$GPOName = $GPODisp
$PrinterName = $PrinterMap.name
$PrinterPath = $PrinterMap.Properties.path
$PrinterAction = $PrinterMap.Properties.action.Replace(“U”,”Update”).Replace(“C”,”Create”).Replace(“D”,”Delete”).Replace(“R”,”Replace”)
$PrinterDefault = $PrinterMap.Properties.default.Replace(“0″,”False”).Replace(“1″,”True”)
$PrinterFilterGroup = $PrinterMap.Filters.FilterGroup.name

Add-Content -Value “$GPOName,$PrinterName,$PrinterPath,$PrinterAction,$PrinterDefault,$PrinterFilterGroup” -Path C:\Temp\GPOPrinterMaps.csv
}
}
}

Exchange – Can’t remove the domain ‘xy.com’ because it’s referenced in an email address policy by the proxy address template ‘smtp:@xy.com’.

Posted on by Hayes Jupe

The current company im working for is a large (for Aus) media company and as such, because basically everyone that works here is into marketing and sales, we just have a ridiculous number of domains.

As part of the cleanup process here – i bought up the huge number of domains we had and that maybe some of them weren’t current – and got back confirmation that many weren’t (approx 120)

The first job was removing them from exchange accepted domains, but i kept getting an error

Can’t remove the domain ‘xy.com’ because it’s referenced in an email address policy by the proxy address template ‘smtp:@xy.com’.

Even though the domains were not anywhere in the existing email address policies.

As usual, ADSIEdit to the rescue

navigate to

CN=Recipient Policies,CN=<Your Org Name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=xy,DC=com

Find the recipient policy which contains an attribute called “disabledGatewayProxy”

Remove each of the domain values from that attribute

Wait a few minutes for replication

then go back to exchange and remove the accepted domain.

Unlicensed OneDrive charges as of Jan 2025 and analysis

Posted on by Hayes Jupe

If you work in a MS-based environment, it is likely you have seen that as of January 27th 2025, you will start getting charged for retained OneDrive data.

https://learn.microsoft.com/en-us/sharepoint/unlicensed-onedrive-accounts?WT.mc_id=365AdminCSH_spo 

The storage cost for the will be $US 0.05/GB/Month – which can add up very quickly.

https://mc.merill.net/message/MC836942

In the environment i’ve recently started working in, there was a guy who was previously working in the role, who i am growing to absolutely despise, who made sure that we would experience maximum hassle when trying to extricate ourselves from his exceedingly poor and short-sighted decisions.

 

Step 1 – Determine the size of the issue

  • Go to SharePoint admin center > Reports > OneDrive accounts
  • You will be presented with the size of your issue immediately. Here is an example from my work tenant.
  • Assuming you have an issue, click “Download report”
  • This report will give us three important things
    • The direct urls for the onedrive sites we are interested in looking at
    • The reason the OneDrive is unlicensed
    • Why the deletion has been blocked

Step 2 – Analysing the report data

  • In the “Unlicensed due to” column you will see 3 options
    • Owner deleted from EntraID
      • Reasonably self-explanatory. The account has been deleted, but importantly, no-one else has been granted ownership over the OneDrive – and it is likely safe to delete
    • License removed by admin
      • License has been removed, but the user still exists. In our environment this seemed to be mainly shared resources that were accidentally allocated a license when they were created.
    • Duplicate account
      • This generally indicates that ownership of the OneDrive has been reallocated to another current user. It can also occur when a user leaves and comes back and the “old” OneDrive is still there due to a retention policy.
  • In the “Delete blocked by” column you also will see 3 options
    • Retention period
      • This is the OneDrive retention period for unlicensed users as specified here
      • https://<YourSiteName-admin.sharepoint.com/_layouts/15/online/AdminHome.aspx#/settings/OrphanedPersonalSitesRetentionPeriod
      • As you can see – ours is set 2555 days…. or 7 years.
    • Retention policy
    • Owner active in Entra ID
      • This means that another user, who is active has been granted access over the OneDrive.
  • For us at least, all entries are blocked by both the OneDrive retention period and the retention policy. Some are (in addition to the other two) also blocked by “Owner active in Entra ID”

Step 3 – an approach to sorting through this mess

  • Take the .csv downloaded in step 1 and open it in Excel
  • Enable auto-filter – and class the deleted OneDrives into 3 categories
    • “Owner deleted from EntraID”
      • In my instance, there was general agreement that where the user has simply been deleted, that we didn’t need to do any further analysis on these accounts. An email would be sent out to business to explain that old OneDrive accounts would no longer be available as of x. The problem is here of course, that many users don’t know if/when they are accessing a file from a shared OneDrive.
    • “License removed by admin”
      • We need to find out more details here, such as if the account is a shared mailbox that was accidentally licensed. Who the owner is. Does the OneDrive contain any data etc.
    • “Duplicate account”
      • These are the OneDrives that are most likely to be in active use and require further analysis

 

Step 4 – Slightly more detailed analysis

  • As per anything that is in anyway related to sharepoint, this is way harder than it needs to be.
  • Filter the Excel document to your desired list (.e.g duplicate accounts)
    • Copy and paste the urls into a txt document and use it to drive the following script
    • Note : This script will grant your admin account “SiteCollection” admin rights! If you need to seek permission to do this from management first – do that before running this script!

# Connect to SharePoint Online Admin Center
$adminSiteUrl = “https://<YourSite>-admin.sharepoint.com”
Connect-SPOService -Url $adminSiteUrl

# Path to the text file with the list of OneDrive URLs
$onedriveUrlsFilePath = “C:\Temp\DuplicateAccount.txt”

# Path to the output CSV file
$outputCsvFilePath = C:\Temp\DuplicateAccount.csv”

# Import the OneDrive URLs from the text file
$onedriveUrls = Get-Content -Path $onedriveUrlsFilePath

# Initialize an empty array to store the results
$onedriveDetails = @()

# Loop through each OneDrive URL
foreach ($onedriveUrl in $onedriveUrls) {
# Get site details (StorageUsageCurrent and LastContentModifiedDate)
Write-host “Getting details for account: $onedriveUrl”
$siteDetails = Get-SPOSite -Identity $onedriveUrl -Detailed

#Grant siteAdmin permissions
Set-SPOUser -Site $oneDriveUrl -LoginName <YourAdminUsername> -IsSiteCollectionAdmin $true

$SiteAdmins = Get-SPOUser -Site $onedriveUrl -Limit All | Where-Object { $_.IsSiteAdmin -eq $true }
$SiteMembers = Get-SPOUser -Site $onedriveUrl -Limit All | Where-Object { $_.IsSiteAdmin -eq $false }

# Store the details in a PowerShell object
$onedriveInfo = [PSCustomObject]@{
URL = $onedriveUrl
Owner = $siteDetails.Owner
StorageUsageCurrent = $siteDetails.StorageUsageCurrent
LastContentModifiedDate = $siteDetails.LastContentModifiedDate
SiteAdmins = ($SiteAdmins | ForEach-Object { $_.LoginName }) -join “; ”
SiteMembers = ($SiteMembers | ForEach-Object { $_.LoginName }) -join “; ”
}

# Add the details to the array
$onedriveDetails += $onedriveInfo
}

# Export the details to a CSV file
$onedriveDetails | Export-Csv -Path $outputCsvFilePath -NoTypeInformation

 

The output from this script will give you a csv with

  • The URL
  • Owner
  • Current usage
  • Last content modified date
  • Login ID’s of accounts that have SiteAdmin permissions
  • Login ID’s of accounts that are site members – meaning that at some stage, the original user has shared a folder or document with one of these users from their OneDrive.

From here – you now are starting to get enough information to track down possible usages of these “old” OneDrives.

Now – if your anything like us – there is waaaay to many permissions in order for anyone to track all these down by contacting the users in question.

Step 5 – Potential “solutions”

  • Now… solutions may be a bit of a strong word here…. so perhaps lets go with “ways of gradually vetting then removing access to data prior to it being deleted to improve your chances of not deleting something important”
  • If someone wants to view a OneDrive contents before being deleted, you can grant them site collection admin via
    • GUI : <TheURLtoTheUsersOneDrive>/_layouts/15/mngsiteadmin.aspx (copy the url from the spreadsheet you created in step1)
    • Powershell : Set-SPOUser -Site <TheURLtoTheUsersOneDrive> -IsSiteCollectionAdmin:$true -LoginName <UPNofUserToGrantAccessToo>
  • Due to a large number of members potentially having access to files in someone’s onedrive – and not knowing that they are accessing it from someone’s OneDrive, we can
    • Record who has access using the script above
    • Perform a scream test for a period of time by removing member access by using the following script

# Connect to SharePoint Online Admin Center
$adminSiteUrl = “https://<YourSite>-admin.sharepoint.com”
Connect-SPOService -Url $adminSiteUrl

# Path to the text file with the list of OneDrive URLs
$onedriveUrlsFilePath = “C:\Temp\Testing2.txt”

# Define the log file path
$logFilePath = “C:\Temp\Testing2.log”

# Import the OneDrive URLs from the text file
$onedriveUrls = Get-Content -Path $onedriveUrlsFilePath

# Function to log the removal of a user
function Log-UserRemoval {
param (
[string]$siteUrl,
[string]$userName
)
$timestamp = Get-Date -Format “yyyy-MM-dd HH:mm:ss”
$logEntry = “$timestamp – Removed user: $userName from site: $siteUrl”

# Append the log entry to the log file
Add-Content -Path $logFilePath -Value $logEntry
Write-Host $logEntry
}

# Loop through each OneDrive URL
foreach ($onedriveUrl in $onedriveUrls) {
Write-host “Getting details for account: $onedriveUrl”
# Define the OneDrive or SharePoint site URL
$siteUrl = $onedriveUrl

# Get all users from the site
$users = Get-SPOUser -Site $siteUrl -Limit All

# Loop through each user and check if they are not a site admin and if their username ends with @<yourDomainSuffix>
foreach ($user in $users) {
if ($user.IsSiteAdmin -eq $false -and $user.LoginName -like “*@<yourDomainSuffix>”) {
# Remove the user from the site if their login ends with <yourDomainSuffix>
Write-Host “Removing user: $($user.LoginName)”
Remove-SPOUser -Site $siteUrl -LoginName $user.LoginName

# Log the removal of the user
Log-UserRemoval -siteUrl $siteUrl -userName $user.LoginName
}
}

}

 

That’s my current best shot at this…. will be interested to hear if any of you have additional/different ways of tackling this mess, while the rest of organisation is like the below:

Other

Other random commands i found helpful

Build a list of all OneDrive urls

from https://learn.microsoft.com/en-us/sharepoint/list-onedrive-urls

$TenantUrl = Read-Host “Enter the SharePoint admin center URL”
$LogFile = [Environment]::GetFolderPath(“Desktop”) + “\OneDriveSites.log”
Connect-SPOService -Url $TenantUrl
Get-SPOSite -IncludePersonalSite $true -Limit all -Filter “Url -like ‘-my.sharepoint.com/personal/'” | Select -ExpandProperty Url | Out-File $LogFile -Force
Write-Host “Done! File saved as $($LogFile).”

In particular, update the “Url -like ‘-my.sharepoint.com/personal/'” to include part of a username you are interested in e.g. “Url -like ‘-my.sharepoint.com/personal/mike'”

 

Lock/Unlock OneDrive’s in order to prevent user access – this can be helpful as a scream test before permanent deletion

connect-sposervice -Url https://<yoursitename>-admin.sharepoint.com

Set-SPOSite <onedrive url> -LockState Unlock

Set-SPOSite <onedrive url> -LockState NoAccess

 

Delete a specific OneDrive (This is useful where a license may have been accidentally allocated to a shared resource)

Remove-SPOSite -Identity <url>

 

WSUS is now deprecated

Posted on by Hayes Jupe

https://techcommunity.microsoft.com/t5/windows-it-pro-blog/windows-server-update-services-wsus-deprecation/ba-p/4250436

Not really surprising – there hasn’t been any meaningful updates to WSUS for a very long time… there are improvements that could be made – but no appetite to make them…. i imagine because the cloud solutions have a direct cost (and are therefore revenue producing) whereas WSUS does not.

The argument could be made that the ability to patch en-mass via products such as WSUS (and by extension, SCCM) is part of what you are paying for when you pay for a Windows license… the same as the the idea that you’re paying for supported product… its all part of the eco-system…. and without support, or patching or sysinternals tools or <insert other tools here> the Windows eco-system becomes much less attractive.

Anyhoo – obviously not the way the MS management see things…. it appears as if anything that doesn’t have a direct revenue stream associated with it is being killed off. Not surprising – but it also really sucks being shoe-horned into a immature platform that doesn’t always fit business or technical needs.

Removing EXO litigation holds and making your mailbox functional again

Posted on by Hayes Jupe

Recently i had a situation where a user with a shared mailbox which had a ludicrous number of items. He recently started in the role and wanted to “start fresh” – but items could not be deleted.

A quick investigation found that litigation hold had been enabled for the mailbox (in addition to our org wide retention policies). It was unclear why the litigation hold had been enabled – and the reasons were lost due to staff turnover.

Finding and fixing the issue included:

  • get-mailbox <identity> | fl *hold*
    • This command will show the status of all holds on that specific mailbox. In my case, i could see that “LitigationHoldDate” and “DelayHoldApplied” were populated
    • In order to remove these i ran
      • get-mailbox <identity> | Set-Mailbox -LitigationHoldEnabled $false
      • get-mailbox <identity> | Set-Mailbox -RemoveDelayHoldApplied
  • After running these steps – i was still not able to delete items from the mailbox, so i ran
    • get-mailboxFolderStatistics <identity> -FolderScope RecoverableItems | FL Name,FolderAndSubfolderSize,ItemsInFolderAndSubfolders
    • and could see that the recoverable items and purges were both at 100GB – meaning that the quota, which also applied to delete items was full – so i could not yet delete anything more
    • In order to speed up the process of the managed folder assistant doing its job, run
      • Start-ManagedFolderAssistant -Identity <identity> -FullCrawl -HoldCleanup
    • After some time, if you re-run the get-mailboxFolderStatistics command, you should see the recoverable items and purges start to come down
  • Since this mailbox was full and is receiving a very high volume of new mail – the 100GB limit was going to be hit again very quickly – so in order to mitigate that for this initial cleanup, i then set
    • Set-Mailbox -Identity <identity> -RetainDeletedItemsFor 1
    • This will only detain deleted items for 1 day in the dumpster before purging
    • I will set this back to 30 days once the initial cleanup is complete and the mailbox is back to “normal” operation.
  • If the mailbox also has a retention policy in addition to the litigation hold
    • get-mailbox -Identity <identity> -Archive | fl *holds*
      • Take note of the ID of the in place hole
    • Connect-IPPSSession
    • Get-RetentionCompliancePolicy | fl Name,ExchangeObjectId
      • Match up the id of the retention policy to a name
    • Set-RetentionCompliancePolicy -Identity <NameOfPolicy> -AddExchangeLocationException <useridentity>
    • Get-Mailbox <identity> | FL DelayHoldApplied,DelayReleaseHoldApplied
    • Set-Mailbox <identity> -RemoveDelayHoldApplied
    • Set-Mailbox <identity> -RemoveDelayReleaseHoldApplied
    • Start-ManagedFolderAssistant -Identity <identity> -FullCrawl -HoldCleanup
    • <wait>
    • To check the status, use
      • get-mailboxFolderStatistics <identity> -FolderScope RecoverableItems | FL Name,FolderAndSubfolderSize,ItemsInFolderAndSubfolders

I’ve read some articles that say that “Start-ManagedFolderAssistant” needs to be run twice to pick up the new settings – but its all a bit of black magic because we cant see what is happening at the backend…. i tend to run it two or three times – as if those articles are right, im speeding things up, if they are wrong, there is no visible harm.

Crowdstrike BSOD and GPO no longer updating

Posted on by Hayes Jupe

After the Crowdstrike BSOD’s on 19/07/2024 – we have seen a significant uptick on clients not refreshing group policy.

The machines in question can be identified via:

  • The last update file date on C:\Windows\System32\GroupPolicy\Machine\registry.pol being on or around 19/07/2024 (some were on the 20th or 21st for us)
  • Event ID 1096 in the system event log with a line similar to “The processing of Group Policy failed. Windows could not apply the registry-based policy settings for the Group Policy object LocalGPO. Group Policy settings will not be resolved until this event is resolved. View the event details for more information on the file name and path that caused the failure”

The fix itself is very simple, delete the file C:\Windows\System32\GroupPolicy\Machine\registry.pol… but in an environment which does not have SCCM on all endpoints (which is incredibly frustrating), the following can be utilised to identify the machines suffering from the issue. The following script also checks for setup log event ID 1015 – indiciating Windows component store corruption… far less common – but we’ve also had some of that (although im less including to think this is Crowdstrike related and more just the poor maintenance of machines)

Obviously you could also add the code to delete the file when found – but at this point, i just needed to identify.

# Define the path to the input file containing the list of machines
$inputFilePath = “<path to txt file with computer list – could also run against AD if you wanted>”

# Define the output file to store the results
$outputFilePath = “<outputpath>\results.csv”

# Import the list of machines from the text file
$machines = Get-Content -Path $inputFilePath

# Initialize an array to hold the results
$results = @()

foreach ($machine in $machines) {
# Trim any leading/trailing whitespace
$machine = $machine.Trim()

# Ping the machine to check if it’s online
if (Test-Connection -ComputerName $machine -Count 1 -Quiet) {
Write-Host “$machine is online.”

# Define the path of the file to check
$filePath = “\\$machine\C$\Windows\System32\grouppolicy\machine\registry.pol”

# Check if the file exists and get the last write time
if (Test-Path -Path $filePath) {
$fileDate = (Get-Item -Path $filePath).LastWriteTime
Write-Host “File found on $machine. Last modified on: $fileDate.”
} else {
Write-Host “File not found on $machine.”
$fileDate = $null
}

# Check for Event ID 1096 in the System log within the last 7 days
$event1096 = Get-WinEvent -ComputerName $machine -FilterHashtable @{LogName=’System’; Id=1096; StartTime=(Get-Date).AddDays(-7)} -ErrorAction SilentlyContinue

# Check for Event ID 1015 in the Setup log within the last 7 days
$event1015 = Get-WinEvent -ComputerName $machine -FilterHashtable @{LogName=’Setup’;Id=1015; StartTime=(Get-Date).AddDays(-7)} -ErrorAction SilentlyContinue

# Determine the status of the events
$event1096Status = if ($event1096) { “Event 1096 Found” } else { “Event 1096 Not Found” }
$event1015Status = if ($event1015) { “Event 1015 Found” } else { “Event 1015 Not Found” }

# Add the result to the array
$results += [PSCustomObject]@{
Machine = $machine
Online = $true
FileDate = $fileDate
Event1096 = $event1096Status
Event1015 = $event1015Status
}
} else {
Write-Host “$machine is offline.”

# Add the result to the array
$results += [PSCustomObject]@{
Machine = $machine
Online = $false
FileDate = $null
Status = “Offline”
Event1096 = “N/A”
Event1015 = “N/A”
}
}
}

# Export the results to a CSV file
$results | Export-Csv -Path $outputFilePath -NoTypeInformation

Write-Host “Results have been saved to $outputFilePath.”

 

RDS Session hosts don’t accept new connections after reboot – despite not being in drain mode

Posted on by Hayes Jupe

Recently – i have had the following scenario:

RDS Farm with 2 x DNS-RR brokers and approx 30 session hosts, all server 2022.

Some session hosts, seemingly randomly, after a reboot will all look ok, but won’t accept any connections.

cycling the session host to not accept new connections/accept new connections would bring the server “back”

After looking through the logs and posting on a few forums (and getting some exceedingly poor responses) – i came to a point where i knew i could implement a “hack” – but would prefer to find the root cause.

To that end, i engaged Andy from https://purerds.org/ – who i’d previously worked with and seems to have that “next level” RDS knowledge – partially for a sanity check that i hadn’t missed something – and partially with the hope that he had seen this before.

The guts of it is:

  • In SQL – the RDSHProperty table shows a “drainmode” of “0” for all servers – so the servers not accepting connections is not recognised by the brokers (as we expected)
  • In SQL – the TargetProperty table shows a “serverMaxActiveSessions” of “10000” for all servers – in line with the above
  • In the log “Microsoft-Windows-TerminalServices-SessionBroker-Client/Operational”
    • We can see the server leave the farm @ 1am (reboot time) with
      • Event Id 1283
      • Remote Desktop Services successfully left a farm on the Connection Broker server <broker1>;<broker2>
    • But no corresponding entry to re-join the farm (unlike the healthy servers)
  • If I restart the service “TermService” on the local server, I get the following events (as expected – but just for the sake of documenting things)
    • EventId 1280 – Remote Desktop Services failed to join the Connection Broker on server <broker1>;<broker2>. Error: Current async message was dropped by async dispatcher, because there is a new message which will override the current one.
    • EventId 1281- Remote Desktop Services successfully joined a farm on the Connection Broker server <broker1>;<broker2>

In the end, we were unable to find a root cause, so i ended up using the following powershell script as a scheduled task on each session host:

$LogPath = “C:\Windows\Temp”
$LogName = “RDSRestartOnJoinFail.log”
$startTime = (Get-Date).AddHours(-12)

#Logging
Function Write-Log {
[CmdletBinding()] param(
[Parameter()] [ValidateNotNullOrEmpty()] [string]$Message,

[Parameter()] [ValidateNotNullOrEmpty()] [ValidateSet(‘Information’,’Warning’,’Error’,’Success’,’Break’)] [string]$Severity = ‘Information’,

[Parameter()] [ValidateNotNullOrEmpty()] [ValidateSet(‘Console’,’LogFile’,’Both’)] [string]$LogType = ‘Console’
)

$Date = (Get-Date).toString(“yyyy/MM/dd HH:mm:ss”)
$LogString = $Date + “, ” + $Severity + “, ” +$Message
If ($LogType -eq “Console” -or $LogType -eq “Both”) {
If ($Severity -eq “Information”) { Write-Host $LogString -foregroundColor Blue}
If ($Severity -eq “Warning”) { Write-Host $LogString -foregroundColor Yellow}
If ($Severity -eq “Error”) { Write-Host $LogString -foregroundColor Red}
If ($Severity -eq “Success”) { Write-Host $LogString -foregroundColor Green}
If ($Severity -eq “Break”) { Write-Host $LogString -foregroundColor White}
}

If ($LogType -eq “LogFile” -or $LogType -eq “Both”) {
Add-Content $LogPath\$LogName -value $LogString }
}

#Main
$Events = Get-WinEvent -FilterHashtable @{ProviderName = “Microsoft-Windows-TerminalServices-SessionBroker-Client”; LogName = “Microsoft-Windows-TerminalServices-SessionBroker-Client/Operational”; id=’1281′; StartTime = $startTime} -ErrorAction SilentlyContinue

if ($events.Count -eq 0) {
Write-Log -Message “No events with ID 1281 found in the past 12 hours – Can assume that machine has not re-joined the farm. Restarting TERMSERV service” -Severity Warning -LogType LogFile
Restart-Service -Name TermService -Force
} else {
Write-Log -Message “$($events.Count) events with ID 1281 found in the past 12 hours – Can assume that machine HAS re-joined the farm” -Severity Information -LogType LogFile
}

Pet insurance australia – just shit…

Posted on by Hayes Jupe

Dogs…. just fluffy balls of awesomeness right ?

Just like we have health insurance, i got pet insurance for our first Golden Retriever – who turned 11 a last month, through Pet Insurance Australia… as they seemed to be ok-ish based on the online reviews… acknowledging that its incredibly difficult to discern a real review from a bot-farm review anymore.

He’s had a full life of playing with other dogs (his favourite), his little human, his therapy dog work and the rest of our family… like most goldens, he’s pretty much universally loved… because he’s fucking awesome and might well be the nicest creature on the planet – ever.

All the way back in 2016, i got pet insurance for him because – risk and risk mitigation. At the time it was around the $500 a year mark.

Fast forward to yesterday (July 2024) – the premiums are now approx $2200 for the upcoming renewal. One one hand, i understand inflation and that his risk profile has changed now he’s older… on the other – isn’t that what i paid premiums for the last 8 years to help cover ?

When i rang to cancel the policy, i got the same old bullshit, including an offer to give us 3 months free… which really sealed the deal for me. If you can offer 3 months for free, then you’re just price gouging (like most corporates at the moment, i’m not saying this is isolated) rather than increasing prices in line with inflation.

Fuck you Pet Insurance Australia…. there aren’t many sacred things left in the world – but the health of doggies everywhere is one of them – you don’t fuck with that…. may you all get bowel cancer and die a long, incredibly painful death.

Moving from Synology to QNAP

Posted on by Hayes Jupe

My Synology 2413+ 12 bay NAS recently died after 12 years of service.

This NAS was primarily used as:

  • an iSCSI backup target for Veeam
  • Video recording for home security cameras
  • Media storage

Overall, i was pretty happy with the unit itself – but as per most companies these days, support was non-existent…. so when i did run into an issue, i was on my own.

Due to that, and Synology not being able to answer what would happen with my surveillance station licenses, i made the decision to go for a QNAP as:

  • It was a little cheaper for better hardware specs (this is in the 8-bay desktop model i was looking at – may be different for other models)
  • QVRPro – the equivalent of Synology surveillance station is free for up to 8 cameras – and i only use 4. There is apparently a 14 day retention time on video at the “free” license level…. and while i would prefer it to be 31 days…. its going to be fine most of the time.

In the ways im interested in, the QNAP has so far proven to be quite good, its setup and joining to an AD domain was simple and painless, adding disks, storage pools and volumes was easy and clear, QVRPro setup had very minor hiccups (more due to my understanding than the software)… but, it hasn’t been all great. The issues i have noticed so far:

  • The lack of a Synology Hybrid RAID equivalent isn’t a disaster, but disappointing…
  • Due to the above, i have purchased some more 8TB disks (previously had a mix of 6TB and 8TB) – the time taken to expand/repair is significant (as expected) – but the poor thing has been the performance of the device while this is occuring. Trying to stream anything during this process has been pointless – with constant dropouts. Having the performance degrade during a repair or expand is not unexpected – but not to the point of drop-outs.

Will be interested to see the performance difference once the rebuild has finished.