Crowdstrike BSOD and GPO no longer updating

After the Crowdstrike BSOD’s on 19/07/2024 – we have seen a significant uptick on clients not refreshing group policy.

The machines in question can be identified via:

  • The last update file date on C:\Windows\System32\GroupPolicy\Machine\registry.pol being on or around 19/07/2024 (some were on the 20th or 21st for us)
  • Event ID 1096 in the system event log with a line similar to “The processing of Group Policy failed. Windows could not apply the registry-based policy settings for the Group Policy object LocalGPO. Group Policy settings will not be resolved until this event is resolved. View the event details for more information on the file name and path that caused the failure”

The fix itself is very simple, delete the file C:\Windows\System32\GroupPolicy\Machine\registry.pol… but in an environment which does not have SCCM on all endpoints (which is incredibly frustrating), the following can be utilised to identify the machines suffering from the issue. The following script also checks for setup log event ID 1015 – indiciating Windows component store corruption… far less common – but we’ve also had some of that (although im less including to think this is Crowdstrike related and more just the poor maintenance of machines)

Obviously you could also add the code to delete the file when found – but at this point, i just needed to identify.


# Define the path to the input file containing the list of machines
$inputFilePath = “<path to txt file with computer list – could also run against AD if you wanted>”

# Define the output file to store the results
$outputFilePath = “<outputpath>\results.csv”

# Import the list of machines from the text file
$machines = Get-Content -Path $inputFilePath

# Initialize an array to hold the results
$results = @()

foreach ($machine in $machines) {
# Trim any leading/trailing whitespace
$machine = $machine.Trim()

# Ping the machine to check if it’s online
if (Test-Connection -ComputerName $machine -Count 1 -Quiet) {
Write-Host “$machine is online.”

# Define the path of the file to check
$filePath = “\\$machine\C$\Windows\System32\grouppolicy\machine\registry.pol”

# Check if the file exists and get the last write time
if (Test-Path -Path $filePath) {
$fileDate = (Get-Item -Path $filePath).LastWriteTime
Write-Host “File found on $machine. Last modified on: $fileDate.”
} else {
Write-Host “File not found on $machine.”
$fileDate = $null
}

# Check for Event ID 1096 in the System log within the last 7 days
$event1096 = Get-WinEvent -ComputerName $machine -FilterHashtable @{LogName=’System’; Id=1096; StartTime=(Get-Date).AddDays(-7)} -ErrorAction SilentlyContinue

# Check for Event ID 1015 in the Setup log within the last 7 days
$event1015 = Get-WinEvent -ComputerName $machine -FilterHashtable @{LogName=’Setup’;Id=1015; StartTime=(Get-Date).AddDays(-7)} -ErrorAction SilentlyContinue

# Determine the status of the events
$event1096Status = if ($event1096) { “Event 1096 Found” } else { “Event 1096 Not Found” }
$event1015Status = if ($event1015) { “Event 1015 Found” } else { “Event 1015 Not Found” }

# Add the result to the array
$results += [PSCustomObject]@{
Machine = $machine
Online = $true
FileDate = $fileDate
Event1096 = $event1096Status
Event1015 = $event1015Status
}
} else {
Write-Host “$machine is offline.”

# Add the result to the array
$results += [PSCustomObject]@{
Machine = $machine
Online = $false
FileDate = $null
Status = “Offline”
Event1096 = “N/A”
Event1015 = “N/A”
}
}
}

# Export the results to a CSV file
$results | Export-Csv -Path $outputFilePath -NoTypeInformation

Write-Host “Results have been saved to $outputFilePath.”

 

Ctrl + Alt + Del in nested RDP/VM sessions

If your anything like me – the amount of environments that you are now accessing via a variety of nested citrix/RDP/VPN/Hyper-V Console/JumpBox’s  etc – well…. its now 100% of what you deal with.

 

One of the pains of nested sessions (apart from the clipboard) is pressing Ctrl + Alt + Del for tasks such as changing your password.

A common method around this is to use the on-screen keyboard

  • launch “osk.exe” from a command line
  • press Ctrl + alt – and then use your mouse to press “del”

However, i’ve always found that little painful – and a few years ago i found a direct command line, then lost it again – and have now found it again! So i am now recording it here – for future me – and anyone else

C:\Windows\explorer.exe shell:::{2559a1f2-21d7-11d4-bdaf-00c04f60b9f0}

Resetting the local admin password on a more locked down server

A client recently had an issue where they had lost the administrator password on their offline root CA.

The well known method of copying cmd.exe over utilman.exe was not working

When trying to reset the administrator password, it appeared to work, but on reboot, the new password was not accepted.

To that end i utilise “net” to add a temporary admin account via

net user <username> /add
net localgroup administrators <username> /add
in addition, the server had been configured to not allow any other user but administrator to show up at the console….. this i hadn’t seen before.
After a bit of poking around, i found that the following registry keys were the culprit.
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\LogonUI] “LastLoggedOnProvider”=””
“LastLoggedOnSAMUser”=””
“LastLoggedOnUser”=””
once those keys were set, i could logon as my temporary admin account.

Microsoft releases security update for new IE zero-day

All I want for Christmas is a new security update to patch a zero day IE exploit …….

Microsoft have today released a new out of band update for an Internet Explorer vulnerability that is currently being abused in the wild. Just in time for all those Admins planning to have some time off and well after any planned change lock out windows have come into effect!

According to a security advisory released, the IE zero-day exploit can allow an attacker to execute malicious code on a user’s computer. An attacker who successfully exploited the vulnerability could gain the same user rights as the current user, and we’re all doing the right thing and NOT granting our users Admin rights, aren’t we!!!!!

In a nice move by Microsoft, for all those IT Admins out there half way out the door for the holiday period, and may not have the time to thoroughly test and deploy the latest hot-fix and cumulative update, the security advisory CVE-2018-8653 also contains workarounds for restricting access to the IE scripting engine, until system administrators can deploy today’s official patch.

Workarounds

The workaround provided by Microsoft, is to simply disable user access to the DLL that is affected (jscript.dll), which is not the default JavaScript engine DLL that Internet Explorer uses (Jscript9.dll). The jscript.dll is only called in a specific manner, in this instance a malicious method, so the workaround  should have minimal impact for general use.

Edit (22/12): Workaround modified slightly by Microsoft (added takeown cmd) and republished, updated below.

Edit (20/12): 15:30 AEDST Microsoft have unpublished the suggested workaround.

Restrict access to JScript.dll For 32-bit systems, enter the following command at an administrative command prompt:

	takeown /f %windir%\system32\jscript.dll
	cacls %windir%\system32\jscript.dll /E /P everyone:N

For 64-bit systems, enter the following command at an administrative command prompt:

	takeown /f %windir%\syswow64\jscript.dll
	cacls %windir%\syswow64\jscript.dll /E /P everyone:N
	takeown /f %windir%\system32\jscript.dll
	cacls %windir%\system32\jscript.dll /E /P everyone:N

Impact of Workaround. By default, IE11, IE10, and IE9 uses Jscript9.dll which is not impacted by this vulnerability. This vulnerability only affects certain websites that utilizes jscript as the scripting engine.

How to undo the workaround. For 32-bit systems, enter the following command at an administrative command prompt:

	cacls %windir%\system32\jscript.dll /E /R everyone

For 64-bit systems, enter the following command at an administrative command prompt:

	cacls %windir%\syswow64\jscript.dll /E /R everyone

Reducing your Risks

As I’m sure you’re all aware, there was another vulnerability advertised to the general public over the Christmas new year period, and if you’ve been following the details, the patches to fix this specific vulnerability have been recalled. The advice from Intel and other vendors currently is, “don’t deploy the patch as it can cause system instability and in some circumstances cause data loss or corruption”. Good stuff!

Update: Intel releases Spectre fix for Skylake CPUs only

Protecting against vulnerabilities like this and many other security threats is a multi-layered approach, if you’ve got these layers of protection in place, then the risk of your computers being impacted by any of these vulnerabilities is greatly reduced.

Removing Admin rights
First and foremost, to protect your network and computers, you should be granting user’s with sufficient rights to do their job, nothing more. In our opinion, users very rarely need Administrative rights over a computer. Users in an Enterprise environment shouldn’t be installing software as they please, not only does this prevent system changes from being made, intentionally or otherwise, it also allows the IT department to maintain control of your software licensing.

One issue we tend to face when suggesting or implementing the removal of Admin rights, tend to be those joyful applications that sing out in protest. Most of these well written applications may simply require write access to the local machine registry hive, or write access to the application install location. You can use tools such as ‘Process Monitor’, system instability can in some circumstances cause data loss or corruption troubleshoot these applications and then granting the users write access to the require locations. This is far more secure than granting blanket Admin rights of the entire computer, or computer fleet!

Application Whitelisting
Not all vulnerabilities or malicious code require administrative access, a user accidentally running a crypto locker application will cause more than enough headaches when all the network shares they have access to become encrypted. This is where Application whitelisting comes in. Using Group Policies AppLocker we can ensure that only authorised applications (e.g. programs, software libraries, scripts and installers) can be executed. The default rules you can create with AppLocker, allow applications installed in the ‘Program Files’ and Windows directories to run without hindrance. You can then extend these rules to allow additional applications to run as needed for your environment, and as you’ve removed Admin rights from your users, they wont have write access to these locations.

Blocking Attachments
By far the most common distribution of malware I’ve experienced has been via E-mail attachments. I’m sure, like me, you’ve lost count of the number of times you’ve told friends, family, users, don’t open emails or attachments you don’t know, but let’s face it, that’s a losing battle, especially when one of these people get infected, and then start sending out emails unknowing to their address list containing a malicious payload. Most malware I’ve seen attached to emails has been either
an executable or script directly attached to an email, or in a zip file attachment, there are very few reasons a standard user would be sending these types of attachments via email, I’d even argue that IT users should also be using alternative methods for transferring these files. It may simply be a case of changing the script file extension to txt, which then at least requires the users interaction before it will run. In the enterprise environment, I strongly suggest setting up rules in your email system to block or quarantine any email with an executable attachment (including scripts) or any zip file attachments that include executable files.

If you’d like any assistance or guidance in implementing any of these measures in your environment, feel free to contact us, we’d be happy to help.

Meltdown and Spectre patches available

Hi all,

For many of you that switched off over the xmas break (like me), you may have missed that there are now patches (released Jan 3rd 2018) for the creatviely (almost bond movie like) named vulnerabilities of “meltdown” and “spectre”.

You can find more detail on these Vulnerabilities  here – https://meltdownattack.com/

Advice for Microsoft client OS’s is here – https://support.microsoft.com/en-us/help/4073119/protect-against-speculative-execution-side-channel-vulnerabilities-in . The page still indicates to “contact your vendor” for microcode updates – which isnt going to overly helpful for standard end-users.

Advice for Microsoft server OS’s is here – https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution. There is additional work required over and above the patch for Remote Desktop and Hyper-V servers. Additionally, Windows server 2008 and 2012 are not yet patched, only Server 2008 R2/2012 R2/2016/1709 – read into that what you will.

The register has a good article (as they do most of time) cutting through the intel PR bullshit. Importantly, there has been various reports of performance impacts after installing these patches – but it is still too early to tell exactly how large/important those perfomrance impacts are.

There are links to many vendor advisories (which in turn have links to updates) @ https://meltdownattack.com/ – which is quite useful.

The patches and additional mitigations are fairly easy to implement if you have patching/management infrastructure in place – but if your company needs any assistance, we’re happy to help too.

Preparing to update for Intel® Management Engine Critical Firmware Update (Intel SA-00086)

Intel released a security advisory yesterday (22/11/2017) advising of vulnerabilities with their management engine firmware – which can he found here

The reason why this is concerning for corporate customers is that basically every PC, Server and laptop that you have, most likely will be exposed, as the vast majority of corporate level hardware contains this hardware.

Intel has provided a detection tool available at https://downloadcenter.intel.com/download/27150. Contained within is a couple of applications, the GUI version will probably be handy for those orgs with only a handful of makes and models – where-as the command line tool will be more useful for larger organisations to run and centralise results via tools such as SCCM.

 

The original page contains links for various vendors update, the downside is that there doesnt seem to be many patches as yet, as per this page (at the time of writing, all Dell entries are marked as “TBD”, where-as Lenovo lists a target date of 24/11/2017)

 

So, what can you do while waiting for the patches to be released?

  • Test your systems using the provided tool to see if they are vulnerable. Testing at least one of each make/model will give you a good idea what you will need to target
  • Setup SCCM collections ready to go, which would entail
    • A collection for each make and model (I imagine many places would have this already)
    • A series of collections which each include one/make model and the criteria “AMT Agent – Flash is NOT equal to <insert version number of patched FW – once released>” – This will enable you quickly identify machines which need to be targeted with the update
    • (Please note the above is an assumption, some vendors may patch this via a BIOS update, in which case, the BIOS version may be the identifier instead)

 

Please post a comment if there are any questions – and ill update thios post once the patches are released – if there are any gotcha’s we run into.

SCEP 2012 vs others – making my blood boil

We’ve had a number of customers recently move to SCEP 2012 from other solutions, particularly mcafee, but also some symantec, sophos etc.

These have generally been customers on an EA, with core cal already purchased and additionally , SCCM 2007 or 2012 already in place for deployment…. so for these enviornments, there is a cost saving with SCEP (as its part of the core cal) and since the SCCM infrastructure is already in place, the management overhead (when other products were in use) is reduced.

The biggest thing i see – and continue to hear from customers is the about the massive performance difference on the workstations after moving from something like Mcafee to SCEP…. Mcafee is a machine crippler, plain and simple. For those of you that have it…. build up two SOE machines, one with Mcafee, one with with SCEP and have a look at the performance difference for yourselves.

Anyhoo – a few days ago, a customer was telling me about independant reports which claim SCEP is quite bad (for various reasons) and how superior the administration of EPO is, the feature set etc etc…. so, i went looking for these independant reports.

This is the first one i found – http://www.mcafee.com/us/resources/demos/endpoint-protection-comparison/McAfee-EP-Microsoft-FEP.swf – and, well, as per the title of this post, my blood boiled.

This is flat out, religious style, mis-information, bullshit and quater truths trying to look legimate… some of the claims in the presentation are just ludicrous.

Now this slidehsow is based on FEP 2010 – but quite frankly, the majority of points have not changed between FEP 2010 and SCEP 2012 (but a couple have)

such as

Administration costs (slide 6) – These figures are just insane… where are these coming from ? 3 times as many new servers to deploy SCEP ? how? 1 server is required to deploy SCEP – and, if the organisation is already using SCCM (which isnt exactly uncommon!) – 0… none, nada, zilch new servers required to deploy. Even if geographically diverse orgs, if SCCM was only being used for SCEP – then 1 server is still only required, as BDP’S (SCCM 2007) or DP’s on a workstation (SCCM 2012) can be used for remote distribution (and this would depend on network speeds/topology etc)

The higher personell costs… well, intentionally vague because of what a ludicrous statement it is….. you need 1 (one!) automatic deployment rule to deploy updates… and in the client policy you set “deploy SCEP” to true…. now clearly this guy doesnt actually use the products he’s talking about, but think of the dumbest tech you possible can – even they could rollout and keep SCEP up to date, with a total admin effort of a few hours (a few mins for an SCCM experienced tech).

The comment about SCCM and forefront being “more complex to administer” – you’ve got to be fucking joking. EPO is definitely one of the better anti-virus administration tools out there – but it, as all software, has its own quirks and complexities too…. trying to clain that SCCM is more complex than EPO…. well ofcourse it is if you have used EPO for 5 years and never used SCCM! The same is true the other way around! Thats just a dodgey arguement.

License fee’s – well that all depends on if you are already licensed via an EA (which is very common in our market) – in which case its bundled in with your other CAL’s….. thats a very large and  important point to miss!

Licenisng for other OS’es – This is a fair point for FEP 2010, but as of SCEP 2012 SP1 (due Jan 2013-ish), SCEP supports MAC and Liunx

Admin and reporting (Slide 8)

EPO group membership is security orientated where-as SCCM collection membership is patch orientated…. just huh ? If i understood what he is getting at here, i could shoot it down…. but it just doesnt make any sense.

distributing updates faster – so… your saying you dont know how to configure SCCM ?

Reporting – No question, Mcafeee reporting is richer than SCEP reporting…. SCEP reporting will continue to get better over time (2012 is better than 2010, obviously)

Consoles (Slide 9)

Requires expertise in 6 different consoles! Thats just a flat out lie. SCEP requires you to know how to use the SCCM console, set policy, deploy software updates, view reports….. i.e. the normal things an SCCM admin already does. If you dont already use SCCM, then sure, you’ll have to learn SCCM, but again, the same is true for EPO!

Tampering and reboots (Slide 10)

“Barrage of windows updates, requiring many reboots” – so are we talking about general windows patching or antivirus here – you cant change topic when convienient. Sure windows updates require reboots, but this is completely seperate to antivirus – if you use mcafee, you still need to patch your machines, and those patches still will need reboots. What a disgusting twisting of the facts.

Tamper proof – “users can tamper with and disable forefront” ? really? So you haven’t configured SCEP to lock down the settings (the same as you need to in EPO) and then your complaining that users can change the settings you havent locked down?

 

There is no question Mcafee (and others) have been around longer, are more mature in some ways (reporting in particular) and have more “features”. I argue, and always will, that the additonal features (such as firewalls, execution prevetion etc) are a pain in the arse for most of our clients…. but sure there are some clients that have valid reasons for using them. The biggest doiwnside to these additional features is the crippling performance impact of mcafee (and others) – and important point which seems to have been left out completely. A number of these features are also already available within the OS and can be configured via group policy… sure its a different tool…. but show me one enterprise IT admin that doesnt know how to use group policy.

If you are a Microsoft-based IT enviornment already – chances are the licenses for SCEP are included in the licensing you already have – if you also already use SCCM for deployment, then you already have the infrastructure and skills to deploy SCEP very quickly and keep it easily updated. So its a very compelling case to look seriously at SCEP at save a big wad of cash….. if your not already licensed (which is unlikley if your an MS based enviornment) and dont have SCCM,  by all means, evaluate the different antivirus solutions for your company to see which ones meet your needs, but do not ever, use an “independent” report such as this one as a justification or to form part of your reasoning…. it is one of the most disgracefully, intentionally inaccurate pieces of “independent” advice I have ever seen.

Symantec infection report for 2011

found this over at anandtech – http://www.dailytech.com/article.aspx?newsid=24613

which links to this PDF report – http://www.symantec.com/content/en/us/enterprise/other_resources/b-istr_main_report_2011_21239364.en-us.pdf

I dont necessarily have a lot of time for Symantec – as obviously, increasing fear drives their business, but if you take the figures in the report as accurate – its quite interesting.

The title of the Anandtech article is clearly made to be provocative – but in this case, its a sentiment that, according to the report, is not only technically accurate, but is represented in the rest of society as well.