8 minutes
Santa Client Troubleshooting
I’ve been meaning to post something like this for a bit now, as I see a fair number of people trying to troubleshoot problems with Santa, and not knowing how to tackle it. I really haven’t encountered issues with a Santa sync server, so, here I’ll stick to the client.
What?
If you don’t know what any of this is, Santa is a binary allow/deny-list system extension for macOS. At minimum, when installed, nothing gets blocked, but you get insight into all process executions via event logs. Even in pure monitor mode, you can block IOCs and other unwanted software. Santa is also incredibly performant. It’s wonderful–you should probably be using it. Newer versions of Santa can also block USB storage. However, you’re probably reading this because you’re already using Santa, but have some questions.
In short, Santa runs on a client and keeps a local database of rules that instructs its behavior. Very often, those rules are synced from a server. In either case, the local rules can become out of sync with what you expect, or with what your server is sending.
From here, I’m assuming a Santa client that syncs with some backend sync server (Zentral, Rudolph, or some custom service), and an ability to push profiles via MDM.
On-Client Report
From the client’s point of view, what is Santa doing?
% santactl status
>>> Daemon Info
Mode | Lockdown
Log Type | file
File Logging | Yes
USB Blocking | No
Watchdog CPU Events | 0 (Peak: 14.02%)
Watchdog RAM Events | 0 (Peak: 131.36MB)
>>> Cache Info
Root cache count | 133
Non-root cache count | 3
>>> Database Info
Binary Rules | 532
Certificate Rules | 91
TeamID Rules | 11
Compiler Rules | 0
Transitive Rules | 0
Events Pending Upload | 0
>>> Watch Items
Enabled | No
>>> Sync Info
Sync Server | https://santa.example.com/sync/yqsqOI/
Clean Sync Required | No
Last Successful Full Sync | 2023/03/22 12:02:46 -0400
Last Successful Rule Sync | 2023/03/22 12:02:46 -0400
Push Notifications | Disconnected
Bundle Scanning | Yes
Transitive Rules | Yes
This shows that the client is running (“Driver Connected”), and is in blocking (“Lockdown”) mode.
Is/Should this Client be in Blocking Mode?
This can be set via a local config, or via a profile likely pushed through your MDM server.
Client Not Running?
Full Disk Approval
Like many other endpoint agents, Santa requires Full Disk Access to do its job. You’ll once again need a TCC profile in place, pushed from your MDM. To check on the client of that profile is there (and substitute your profile name in the grep statement):
% sudo profiles show | grep -A 18 "Santa Full Disk Access"
Password:
_computerlevel[7] attribute: name: Santa Full Disk Access (new)
_computerlevel[7] attribute: configurationDescription: Santa FDA policy.
_computerlevel[7] attribute: installationDate: 2022-11-29 23:35:19 +0000
_computerlevel[7] attribute: organization: Example, LLC
_computerlevel[7] attribute: profileIdentifier: 9496D586-53AC-4ECD-AC3E-EF485BA47987
_computerlevel[7] attribute: profileUUID: 9496D586-53AC-4ECD-AC3E-EF485BA47987
_computerlevel[7] attribute: profileType: Configuration
_computerlevel[7] attribute: removalDisallowed: TRUE
_computerlevel[7] attribute: version: 1
_computerlevel[7] attribute: containsComputerItems: TRUE
_computerlevel[7] attribute: installedByMDM: TRUE
_computerlevel[7] attribute: internaldata: TRUE
_computerlevel[7] payload count = 1
_computerlevel[7] payload[1] name = Privacy Preferences Policy Control
_computerlevel[7] payload[1] description =
_computerlevel[7] payload[1] type = com.apple.TCC.configuration-profile-policy
_computerlevel[7] payload[1] organization = Example, LLC
_computerlevel[7] payload[1] identifier = 65F18A6E-A39F-438F-8145-86184B0A84D9
_computerlevel[7] payload[1] uuid = 65F18A6E-A39F-438F-8145-86184B0A84D9
If not present, the Santa client will not be able to perform its functions.
System Extension Approval
Like other extensions, Santa requires macOS TCC approval to load into the system. Check on the client if it’s present:
% sudo profiles show | grep -A 18 "Santa System Extension"
_computerlevel[8] attribute: name: Santa System Extension Approval
_computerlevel[8] attribute: configurationDescription: Approve the santa system extension based on team ID (EQHXZ8M8AV).
_computerlevel[8] attribute: installationDate: 2022-11-29 23:35:17 +0000
_computerlevel[8] attribute: organization: Example, LLC
_computerlevel[8] attribute: profileIdentifier: ADA0FD16-6F5D-411B-9B7B-B6C6F1BA3DA7
_computerlevel[8] attribute: profileUUID: ADA0FD16-6F5D-411B-9B7B-B6C6F1BA3DA7
_computerlevel[8] attribute: profileType: Configuration
_computerlevel[8] attribute: removalDisallowed: TRUE
_computerlevel[8] attribute: version: 1
_computerlevel[8] attribute: containsComputerItems: TRUE
_computerlevel[8] attribute: installedByMDM: TRUE
_computerlevel[8] attribute: internaldata: TRUE
_computerlevel[8] payload count = 1
_computerlevel[8] payload[1] name = System Extensions
_computerlevel[8] payload[1] description =
_computerlevel[8] payload[1] type = com.apple.system-extension-policy
_computerlevel[8] payload[1] organization = Example, LLC
_computerlevel[8] payload[1] identifier = 4EFB36B2-CE40-4138-BF60-61AC1ED499EC
_computerlevel[8] payload[1] uuid = 4EFB36B2-CE40-4138-BF60-61AC1ED499EC
If not present, Santa won’t start.
Also, ensure the extension is approved and activated:
* * EQHXZ8M8AV com.google.santa.daemon (2023.2/2023.2.510496050) santad [activated enabled]
Is Santa Running?
In addition to sudo santactl status above, Santa will be present in the process list:
% sudo ps ax | grep -i \[s\]anta [main=]
804 ?? Ss 0:00.03 /Applications/Santa.app/Contents/MacOS/santametricservice --syslog
872 ?? Ss 2:42.76 /Library/SystemExtensions/BB79CC2F-A6BF-4E0D-9021-9511F2953E21/com.google.santa.daemon.systemextension/Contents/MacOS/com.google.santa.daemon
1227 ?? Ss 0:56.11 /Applications/Santa.app/Contents/MacOS/santasyncservice --syslog
1926 ?? S 0:00.28 /Applications/Santa.app/Contents/MacOS/Santa --syslog
If not, check that Santa is on disk, and ensure permissions are correct:
% ls -l /Applications/Santa.app/Contents/MacOS/Santa
-rwxr-xr-x 1 root wheel 1086560 Feb 17 15:53 /Applications/Santa.app/Contents/MacOS/Santa
You’ve checked everything else: it’s on-disk, extension is present and approved, etc.…try rebooting.
Santa Client Running, but Not Behaving as Expected
Santa is ready, but not configured
Santa gets its config from a profile. See if it’s installed:
% plutil -p /Library/Managed\ Preferences/com.google.santa.plist
{
"BannedBlockMessage" => "The following application has been blocked from executing"
"ClientMode" => 1
"EnableBadSignatureProtection" => 0
"EnablePageZeroProtection" => 1
"EnableSysxCache" => 1
"FileChangesRegex" => "^/(?!(?:private/tmp|Library/(?:Caches|Managed Installs/Logs"
"MachineIDKey" => "hwuuid"
"MachineIDPlist" => "/etc/info.plist"
"MachineOwnerKey" => "owner"
"MachineOwnerPlist" => "/etc/info.plist"
"ModeNotificationLockdown" => "Switching into Lockdown mode"
"ModeNotificationMonitor" => "Switching into Monitor mode"
"SyncBaseURL" => "https://santa.example.com/sync/yqsqOI/"
"UnknownBlockMessage" => "The following application has been blocked from executing<br/>
because it is unknown."
}
This profile will be pushed from your MDM.
Santa is blocking ${APPLICATION}
This will only happen for machines in “Lockdown” mode, with the exception of explicit blocks (Santa respects the deny list, even in monitor mode). In lockdown mode, this means that the application is unknown and has no associated rule. In any case, the fileinfo verb will tell you why Santa is making a particular decision:
% sudo santactl fileinfo /Applications/Santa.app
Path : /Applications/Santa.app/Contents/MacOS/Santa
SHA-256 : b61a569104637861f972894ab3b1d0ce66ab0bf8e387af6a1c777d76b958e34a
SHA-1 : bf3dbf14c66ac9967577a65d7d3c16b2552e5307
Bundle Name : Santa
Bundle Version : 2023.2.510496050
Bundle Version Str : 2023.2
Team ID : EQHXZ8M8AV
Type : Executable (arm64, x86_64)
Code-signed : Yes
Rule : Allowed (Binary)
Signing Chain:
1. SHA-256 : 0bda2aca4b967fd15bb6840c54de1cc230921efd1a181a6f0fc814ada3ffaa4f
SHA-1 : 765bb3620a0f7a33500da39b20122b1cec41140f
Common Name : Developer ID Application: Google LLC (EQHXZ8M8AV)
Organization : Google LLC
Organizational Unit : EQHXZ8M8AV
Valid From : 2022/02/08 17:32:55 -0500
Valid Until : 2027/02/01 17:12:15 -0500
2. SHA-256 : 7afc9d01a62f03a2de9637936d4afe68090d2de18d03f29c88cfb0b1ba63587f
SHA-1 : 3b166c3b7dc4b751c9fe2afab9135641e388e186
Common Name : Developer ID Certification Authority
Organization : Apple Inc.
Organizational Unit : Apple Certification Authority
Valid From : 2012/02/01 17:12:15 -0500
Valid Until : 2027/02/01 17:12:15 -0500
3. SHA-256 : b0b1730ecbc7ff4505142c49f1295e6eda6bcaed7e2c68c5be91b5a11001f024
SHA-1 : 611e5b662c593a08ff58d14ae22452d198df6c60
Common Name : Apple Root CA
Organization : Apple Inc.
Organizational Unit : Apple Certification Authority
Valid From : 2006/04/25 17:40:36 -0400
Valid Until : 2035/02/09 16:40:36 -0500
In this example, I’m checking the Santa application itself, which is just for demonstration: substitute the path to the relevant application for your case. In this output, the “Rule” key tells you why Santa made its decision.
Corrupted/Out of Sync Ruleset
Every now and then, the client won’t sync current rules from the server, particularly after getting the profile changed while running (pointing it to a new ruleset). The symptoms for this look like applications getting blocked, even when there is a server-side allow rule. This can be cleared up with a clean sync:
% sudo santactl sync --clean
Preflight starting
Clean sync requested by user
Performing request, attempt 1
Server Trust: /O=(null)/OU=(null)/CN=santa.example.com/SHA-1=28a1a2889491974d915e09dc4b3e2fd5
Clean sync requested by server
Preflight complete
Event upload starting
Event upload complete
Rule download starting
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 50 rules
Performing request, attempt 1
Received 34 rules
Performing request, attempt 1
Received 0 rules
Processed 634 rules
Rule download complete
Postflight starting
Performing request, attempt 1
Postflight complete
Sync completed successfully
In very rare cases, a local rule database will be corrupt beyond a clean sync. There will be an error message produced in such cases:
% sudo santactl sync --clean
password:
Received 50 rules
Received 50 rules
Received 59 rules
Received 50 rules
Received 11 rules
Received 0 rules
Failed to add rule(s) to database: A database error occurred while inserting/replacing a rule
Rule download failed. aborting run
In this case, we can stop the daemon, remove the rules, and then start the daemon. (This is the nuclear option, and is last in this list as it’s the last thing you should reach for):
% sudo /Applications/Santa.app/Contents/MacOS/Santa --unload-system-extension
sudo rm /var/db/santa/rules.db*
sudo /Applications/Santa.app/Contents/MacOS/Santa --load-system-extension
If you’re thinking of doing this in the background over ssh while helping an end-user, that first command will cause a GUI popup for authentication.
The second line removes the rules database and its journal. The final line re-loads Santa. It will take a few seconds to re-sync all of the rules. While that is happening, background activity may get flagged by Santa, so, ignore popups during this (short amount) of time.
The End?
I hope that helps someone on their Santa journey. If you have other tips or notes, I’m happy to add in more detail here.
1578 Words
2023-03-22 20:16