Nezha Monitoring System Intrusion Incident Response Report

A complete record of a real-world mass intrusion incident involving 40 VPS instances: from vulnerability discovery to backdoor cleanup, root cause analysis, recurrence handling, and automated hardening.

Incident Time: 2026-06-19 to 2026-06-20 Scope: 40 servers, Nezha dashboard v2.2.3 on the master, 34 agents implanted with backdoors Nature: Exploitation of low-version Nezha security vulnerabilities to deploy persistent backdoors

1. Incident Background

I have 40 VPS instances distributed across multiple cloud providers (Tencent Cloud, Alibaba Cloud, Huawei Cloud, RackNerd, NetCup, DMIT, etc.), all connected to the same Nezha monitoring panel. The master is deployed on Tencent Cloud Singapore (43.156.17.202:8008), running dashboard v2.2.3.

On 2026-06-19, I noticed that Nezha had released several security advisories involving critical vulnerabilities in older versions, such as remote command execution and path traversal leaking secrets. I needed to confirm:

1. Are my dashboard and 40 agents affected?
2. Have attackers already exploited the vulnerabilities to compromise my servers?
3. If compromised, what backdoors were planted? How to clean them?

2. Vulnerability Analysis

Affected Versions

Nezha's official security advisories indicate that the key fix cutoff is 2.2.0:

• Dashboard/Agent < 2.2.0: Affected by multiple medium-severity vulnerabilities
• Older versions accumulate high/critical severity vulnerabilities

Key CVEs

Attack Chain

Attacker gains panel access
  ├─ Method 1: Exploit CVE-2026-53519 path traversal to leak jwt_secret_key → forge admin JWT
  ├─ Method 2: Exploit CVE-2026-46716 cron interface to execute commands directly
  └─ Method 3: Weak password / leaked panel credentials
        ↓
Execute arbitrary commands on Agent nodes via panel's remote command capabilities (cron / terminal / force-update)
        ↓
Deploy persistent backdoor

3. Version Audit

Dashboard Version

The master dashboard is deployed via Docker. Version confirmation:

docker exec nezha-dashboard /dashboard/app -v
# Output: 2.2.3

dashboard 2.2.3 > 2.2.0, not affected by the advisories.

Agent Versions (40 instances)

Here I hit a major pitfall: host.version returned by the public WebSocket is null under guest view; you must use an admin JWT to see the real version.

I eventually obtained the versions of all 40 agents through these steps:

1. Read jwt_secret_key and the key_id of an unexpired session from the master SQLite database
2. Generate an admin JWT using Go + sqids-go (Nezha uses idcodec to sqids-encode user_id)
3. Connect to the admin WebSocket with the JWT and read host.version

Results:

• Master agent: 2.2.2 (latest)
• 31 online agents: 2.2.2 (latest)
• 5 online agents: 1.9.7 (affected by vulnerabilities)
• 4 offline agents: version unknown

5 old-version agents: Gouyun-MG-V6, hostyun-us, GreenCloud-Osaka, Gouyun-MG-V4, racknerd-DC02.

4. First Round Cleanup: sysmon Backdoor

Discovery

During a deep read-only check on the master, I found /etc/systemd/system/sysmon.service and sysmon-guard.timer. Further investigation revealed a complete backdoor system:

Backdoor Structure:

/usr/local/sysmon/              ← Fake Nezha agent binary + config.yml
  └─ config.yml                 ← server: 51.254.44.35:8008 (attacker's panel)
/usr/local/.sysmon-guard/       ← Guardian process
/usr/local/lib/libsysmon.so     ← Preload injection library
/etc/systemd/system/sysmon.service
/etc/systemd/system/sysmon-guard.service
/etc/systemd/system/sysmon-guard.timer   ← 30-second self-healing trigger
/etc/ld.so.preload              ← Loads libsysmon.so

Backdoor Behavior:

• Disguised as a sysmon system monitoring service, actually runs a binary similar to Nezha Agent
• Config file server: points to the attacker-controlled Nezha panel 51.254.44.35:8008
• sysmon-guard.timer checks every 30 seconds; if sysmon is killed, it restarts it
• /etc/ld.so.preload loads libsysmon.so for global hooks (likely to hide processes/files)
• Logs show the guardian process executes SELF-HEAL: Re-installing cron every 30 seconds

Backdoor Design with Environment Variables:

STEALTH_NAME=sysmon
INSTALL_DIR=/usr/local/sysmon
GUARD_DIR=/usr/local/.sysmon-guard
LIB_PATH=/usr/local/lib/libsysmon.so
PRELOAD_FILE=/etc/ld.so.preload

Cleanup

The cleanup order is critical (wrong order allows the guardian to rebuild files):

1. Stop and disable sysmon-guard.timer, sysmon-guard.service, sysmon.service
2. kill -9 all sysmon-related processes
3. Delete systemd unit files
4. Remove libsysmon.so reference from /etc/ld.so.preload
5. Delete /usr/local/lib/libsysmon.so
6. Delete /usr/local/sysmon and /usr/local/.sysmon-guard directories
7. systemctl daemon-reload

Post-cleanup verification: sysmon backdoor on master completely removed, no residual processes/files/outbound connections.

5. Batch Cleanup of 40 Instances

Used paramiko thread pool (10 concurrent) for batch cleanup across all servers. Passwords were kept only in memory, never written to disk.

Cleanup Results

• Master + most infected nodes: sysmon backdoor cleaned
• 3 SSH timeouts: raksmart-SanJose, YunxiHuanjing, geelinx-b; require manual handling
• Agent version upgrade: Upgraded 5 instances from 1.9.7 to 2.2.2 via panel force-update API + manual SSH

Pitfalls in Upgrading Agents

• force-update API is ineffective for agents with disable_force_update: true
• Master showing old version doesn't necessarily mean the binary is old; the process might not have restarted
• Some machines had multiple nezha-agent services, only one was restarted
• hostyun-us IP changed from 45.155.222.112 to 207.57.137.18

6. Recurrence! Finding the True Root Cause

After cleaning sysmon, I performed a full re-audit and found 0 recurrence of the sysmon backdoor, but discovered something more dangerous.

Second Config Backdoor

In the /opt/nezha/agent/ directory on 34 servers, alongside the legitimate config.yml, there were randomly named config-XXXXX.yml files:

Key Evidence:

1. data.nat.tf resolves to 43.156.17.202 (my master), confirming config.yml is legitimate
2. 207.58.173.192:8088 is currently unreachable, but the config and process still exist
3. A malicious agent process (PID 911, config-etp0h.yml → 207.58.173.192:8088) was running on the master
4. All 34 malicious config filenames had random 5-character suffixes, indicating automated batch deployment

Why This Is More Dangerous Than sysmon

This was the true root cause of recurrence: The attacker didn't deploy an independent binary, but used the legitimate nezha-agent binary + an additional config file. Even after cleaning sysmon, the attacker could still maintain remote command execution capability on 34 servers through the legitimate nezha-agent + malicious config.

7. Second Round Cleanup: Second Config Backdoor

Cleanup Logic

# 1. Find all config-*.yml (malicious configs with random suffixes)
for cfg in /opt/nezha/agent/config-*.yml; do
  # 2. Find and disable corresponding services
  for svc in $(systemctl list-units | grep nezha-agent-); do
    svc_cfg=$(systemctl cat "$svc" | grep -oE "/opt/nezha/agent/config[^ ]*\.yml")
    if [ "$svc_cfg" = "$cfg" ]; then
      systemctl stop "$svc"
      systemctl disable "$svc"
      rm -f "/etc/systemd/system/$svc"
    fi
  done
  # 3. Kill processes
  pkill -9 -f "$(basename $cfg .yml)"
  # 4. Delete config
  rm -f "$cfg"
done
systemctl daemon-reload

# 5. Harden legitimate config.yml
sed -i 's/^disable_command_execute:.*/disable_command_execute: true/' /opt/nezha/agent/config.yml
systemctl restart nezha-agent.service

Cleanup Results

• 37 instances successfully cleaned: 35 malicious configs + 35 malicious services removed
• 3 timeouts: raksmart-SanJose, YunxiHuanjing, geelinx-b
• All successfully cleaned servers: AFTER state shows only config.yml + nezha-agent.service, network checks all CLEAN

8. Deep Re-Audit

After cleanup, I performed a 19-point deep IOC scan on all 40 servers:

1. sysmon backdoor files
2. ld.so.preload injection
3. Second config backdoor
4. Legitimate config hardening status
5. Nezha service list
6. Cryptominer processes
7. sysmon-related processes
8. Attacker network connections
9. Cron persistence
10. Suspicious systemd services
11. SSH authorized_keys
12. Login records
13. SUID files
14. Hidden files in /tmp
15. Suspicious timers
16. CPU usage
17. Abnormal listening ports
18. rc.local/profile.d
19. Kernel modules

Results

Conclusion: No recurrence of sysmon backdoor or second config backdoor. No new trojans or hacker remnants found.

9. Automated Hardening

1. Disable Remote Command Capability

All 34 Nezha nodes' config.yml have been set:

disable_command_execute: true   # Disable remote command execution (terminal/webssh)
disable_nat: true               # Disable NAT traversal

2. Dashboard Auto-Update (Watchtower)

Deployed Watchtower container for automatic dashboard updates:

docker run -d \
  --name watchtower \
  --restart always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --cleanup \
  --interval 3600 \
  nezha-dashboard

• Checks ghcr.io/nezhahq/nezha:latest for new images every hour
• Automatically pulls and restarts the nezha-dashboard container when a new version is available
• Only monitors nezha-dashboard, does not affect other containers
• After a new GitHub release, auto-updates within at most 1 hour, no manual intervention needed

3. Agent Auto-Update

Modified agent config:

disable_auto_update: false      # Enable auto-update
disable_force_update: false     # Allow master to force updates
disable_command_execute: true   # Keep remote command disabled

After a new agent version is released on GitHub, each node updates automatically, no manual intervention needed.

10. Root Cause Analysis of Recurrence

Backdoor reappearing after cleanup is usually due to one of the following:

1. ★Second config backdoor not cleaned (most stealthy): Cleaned sysmon but didn't check for config-XXXXX.yml; attacker maintains control via legitimate nezha-agent + malicious config. This was the true root cause of recurrence in this incident.
2. Residual self-healing chain: Any remnant of sysmon-guard.timer / cron / ld.so.preload / hidden script
3. Attacker still has login access: Any leaked root password, SSH key, panel account, Agent Secret, or JWT Secret
4. Nezha remote command not disabled in time: Attacker obtained panel token during exposure of low version, can continue issuing commands
5. Other panel vulnerabilities on the same machine: 1Panel / Baota / x-ui / MCSManager / phpMyAdmin / MySQL / Redis exposed to the public
6. Wrong cleanup order: Deleted files without stopping the guardian first → guardian rebuilds files
7. Credentials not rotated: Root password / SSH key not changed after cleanup

11. Lessons Learned

1. Don't Check for Only One Type of Backdoor

The sysmon backdoor has obvious characteristics (independent path + ld.so.preload), making it easy to find and clean. But the second config backdoor is extremely stealthy (legitimate binary + mixed in config directory). After cleaning sysmon, you must re-audit for config-XXXXX.yml.

2. Cleanup Order Matters

Stop guardian → Kill processes → Delete files → Clear ld.so.preload → daemon-reload

If the order is wrong, the guardian will rebuild the files, making the cleanup useless.

3. Pitfall in Reading Nezha Versions

host.version from the public WebSocket is null under guest view; you must use an admin JWT to see the real version. This can lead to misjudging "all agent versions unknown" and delaying the discovery of old-version nodes.

4. Windows SSH Chinese Paths

OpenSSH on Windows handles private key paths containing Chinese characters/spaces unreliably. You must copy to an ASCII temporary path + use icacls to tighten ACLs + use the system's built-in C:\Windows\System32\OpenSSH\ssh.exe.

5. Use Paramiko for Batch Operations

When the system lacks sshpass/expect, paramiko is the best choice for batch SSH operations. Passwords are kept only in memory, never written to disk.

6. Auto-Update Is Important

The root cause of this incident was the exploitation of low-version vulnerabilities. After deploying Watchtower and enabling agent auto-update, future GitHub releases will be automatically updated, eliminating the need for manual intervention.

7. Security Hardening Checklist

• Upgrade Dashboard to latest version (2.2.3)
• Upgrade all Agents to latest version (2.2.2)
• Clean sysmon backdoor
• Clean second config backdoor
• Disable remote command execution (disablecommandexecute: true)
• Deploy Watchtower for Dashboard auto-update
• Enable Agent auto-update
• Rotate root password
• Rotate SSH key
• Rotate Nezha panel password
• Rotate Agent Secret
• Rotate JWT Secret
• Close unnecessary public panel access

12. Open Source Tools

The cleanup scripts developed during this incident have been open-sourced:

• Repository: https://github.com/motao123/clean_malware_and_harden_nezha
• Full Cleanup Script: clean_malware_and_harden_nezha.sh (covers three types of backdoors)
• Lightweight Version: manual_clean_second_config.sh (cleans only the second config backdoor)

One-Click Usage

curl -fsSL https://raw.githubusercontent.com/motao123/clean_malware_and_harden_nezha/main/clean_malware_and_harden_nezha.sh -o /root/clean.sh
bash /root/clean.sh

Mass Check Commands

If you want to check if your own servers are compromised:

# sysmon backdoor check
ls -ld /usr/local/sysmon /usr/local/.sysmon-guard /usr/local/lib/libsysmon.so /etc/systemd/system/sysmon* /etc/ld.so.preload 2>/dev/null

# Second config backdoor check
ls /opt/nezha/agent/config-*.yml 2>/dev/null && echo "WARNING: second config backdoor detected"

# Cryptominer check
ps auxww | grep -Ei "sysmon|kworker|kdevtmpfsi|kinsing|xmrig" | grep -v grep

# Attacker connection check
ss -tnp | grep -E "51.254.44.35|207.58.173.192"

13. Incident Timeline

2026-06-19  Attacker exploits low-version vulnerabilities to deploy sysmon backdoor + second config backdoor
2026-06-20 00:30  Discovered Nezha security advisories, began investigation
2026-06-20 00:45  Confirmed master dashboard 2.2.3 unaffected, 5 agents at 1.9.7
2026-06-20 00:51  Discovered sysmon backdoor on master, began cleanup
2026-06-20 01:00  sysmon backdoor cleanup completed (master)
2026-06-20 01:15  Batch cleanup of sysmon backdoor on 40 instances
2026-06-20 01:30  Batch upgrade of old agents to 2.2.2
2026-06-20 02:00  hostyun-us IP changed, manual fix applied
2026-06-20 10:25  First recurrence audit: sysmon 0 recurrence, but discovered 34 instances with second config backdoor
2026-06-20 11:00  Located recurrence root cause: second config backdoor (config-XXXXX.yml → 207.58.173.192:8088)
2026-06-20 11:47  Batch cleanup of second config backdoor (37 instances successful)
2026-06-20 11:52  Updated cleanup script and pushed to GitHub
2026-06-20 12:00  Deep audit of 40 instances (all 19 checks passed)
2026-06-20 12:07  Deployed Watchtower + enabled auto-update
2026-06-20 12:13  Wrote this document

Summary

The core lesson from this incident: Cleaning one type of backdoor does not mean all backdoors are cleaned. The sysmon backdoor has obvious characteristics and is easy to find, but the second config backdoor uses a legitimate binary mixed in the config directory, making it extremely stealthy. If not for the deep audit, the attacker could still maintain remote command execution capability on 34 servers through the legitimate nezha-agent.

Security is a continuous process, not a one-time operation. Cleaning backdoors is only the first step; you also need to: upgrade versions, disable remote capabilities, enable auto-updates, rotate credentials, and close unnecessary public access points. Only full-chain hardening can prevent future intrusions.

This document is based on a real incident. Related cleanup tools have been open-sourced: https://github.com/motao123/clean_malware_and_harden_nezha

Original article: https://github.com/motao123/clean_malware_and_harden_nezha/blob/0772009f3980f8fa3410fe980cd4ab3f3cbd0681/incident-report.md

https://lenghang.com/archives/nezha-jian-kong-xi-tong-ru-qin-shi-jian-ying-ji-xiang-ying-shi-lu