The short version: A website on one of our shared servers had been getting hacked over and over for months. Every time we cleaned it up, the attacker was back within hours. We finally caught the channel they were using to get back in — a Web Disk account on the cPanel that had been planted a year ago, completely separate from the cPanel password we'd been resetting and the 2FA we'd been enabling. The site is now clean, the door is closed, and the lessons from this one are worth sharing.
First, an important piece of context: the website in question is on a hosting account that was migrated to Cybersalt from another host. The customer came to us because they wanted better security and more attentive management — they'd been getting hacked at their previous host and assumed moving to us would solve it. The cPanel account was isolated on its own under our usual setup (a shared cPanel server, but every account in its own user, its own file space, its own resources), and over the months we'd been helping them clean and re-clean it. So this story is not "Cybersalt's server got hacked." It's "Cybersalt inherited a quietly-compromised account and finally, today, found the persistence channel the attacker had left behind from before we ever had it."
We'd been at this for months. Every time we found the latest round of malicious files, we cleaned them up, rotated the relevant passwords, hardened the configuration a little further, and watched the site for a few days. It would behave. Then, sometimes within hours and sometimes a week or two later, fresh malware would appear and the same cleanup cycle would start over.
Along the way we tried just about everything you'd think to try. We reset the cPanel main account password — twice. We reset the email-account passwords. We audited the website's content-management system for unauthorized administrator accounts and for unfamiliar API tokens. We deployed a file-modification trap meant to catch the attacker the next time they wrote anything. We had the host's daily malware scanner running and surfacing every webshell it could see. The customer turned on two-factor authentication for their cPanel account. Each one of those steps made sense given what we knew. None of them stopped the re-hacks.
By the time we got to the morning of this incident, "what are we missing?" was the only honest framing left.
Read the full technical breakdown ↓
What we did about it
- We set up a small monitoring script that watches the website's main
index.phpfile and emails us the moment its fingerprint (technically, its MD5 hash) changes. The idea was deliberately simple: instead of finding out about a re-hack hours or days after the fact, we'd catch the attacker in the act and have a fresh trail to follow. - The alert fired one morning at 10:20. The file had been modified four minutes earlier. We replaced it with a clean copy as usual — but this time, before doing anything else, we pulled the website's access logs to see which request had caused the change.
- There was no such request. Zero website hits at the moment of modification. No FTP login. No SSH login. No cPanel login. The file had been changed silently, on a channel that didn't touch any log we'd been checking.
- That mismatch was the breakthrough. It told us the attacker wasn't coming in through the website at all. They were authenticating to a parallel cPanel service called Web Disk — a remote file-access protocol that lets a user mount their hosting space like a network drive — completely separate from the website and from the email accounts we'd been securing.
- We found a Web Disk user on the account that had been quietly sitting there for exactly one year. We deleted it.
- We cleaned up the files the attacker had dropped that morning, plus an older set of attacker files we discovered tucked inside a benign-looking subdirectory three weeks earlier. We restored the website's proper security configuration, and blocked the attacker's payload server at the network level so even any leftover code that tried to phone home now can't.
If you're a Cybersalt hosting customer reading this — your account isn't affected. This was one specific account where the attacker had a foothold from before the account moved to us, and the pattern hasn't surfaced anywhere else on our hosting since. The Web Disk audit we ran on the shared server hosting that account confirmed no other customer account on it has anything similar. If you'd like us to spot-check your own hosting setup or talk through what we found, get in touch.
The technical breakdown
The rest of this post is for fellow IT and security folks who want the receipts. If you came for the summary, you're done — thanks for reading.
Discovery
A few weeks before this incident, we'd had a more elaborate file-modification trap running on this same account. It used PHP's auto_prepend_file directive plus inotifywait to watch for any PHP process spawning at the moment of write, with chattr +i immutability flags on the most-targeted files as a backstop. It was clever, and it never fired once in six weeks. Three reasons, all of which only became obvious in hindsight: the watch surface was tied to a specific attacker pattern we'd inferred from an earlier cleanup; the immutability flag on index.php would have silently broken the next Joomla core upgrade; and the whole design assumed the attacker was writing files via PHP — which, as we'd find out, was the wrong assumption.
So a couple of weeks before today we replaced it with the dumbest possible monitor: a 20-line shell script, scheduled by cron every minute, computing the MD5 of index.php and emailing if the hash changed. No clever predicates, no assumptions about how the file was being modified. Just "did this file change since the last check? Yes? Tell me." That turned out to be exactly what we needed.
The watcher fired at 10:20:01 in the morning. The file modification time was 10:15:41 — four minutes earlier. The cron interval and a small queueing delay accounted for the gap. The new file on disk was the standard cloaking shell the attacker had been planting on this site for months: 22 KB of obfuscated PHP whose only job was to serve a Chinese counterfeit-e-commerce page to search-engine crawlers while leaving normal visitors with the real site.
The change we made compared to every previous cleanup: instead of replacing the file with a clean copy and moving on, we preserved the malicious copy and went straight to the access logs.
Here's the watcher itself, genericized for re-use. Adjust the mail line for whatever notification channel you actually have wired up — local sendmail, a Mailgun/Postmark cURL POST, an ssmtp config, whatever you trust to deliver an alert at 3 a.m. The mail call is the only host-specific bit.
#!/bin/bash
# Cron every minute. Email when index.php's MD5 changes.
WATCHFILE="/home/USERNAME/public_html/index.php"
HASHFILE="/root/.USERNAME_index_md5"
ALERT_EMAIL="This email address is being protected from spambots. You need JavaScript enabled to view it. "
CURRENT_HASH=$(md5sum "$WATCHFILE" | awk '{print $1}')
if [ ! -f "$HASHFILE" ]; then
echo "$CURRENT_HASH" > "$HASHFILE"
exit 0
fi
if [ "$CURRENT_HASH" != "$(cat "$HASHFILE")" ]; then
{
echo "index.php has changed."
echo
stat "$WATCHFILE"
echo
echo "Recent access log (last 50 lines):"
tail -50 /home/USERNAME/access-logs/example.com-ssl_log
} | mail -s "ALERT: index.php changed" "$ALERT_EMAIL"
echo "$CURRENT_HASH" > "$HASHFILE"
fi
Findings
Step 1 — no matching request. The first thing we did was align the file's modification timestamp with the access logs. The file had changed at 10:15:41 EDT. We checked the SSL access log, the plain-HTTP access log, the FTP transfer log (/var/log/xferlog), SSH and SFTP authentication records in /var/log/secure, and the cPanel UI's own access log. Nothing matched. No request to the website, no upload session, no login of any kind landed in that one-minute window.
This was the moment the investigation pivoted. Every cleanup we'd done for months had been built on the unstated assumption that an attack ends with a request — somewhere — to the website. That assumption had just broken.
Step 2 — narrow the search. We listed every recently-modified PHP file or .htaccess on the account, filtering out the noise from Joomla's own cache and log directories:
find /home/<account>/public_html -type f \( -name '*.php' -o -name '.htaccess' \) -mtime -2 \
| grep -vE '/cache/|/logs/'
That command returned seven files — and every one of the seven was attack-related. Two webshells (a small file-manager and a 98 KB obfuscated PHP shell), one cloaking script, and four stripped-to-minimum .htaccess files in three different web roots. Without the -mtime -2 window and the cache/log filter, this same find on a freshly-upgraded Joomla site would have returned roughly 1.6 MB of legitimate core files and buried the signal completely. The filter is the trick.
Step 3 — the timestamps gave it away. The three .htaccess files lived in three different web roots — the production vhost, a development subdomain, and a test subdomain. Their creation timestamps matched to the microsecond. Identical down to the sub-second. No human types that fast across three directories, and no web request can write to three separate vhost roots at once unless the writer has filesystem-level access at the cPanel-user layer — not just permission to upload through one site.
That was the second signal that we weren't dealing with a normal web attack.
Step 4 — the WebDAV lock file. The third signal came from a file we'd never had occasion to look at before:
/home/<account>/.cpanel/webdav_locks.bin
Modified at 10:17:00 EDT — 36 seconds after the malicious index.php write. That file gets touched any time a WebDAV session completes a lock operation. Its mtime is one of the few residues a Web Disk session leaves on the account.
Step 5 — the planted user. The last step was confirmation. We looked at the per-account WebDAV password store:
sudo cat /home/<account>/etc/webdav/passwd
sudo cat /home/<account>/etc/webdav/shadow
A single Web Disk user. The name was deliberately bland — the kind of name you'd skim past on a list. Read-write access to the entire public_html. SHA-512 password hash. The passwd and shadow files were both stamped with a modification date one year prior to the day we found them — a quiet, year-old foothold that had outlived every credential rotation, every Joomla upgrade, every malware scan, and the trap we'd built specifically to catch the next attack on this account.
Timeline of the day, reconstructed from the logs and file mtimes:
~1 year prior Web Disk user planted on the account
~3 weeks prior First cloak-and-shell drop in a benign-looking subdirectory
— survived every subsequent cleanup
Day-of 01:41 .htaccess in 3 vhost roots rewritten in parallel (WebDAV)
Day-of 08:21 Two webshells dropped in the document root (WebDAV)
Day-of 08:21 Attacker connects via normal HTTPS from an offshore IP
to verify the shells respond
Day-of 09:04 Older beachhead's cloak script rewritten with the
current payload (WebDAV)
Day-of 10:15:41 Root index.php rewritten — this is what tripped the
file watcher (WebDAV)
Day-of 10:16 Attacker GETs the file-manager webshell, confirms
persistence
Day-of 10:16+ Search-engine crawlers start receiving the 347 KB
cloaked HTML
Day-of 10:17 Final WebDAV lock release — webdav_locks.bin mtime
updates
Day-of 10:20:01 File watcher emails the alert
Why this defeated every prior protection
Four reasons the Web Disk channel was so resistant to everything we'd been doing:
- Separate password store. Web Disk authenticates via the
cpdavddaemon against the per-account file~/etc/webdav/shadow. That hash is completely independent of the cPanel main account password. Rotating the cPanel password — which we'd done multiple times — leaves the Web Disk shadow untouched. - 2FA doesn't apply. On this CloudLinux build of cPanel, the 2FA gate protects the cPanel UI login (port 2083) but does not enforce on
cpdavd. The customer had 2FA enabled correctly. It just wasn't on the channel the attacker was actually using. - No per-domain access log.
cpdavdwrites its session activity to/usr/local/cpanel/logs/cpdavd_session_log, not to the per-domain/usr/local/apache/domlogs/<domain>log. Every forensic check we'd done over months had been on the per-domain logs — and the activity wasn't there because it was never going to be there. - The cpdavd session log doesn't record usernames. Even if you do look at the
cpdavdsession log, it shows source IP + session ID + ports — but not which Web Disk user authenticated. So even targeted forensics on the right log file can't directly tie a session to a specific user. You can infer from timing, but you can't prove it from the cpdavd log alone.
The combination of those four properties is what made this so hard to find: every channel we'd checked came up empty for the same reason — the attacker was on a fifth channel.
Remediation
# 1. Forensic snapshot before anything destructive
mkdir -p /root/<account>-webdav-incident-$(date +%F)
cp -p /home/<account>/etc/webdav/passwd \
/home/<account>/etc/webdav/shadow \
/home/<account>/.cpanel/webdav_locks.bin \
/root/<account>-webdav-incident-$(date +%F)/
tar czf /root/<account>-webdav-incident-$(date +%F).tar.gz \
-C /root <account>-webdav-incident-$(date +%F)/
# 2. Remove the Web Disk user — all four artifacts
> /home/<account>/etc/webdav/passwd
> /home/<account>/etc/webdav/shadow
rm -f /home/<account>/.cpanel/webdav_locks.bin
rm -f /home/<account>/etc/webdav/@pwcache/<user>
# 3. Remove the dropped webshells and any cloak directories
rm -f /home/<account>/public_html/<shell-names>
rm -rf /home/<account>/public_html/<cloak-dir>/
# 4. Restore the proper .htaccess from the platform's stock template
cp -p /home/<account>/public_html/htaccess.txt \
/home/<account>/public_html/.htaccess
chown <account>:<account> /home/<account>/public_html/.htaccess
# 5. Block the cloak host server-wide (IPv4 AND IPv6)
printf '0.0.0.0 cloak-host.example\n::1 cloak-host.example\n' >> /etc/hosts
Two things in that cleanup script are worth calling out specifically — neither is obvious, and both have bitten us.
The @pwcache file is the fourth artifact. A surprising amount of online guidance for "delete a cPanel Web Disk user from the command line" says to truncate etc/webdav/passwd and etc/webdav/shadow and you're done. That's wrong. cPanel also caches the user's password hash in etc/webdav/@pwcache/<username> for performance, and that cache file survives the truncate-the-two-files approach. Until it's also removed, the deleted user can still authenticate. The WHM UI's "Delete Web Disk Account" button handles all four artifacts atomically — passwd, shadow, the lock file, and the pwcache entry. Doing it from the shell, you have to be explicit about all four. We learned this the hard way when an earlier manual cleanup on an unrelated account left the cache file behind and a follow-up audit caught it.
The IPv4-plus-IPv6 hosts pair is non-optional. Sinkholing a hostname via /etc/hosts with 0.0.0.0 works fine for an IPv4-only target. The moment the destination sits behind Cloudflare — or any modern CDN — the hostname resolves over both IPv4 and IPv6, and a single-stack hosts entry silently leaks across to the IPv6 path. Always pair the 0.0.0.0 line with a matching ::1 line for the same hostname, and verify with getent hosts <name>. The expected output is two loopback addresses and nothing else. If any non-loopback address comes back, the block is still leaking.
What I learned / would do differently
Three carry-forward lessons:
SYMPTOMS ≠ CAUSE. When a site keeps getting hacked despite credential rotation and 2FA, the working hypothesis should be: there's a second, silent authentication channel we haven't checked yet. Don't conclude "false alarm" from "I disproved each surface symptom" — a real compromise on a different vector is the default assumption until ruled out. On any future recurring-compromise report, the first commands now are:
sudo cat /home/<user>/etc/webdav/passwd
sudo cat /home/<user>/etc/webdav/shadow
sudo find /home/<user> -name webdav_locks.bin -printf '%TY-%Tm-%Td %TH:%TM %p\n'
sudo find /home/<user>/etc/webdav/@pwcache/ -type f 2>/dev/null
Server-wide Web Disk audits should be routine. After resolving the incident, we ran the equivalent of find /home -maxdepth 4 -path '*/etc/webdav/passwd' -size +0c on the shared cPanel server hosting the account. Two other customers on it had legitimate Web Disk users — both are confirmed legitimate, but the audit also surfaces the 2FA-bypass risk worth flagging to those customers: any Web Disk credential is functionally an SSH-key-equivalent shortcut into the file system, separate from cPanel's 2FA. A quarterly cron that diffs the live Web Disk user list against an allowlist and emails on changes is worth wiring up.
A simple MD5-on-cron file watcher is more useful than an elaborate trap. The earlier attempt at a smarter trap on this same site never fired once in six weeks because it watched for processes-spawning-PHP at the moment of write — which doesn't catch a write that doesn't go through PHP. The replacement 20-line script that just asks "did the hash change?" caught the next attack on the first try. Don't out-engineer the simple solution.
← Back to the plain-English summary
Add comment