The short version: For about five days this past week, websites hosted on our shared servers were occasionally throwing random errors — a photo wouldn't load, a page would refuse to come up, the editor in a site's admin area would refuse to save an article. The failure rate was around one in every ten requests, which is high enough to be irritating but low enough that nobody could point at it and say "the site is down." We tracked it down today to a routine Apache web-server software update, released on May 8, which accidentally exposed a pair of long-dormant security rules to a subtle bug in how the new version handles modern browser connections. The fix took two rounds — disabling the obviously-firing rule cleared most of the noise, then a sibling rule the first one had been masking became the new active denier and had to be disabled as well. After the second round, every site on the affected servers is back to normal.
If you're a Cybersalt hosting client and your site has been feeling a little off this past week — pictures sometimes missing, an article refusing to save, a page randomly showing a 500 error — that was this. It's resolved now. No data was lost, and there's nothing on your end you need to do.
Read the full technical breakdown ↓
What we did about it
The cause was a pair of security rules that, until last week, had been silently doing nothing useful for years. The rules were originally written to catch a very specific class of attack against web servers — but a small change in how the new Apache version internally handles modern web traffic caused them to start mistakenly catching ordinary browser requests as well. Whichever request happened to land on the affected internal path would be denied with a 500-error response.
The fix took two rounds. We disabled the first rule across both of our shared hosting boxes and reloaded the web server, and the obvious symptoms cleared. But a sibling rule — sitting thirteen lines below the first one in the same configuration file, doing essentially the same job in a slightly more paranoid way — had been masked all along by the first rule firing earlier. With the first rule disabled, the sibling took over as the active denier, and a real visitor of one of our client sites hit a failed page load before we realized the problem wasn't fully resolved. Once we identified the second rule, the same kind of fix applied — disable, reload — and everything came back clean. We then reviewed every other rule in the same configuration file to make sure no further siblings were lurking.
Going forward, we're keeping a closer eye on the cPanel-managed Apache update channel. Routine cPanel updates do not normally cause this kind of trouble — this one was unusually subtle because the bug only surfaces on a specific request-handling path and only when one of a dozen-or-so security rules is active. But "routine" is not the same as "harmless," and we'll flag the next big Apache version bump for proactive testing rather than waiting for client reports. We'll also pair-check vendor rulesets more carefully — a rule whose own author calls it "redundant paranoia" sitting right next to the rule it duplicates is exactly the kind of thing that's easy to miss on a first pass.
If you noticed anything specific that still doesn't feel right with your site, give us a shout — we'd rather hear about it than have it linger.
The technical breakdown
The rest of this post is for fellow web-server and security folks who want the receipts. If you came for the summary, you're done — thanks for reading.
The signal
The first reports came in as three unrelated-looking incidents over a couple of days:
- One client's Joomla admin editor stopped saving articles cleanly. Sometimes the Save button worked, sometimes it spun and threw a JavaScript console error.
- Another client's site, on a different one of our shared servers, reported missing photos and modules that appeared to be unconfigured (they weren't — they just hadn't rendered).
- Sporadic browser-console errors on Joomla
/administrator/login pages —core.min.jsfailing to load, then a cascade of "Joomla is not defined" follow-ons.
What tied the three together: every symptom was intermittent, not consistent. Refresh the page and it would work. Refresh again and a different asset would 500. Visit the admin login and the page would render but the JS dependency chain would be missing a piece.
The audit log
A grep against /etc/apache2/logs/error_log for the affected hostname revealed dozens of ModSecurity denials all firing on a single rule:
[Wed May 13 ...] [security2:error] [pid ...:tid ...] [remote 2001:569:...:0]
ModSecurity: Access denied with code 500 (phase 2).
Match of "rx ^$" against "REQUEST_HEADERS:Transfer-Encoding" required.
[file "/etc/apache2/conf.d/modsec2.liquidweb.conf"] [line "179"]
[id "340004"] [rev "1"] [msg "Dis-allowed Transfer Encoding"]
[severity "CRITICAL"] [hostname "..."] [uri "/media/system/js/core.min.js"]The rule, 340004, is part of the LiquidWeb-shipped ModSecurity ruleset that comes pre-installed with cPanel/WHM. Its job is to deny any request whose Transfer-Encoding request header is non-empty. That's a reasonable rule — Transfer-Encoding is a hop-by-hop header used for chunked request body framing, and a normal browser GET for a static JS file doesn't (and shouldn't) carry one. Historically the rule only caught actual HTTP smuggling attempts and misbehaving proxies.
The clue that tied the room together
Every audit-log entry showed the source port as :0.
That's the tell. A real TCP connection has an ephemeral source port somewhere in the 12000–65000 range. When mod_http2 dispatches HTTP/2-multiplexed streams to downstream Apache modules, it reports the source port as :0 because the actual TCP port is shared across all streams on that connection. So every :0 in the audit log was an HTTP/2-multiplexed-stream request — and only HTTP/2-multiplexed-stream requests were getting denied. Plain HTTP/1.1 (and HTTP/2 single-stream) traffic was passing through fine. That explained the intermittent failure rate immediately: a single browser page load issues twenty-or-so asset requests across a multiplexed HTTP/2 connection, and the ones that happen to land on the affected internal path get denied.
It also made the timeline question concrete: when did mod_http2 change?
The timeline
2026-03-31 /etc/apache2/conf.d/modsec2.liquidweb.conf last modified.
(Rules 340004 and 300003 have both been live since at
least this date.)
2026-05-08 cPanel/EasyApache automatic update at 01:40 AM:
ea-apache24 2.4.66-4 → 2.4.67-1
ea-apache24-mod_http2 2.4.66-4 → 2.4.67-1
(...and the other ea-apache24-mod_* siblings.)
2026-05-09 First client-side symptoms — "feels off," JCE editor blips,
missing photos. Reports slow to escalate because the failure
rate is only ~10% per request.
2026-05-13 (morning)
Tied to ModSec rule 340004. First-round fix attempted via
`whmapi1 modsec_disable_rule` against both servers. WHM
rules-list UI shows rule as Disabled, audit-log 340004
hits stop. Apparent success — actually wasn't.
2026-05-13 (evening)
Residual symptoms still surface on the same kinds of pages.
Forensic audit reveals two real problems: (1) the whmapi1
call never persisted to disk (cPanel datastore showed
disabled_rules empty), so the "disable" was effectively
in-memory-only; (2) a sibling rule 300003 in the same file,
with the same Transfer-Encoding match pattern, had been
masked by 340004 firing first and now took over.
Fix re-done correctly: `SecRuleRemoveById 340004` and
`SecRuleRemoveById 300003` added to
/etc/apache2/conf.d/modsec2/exclude.conf on both servers,
hard Apache restart. Verified 10/10 success on HTTP/2
burst tests against canary URLs.The rule file sat unchanged for over five weeks before the Apache update. The Apache update sat live for five days before clients reported in. By the time the symptoms were visible enough to chase, the linkage to the May 8 update was non-obvious.
What changed in mod_http2 2.4.66 → 2.4.67
When mod_http2 receives an HTTP/2 stream and translates it into the internal request record that downstream modules (mod_security, the response handler, etc.) consume, it has to synthesize HTTP/1.1-style request headers. The 2.4.66 build did not set a Transfer-Encoding header on the synthesized record. The 2.4.67 build does — it sets it to chunked on certain request shapes as part of the framing translation, even though the original HTTP/2 stream didn't carry one. (HTTP/2 doesn't use a Transfer-Encoding request header at all — request body framing happens at the stream layer instead.)
Rule 340004 evaluates the request after mod_http2 has finished translating but before mod_security would normally know it was an HTTP/2-origin request. From mod_security's vantage point, the request "has" a Transfer-Encoding: chunked header, the rule's ^$ (empty-string) requirement isn't satisfied, and the request is denied with HTTP 500.
Neither side is "wrong" in isolation. Rule 340004 catches exactly what it was designed to catch. mod_http2 synthesizing a Transfer-Encoding header on internal translation is a defensible implementation choice. The two are just incompatible in this combination.
The fix that actually worked (after the first one didn't)
First attempt (didn't persist — flagged here so other ops folks avoid the same trap):
whmapi1 modsec_disable_rule config=modsec2.liquidweb.conf id=340004
/scripts/restartsrv_httpd --gracefulThe WHM UI shows the rule as Disabled with a strikethrough in the rules-list view, and the audit-log entries for that rule ID stop. Both of which made the change look successful. The audit revealed that /var/cpanel/modsec_cpanel_conf_datastore — the file the rebuild process actually consumes — still showed disabled_rules: {} empty. The whmapi1 call had clearly been processed in-memory but didn't write to disk. Whatever the exact mechanism, the disable wouldn't have survived the next config rebuild, and on its face it didn't even appear to be effective: rule 340004 stopped firing not because the disable took effect, but because the sibling rule 300003 was now firing on the same requests first and short-circuiting the rule chain. That detail came out only on closer audit.
Gotcha if you do use the whmapi1 approach anyway: the command requires both an id= and a config= parameter. Omit config= and it returns:
result: 0
reason: 'API failure: ... Provide the "config" parameter for the
"Whostmgr::API::1::ModSecurity::modsec_disable_rule" function.'Easy to miss because the failure mode is informative but the operator-typed command looked correct.
Second attempt (the one that actually fixed it):
cPanel ships an explicit user-override file for the LiquidWeb vendor ruleset — /etc/apache2/conf.d/modsec2/exclude.conf. The file is Include'd at line 748 of modsec2.liquidweb.conf, after all the vendor rule definitions, which is the right position for SecRuleRemoveById directives — they can only remove rules that have already been parsed earlier in the chain. exclude.conf already contained 152 working SecRuleRemoveById entries from past LW-rule overrides, which is how we knew the file was the right layer.
Two directives appended, each with an explanatory comment block:
# Disable LiquidWeb rule 340004 — Apache 2.4.67 mod_http2 synthesizes a
# Transfer-Encoding header on HTTP/2 internal stream translation, which
# rule 340004 ("Dis-allowed Transfer Encoding") was never designed to
# handle. Must live in exclude.conf, NOT custom.conf — custom.conf is
# included at line 23 of modsec2.liquidweb.conf (before rule 340004 is
# defined at line 179), so a SecRuleRemoveById there is a silent no-op.
# exclude.conf is included at line 748 (after rule defs).
SecRuleRemoveById 340004
# Disable LiquidWeb rule 300003 — sibling of 340004. Same root cause:
# Apache 2.4.67 mod_http2 synthesizes Transfer-Encoding: chunked on
# HTTP/2 internal stream translation, and 300003 matches the literal
# "chunked" pattern while 340004 matches non-empty. LW's own comment
# at line 191 says "the rule before this one should cover this, but
# hey paranoia is cheap" — i.e. 300003 is redundant once 340004 is
# in place.
SecRuleRemoveById 300003Followed by a hard restart of Apache:
/scripts/restartsrv_httpd(Hard restart rather than --graceful because modsec rule-loading state, in our experience on this build, is more reliably refreshed by a full stop/start cycle.)
Don't edit the vendor file directly. /etc/apache2/conf.d/modsec2.liquidweb.conf is owned by LiquidWeb's ruleset-refresh mechanism — any hand edit gets reverted the next time LW pushes an update. exclude.conf is cPanel's user-override layer for the same ruleset and persists through refreshes.
Include-order matters. First exclude-attempt during this incident actually went into /etc/apache2/conf.d/modsec2/custom.conf rather than exclude.conf. custom.conf is Include'd at line 23 of modsec2.liquidweb.conf — before any of the vendor rule definitions. A SecRuleRemoveById directive there is a silent no-op against any later-defined rule. Took longer than it should have to spot because there's no error message — the directive parses fine, it just doesn't apply.
Verification
After the second-round restart, on each server:
grep -c 'id "340004"' /etc/apache2/logs/error_logagainst fresh log entries returned zero.grep -c 'id "300003"' /etc/apache2/logs/error_logagainst fresh log entries returned zero.- Ten-request HTTP/2 burst tests against three canary asset URLs (the exact URLs that had been failing intermittently before) —
media/system/js/core.min.json two affected sites,media/vendor/accessibility/js/accessibility.min.json a third — returned 10/10 200 responses on every burst. - Pre-fix, the same test had reproduced the ~40% failure rate consistently. Post-fix, repeated bursts (HTTP/2 and HTTP/1.1) all clean.
Lessons filed away
Routine ≠ harmless. cPanel/EasyApache updates land automatically and we generally trust them to be benign. This one was — but its interaction with a separate vendor's ruleset wasn't. The next major Apache version bump is now on our "test against the full ModSec ruleset before assuming it's quiet" list.
The intermittent-failure tax is real. A 10% failure rate is more confusing to chase than a 100% failure rate. Clients adapt around it without realizing they're adapting — they refresh and it works, so they don't report it — which delays the trigger that would surface the problem. Worth keeping a low threshold for "feels weird, multiple unrelated reports in a short window" as a category-1 signal.
:0 in a mod_security audit log is the HTTP/2-stream tell. When the same rule fires only on HTTP/2 traffic and never on HTTP/1.1 traffic, you'll see it in the source-port field before you'll find it anywhere else. Worth committing to muscle memory.
whmapi1 modsec_disable_rule is not always a persistent disable. The WHM rules-list UI showing a rule as Disabled does not mean the disable lives on disk — verify by inspecting /var/cpanel/modsec_cpanel_conf_datastore after the call. If the disabled_rules map there is empty, the disable didn't actually write through. The reliable mechanism is SecRuleRemoveById in /etc/apache2/conf.d/modsec2/exclude.conf, which is part of cPanel's user-override layer and persists through every vendor refresh.
SecRuleRemoveById is order-sensitive. It only removes rules that have already been parsed earlier in the include chain. On a stock cPanel build, that means custom.conf (included before vendor rule definitions) is the wrong location and exclude.conf (included after) is the right one. The parser doesn't error on a too-early SecRuleRemoveById — it just silently no-ops, which is the worst kind of failure mode to debug.
When a vendor's own inline comments call a rule "redundant paranoia" against the same condition another nearby rule covers, disable the pair, not just one. LW's "paranoia is cheap" comment above rule 300003 was the explicit tell that 300003 and 340004 are a redundant pair. With 340004 disabled, 300003 became the active denier and the symptoms returned within hours. A pair-aware audit at first-fix time would have saved a round trip.
HTTP/2 vs HTTP/1.1 is the unambiguous mod_http2-regression diagnostic. When you've got intermittent 500s on a recently-updated Apache box, the very first test to run is curl --http1.1 <failing-url> against curl <failing-url> (which defaults to HTTP/2 over TLS) — a clean 100% vs ~40% split between the two protocols pins the cause to the HTTP/2 stream-translation path and saves you from chasing rabbit holes in client templates, browser caches, or CDN configurations. Cheaper than every other diagnostic.
← Back to the plain-English summary
Add comment