The short version: For most of one day this week, websites on two of our servers kept blinking offline for a minute or two at a time and then recovering on their own. It was not a hack and it was not an attack. A popular Joomla add-on shipped an important update that day, and a lot of site owners went to install it around the same time. The way that update installed itself happened to disagree with one behind-the-scenes speed setting on our servers. Turning that setting off made the problem stop completely.
If you had a site with us that day, here is what you might have seen: a page that was slow to load or timed out, then loaded fine when you tried again a minute later. Nothing was damaged, nothing was lost, and no visitor information was ever at risk. The sites were not "down" in the usual sense. They were waiting in line behind an update that could not finish.
What we changed is a small network setting that only affects how our servers talk to Cloudflare, the service that sits in front of the sites and delivers them quickly around the world. Visitors still get the same fast, modern connection they always did. The change only touched one internal hop that most people never see.
Read the full technical breakdown โ
What actually happened
Recently a popular Joomla extension released an important update. That was a good thing; it closed real security holes, so a lot of site owners moved to install it right away. But two side effects arrived together.
First, the downloads were slow for everyone, because so many sites were pulling the same update from the vendor at once.
Second, and this is the one that took sites offline: the installer had a subtle flaw in how it protected itself from running twice at the same time.
The symptom that didn't add up
The servers looked idle. CPU was low, memory was fine, and yet sites would not respond. When we looked at the PHP worker processes (the small programs that actually build each page), dozens of them were sitting asleep, using no CPU, waiting on nothing we could see, and never finishing. Restarting the web server did nothing. Restarting the PHP process manager cleared it instantly, and then it would slowly fill up again.
That pattern (lots of stuck idle workers, almost no load, fixed only by restarting PHP-FPM specifically) is the fingerprint of processes blocked on a lock.
The lock that never let go
To avoid corrupting itself if two installs ran at once, the installer took an exclusive file lock before doing its work, something like this:
$lock = fopen($lock_file, 'c');
flock($lock, LOCK_EX); // wait here until the lock is free, forever if needed
// ... then unpack and install a fairly large package ...The problem is in that comment. LOCK_EX on its own waits forever. There is no timeout, and no "give up if this takes too long" option. On an idle machine that is harmless, because the first install finishes in about a second and releases the lock. On a busy machine during a global update rush, the install took long enough to cross a different limit: Cloudflare gives an origin server 100 seconds to respond before it returns a "524" timeout to the browser.
So the sequence that took sites down was:
- Someone starts an update. The package is large and the server is busy, so it runs long.
- At 100 seconds Cloudflare gives up and shows a timeout page, but the install is still running on the server, still holding the lock.
- The person understandably refreshes or clicks update again. That second attempt reaches
flock()and waits for a lock the first attempt still holds. - Every retry adds one more PHP worker stuck waiting forever. They pile up until no free workers remain, and now every site on that server is waiting in the same line.
Where HTTP/2 came in
Turning off one setting, HTTP/2 between our servers and Cloudflare, is what actually stabilized everything, so it is worth explaining why a stuck lock cared about a network protocol.
HTTP/2 can carry many requests down a single connection at the same time (it is called multiplexing, and for normal traffic it is genuinely faster). During this incident that speed worked against us. It delivered all those update retries and background admin requests to the PHP layer at once, so the pile of stuck workers grew faster than it could ever drain. Switching that internal hop back to HTTP/1.1, which sends requests more one at a time, slowed the pile-up enough that it stopped happening. It did not fix the underlying lock, but it removed the amplifier.
# /etc/apache2/conf.d/http2.conf
Protocols http/1.1followed by a graceful reload. On a server where we cannot edit that config directly, the same result comes from one toggle in Cloudflare: Network โ "HTTP/2 to Origin" โ Off.
One important point: this only changes the connection between our servers and Cloudflare's network. Visitors still reach Cloudflare over HTTP/2 and HTTP/3 exactly as before, so there is no speed loss for anyone browsing the sites.
What we passed along
Because this was a flaw in someone else's software rather than in our servers, the change on our end is a stopgap. We wrote up the full mechanism and sent it to the extension's author, with a suggested fix: have the installer wait a bounded amount of time for the lock and then fail with a clear message, instead of waiting forever.
$tries = 0;
while (!flock($lock, LOCK_EX | LOCK_NB)) {
if (++$tries > 30) {
throw new RuntimeException('Another installation is already in progress. Please wait and try again.');
}
sleep(1);
}What we took away from it
Three things worth remembering, whether you run one Joomla site or a hundred:
- When a slow install times out in the browser, do not immediately retry it. The install is very likely still running on the server. Wait a couple of minutes and reload the extensions list to see whether the version already updated. Retrying is what stacks the problem.
- When many unrelated sites on different servers all slow down at the same moment, look outward before you look at your own configuration. A shared dependency (an update server, a CDN, a release everyone is pulling at once) is a likelier culprit than something on your own box.
- Restart the right thing. Restarting the web server is the reflex, but when the stuck component is the PHP process manager, only restarting that clears it.
$lock = fopen($lock_file, 'c');
flock($lock, LOCK_EX); // wait here until the lock is free, forever if needed
// ... then unpack and install a fairly large package ...
Add comment