Why the blog was slow: CPU throttling, not bandwidth

This blog felt sluggish right after we added an outbound bandwidth cap. The cap was innocent: Ghost was being CPU-throttled. Here's how we measured it and what fixed it.

Share

Right after we shipped the per-app outbound bandwidth cap, this blog started to feel slow. Pages took between half a second and 1.4 seconds before the first byte arrived. The timing made the new cap the obvious suspect. It wasn't.

Ruling out bandwidth

The quickest check was to time two requests through the same path. One was the home page, which Ghost renders on every request. The other was a static CSS file, which Ghost only has to send.

curl -s -o /dev/null -w "ttfb=%{time_starttransfer}\n" https://blog.picopaas.xyz/
curl -s -o /dev/null -w "ttfb=%{time_starttransfer}\n" https://blog.picopaas.xyz/assets/built/screen.css

The 54 KB stylesheet arrived in about 120 ms, which is roughly one network round trip. The home page is only 28 KB, but it took 450 to 1,400 ms. A 20 Mbit/s cap moves 28 KB in about 11 ms. Most of the wait was therefore Ghost building the page, not sending it.

Finding the real cause

Every container's CPU accounting is exposed in its cgroup. We read /sys/fs/cgroup/cpu.stat in the Ghost pod, loaded the home page eight times, and read it again.

  • Each page render used about 200 ms of CPU time.
  • Ghost's CPU limit was 250m, a quarter of a core.
  • The kernel throttled the container in 57 of the 71 scheduling periods during the test. That added roughly 350 ms of waiting to every page.

At a quarter of a core, a render that needs 200 ms of CPU spreads out across most of a second. The pod had also just restarted for the bandwidth rollout, so Node's JIT was still cold and every render cost a little more than usual. That made things worse, but it wasn't the root cause.

The fix

We raised Ghost's CPU limit to 500m using the new per-app admin override and left the request at 64m. The app can now burst while it renders a page without reserving more capacity on the node. The node was idling at about 10% CPU, so there was plenty of headroom. Time to first byte dropped to 0.2–0.45 s. There is still some throttling, so a higher limit would shave off a little more.

Takeaways

  • Measure before you blame the newest change. Compare a static file with a dynamic page on the same host. That one comparison tells you whether the time goes to the network or to the server.
  • cpu.stat is the ground truth for throttling. Look at nr_throttled and throttled_usec. Average CPU graphs hide this, because a throttled app can look nearly idle.
  • Server-rendered apps want burst room. Apps like Ghost do very little most of the time and then need CPU in short spikes. A low request with a higher limit suits them well. Admins can now set that per app without touching the platform defaults.