# How we responded to an internet-wide web server vulnerability in 1 day

URL: https://www.pushwoosh.com/blog/nginx-http2-vulnerability-response/
Language: en
Published: 2026-11-20
Author: Pushwoosh Team
Categories: Pushwoosh news, Security & compliance

> When 2 critical nginx HTTP/2 vulnerabilities went public, we patched every internet-facing Pushwoosh nginx instance in 1 day. Here's how one engineer and AI agents did it.

On November 18, 2026, security researchers disclosed 2 critical vulnerabilities in nginx, one of the most widely used web servers on the internet. Every nginx release from the past decade is affected, so companies around the world running it with HTTP/2 enabled had to update.

The next morning, one of our DevOps engineers shared the [Wiz Research breakdown](https://www.wiz.io/blog/nginx-http2-rce-vulnerabilities) with the team and asked which of our systems needed the update. By the end of that working day, every internet-facing nginx instance serving the Pushwoosh platform was updated and verified. An internal audit the next day found no signs of exploitation.

Pushwoosh is an omnichannel customer engagement platform, and our customers' apps send API traffic through our edge around the clock. When a web server bug like this goes public, how quickly we close it matters to every one of them. Here is what was disclosed, what we did, how one engineer and a set of AI agents split the work, and what we learned.

## What was disclosed

| CVE | What it is | Affected nginx OSS | Fixed in |
| --- | --- | --- | --- |
| CVE-2026-123007 | Integer underflow in HTTP/2 flow control (`send_window`) | 1.9.5 to 1.31.8 | 1.30.7 (stable), 1.31.9 (mainline) |
| CVE-2026-123633 | Use-after-free of an HTTP/2 connection in reverse-proxy mode | 1.19.7 to 1.31.8 | 1.30.7 (stable), 1.31.9 (mainline) |

Both are rated critical with a CVSS 4.0 score of 9.2. The same release also closed 8 more CVEs in HTTP/3, the mail proxy, the DNS resolver and stream modules.

Exploitation is hard: it needs a memory leak and careful heap grooming, and Wiz did not publish its exploit code, although some security vendors soon reported public exploits. We treated it as urgent anyway. The bugs allow remote code execution without authentication, HTTP/2 was on across almost all our public endpoints, and TLS terminators keep private keys in memory. We decided to patch within 1 to 2 days instead of waiting for the next maintenance window.

👉 Full technical analysis: [NGINX HTTP/2 RCE vulnerabilities, Wiz Research](https://www.wiz.io/blog/nginx-http2-rce-vulnerabilities)

## Our response, step by step

### 1. Map the exposure first

Before touching anything, we swept the whole fleet in read-only mode: edge routing, TLS termination, internal services and the Kubernetes ingress controllers. For every instance we recorded the nginx version and whether HTTP/2 was on.

That gave us a priority list built on actual exposure. It also turned up a few internal hosts on nginx versions that were years old.

### 2. Find the hidden dependencies

Upgrading nginx is rarely just upgrading nginx.

Our builds ship with custom dynamic modules for metrics, geolocation and WAF, and nginx only loads a module built for its exact version. Every module had to be rebuilt for 1.30.7, on every CPU architecture we run.

ARM was the awkward part: our dedicated ARM build runner had been retired a week earlier. We moved those builds to emulated cross-builds in shared CI. Each one took up to 25 minutes instead of a couple, which was annoying, but it needed no new hardware.

### 3. Backport when an upgrade is not an option

Our Kubernetes ingress controller is built on a frozen upstream that pins its nginx version and carries dozens of its own patches. Moving it to nginx 1.30 would have been a separate project.

So we took the official security diff between nginx 1.30.6 and 1.30.7 and applied it to the controller's nginx. It went on cleanly over all the upstream patches, and we checked the one fuzzy spot by hand. The version string still says 1.27.1, while the HTTP/2 code underneath matches 1.30.7.

### 4. Ship through review, roll out by blast radius

Every change went through a merge request with automated code and security review: module packages, the install role, the ingress image, the version pins. Hosts managed by our playbooks were changed only through those merge requests.

We rolled out 1 host at a time, starting where a failure would hurt least. First came a single edge node in one data center, checked from outside over HTTP/2. Then the rest of that data center, the other data centers, TLS terminators and cloud hosts, with the access bastion last.

On every host the playbook ran the config test before and after, restarted nginx only if the binary had changed, and confirmed the new binary was running with all modules loaded, a working TLS handshake and a clean error log.

Kubernetes ingress went out through Helm. Before the apply we compared the live release against git and ran a server-side dry run, and the new controllers replaced the old ones one node at a time.

### 5. Close the long tail

A few hosts lived outside our main automation, mostly internal tools and VPN nodes on distro packages. Turning HTTP/2 off would have been the quick, officially recommended stopgap. Fixed packages turned out to exist for every OS version we use, so we upgraded these hosts to 1.30.7 as well, one at a time and with a config backup taken first.

They came with their own surprises. Old distro module packages had to go first, or the new nginx failed its config test. The old process kept serving traffic throughout, so nothing went down. A few nodes also needed a newer OpenSSL before the upgrade.

The find we're most glad about: 1 provisioning role had an old default nginx version and allowed downgrades. Its next routine run would have quietly rolled the fix back. A separate merge request now pins the version in code.

💡 After any manual fix, check that your own automation won't undo it.

## How the work was organized

1 DevOps engineer ran the response together with AI agents, each responsible for a narrow part of the process: execution, task routing, code review, releases and access approval. The agents handled the repetitive work that is easy to verify, such as discovery, builds, merge requests, rollout and post-change checks.

Decisions stayed with the engineer. Scope, risk and production access were his calls, and the scope only grew after his explicit go in the team thread.

Access followed the same model. Elevated access was granted per stage of the work, not per command, only after the engineer approved it, and it expired when the working window closed.

This split is what made a one-day turnaround possible: 1 person could cover the whole fleet, while every change still went through review and a human decision.

## The result

| When | What happened |
| --- | --- |
| Nov 18 | F5 releases fixes, vulnerabilities disclosed publicly |
| Nov 19, morning | Engineer shares the Wiz analysis, exposure mapping starts |
| Nov 19, midday | Modules rebuilt for all architectures, ingress image backported, changes reviewed and merged |
| Nov 19, afternoon | Staged rollout: edge, TLS terminators, cloud hosts, bastion, internal tools and VPN nodes |
| Nov 19, end of day | Every internet-facing nginx instance serving the Pushwoosh platform patched and verified |
| Nov 20 | Internal audit completed with no signs of exploitation, security notice published on the status page |

- More than 30 hosts moved to nginx 1.30.7 and checked live, including ARM machines.
- Ingress in both Pushwoosh production Kubernetes clusters runs on the backported image.
- An internal audit completed on November 20 found no signs of exploitation.

## What this means for Pushwoosh customers

No action is required on your side for the Pushwoosh platform, and our Mobile and Web SDKs are not affected. The details are in our [security notice on the status page](https://status.pushwoosh.com/announcements), and the full Security Advisory PW-SA-2026-003 is available on request at support@pushwoosh.com.

If your own systems run nginx, NGINX Plus or ingress-nginx with HTTP/2 enabled, we recommend updating to nginx 1.30.7, 1.31.9 or later.

## What we'd tell any team

1. **Know your exposure before you need it.** Our first hour went to answering "which systems are affected". With an up-to-date inventory of versions and settings, that takes minutes.
2. **Dynamic modules are a hidden dependency of every patch.** Budget rebuild time for each module and each architecture before you schedule the upgrade, especially for an architecture with no dedicated build runner.
3. **Backporting a security diff is a legitimate tool.** If a frozen upstream blocks the upgrade, applying the official fix to the version you already run can close the risk the same day.
4. **Make sure automation won't undo your fix.** Look for roles with a default version and downgrades allowed. One of ours would have reverted a day of work on its next routine run.
5. **Keep people on scope decisions, and keep access time-boxed.** 1 time-boxed access grant per stage, not per command, means fewer approval clicks, and the access disappears when the work is done.

## Protect customer trust with Pushwoosh

Responding to a critical CVE isn't about how fast someone can type. It's about having the inventory, the pipeline and the review process ready before the advisory lands.

We apply the same discipline to everything that runs through Pushwoosh, including your messages and user data. If you want details on how we handle security for your app, talk to our team.

Have a security question?
