<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>EC2 on Fabio Lopes</title><link>https://fabiolopes.page/tags/ec2/</link><description>Recent content in EC2 on Fabio Lopes</description><generator>Hugo -- gohugo.io</generator><language>en-US</language><copyright>Fabio M. Lopes</copyright><lastBuildDate>Tue, 27 Oct 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://fabiolopes.page/tags/ec2/index.xml" rel="self" type="application/rss+xml"/><item><title>PHP2Shell on AWS: Incident Response and the Limits of Boot-Time Patching</title><link>https://fabiolopes.page/p/php2shell-aws-incident-response-lessons/</link><pubDate>Tue, 27 Oct 2026 00:00:00 +0000</pubDate><guid>https://fabiolopes.page/p/php2shell-aws-incident-response-lessons/</guid><description>&lt;p&gt;&lt;em&gt;How a critical PHP advisory became a cross-application incident response, and why the hardest lesson was about the way new servers are built.&lt;/em&gt;&lt;/p&gt;&#10;&#10;&#10;&#10;&lt;aside class="engineering-callout engineering-callout--note" aria-label="Scope of this account"&gt;&#10; &lt;p class="callout-label"&gt;Scope of this account&lt;/p&gt;&#10; &lt;div&gt;&lt;p&gt;This is a sanitized account of an operational response. Application names, customer identities, internal addresses, hostnames, and infrastructure identifiers have been omitted. The examples describe engineering patterns rather than a complete internal runbook. Observed actions are distinguished from future improvements.&lt;/p&gt;&#10;&lt;/div&gt;&#10;&lt;/aside&gt;&#10;&#10;&lt;h2 id="the-advisory-was-only-the-beginning"&gt;&lt;a href="#the-advisory-was-only-the-beginning" class="header-anchor"&gt;&lt;/a&gt;The advisory was only the beginning&#10;&lt;/h2&gt;&lt;p&gt;On October 21, 2026, the PHP project announced security releases addressing &lt;strong&gt;CVE-2026-109618&lt;/strong&gt; and &lt;strong&gt;CVE-2026-109619&lt;/strong&gt;, two critical vulnerabilities commonly referred to as &lt;strong&gt;PHP2Shell&lt;/strong&gt;. The vulnerabilities could enable unauthenticated remote code execution in affected deployments. The security updates included PHP &lt;strong&gt;8.4.27&lt;/strong&gt;, &lt;strong&gt;8.3.36&lt;/strong&gt;, &lt;strong&gt;8.2.35&lt;/strong&gt;, and &lt;strong&gt;8.5.12&lt;/strong&gt;. The PHP project warned that temporary mitigations were incomplete and updating was essential. See the &lt;a class="link" href="https://www.php.net/archive/2026.php" target="_blank" rel="noopener"&#10; &gt;PHP announcement&lt;/a&gt; and &lt;a class="link" href="https://github.com/NCSC-NL/php2shell" target="_blank" rel="noopener"&#10; &gt;NCSC-NL operational information&lt;/a&gt;.&lt;/p&gt;&#10;&#10;&#10;&#10;&lt;aside class="engineering-callout engineering-callout--note" aria-label="Advisory status as of October 27, 2026"&gt;&#10; &lt;p class="callout-label"&gt;Advisory status as of October 27, 2026&lt;/p&gt;&#10; &lt;div&gt;&lt;p&gt;These are the fixed releases for the two original CVEs, not a guarantee against every PHP issue. On October 23, PHP also reported a conditional session-upload-progress RCE path, &lt;strong&gt;CVE-2026-115560&lt;/strong&gt;, and recommended &lt;code&gt;session.upload_progress.enabled=0&lt;/code&gt; pending a further fix. Review the &lt;a class="link" href="https://www.php.net/archive/2026.php" target="_blank" rel="noopener"&#10; &gt;updated PHP guidance&lt;/a&gt; for applicability. This account does not establish whether that setting was changed during our response.&lt;/p&gt;&#10;&lt;/div&gt;&#10;&lt;/aside&gt;&#10;&#10;&lt;p&gt;Our team supported multiple applications, not one isolated PHP installation. Their deployment models differed: some ran directly on &lt;strong&gt;Amazon EC2&lt;/strong&gt; instances in an &lt;strong&gt;Auto Scaling Group (ASG)&lt;/strong&gt;, while others ran as containers on &lt;strong&gt;Amazon Elastic Container Service (ECS)&lt;/strong&gt; backed by EC2 capacity. We also encountered a legacy application module running &lt;strong&gt;PHP 7.4&lt;/strong&gt;, which could not be upgraded within the emergency response window.&lt;/p&gt;&#10;&lt;p&gt;That made the incident a portfolio-wide coordination problem. We had to identify likely PHP endpoints, reduce exposure, verify actual deployed runtimes, work with application owners, and choose remediation paths appropriate to each workload.&lt;/p&gt;&#10;&lt;p&gt;One issue turned out to be bigger than PHP2Shell itself: in an EC2-based application, &lt;strong&gt;User Data installed the latest available packages at boot&lt;/strong&gt;, making scaling and replacement dependent on live external package repositories. During the emergency, those repository requests repeatedly timed out.&lt;/p&gt;&#10;&lt;p&gt;The incident highlighted two linked concerns: &lt;strong&gt;how quickly we could remediate exposed workloads&lt;/strong&gt;, and &lt;strong&gt;whether our infrastructure could reliably reproduce a patched state under pressure&lt;/strong&gt;.&lt;/p&gt;&#10;&lt;h2 id="1-start-wide-identify-potential-exposure-across-applications"&gt;&lt;a href="#1-start-wide-identify-potential-exposure-across-applications" class="header-anchor"&gt;&lt;/a&gt;1. Start wide: identify potential exposure across applications&#10;&lt;/h2&gt;&lt;p&gt;We began with a lightweight Python utility that requested a list of application URLs and searched the responses for hints that PHP might be in use. The checks included:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;code&gt;X-Powered-By&lt;/code&gt; response headers that mention PHP.&lt;/li&gt;&#10;&lt;li&gt;Cookies such as &lt;code&gt;PHPSESSID&lt;/code&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;code&gt;.php&lt;/code&gt; in a final URL path or links and form actions in HTML.&lt;/li&gt;&#10;&lt;li&gt;PHP-related error patterns in the response body.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The utility produced an initial set of investigation targets, not a definitive vulnerability inventory. A PHP application can conceal all of those indicators behind a proxy or application framework; a PHP-looking URL can also exist without an affected PHP runtime. The HTTP response cannot reliably establish a deployed runtime version or whether either CVE is exploitable.&lt;/p&gt;&#10;&lt;p&gt;We used the scan for &lt;strong&gt;triage&lt;/strong&gt;, then followed it with infrastructure-level checks. A negative or inconclusive scan result could not safely clear an application. This distinction matters in a rapidly moving incident: a discovery shortcut becomes dangerous if it is mistaken for proof of safety.&lt;/p&gt;&#10;&lt;p&gt;The discovery scripts are not included in this article. A separate, sanitized repository would need documentation, example input, and clear restrictions to systems the operator is authorized to test. Their purpose is &lt;strong&gt;technology-discovery assistance&lt;/strong&gt;, not exploit testing or vulnerability confirmation.&lt;/p&gt;&#10;&lt;h2 id="2-contain-exposure-while-remediation-is-prepared"&gt;&lt;a href="#2-contain-exposure-while-remediation-is-prepared" class="header-anchor"&gt;&lt;/a&gt;2. Contain exposure while remediation is prepared&#10;&lt;/h2&gt;&lt;p&gt;As we identified potentially affected entry points, we applied temporary access restrictions at the traffic-control layer, including load-balancer-level controls. This reduced reachability while the teams investigated deployments and prepared updates.&lt;/p&gt;&#10;&lt;p&gt;Containment decisions required care. Blocking an endpoint can protect it from untrusted traffic but can also interrupt legitimate users. Access restrictions therefore had to be coordinated with the teams responsible for each application and treated as temporary risk reduction, not as evidence that the underlying code was safe.&lt;/p&gt;&#10;&lt;h3 id="a-legacy-php-74-module-required-a-different-decision"&gt;&lt;a href="#a-legacy-php-74-module-required-a-different-decision" class="header-anchor"&gt;&lt;/a&gt;A legacy PHP 7.4 module required a different decision&#10;&lt;/h3&gt;&lt;p&gt;During the inventory, we found &lt;strong&gt;one module of an application still running PHP 7.4&lt;/strong&gt;. That branch had long been out of upstream support, and a compatible upgrade could not be delivered during the emergency response window.&lt;/p&gt;&#10;&lt;p&gt;For this exceptional release, PHP also published minimal source backports for 7.4 and other unsupported branches. Those do not restore upstream support or automatically provide deployable vendor packages. Evaluating and validating such a build is separate work; this account does not claim that we did so.&lt;/p&gt;&#10;&lt;p&gt;The organization&amp;rsquo;s direction was to &lt;strong&gt;restrict access to components that could not be patched promptly&lt;/strong&gt;. For this module, the selected control was an &lt;strong&gt;AWS WAF web ACL rule&lt;/strong&gt;: requests to the specific affected URL were restricted to an &lt;strong&gt;allowlist of authorized client IP addresses&lt;/strong&gt;.&lt;/p&gt;&#10;&lt;p&gt;This maintained access for the intended clients while reducing the public attack surface. It did &lt;strong&gt;not&lt;/strong&gt; repair PHP 7.4, remove the possibility of attacks from allowed networks, or substitute for a supported runtime. Long-term work remains necessary to upgrade, replace, or retire the module.&lt;/p&gt;&#10;&#10;&#10;&#10;&lt;aside class="engineering-callout engineering-callout--constraint" aria-label="Operational constraint"&gt;&#10; &lt;p class="callout-label"&gt;Operational constraint&lt;/p&gt;&#10; &lt;div&gt;&lt;p&gt;An IP allowlist only provides the intended containment when the protected path rejects non-allowed clients, rule ordering preserves that restriction, and alternate routes cannot bypass the control. An allow rule alone is insufficient if unmatched traffic is otherwise accepted. This is a validation requirement, not a claim that every possible bypass was tested during the response. See the &lt;a class="link" href="https://docs.aws.amazon.com/waf/latest/developerguide/web-acl.html" target="_blank" rel="noopener"&#10; &gt;AWS WAF documentation&lt;/a&gt;.&lt;/p&gt;&#10;&lt;/div&gt;&#10;&lt;/aside&gt;&#10;&#10;&lt;p&gt;The legacy finding also exposed an inventory problem: &lt;strong&gt;a runtime reaching end of life should be visible before an emergency forces a decision about it&lt;/strong&gt;.&lt;/p&gt;&#10;&lt;h2 id="3-verify-runtimes-in-ecs-workloads-not-just-on-their-ec2-hosts"&gt;&lt;a href="#3-verify-runtimes-in-ecs-workloads-not-just-on-their-ec2-hosts" class="header-anchor"&gt;&lt;/a&gt;3. Verify runtimes in ECS workloads, not just on their EC2 hosts&#10;&lt;/h2&gt;&lt;p&gt;A second part of the response concerned containerized applications running on &lt;strong&gt;Amazon ECS with EC2 capacity&lt;/strong&gt;.&lt;/p&gt;&#10;&lt;p&gt;I used an additional script to assist the investigation across the EC2 infrastructure associated with the ECS clusters and their projects. The goal was to locate relevant workloads, help determine their PHP versions, and direct follow-up to the appropriate application owners.&lt;/p&gt;&#10;&lt;p&gt;There is an important technical distinction here: &lt;strong&gt;the PHP version installed on an ECS container host is not necessarily the PHP version running in an application container&lt;/strong&gt;. For a containerized workload, the relevant evidence is the runtime inside the deployed container or the exact application image that the active ECS task runs. Host-level discovery alone cannot establish that.&lt;/p&gt;&#10;&lt;p&gt;The defensible validation path is therefore:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Identify the running ECS services and tasks associated with the application.&lt;/li&gt;&#10;&lt;li&gt;Record their deployed task definitions and container image references.&lt;/li&gt;&#10;&lt;li&gt;Inspect the relevant container runtime or independently verify the packages in the exact image artifact.&lt;/li&gt;&#10;&lt;li&gt;Compare the findings against the vendor&amp;rsquo;s affected and fixed builds.&lt;/li&gt;&#10;&lt;li&gt;Recheck the &lt;strong&gt;running tasks after deployment&lt;/strong&gt;, not merely the image intended for release.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;An ECS task definition identifies the container configuration, including its image reference. The investigation must connect that definition to the tasks actually serving traffic; see &lt;a class="link" href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html" target="_blank" rel="noopener"&#10; &gt;Amazon ECS task definitions&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;The description above is the verification standard I would apply. It should not be read as a claim that every one of these steps was automated by the investigation script during the incident.&lt;/p&gt;&#10;&lt;h3 id="rebuilding-corrected-container-images"&gt;&lt;a href="#rebuilding-corrected-container-images" class="header-anchor"&gt;&lt;/a&gt;Rebuilding corrected container images&#10;&lt;/h3&gt;&lt;p&gt;Once the affected application teams had been notified, some remediation paths were relatively straightforward. Corrected PHP base images were already available by the time those workloads were handled, so the teams could rebuild the application images and deploy the new artifacts through ECS.&lt;/p&gt;&#10;&lt;div class="code-example"&gt;&#10; &lt;div class="code-toolbar"&gt;&lt;span&gt;Source code&lt;/span&gt;&lt;span class="code-language"&gt;text&lt;/span&gt;&lt;/div&gt;&#10; &lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Security advisory&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Portfolio triage and application ownership&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Identify ECS workloads and check deployed PHP runtime&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Responsible team rebuilds application image&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;with corrected PHP base image&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Publish and deploy updated artifact through ECS&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Verify new tasks and their PHP runtime&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;p&gt;Rebuilding against a newly fixed base image is useful, but it is not sufficient proof by itself. A build may reuse stale layers, a mutable image tag may still point to an unexpected artifact, or an ECS service may continue running older tasks during a rollout. A stronger process uses immutable image digests where practical, records artifact provenance, confirms deployment convergence, and checks the version actually executing.&lt;/p&gt;&#10;&lt;p&gt;This episode reinforced the advantage of having a repeatable build-and-deploy path for application dependencies. &lt;strong&gt;The image is the remediation unit&lt;/strong&gt;, and validation must follow that artifact into production.&lt;/p&gt;&#10;&lt;h2 id="4-a-different-remediation-path-php-directly-on-ec2"&gt;&lt;a href="#4-a-different-remediation-path-php-directly-on-ec2" class="header-anchor"&gt;&lt;/a&gt;4. A different remediation path: PHP directly on EC2&#10;&lt;/h2&gt;&lt;p&gt;Another production application ran PHP directly on EC2 instances managed by an &lt;strong&gt;Auto Scaling Group&lt;/strong&gt;. Its operating system packages, including PHP, were installed using an external RPM repository during instance initialization.&lt;/p&gt;&#10;&lt;p&gt;We first checked the PHP CLI version on the instances:&lt;/p&gt;&#10;&lt;div class="code-example"&gt;&#10; &lt;div class="code-toolbar"&gt;&lt;span&gt;Source code&lt;/span&gt;&lt;span class="code-language"&gt;bash&lt;/span&gt;&lt;/div&gt;&#10; &lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;php --version&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&#10;&#10;&#10;&lt;aside class="engineering-callout engineering-callout--constraint" aria-label="CLI evidence is not web-runtime evidence"&gt;&#10; &lt;p class="callout-label"&gt;CLI evidence is not web-runtime evidence&lt;/p&gt;&#10; &lt;div&gt;&lt;p&gt;&lt;code&gt;php --version&lt;/code&gt; identifies the CLI executable selected by the current shell. PHP-FPM or an Apache module can use a different installation, and existing workers can retain the old runtime until restarted or replaced. Record the installed package build and verify the web-serving processes separately. The CLI checks below are evidence of the reported version, not complete proof of the runtime handling HTTP requests.&lt;/p&gt;&#10;&lt;/div&gt;&#10;&lt;/aside&gt;&#10;&#10;&lt;p&gt;The checked instances reported &lt;strong&gt;PHP 8.4.26&lt;/strong&gt;, an upstream release affected by the advisory. Distribution and LTS vendors can backport security fixes without adopting the newest upstream version number, so package provenance and the vendor&amp;rsquo;s advisory also matter. We traced this installation to the &lt;strong&gt;Remi repository&lt;/strong&gt; and checked whether its corrected &lt;strong&gt;8.4.27&lt;/strong&gt; packages were available:&lt;/p&gt;&#10;&lt;div class="code-example"&gt;&#10; &lt;div class="code-toolbar"&gt;&lt;span&gt;Source code&lt;/span&gt;&lt;span class="code-language"&gt;bash&lt;/span&gt;&lt;/div&gt;&#10; &lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1&#10;&lt;/span&gt;&lt;span class="lnt"&gt;2&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;sudo dnf clean expire-cache&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;sudo dnf --showduplicates list php-cli&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;p&gt;These commands refreshed package metadata and displayed available PHP CLI package versions. They were &lt;strong&gt;diagnostic steps&lt;/strong&gt;; by themselves they did not apply or verify an update.&lt;/p&gt;&#10;&lt;p&gt;The package names are specific to that installation. Parallel PHP installations and repository-specific package families can use different names; checking only &lt;code&gt;php-cli&lt;/code&gt; does not account for every installed runtime or SAPI.&lt;/p&gt;&#10;&lt;p&gt;We confirmed that the corrected version was available. However, requests to the repository repeatedly timed out. We suspected that elevated demand after the public disclosure might be contributing, but &lt;strong&gt;the cause of the timeouts was not established&lt;/strong&gt;. Network routing, mirror availability, and other causes remained possible.&lt;/p&gt;&#10;&lt;p&gt;After further attempts, we succeeded in updating one running instance. We then used the &lt;strong&gt;Auto Scaling Group to replace instances&lt;/strong&gt;, checking the version on the newly started machines and repeating the replacement process until the checked instances reported:&lt;/p&gt;&#10;&lt;div class="code-example"&gt;&#10; &lt;div class="code-toolbar"&gt;&lt;span&gt;Source code&lt;/span&gt;&lt;span class="code-language"&gt;text&lt;/span&gt;&lt;/div&gt;&#10; &lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;PHP 8.4.27&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;p&gt;This met the immediate version-update objective &lt;strong&gt;for the instances we verified&lt;/strong&gt;. Patching one host would not have been enough: in an ASG, any new instance must also receive the corrected runtime. Likewise, a verified package version is not proof that exploitation never occurred; compromise assessment requires relevant logs, telemetry, and a separate security investigation. We did not establish evidence of successful exploitation from the incident information available for this account.&lt;/p&gt;&#10;&lt;p&gt;This describes the replacement work performed, not a recommendation to terminate production instances without a rollout plan. A planned refresh should account for healthy capacity, warm-up, application readiness and rollback. See &lt;a class="link" href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html" target="_blank" rel="noopener"&#10; &gt;EC2 Auto Scaling Instance Refresh&lt;/a&gt;.&lt;/p&gt;&#10;&lt;h2 id="5-the-deeper-issue-new-ec2-instances-depended-on-a-live-repository"&gt;&lt;a href="#5-the-deeper-issue-new-ec2-instances-depended-on-a-live-repository" class="header-anchor"&gt;&lt;/a&gt;5. The deeper issue: new EC2 instances depended on a live repository&#10;&lt;/h2&gt;&lt;p&gt;The difficulty with package downloads revealed a broader infrastructure weakness.&lt;/p&gt;&#10;&lt;p&gt;The application&amp;rsquo;s &lt;strong&gt;EC2 User Data&lt;/strong&gt; script installed required packages at instance startup and generally requested their &lt;strong&gt;latest available versions&lt;/strong&gt;. Essential runtime state was therefore assembled during boot rather than embedded in a previously built and tested machine image.&lt;/p&gt;&#10;&lt;p&gt;For Linux instances, user-data scripts normally run on the first boot unless configured otherwise; an ordinary reboot is not the same as launching a replacement instance. The dependency here was the initialization of &lt;strong&gt;new ASG instances&lt;/strong&gt;. See &lt;a class="link" href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html" target="_blank" rel="noopener"&#10; &gt;EC2 User Data&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;A simplified version of the dependency looked like this:&lt;/p&gt;&#10;&lt;div class="code-example"&gt;&#10; &lt;div class="code-toolbar"&gt;&lt;span&gt;Source code&lt;/span&gt;&lt;span class="code-language"&gt;text&lt;/span&gt;&lt;/div&gt;&#10; &lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Auto Scaling Group requests capacity&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Launch new EC2&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Execute User Data&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Reach external RPM repositories&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; +------+------+&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Available Timeout/failure&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Install packages Bootstrap may fail&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | or be incomplete&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Start application&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Pass readiness checks&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;p&gt;Even when two instances launch from the same AMI and Launch Template, this model can produce different outcomes:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;A repository may publish new package builds between launches, yielding different runtimes.&lt;/li&gt;&#10;&lt;li&gt;An upstream package change may introduce an untested regression or incompatible configuration.&lt;/li&gt;&#10;&lt;li&gt;Repository or network failures may prevent a replacement instance from starting the application.&lt;/li&gt;&#10;&lt;li&gt;Unless bootstrap errors are propagated and health checks test the application, an instance may appear healthy at the EC2 layer without being ready to serve requests.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The last point is a &lt;strong&gt;possible failure mode&lt;/strong&gt;, not a claim that all instances in this incident passed health checks while misconfigured.&lt;/p&gt;&#10;&lt;p&gt;An ASG can launch replacement compute capacity. It cannot, by itself, make external repositories available or guarantee that an application&amp;rsquo;s initialization script succeeded. In a bad failure sequence, repeated instance replacement can amplify the dependency rather than resolve it.&lt;/p&gt;&#10;&lt;p&gt;I did not originally design this bootstrap script. The purpose of documenting the finding is not to assign blame to its author: a previously workable provisioning choice had become a measurable reliability risk, and the incident exposed it under particularly unfavorable conditions.&lt;/p&gt;&#10;&lt;h3 id="why-install-latest-at-boot-is-not-a-patching-strategy"&gt;&lt;a href="#why-install-latest-at-boot-is-not-a-patching-strategy" class="header-anchor"&gt;&lt;/a&gt;Why “install latest at boot” is not a patching strategy&#10;&lt;/h3&gt;&lt;p&gt;Fetching the newest packages might sound like a way to keep servers secure. It also means that the exact software released to production is not fully known until the machine starts. Under normal scaling this reduces reproducibility; during an incident it makes rollback, troubleshooting, and capacity recovery more uncertain.&lt;/p&gt;&#10;&#10;&#10;&#10;&lt;aside class="engineering-callout engineering-callout--decision" aria-label="Separate patch delivery from capacity recovery"&gt;&#10; &lt;p class="callout-label"&gt;Separate patch delivery from capacity recovery&lt;/p&gt;&#10; &lt;div&gt;&lt;p&gt;Pinning a package version helps with reproducibility but &lt;strong&gt;does not solve availability&lt;/strong&gt; if every new instance still depends on a remote repository. Baking packages into an AMI reduces boot-time dependencies but requires an image lifecycle capable of handling security releases quickly. A sound design needs both controlled artifacts &lt;strong&gt;and&lt;/strong&gt; a prompt, testable patching pipeline.&lt;/p&gt;&#10;&lt;/div&gt;&#10;&lt;/aside&gt;&#10;&#10;&lt;h2 id="6-follow-up-engineering-work-identified-not-yet-completed"&gt;&lt;a href="#6-follow-up-engineering-work-identified-not-yet-completed" class="header-anchor"&gt;&lt;/a&gt;6. Follow-up engineering work: identified, not yet completed&#10;&lt;/h2&gt;&lt;p&gt;The emergency work focused on containment and verified runtime remediation. The bootstrap redesign and the improvements below are &lt;strong&gt;follow-up proposals&lt;/strong&gt;, not actions already completed during the incident.&lt;/p&gt;&#10;&lt;p&gt;This builds on the ownership and lifecycle questions in &lt;a class="link" href="https://fabiolopes.page/p/beyond-day-1-terraform-rethinking-platform-operations-in-my-environment/" &gt;Beyond Day-1 Terraform&lt;/a&gt;: a declared infrastructure configuration is only part of what determines the software that ultimately serves traffic.&lt;/p&gt;&#10;&lt;h3 id="build-and-validate-versioned-amis"&gt;&lt;a href="#build-and-validate-versioned-amis" class="header-anchor"&gt;&lt;/a&gt;Build and validate versioned AMIs&#10;&lt;/h3&gt;&lt;p&gt;A stronger approach for the EC2-based application would put essential runtime packages into a tested AMI, using &lt;strong&gt;EC2 Image Builder&lt;/strong&gt;, &lt;strong&gt;Packer&lt;/strong&gt;, or equivalent automation.&lt;/p&gt;&#10;&lt;div class="code-example"&gt;&#10; &lt;div class="code-toolbar"&gt;&lt;span&gt;Source code&lt;/span&gt;&lt;span class="code-language"&gt;text&lt;/span&gt;&lt;/div&gt;&#10; &lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Source-controlled image configuration&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Build AMI with approved PHP/packages&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Security and application smoke tests&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Publish immutable, versioned AMI&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Update ASG Launch Template version&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Controlled instance refresh / rollout&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Validate running application + rollback readiness&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&#10;&lt;p&gt;This would make instance launches less dependent on third-party repositories and align the deployment lifecycle with a known runtime baseline. The trade-off is additional image-build and update automation. Without a disciplined rebuild process, an immutable image can just as easily become an immutable vulnerability.&lt;/p&gt;&#10;&lt;h3 id="make-remaining-bootstrap-tasks-failure-aware"&gt;&lt;a href="#make-remaining-bootstrap-tasks-failure-aware" class="header-anchor"&gt;&lt;/a&gt;Make remaining bootstrap tasks failure-aware&#10;&lt;/h3&gt;&lt;p&gt;Some configuration may still need to happen at boot. For those tasks, the desired properties are:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Explicit failure on a missing critical dependency, rather than silently continuing.&lt;/li&gt;&#10;&lt;li&gt;Bounded retries with exponential backoff and jitter for transient failures.&lt;/li&gt;&#10;&lt;li&gt;Centralized bootstrap logs and actionable failure signals.&lt;/li&gt;&#10;&lt;li&gt;Startup and health checks that only mark the application ready after essential steps succeed.&lt;/li&gt;&#10;&lt;li&gt;A defined way to replace unhealthy instances without inducing uncontrolled replacement loops.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h3 id="strengthen-inventory-and-deployment-evidence"&gt;&lt;a href="#strengthen-inventory-and-deployment-evidence" class="header-anchor"&gt;&lt;/a&gt;Strengthen inventory and deployment evidence&#10;&lt;/h3&gt;&lt;p&gt;Across both EC2 and ECS, the next response should not require rebuilding the runtime inventory from scratch. Useful records would connect &lt;strong&gt;application owner, endpoint, deployment model, runtime version, artifact digest or AMI ID, support status, and deployment date&lt;/strong&gt;.&lt;/p&gt;&#10;&lt;p&gt;For ECS, the inventory must lead from service to running task and immutable application image. For direct EC2 deployments, it must lead from ASG and Launch Template to the actual installed runtime. Periodic checks should alert on end-of-life branches before an urgent advisory turns them into a containment problem.&lt;/p&gt;&#10;&lt;h3 id="rehearse-dependency-failures"&gt;&lt;a href="#rehearse-dependency-failures" class="header-anchor"&gt;&lt;/a&gt;Rehearse dependency failures&#10;&lt;/h3&gt;&lt;p&gt;A controlled test should deliberately make repositories unavailable, slow down package retrieval, cause an installation step to fail, and prevent an application service from starting. The outcome to measure is not just whether EC2 launches, but whether the system maintains or regains &lt;strong&gt;healthy application capacity&lt;/strong&gt; and whether operators can see why recovery failed.&lt;/p&gt;&#10;&lt;h2 id="7-what-the-different-outcomes-taught-us"&gt;&lt;a href="#7-what-the-different-outcomes-taught-us" class="header-anchor"&gt;&lt;/a&gt;7. What the different outcomes taught us&#10;&lt;/h2&gt;&lt;p&gt;This was not a single patch applied the same way everywhere. It was one security event with several operational responses:&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Situation&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Action during the response&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;What remains important&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Broad application portfolio&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Used HTTP-based discovery as initial triage, followed by infrastructure investigation&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Avoid treating a negative heuristic scan as proof of safety&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Potentially exposed endpoints&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Restricted access through traffic controls while remediation proceeded&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Account for legitimate traffic and remove temporary restrictions only when appropriate&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;ECS applications&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Coordinated with owners to rebuild and redeploy images using corrected PHP bases&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Verify the &lt;strong&gt;deployed container runtime&lt;/strong&gt;, not the host PHP version or build intention&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;EC2 application in an ASG&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Retrieved PHP 8.4.27 from Remi, replaced instances, and checked resulting PHP versions&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Make the patched state reproducible for every future replacement&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Legacy PHP 7.4 module&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Restricted its URL through an AWS WAF IP allowlist&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Migrate or retire the unsupported runtime; WAF is containment, not a patch&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;EC2 bootstrap design&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Identified repository timeouts and nondeterministic boot-time package installation&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Move toward versioned images, resilient bootstrap, and application-level readiness checks&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;There are two practical lessons I would carry into the next incident.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;First, security response begins with trustworthy deployment visibility.&lt;/strong&gt; A URL, an EC2 host, an ECS service, a container image, and a PHP process are different objects. They must be connected carefully before declaring exposure or remediation complete.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Second, recovery is only as reliable as the artifacts and dependencies needed to rebuild capacity.&lt;/strong&gt; Autoscaling solves part of the compute lifecycle, while image management, bootstrap behavior, repository availability, and readiness checks determine whether that new capacity is useful.&lt;/p&gt;&#10;&lt;h2 id="final-thoughts"&gt;&lt;a href="#final-thoughts" class="header-anchor"&gt;&lt;/a&gt;Final thoughts&#10;&lt;/h2&gt;&lt;p&gt;The most revealing part of this incident was not the command used to display a PHP version. It was the contrast between two delivery models under time pressure.&lt;/p&gt;&#10;&lt;p&gt;For ECS workloads, a corrected runtime could be packaged into a new application image and deployed through established workflows. For one traditional EC2 application, replacement capacity still depended on external package downloads at boot. That dependency repeatedly got in the way of emergency maintenance and remains a candidate for architectural improvement.&lt;/p&gt;&#10;&lt;p&gt;The legacy PHP 7.4 module added another dimension: sometimes an upgrade is not available within the response window, and disciplined containment is necessary while a longer-term migration is planned.&lt;/p&gt;&#10;&lt;p&gt;We verified the corrected PHP version on the EC2 instances reviewed, coordinated container-image remediation with application teams, and reduced exposure where immediate patching was not feasible. Those are concrete outcomes. The next engineering challenge is to make inventory, build artifacts, and infrastructure replacement more predictable, so that the next critical advisory requires less discovery and less improvisation.&lt;/p&gt;&#10;&lt;hr&gt;&#10;&lt;h2 id="references"&gt;&lt;a href="#references" class="header-anchor"&gt;&lt;/a&gt;References&#10;&lt;/h2&gt;&lt;ol&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://www.php.net/archive/2026.php" target="_blank" rel="noopener"&#10; &gt;PHP Project, security release announcements, October 21, 2026&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://github.com/NCSC-NL/php2shell" target="_blank" rel="noopener"&#10; &gt;NCSC-NL, PHP2Shell operational information (CVE-2026-109618 and CVE-2026-109619)&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://www.php.net/releases/8_4_27.php" target="_blank" rel="noopener"&#10; &gt;PHP 8.4.27 release announcement&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html" target="_blank" rel="noopener"&#10; &gt;AWS, Run commands on your Linux instance at launch (User Data)&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html" target="_blank" rel="noopener"&#10; &gt;AWS, EC2 Auto Scaling Instance Refresh&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://docs.aws.amazon.com/waf/latest/developerguide/web-acl.html" target="_blank" rel="noopener"&#10; &gt;AWS, AWS WAF web ACLs&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html" target="_blank" rel="noopener"&#10; &gt;AWS, Amazon ECS task definitions&lt;/a&gt;&lt;/li&gt;&#10;&lt;/ol&gt;&#10;</description></item></channel></rss>