Articles on Fabio Lopes

PHP2Shell on AWS: Incident Response and the Limits of Boot-Time Patching

Tue, 27 Oct 2026 00:00:00 +0000

How a critical PHP advisory became a cross-application incident response, and why the hardest lesson was about the way new servers are built.

The advisory was only the beginning

On October 21, 2026, the PHP project announced security releases addressing CVE-2026-109618 and CVE-2026-109619, two critical vulnerabilities commonly referred to as PHP2Shell. The vulnerabilities could enable unauthenticated remote code execution in affected deployments. The security updates included PHP 8.4.27, 8.3.36, 8.2.35, and 8.5.12. The PHP project warned that temporary mitigations were incomplete and updating was essential. See the PHP announcement and NCSC-NL operational information.

Our team supported multiple applications, not one isolated PHP installation. Their deployment models differed: some ran directly on Amazon EC2 instances in an Auto Scaling Group (ASG), while others ran as containers on Amazon Elastic Container Service (ECS) backed by EC2 capacity. We also encountered a legacy application module running PHP 7.4, which could not be upgraded within the emergency response window.

That made the incident a portfolio-wide coordination problem. We had to identify likely PHP endpoints, reduce exposure, verify actual deployed runtimes, work with application owners, and choose remediation paths appropriate to each workload.

One issue turned out to be bigger than PHP2Shell itself: in an EC2-based application, User Data installed the latest available packages at boot, making scaling and replacement dependent on live external package repositories. During the emergency, those repository requests repeatedly timed out.

The incident highlighted two linked concerns: how quickly we could remediate exposed workloads, and whether our infrastructure could reliably reproduce a patched state under pressure.

1. Start wide: identify potential exposure across applications

We began with a lightweight Python utility that requested a list of application URLs and searched the responses for hints that PHP might be in use. The checks included:

X-Powered-By response headers that mention PHP.
Cookies such as PHPSESSID.
.php in a final URL path or links and form actions in HTML.
PHP-related error patterns in the response body.

The utility produced an initial set of investigation targets, not a definitive vulnerability inventory. A PHP application can conceal all of those indicators behind a proxy or application framework; a PHP-looking URL can also exist without an affected PHP runtime. The HTTP response cannot reliably establish a deployed runtime version or whether either CVE is exploitable.

We used the scan for triage, then followed it with infrastructure-level checks. A negative or inconclusive scan result could not safely clear an application. This distinction matters in a rapidly moving incident: a discovery shortcut becomes dangerous if it is mistaken for proof of safety.

The discovery scripts are not included in this article. A separate, sanitized repository would need documentation, example input, and clear restrictions to systems the operator is authorized to test. Their purpose is technology-discovery assistance, not exploit testing or vulnerability confirmation.

2. Contain exposure while remediation is prepared

As we identified potentially affected entry points, we applied temporary access restrictions at the traffic-control layer, including load-balancer-level controls. This reduced reachability while the teams investigated deployments and prepared updates.

Containment decisions required care. Blocking an endpoint can protect it from untrusted traffic but can also interrupt legitimate users. Access restrictions therefore had to be coordinated with the teams responsible for each application and treated as temporary risk reduction, not as evidence that the underlying code was safe.

A legacy PHP 7.4 module required a different decision

During the inventory, we found one module of an application still running PHP 7.4. That branch had long been out of upstream support, and a compatible upgrade could not be delivered during the emergency response window.

For this exceptional release, PHP also published minimal source backports for 7.4 and other unsupported branches. Those do not restore upstream support or automatically provide deployable vendor packages. Evaluating and validating such a build is separate work; this account does not claim that we did so.

The organization’s direction was to restrict access to components that could not be patched promptly. For this module, the selected control was an AWS WAF web ACL rule: requests to the specific affected URL were restricted to an allowlist of authorized client IP addresses.

This maintained access for the intended clients while reducing the public attack surface. It did not repair PHP 7.4, remove the possibility of attacks from allowed networks, or substitute for a supported runtime. Long-term work remains necessary to upgrade, replace, or retire the module.

The legacy finding also exposed an inventory problem: a runtime reaching end of life should be visible before an emergency forces a decision about it.

3. Verify runtimes in ECS workloads, not just on their EC2 hosts

A second part of the response concerned containerized applications running on Amazon ECS with EC2 capacity.

I used an additional script to assist the investigation across the EC2 infrastructure associated with the ECS clusters and their projects. The goal was to locate relevant workloads, help determine their PHP versions, and direct follow-up to the appropriate application owners.

There is an important technical distinction here: the PHP version installed on an ECS container host is not necessarily the PHP version running in an application container. For a containerized workload, the relevant evidence is the runtime inside the deployed container or the exact application image that the active ECS task runs. Host-level discovery alone cannot establish that.

The defensible validation path is therefore:

Identify the running ECS services and tasks associated with the application.
Record their deployed task definitions and container image references.
Inspect the relevant container runtime or independently verify the packages in the exact image artifact.
Compare the findings against the vendor’s affected and fixed builds.
Recheck the running tasks after deployment, not merely the image intended for release.

An ECS task definition identifies the container configuration, including its image reference. The investigation must connect that definition to the tasks actually serving traffic; see Amazon ECS task definitions.

The description above is the verification standard I would apply. It should not be read as a claim that every one of these steps was automated by the investigation script during the incident.

Rebuilding corrected container images

Once the affected application teams had been notified, some remediation paths were relatively straightforward. Corrected PHP base images were already available by the time those workloads were handled, so the teams could rebuild the application images and deploy the new artifacts through ECS.

Source codetext


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Security advisory
 |
 v
Portfolio triage and application ownership
 |
 v
Identify ECS workloads and check deployed PHP runtime
 |
 v
Responsible team rebuilds application image
with corrected PHP base image
 |
 v
Publish and deploy updated artifact through ECS
 |
 v
Verify new tasks and their PHP runtime

Rebuilding against a newly fixed base image is useful, but it is not sufficient proof by itself. A build may reuse stale layers, a mutable image tag may still point to an unexpected artifact, or an ECS service may continue running older tasks during a rollout. A stronger process uses immutable image digests where practical, records artifact provenance, confirms deployment convergence, and checks the version actually executing.

This episode reinforced the advantage of having a repeatable build-and-deploy path for application dependencies. The image is the remediation unit, and validation must follow that artifact into production.

4. A different remediation path: PHP directly on EC2

Another production application ran PHP directly on EC2 instances managed by an Auto Scaling Group. Its operating system packages, including PHP, were installed using an external RPM repository during instance initialization.

We first checked the PHP CLI version on the instances:

Source codebash

1

php --version

The checked instances reported PHP 8.4.26, an upstream release affected by the advisory. Distribution and LTS vendors can backport security fixes without adopting the newest upstream version number, so package provenance and the vendor’s advisory also matter. We traced this installation to the Remi repository and checked whether its corrected 8.4.27 packages were available:

Source codebash

1
2

sudo dnf clean expire-cache
sudo dnf --showduplicates list php-cli

These commands refreshed package metadata and displayed available PHP CLI package versions. They were diagnostic steps; by themselves they did not apply or verify an update.

The package names are specific to that installation. Parallel PHP installations and repository-specific package families can use different names; checking only php-cli does not account for every installed runtime or SAPI.

We confirmed that the corrected version was available. However, requests to the repository repeatedly timed out. We suspected that elevated demand after the public disclosure might be contributing, but the cause of the timeouts was not established. Network routing, mirror availability, and other causes remained possible.

After further attempts, we succeeded in updating one running instance. We then used the Auto Scaling Group to replace instances, checking the version on the newly started machines and repeating the replacement process until the checked instances reported:

Source codetext

1

PHP 8.4.27

This met the immediate version-update objective for the instances we verified. Patching one host would not have been enough: in an ASG, any new instance must also receive the corrected runtime. Likewise, a verified package version is not proof that exploitation never occurred; compromise assessment requires relevant logs, telemetry, and a separate security investigation. We did not establish evidence of successful exploitation from the incident information available for this account.

This describes the replacement work performed, not a recommendation to terminate production instances without a rollout plan. A planned refresh should account for healthy capacity, warm-up, application readiness and rollback. See EC2 Auto Scaling Instance Refresh.

5. The deeper issue: new EC2 instances depended on a live repository

The difficulty with package downloads revealed a broader infrastructure weakness.

The application’s EC2 User Data script installed required packages at instance startup and generally requested their latest available versions. Essential runtime state was therefore assembled during boot rather than embedded in a previously built and tested machine image.

For Linux instances, user-data scripts normally run on the first boot unless configured otherwise; an ordinary reboot is not the same as launching a replacement instance. The dependency here was the initialization of new ASG instances. See EC2 User Data.

A simplified version of the dependency looked like this:

Source codetext


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


Auto Scaling Group requests capacity
 |
 v
 Launch new EC2
 |
 v
 Execute User Data
 |
 v
 Reach external RPM repositories
 |
 +------+------+
 | |
 Available Timeout/failure
 | |
 v v
 Install packages Bootstrap may fail
 | or be incomplete
 v
 Start application
 |
 v
 Pass readiness checks

Even when two instances launch from the same AMI and Launch Template, this model can produce different outcomes:

A repository may publish new package builds between launches, yielding different runtimes.
An upstream package change may introduce an untested regression or incompatible configuration.
Repository or network failures may prevent a replacement instance from starting the application.
Unless bootstrap errors are propagated and health checks test the application, an instance may appear healthy at the EC2 layer without being ready to serve requests.

The last point is a possible failure mode, not a claim that all instances in this incident passed health checks while misconfigured.

An ASG can launch replacement compute capacity. It cannot, by itself, make external repositories available or guarantee that an application’s initialization script succeeded. In a bad failure sequence, repeated instance replacement can amplify the dependency rather than resolve it.

I did not originally design this bootstrap script. The purpose of documenting the finding is not to assign blame to its author: a previously workable provisioning choice had become a measurable reliability risk, and the incident exposed it under particularly unfavorable conditions.

Why “install latest at boot” is not a patching strategy

Fetching the newest packages might sound like a way to keep servers secure. It also means that the exact software released to production is not fully known until the machine starts. Under normal scaling this reduces reproducibility; during an incident it makes rollback, troubleshooting, and capacity recovery more uncertain.

6. Follow-up engineering work: identified, not yet completed

The emergency work focused on containment and verified runtime remediation. The bootstrap redesign and the improvements below are follow-up proposals, not actions already completed during the incident.

This builds on the ownership and lifecycle questions in Beyond Day-1 Terraform: a declared infrastructure configuration is only part of what determines the software that ultimately serves traffic.

Build and validate versioned AMIs

A stronger approach for the EC2-based application would put essential runtime packages into a tested AMI, using EC2 Image Builder, Packer, or equivalent automation.

Source codetext


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


Source-controlled image configuration
 |
 v
Build AMI with approved PHP/packages
 |
 v
Security and application smoke tests
 |
 v
Publish immutable, versioned AMI
 |
 v
Update ASG Launch Template version
 |
 v
Controlled instance refresh / rollout
 |
 v
Validate running application + rollback readiness

This would make instance launches less dependent on third-party repositories and align the deployment lifecycle with a known runtime baseline. The trade-off is additional image-build and update automation. Without a disciplined rebuild process, an immutable image can just as easily become an immutable vulnerability.

Make remaining bootstrap tasks failure-aware

Some configuration may still need to happen at boot. For those tasks, the desired properties are:

Explicit failure on a missing critical dependency, rather than silently continuing.
Bounded retries with exponential backoff and jitter for transient failures.
Centralized bootstrap logs and actionable failure signals.
Startup and health checks that only mark the application ready after essential steps succeed.
A defined way to replace unhealthy instances without inducing uncontrolled replacement loops.

Strengthen inventory and deployment evidence

Across both EC2 and ECS, the next response should not require rebuilding the runtime inventory from scratch. Useful records would connect application owner, endpoint, deployment model, runtime version, artifact digest or AMI ID, support status, and deployment date.

For ECS, the inventory must lead from service to running task and immutable application image. For direct EC2 deployments, it must lead from ASG and Launch Template to the actual installed runtime. Periodic checks should alert on end-of-life branches before an urgent advisory turns them into a containment problem.

Rehearse dependency failures

A controlled test should deliberately make repositories unavailable, slow down package retrieval, cause an installation step to fail, and prevent an application service from starting. The outcome to measure is not just whether EC2 launches, but whether the system maintains or regains healthy application capacity and whether operators can see why recovery failed.

7. What the different outcomes taught us

This was not a single patch applied the same way everywhere. It was one security event with several operational responses:

Situation	Action during the response	What remains important
Broad application portfolio	Used HTTP-based discovery as initial triage, followed by infrastructure investigation	Avoid treating a negative heuristic scan as proof of safety
Potentially exposed endpoints	Restricted access through traffic controls while remediation proceeded	Account for legitimate traffic and remove temporary restrictions only when appropriate
ECS applications	Coordinated with owners to rebuild and redeploy images using corrected PHP bases	Verify the deployed container runtime, not the host PHP version or build intention
EC2 application in an ASG	Retrieved PHP 8.4.27 from Remi, replaced instances, and checked resulting PHP versions	Make the patched state reproducible for every future replacement
Legacy PHP 7.4 module	Restricted its URL through an AWS WAF IP allowlist	Migrate or retire the unsupported runtime; WAF is containment, not a patch
EC2 bootstrap design	Identified repository timeouts and nondeterministic boot-time package installation	Move toward versioned images, resilient bootstrap, and application-level readiness checks

There are two practical lessons I would carry into the next incident.

First, security response begins with trustworthy deployment visibility. A URL, an EC2 host, an ECS service, a container image, and a PHP process are different objects. They must be connected carefully before declaring exposure or remediation complete.

Second, recovery is only as reliable as the artifacts and dependencies needed to rebuild capacity. Autoscaling solves part of the compute lifecycle, while image management, bootstrap behavior, repository availability, and readiness checks determine whether that new capacity is useful.

Final thoughts

The most revealing part of this incident was not the command used to display a PHP version. It was the contrast between two delivery models under time pressure.

For ECS workloads, a corrected runtime could be packaged into a new application image and deployed through established workflows. For one traditional EC2 application, replacement capacity still depended on external package downloads at boot. That dependency repeatedly got in the way of emergency maintenance and remains a candidate for architectural improvement.

The legacy PHP 7.4 module added another dimension: sometimes an upgrade is not available within the response window, and disciplined containment is necessary while a longer-term migration is planned.

We verified the corrected PHP version on the EC2 instances reviewed, coordinated container-image remediation with application teams, and reduced exposure where immediate patching was not feasible. Those are concrete outcomes. The next engineering challenge is to make inventory, build artifacts, and infrastructure replacement more predictable, so that the next critical advisory requires less discovery and less improvisation.

References

Beyond Day-1 Terraform: Rethinking Platform Operations in My Environment

Sun, 25 Oct 2026 00:00:00 +0000

Introduction: The Day-2 Challenge

I’ve been thinking about a challenge in the infrastructure environment I work with: how to keep Terraform relevant after the initial provisioning phase, without interfering with ongoing application deployments or resources managed by other teams.

Provisioning infrastructure with Terraform is relatively straightforward when the boundaries are clear and the environment is being built from scratch. The situation becomes more complicated once applications are running, multiple teams share responsibilities, and operational changes start happening through different workflows.

In our context, we operate across multiple AWS accounts, with workloads involving AWS Outposts, ECS, and supporting infrastructure. Networking, security, platform operations, and application delivery have different ownership boundaries.

This introduces a practical question:

How can we continue managing infrastructure through Terraform while respecting the operational autonomy of the teams responsible for other parts of the environment?

I don’t think there’s a single answer that applies to every organization. What follows are some of the challenges I’ve encountered, the architectural decisions I’m exploring, and the trade-offs I consider relevant to our environment.

1. Understanding the Environment and Its Boundaries

Our environment follows a multi-account model, with separate AWS accounts for development, pre-production, and production.

Several teams participate in operating the platform:

Network team: Responsible for core networking components such as VPCs, subnets, and connectivity.
Security team: Responsible for centralized security policies, edge protection, and components such as the F5 WAF.
Platform team: Responsible for infrastructure components supporting application workloads, including ECS, load balancers, databases, and infrastructure automation.
Application teams and external partners: Responsible for developing and delivering applications, with production access mediated through internal processes.

These responsibilities introduce an important distinction: not everything an application depends on should necessarily be managed by the same Terraform configuration.

The challenge is not simply technical. It is also about respecting ownership boundaries.

Decision 1: Reference Infrastructure Owned by Other Teams

In our case, some infrastructure components already exist and are managed independently of the application platform.

For example, the networking team may own the VPC and subnets used by our workloads. Having our Terraform configuration manage those resources directly would introduce overlapping responsibilities.

For this reason, I prefer referencing externally managed resources through Terraform data sources when the corresponding AWS provider supports them.

Source codehcl


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


data "aws_vpc" "platform_vpc" {
 filter {
 name = "tag:Name"
 values = ["PROD-NETWORK-VPC"]
 }
}

data "aws_subnets" "application" {
 filter {
 name = "vpc-id"
 values = [data.aws_vpc.platform_vpc.id]
 }

 tags = {
 Tier = "Private-App"
 }
}

This allows our Terraform configuration to consume existing infrastructure without assuming responsibility for its lifecycle.

Of course, using data sources doesn’t eliminate dependencies. If another team changes or removes a referenced resource, our deployments may still be affected.

But in our organizational structure, this separation better reflects how responsibilities are actually distributed.

2. Separating Infrastructure Management from Application Delivery

Another challenge comes from how Terraform interacts with ECS application deployments.

In our environment, application teams deliver new container images through internal GitLab CI/CD pipelines. These pipelines can register new ECS task definition revisions and update running services.

Terraform, however, may still contain references to the task definitions used during initial provisioning.

This creates a potential conflict.

Consider the following scenario:

Terraform provisions an ECS service using an initial task definition.
An application pipeline deploys a new version and updates the ECS service.
A platform engineer later runs Terraform to modify an unrelated infrastructure component.
Terraform detects that the ECS service differs from its configuration and may attempt to restore the previously declared task definition.

This is not necessarily a Terraform problem. It is a consequence of having two different workflows attempting to control the same resource attribute.

Decision 2: Define Which Workflow Owns Each Attribute

For our environment, I’m considering a clearer separation between the infrastructure lifecycle and the application release lifecycle.

Terraform remains responsible for provisioning and managing infrastructure components such as ECS clusters, capacity providers, networking configuration, load balancers, and the initial service structure.

Application delivery pipelines manage new task definition revisions and ECS service deployments.

One way to support this arrangement is to use Terraform’s ignore_changes lifecycle argument:

Source codehcl


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


resource "aws_ecs_service" "application" {
 name = "app-core-service"
 cluster = aws_ecs_cluster.platform.id
 task_definition = aws_ecs_task_definition.initial.arn
 desired_count = 2

 lifecycle {
 ignore_changes = [
 task_definition,
 desired_count
 ]
 }
}

In this configuration, Terraform does not attempt to reconcile changes to these attributes for an existing service during normal planning.

There is an important trade-off here.

By ignoring changes to task_definition, we are intentionally giving up Terraform’s ability to detect and correct differences in that attribute. The deployment pipeline becomes responsible for managing it.

The same applies to desired_count, which should only be ignored if another mechanism, such as Application Auto Scaling or an operational workflow, is expected to manage the running task count.

3. Security Groups That Reflect the Actual Traffic Flow

Security group management is another area where ownership and infrastructure design intersect.

Our application traffic follows a path that includes an institutional F5 WAF before reaching the AWS Application Load Balancer and ECS workloads.

A simplified representation is:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


Client
 |
 v
F5 WAF
 |
 v
Application Load Balancer
 |
 v
ECS Services
 |
 +----> Database
 |
 +----> Shared Storage

Given this topology, my preference is to make security group rules reflect the actual communication requirements between components.

For example, where the network architecture supports it:

The ALB accepts application traffic from the expected WAF endpoints.
ECS tasks accept application traffic from the ALB security group.
Database services accept connections only from the application components that require access.
Shared storage access is limited to the appropriate workloads and protocols.

This approach helps reduce unnecessarily broad ingress permissions.

However, it also requires understanding the infrastructure outside AWS, particularly how traffic reaches the ALB through the F5 appliances.

Depending on the network architecture, preserving source addresses, NAT behavior, routing, and the availability of security group references can influence the implementation.

For this reason, I don’t consider a generic security group template sufficient.

The rules need to represent the actual traffic flow of our environment, not just an idealized architecture diagram.

4. Reducing Manual Work in Application Deployments

One operational concern in our setup is the separation between application delivery and production infrastructure access.

External application partners should not require broad AWS production permissions just to release a new version of their applications.

At the same time, making every release dependent on a manual intervention from the infrastructure team introduces unnecessary operational work.

The challenge is finding a balance between controlled access and deployment autonomy.

Decision 3: Use Internal CI/CD as the Deployment Boundary

An approach that makes sense for our environment is to use internal GitLab pipelines as the controlled interface for production deployments.

The intended workflow is:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


Application Change
 |
 v
GitLab CI - Build & Test
 |
 v
Container Image Publication
 |
 v
Image Promotion / Synchronization
 |
 v
Production Approval Gate
 |
 v
Register ECS Task Revision
 |
 v
Update ECS Service

Under this model, application teams interact with the deployment process rather than directly managing AWS production resources.

The pipeline can use scoped IAM permissions, with short-lived credentials through OIDC where supported and configured.

An approval gate can be applied according to our production change-control requirements.

Terraform remains responsible for the infrastructure configuration, while the deployment pipeline handles application revisions.

This separation does not eliminate the need for monitoring, rollback procedures, or permission management.

But it potentially reduces repetitive operational tasks while maintaining the access boundaries required by our organization.

5. Making Terraform Changes More Predictable

The main reason I want to revisit our Terraform practices is not simply to automate more resources.

It is to make subsequent infrastructure changes easier to understand, review, and execute.

In a production environment, a Terraform plan that contains unexpected changes to databases, application revisions, and networking resources is difficult to approve with confidence.

A few practices appear particularly relevant to our situation.

Separating Terraform State by Operational Responsibility

I prefer separating Terraform state according to meaningful infrastructure boundaries.

For example:

Source codePlain text


1
2
3
4
5
6


terraform-platform/
└── environments/
 └── prod/
 ├── 01-base-iam/
 ├── 02-datastores/
 └── 03-compute-ecs/

This structure is only illustrative. The appropriate boundaries depend on resource dependencies, ownership, and how changes are approved.

Separating state can reduce the scope of an individual plan and make changes easier to review.

It does not guarantee complete isolation, however. Resources managed in different states may still depend on one another through IDs, outputs, networking, or application behavior.

The goal is to reduce unnecessary coupling, not to assume that independent state files imply independent systems.

Protecting Stateful Resources

For infrastructure such as production databases, accidental resource replacement is a concern.

Terraform’s prevent_destroy lifecycle argument is one safeguard worth considering.

Source codehcl


1
2
3
4
5
6
7


resource "aws_db_instance" "production" {
 identifier = "prod-db-postgres"

 lifecycle {
 prevent_destroy = true
 }
}

This prevents Terraform from planning the destruction of that resource while the lifecycle rule remains in the configuration.

However, it is not a substitute for AWS deletion protection, backups, access controls, or change reviews.

In particular, removing the resource configuration may also remove the lifecycle protection, so it should not be treated as an absolute safeguard.

For our environment, I see it as one layer in a broader production protection strategy.

Detecting Unexpected Changes

Another practice I’m evaluating is running automated Terraform plans to identify differences between the declared configuration and the current infrastructure.

For instance:

Source codebash

1

terraform plan -detailed-exitcode

The exit codes can distinguish between an unchanged configuration, a plan containing proposed changes, and an execution error.

One limitation is that not every difference represents unintended drift. Some changes may be expected, while others result from configuration updates rather than manual infrastructure modifications.

A dedicated refresh-only plan can help identify changes in remote infrastructure:

Source codebash

1

terraform plan -refresh-only -detailed-exitcode

In either case, the results still require interpretation.

The objective is to make unexpected changes visible before they become part of a production deployment.

6. The Trade-Offs I’m Willing to Make

Looking at these decisions together, the underlying theme is ownership.

In our environment, there are several competing objectives:

Infrastructure should remain reproducible and manageable through code.
Application teams need a practical way to release software.
Network and security teams must retain control over their respective components.
Production changes need to remain auditable and predictable.

These objectives do not always align automatically.

Using ignore_changes means accepting that some resource attributes will no longer be reconciled by Terraform.

Separating state means introducing explicit dependencies between independently managed infrastructure layers.

Using CI/CD for production deployment means investing in secure pipeline identities, approval mechanisms, and recovery procedures.

None of these choices comes without operational cost.

But for the environment I’m working with, I believe these trade-offs are worth examining because they address the actual boundaries between our teams and systems.

Final Thoughts

My intention is not to argue that every organization should structure Terraform this way.

An environment where a single platform team owns networking, infrastructure, and application delivery may benefit from a completely different approach.

The same applies to organizations using Kubernetes, GitOps-based workflows, or other deployment mechanisms.

What I’m trying to improve in our context is the relationship between infrastructure provisioning and ongoing platform operations.

Terraform should not necessarily stop being useful once the environment is created. At the same time, keeping Terraform involved in Day-2 operations requires deciding which parts of the infrastructure it should continue to control.

For me, the most important takeaway is that Day-2 infrastructure management is as much about defining ownership boundaries as it is about writing Terraform code.

The next step is to validate these boundaries through actual operational changes, review the resulting Terraform plans, and adjust the approach based on what works in our environment.

That is the direction I’m pursuing: not a universal Terraform architecture, but a more deliberate and predictable way of operating the platform I work with.

Production Engineering for Single-Node Workloads: Controlled Releases, SQLite Backups, and Native ARM Builds

Fri, 23 Oct 2026 00:00:00 +0000

I originally built IBB Kids as a volunteer project to streamline weekly classroom material management for a non-profit community. Teachers needed instant access to semester schedules, locations, and lesson links without digging through shared spreadsheets.

While the functional requirement was straightforward, the operational constraint was non-negotiable: the system had to run at zero cost on an ARM-based cloud instance with near-zero maintenance overhead, while remaining reliable enough to be operated by non-technical volunteers.

Ten months after its initial launch, the core challenge shifted from feature development to production engineering: managing JWT session revocation, building native C++ modules for the target architecture, controlling releases, and verifying consistent database backups.

This post documents the architectural decisions, resilience patterns, and operational lessons learned while running a single-node SQLite workload in production up to October 2026.

Architectural Steering & AI-Augmented Engineering

To maintain high delivery velocity while keeping my primary focus on system reliability, state consistency, and platform security, I adopted an AI-augmented software engineering workflow:

Architectural Steering & Systems Design (Human): I designed the system topology, multi-environment isolation model, database state transition rules, JWT session versioning schemes, SQLite WAL backup strategies, native C++ binding validation scripts, and CI/CD rollback procedures.
Code Implementation (Codex): I leveraged OpenAI Codex as an execution engine to translate my technical specifications, API schemas, and architectural constraints into full-stack TypeScript (React + Express + SQLite) code.
Validation & Production Hardening (Human): I established the operational quality gates—including unit/integration testing with isolated in-memory SQLite instances, static security analysis via Semgrep SAST, build-id health checks, and native binding checks for ARM64 Linux runtimes.

This division of responsibilities allowed me to act as a Tech Lead / Systems Architect, focusing strictly on system resilience and operability without spending hundreds of hours writing boilerplate application code.

System Architecture: Single-Node Edge Topology

The application relies on a lightweight, decoupled stack designed to maximize performance on low-spec hardware while enforcing strict domain boundary separation:

Frontend: React with TypeScript, bundled by Vite, styled via Tailwind CSS v4.
Backend: Express API server hosting static production assets.
Database: SQLite running locally via better-sqlite3.
Process Management: PM2 enforcing daemon isolation.
Edge Proxy: Nginx managing HTTPS termination via Certbot/Let’s Encrypt certificates on Oracle Linux ARM.

Source codePlain text


1
2
3
4
5
6
7


Browser (React Single-Page App)
 ↓ HTTPS (Port 443)
Nginx (TLS Termination & Reverse Proxy)
 ↓ HTTP Proxy (Internal Loopback)
Express Server (JWT Auth, REST API, Static Asset Delivery)
 ↓ C++ Native Bindings (better-sqlite3)
SQLite Database (WAL Mode: Schedule, Materials, Credentials)

In development, Vite proxies /api calls to Express. In production, Vite is not running; Express directly serves optimized static assets alongside the API endpoints.

Native C++ Bindings Across Environments

Developing on Windows via WSL2 (x86_64) and deploying to Oracle Linux ARM (ARM64) introduced native-module compatibility problems—specifically when loading native C++ modules like better-sqlite3 and bcrypt.

A common failure mode in multi-platform deployments is the dreaded invalid ELF header error, caused by attempting to execute C++ binaries compiled for x86_64 inside an ARM64 Linux runtime (or sharing node_modules across WSL2 and host Windows filesystems).

Rules for Native Module Isolation

Strict Environment Boundary: Never share or mount node_modules between Windows, WSL2, or remote Linux servers.
Version Pinning: Enforce Node.js runtime alignment using .nvmrc (pinned to Node 22 LTS) and commit explicit lockfiles (package-lock.json).
Pre-flight Native Compilation Check: Introduced a dedicated npm run prepare-native lifecycle script during the build step:

Source codebash


1
2
3
4
5
6


nvm install
nvm use
npm ci
npm run prepare-native
npm test
npm run build

Rebuilding on the ARM server is native compilation for that server. Cross-compilation would mean building for ARM on a different target architecture with a suitable toolchain.

The prepare-native script attempts to initialize an in-memory SQLite instance and test bcrypt hash routines. If native bindings fail to load due to an architecture mismatch, the script automatically invokes node-gyp to recompile C++ bindings against local system tools (using GCC Toolset and Python 3.12) before validating the result.

SQLite in Production: WAL Mode, Backup Integrity, and OCI Sync

With WAL enabled, committed changes may still be in the WAL rather than the main database file. Copying only a live .sqlite file can miss those changes or capture an inconsistent copy. A backup must preserve a consistent committed state across SQLite’s storage files.

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[ Live Application Writes ]
 ↓
 SQLite DB File + WAL Journal
 ↓
[ SQLite Online Backup API ] ---> Copies Consistent Committed State
 ↓
[ PRAGMA Integrity Checks ] ---> Checks Structure & Foreign Keys
 ↓
[ Compressed Local Snapshot ]
 ↓
[ OCI Instance Principal ] ---> Authenticated Direct Stream to Private Object Storage

1. Online Backup API Execution

Backups use the SQLite Online Backup API, exposed via better-sqlite3, to create a consistent database copy. It reads through SQLite, including committed data held in the WAL. It does not commit pending transactions or require a manual flush of uncommitted frames. Incremental backup steps acquire the locks needed for those steps. See the SQLite backup documentation.

2. Automated Integrity Validation

Before any backup is marked valid, the pipeline executes rigorous structural checks:

Source codesql

1
2

PRAGMA integrity_check;
PRAGMA foreign_key_check;

If either query returns errors, the snapshot is instantly discarded, and an alert is raised.

3. Keyless Cloud Replication via OCI Instance Principals

Once validated locally, daily backup archives are streamed directly to a private Oracle Cloud Infrastructure (OCI) Object Storage bucket.

To prevent security credential leaks, the VM relies on OCI Instance Principals (IAM dynamic group policies attached to the VM instance). No static private keys, AWS-style secret pairs, or cloud tokens exist in .env files or application code.

4. Controlled Restore and Session Invalidation

Restoring a backup automatically triggers:

File validation and integrity checks on the candidate snapshot.
A compulsory safety backup of the current live database.
Entering a maintenance window, stopping the application and other database writers, and closing their database connections.
Replacing the database using the restore procedure, handling WAL/SHM files as part of that offline replacement, and invalidating sessions.
Application restart and automated /health check validation.

Process Management and Controlled Releases

Relying on git pull and in-place npm restart on a production server creates a window of downtime and risks leaving the application broken if native compilation or build steps fail halfway through.

Why I Chose PM2 Fork Mode

I chose one application process to simplify this deployment. That was an operational choice, not a SQLite requirement:

Write Contention: SQLite supports multiple processes on the same host, with one writer at a time. Keeping one application process simplified transaction management for this workload. Multiple workers can be valid, but need short transactions, busy handling and workload testing. Contention alone is not evidence of database corruption. See SQLite WAL concurrency.
Runtime and Loader Configuration: Using tsx and PM2 together requires a tested interpreter and loader configuration. Fork mode reduced the number of moving parts in this setup; loader failures should not be generalized to every cluster-mode deployment.
Explicit Runtime Selection: Recording the Node version and checking the interpreter used by PM2 makes runtime upgrades easier to diagnose. The daemon and application configuration still need to be checked together.

Versioned Release Directories

Deployments use versioned release directories to prepare a candidate without overwriting the currently deployed files:

Source codePlain text


1
2
3
4
5
6
7
8


/opt/ibbkids/
├── current -> /opt/ibbkids/releases/20261023_140000/
├── shared/
│ ├── .env
│ └── database.sqlite
└── releases/
 ├── 20261020_091500/
 └── 20261023_140000/ (Active Version)

Deployment Lifecycle Execution

Checkout SHA: The CI/CD runner fetches the exact commit SHA into a new folder under /releases/<timestamp>.
Build & Prepare: Dependencies are installed (npm ci), C++ native bindings are compiled (npm run prepare-native), unit tests run against an isolated SQLite instance, and Vite builds static assets.
Controlled Process Replacement: The single application process is restarted against the candidate release using the configured Node runtime. This creates an interruption; its duration needs measurement.
Healthcheck Probe Verification: The deployment script polls the /health endpoint. The endpoint returns a unique JSON payload containing the release commit SHA:

Source codejson


1
2
3
4
5
6


{
 "status": "ok",
 "environment": "production",
 "deployId": "sha-8f2a1b9c",
 "dbStatus": "connected"
}

Release Acceptance: Accept the candidate only if its health response contains the expected deployId. An atomic update of current selects a release directory, but does not by itself replace a running process or change Nginx’s HTTP upstream.
Rollback: If validation fails, return to the previous code release and restart it. This only works if the database schema remains compatible. A code rollback does not undo migrations or writes made by the candidate.

A zero-interruption design would additionally require separate candidate and active endpoints, a health-gated proxy switch, connection draining and compatible database changes. None of those properties follows from a symlink switch alone. PM2 documents zero-downtime reload for cluster-mode applications; this single-process procedure should be evaluated as a controlled restart.

Stateful Security & Deterministic Session Invalidation

JWTs are inherently stateless, which presents a security challenge: if an administrative credential is changed, previously issued JWTs remain valid until their natural expiration.

Session Versioning via `auth_state`

To solve this without adding a heavy Redis layer, I implemented a database-backed session versioning scheme:

Source codesql


1
2
3
4
5


CREATE TABLE auth_state (
 profile TEXT PRIMARY KEY,
 session_version INTEGER NOT NULL DEFAULT 1,
 updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

Every issued JWT payload includes the current sessionVersion for that profile.
Incoming requests verify the JWT signature, allowed algorithm, expiry and configured issuer/audience before checking sessionVersion against auth_state. Decoding alone is not authentication.
When a user updates their password, a SQLite transaction updates the bcrypt password hash and increments session_version:

Source codesql


1
2
3
4


BEGIN TRANSACTION;
UPDATE credentials SET password_hash = ? WHERE profile = ?;
UPDATE auth_state SET session_version = session_version + 1 WHERE profile = ?;
COMMIT;

Tokens carrying the previous version are rejected on their next validation. Every protected request must perform that check; caches and long-lived connections require their own invalidation policy.

A database restore can roll the version counter back and accidentally make an old token valid again. A restore procedure must also rotate the signing key or replace a session epoch held outside the restored database. Incrementing a restored counter alone is insufficient.

CI/CD Security Analysis (Semgrep SAST)

The GitHub Actions pipeline includes a dedicated security job running Semgrep Community Edition (Python 3.12):

Scans pull requests and pushes to dev/develop branches.
Enforces rules for hardcoded secrets, SQL injection vectors, and unsafe path interpolations.
Exports results as SARIF artifacts for historical audit without relying on paid third-party security dashboards.

Operational Evolution After 10 Months (October 2026 Review)

Returning to the system after 10 months of real-world use provided concrete feedback from the volunteers operating the application:

Operational Dimension	Initial Design (Dec 2025)	Hardened Architecture (Oct 2026)
Material Management	Teacher and Parent material links	Added Visual Aids tracking across database, APIs, and UI layers
Data Granularity	Single count of incomplete lessons	Split tracking for missing Teacher, Parent, and Visual materials
Runtime & PM2 Isolation	Hybrid Node 20 / Node 22 PM2 daemons	Unified across environments on Node 22 LTS (ARM64) using PM2 `fork` mode
Database Purging	Manual SQL execution	Password verification, explicit confirmation and a consistent safety backup before destructive changes
Backup Redundancy	Local disk snapshots only	Local integrity-verified snapshots + OCI Object Storage sync via IAM Instance Principals
Test & Security Coverage	Basic build checks	Unit tests and Semgrep checks; counts vary with the revision and should be read from the matching CI run

Key Takeaways for SREs and Platform Engineers

Recovery and Availability Are Different: Backups and versioned releases improve recovery, but one host remains a single failure domain. Host maintenance or failure can take the application offline. Record recovery tests and measured downtime before claiming an availability target.
Treat AI as a Force Multiplier, Not a Black Box: Delegating code generation to tools like Codex works best when the engineer owns the system architecture, boundary constraints, test design, and failure modes.
Choose Process Management Deliberately: One process simplified this workload. SQLite can also serve multiple local processes; concurrency settings, transaction length and observed contention should guide that choice.
Test Restores, Not Just Backup Creation: Create a consistent backup through SQLite, check its integrity, and periodically restore it into an isolated environment. Integrity checks alone do not demonstrate that the application can recover within the required time.

Revision note

October 2026: corrected the explanations of WAL backups, concurrency and native compilation. This article describes a single-host operational design, not a demonstrated zero-downtime or highly available deployment. Availability, restore duration and CI counts need evidence from the corresponding release.

References

Automating Ceph Storage Forecasting with Grafana Mimir

Wed, 17 Sep 2025 00:00:00 +0000

Context and decisions at the time

Our team urgently needed this information for the hardware acquisition process, so I built what was essential and put it to work as fast as possible. Since I’m no expert on analytics, my colleague provided the logic and I first built everything as a Jupyter Notebook, and later moved it to a place where it could be scheduled and had direct access to Mimir.

That need shaped the decisions: integrate collection, processing, forecasting and communication into one workflow; use a simple, understandable projection to support planning; and deliver charts and summaries through Teams without preparing each report manually. More sophisticated models and other improvements could follow after the urgent delivery.

Intro

Capacity planning for distributed storage clusters is a constant challenge, especially in environments where rapid data growth can quickly turn into capacity risks or urgent procurement scenarios. At SERPRO, I developed a Python-based solution that automates the collection of Ceph usage metrics from Grafana Mimir, analyzes historical utilization trends, applies outlier correction, and projects forecasted consumption up to a year ahead for multiple sites and device classes. The system not only visualizes capacity usage with dynamic, outlier-robust trends and threshold indicators but also generates concise summary blocks—automatically posted into Microsoft Teams channels in structured Adaptive Cards. By integrating data science with seamless team communication, this workflow provides actionable insights that help our team act proactively on storage forecasting challenges.

What was done

The core objective of this project is to automate capacity forecasting for Ceph storage clusters by integrating data retrieval, processing, visualization, and reporting into a single Python workflow. The process starts by querying Grafana Mimir via its Prometheus API to fetch historical and recent metrics for raw and used bytes across device classes and sites (in this case, São Paulo and Brasília). The workflow uses structured PromQL queries to collect not only storage usage but also device classification for more granular analysis.

Once the raw metric data is gathered, it is merged into a unified pandas DataFrame. The method handles gaps and missing values through daily interpolation, ensuring the time series needed for statistical analysis and plotting remain continuous and regularized. Outlier treatment comprises delta computation and the application of a rolling standard deviation filter. Sudden, anomalous jumps in daily usage (likely due to transient glitches or exceptional events) are replaced by the median daily delta, ensuring that longer-term trends are not skewed by momentary artifacts.

Forecasting is handled through a simple, robust approach: the system projects usage growth forward using the historical median daily increase, assuming linear growth for the next 365 days. This matches the practical needs for capacity planning where quick, understandable trends trump overfitting or overcomplex modeling.

For each dimension (site and device class), the script generates forecast graphs with matplotlib and seaborn, overlaying current usage, projections, and colored thresholds (75%, 85%, and 100% of available capacity) for visual clarity. Alongside plots, the workflow prepares a text summary containing key metrics such as projected fill dates, growth rates, and estimated time to reach critical thresholds.

Finally, everything is packaged and sent to Microsoft Teams via webhook, using Adaptive Cards for structured, readable inline dashboards. The automation ensures that these insights are always up to date and readily available to stakeholders, without manual intervention or separate reporting pipelines.

Data Treatment and Forecasting Approach

The data treatment phase is crucial for ensuring that the usage forecasts are reliable and robust. The raw metrics retrieved from Grafana Mimir often contain noise, missing values, and occasional outliers—irregular spikes or drops that do not represent typical usage patterns but can distort trend analysis if left uncorrected.

To address this, the workflow first aligns the time series data by resampling it to a daily frequency and fills any missing data points via linear interpolation. This guarantees a continuous timeline, which is essential for consistent statistical calculations and regression modeling.

Outlier handling focuses on the daily usage increments (the difference between consecutive days). The script calculates these deltas for usage metrics and computes their rolling standard deviation to statistically identify abnormal jumps. Values exceeding this threshold are considered outliers and replaced with the median delta value calculated from the non-outlier data. This method smooths erratic fluctuations while preserving the underlying growth trend.

By repairing outliers at the delta level rather than raw usage values directly, the model maintains the cumulative consistency of the metric. The corrected deltas are then cumulatively summed to reconstruct a corrected usage time series. This approach effectively mitigates distortion from anomalous data points without discarding real growth signals.

For forecasting, the corrected historical increments are used to project future growth linearly over a defined horizon (365 days). The median daily delta serves as a stable, robust estimate of expected usage increase per day. While more sophisticated models exist, this method balances simplicity and practicality, yielding easily interpretable results and forecasts useful for capacity planning. It avoids overfitting that can occur with complex models on noisy real-world operational data.

Overall, this combination of careful interpolation, statistically informed outlier correction, and straightforward median-based linear forecasting creates a solid foundation for understanding and anticipating Ceph storage consumption trends with minimal computational complexity and high transparency.

The code

Disclaimer: Some variables will be in portuguese.

forecast.pypython · 412 linesDownload


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412


import os
import io
import json
import base64
import logging
import urllib3
import requests
import pandas
from datetime import datetime, timedelta, timezone
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import matplotlib.ticker as ticker

output_dir = '/srv/scripts'
TiB = 1099511627776 # Is this really necessary?

logging.basicConfig(
 level=logging.INFO,
 format='%(asctime)s - %(levelname)s - %(message)s',
 filename=os.path.join(output_dir,'capacidade-forecast.log'),
 filemode='w'
)
logger = logging.getLogger(__name__)
logger.info(f"-----Inicio-----")

## 1. Query Configuration
logger.info("1. Configuração da Query")
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

url = "https://mimir-endpoint/prometheus/api/v1/query_range"
headers = {
 'X-Scope-OrgID': 'anonymous'
}

queries = {
 "Capacidade SPO": "sum by (device_class) (ceph_osd_stat_bytes{site='spo'} + on(site,ceph_daemon) group_left(device_class) (0*ceph_osd_metadata{site='spo'}))",
 "Capacidade BSA": "sum by (device_class) (ceph_osd_stat_bytes{site='bsa'} + on(site,ceph_daemon) group_left(device_class) (0*ceph_osd_metadata{site='bsa'}))",
 "Uso SPO": "sum by (device_class) ((ceph_osd_stat_bytes_used{site='spo'}) + on(site,ceph_daemon) group_left(device_class) (0*ceph_osd_metadata{site='spo'}))",
 "Uso BSA": "sum by (device_class) ((ceph_osd_stat_bytes_used{site='bsa'}) + on(site,ceph_daemon) group_left(device_class) (0*ceph_osd_metadata{site='bsa'}))"
}

current_utc = datetime.now(timezone.utc)
end_date = (current_utc).replace(hour=0, minute=0, second=0, microsecond=0)
start_date = datetime(2024, 9, 27, 0, 0, 0) # No data prior to this date

params_template = {
 "start": start_date.strftime("%Y-%m-%dT%H:%M:%SZ"),
 "end": end_date.strftime("%Y-%m-%dT%H:%M:%SZ"),
 "step": "1d"
}

## 2. Query Execution
logger.info("2. Execução da Query")
dataframe = pandas.DataFrame()

for query_name, query in queries.items():
 params = params_template.copy()
 params["query"] = query
 response = requests.get(url, params=params, headers=headers, verify=False)
 try:
 response.raise_for_status()
 mimir_data = response.json()['data']['result']
 if not mimir_data:
 logger.error(f"No data returned for query: {query_name}")
 continue
 for result in mimir_data:
 if 'device_class' not in result['metric']:
 continue
 device_class = result['metric']['device_class']
 temp_df = pandas.DataFrame(result['values'], columns=['timestamp', f'{query_name} {device_class}'])
 temp_df['timestamp'] = pandas.to_datetime(temp_df['timestamp'], unit='s')
 if dataframe.empty:
 dataframe = temp_df
 else:
 dataframe = pandas.merge(dataframe, temp_df, on='timestamp', how='outer')
 except requests.exceptions.HTTPError as err:
 logger.error(f"HTTP error for query '{query_name}': {err}\nResponse body: {response.text}")
 except KeyError:
 logger.error(f"Unexpected response format for query '{query_name}': {response.json()}")

# Save as CSV
#output_file = 'device_data.csv'
#dataframe.to_csv(output_file, index=False)

## 3. Treat the Data
logger.info("3. Tratamento dos Dados")
# Define the index
dataframe = dataframe.rename(columns={'timestamp': 'Time'})
dataframe.set_index('Time', inplace=True)

# Convert values to numeric
numeric_columns = dataframe.columns.drop('Time') if 'Time' in dataframe.columns else dataframe.columns
for col in numeric_columns:
 dataframe[col] = pandas.to_numeric(dataframe[col], errors='coerce')

# Interpolate missing values
dataframe = dataframe.resample('D').interpolate(method='linear')
dataframe.reset_index(inplace=True)

## Treat the Outliers
cols_uso = [col for col in dataframe.columns if col.startswith('Uso')]
std_dev = {}
medianas = {}

# Calculate Delta and Standard Deviation
for col in cols_uso:
 _, site, device_class = col.split(' ')
 delta_col = f'Delta {site} {device_class}'
 delta_corrigido_col = f'Delta Corrigido {site} {device_class}'
 uso_corrigido_col = f'Uso Corrigido {site} {device_class}'
 
 # Calculate Delta
 dataframe[delta_col] = dataframe[col].diff()
 
 # Calculate Standard Deviation
 std_dev = dataframe[delta_col].iloc[1:].std()
 logger.debug(f"Desvio Padrão {site} {device_class}: {std_dev/TiB:.2f} TiB")
 
 # Apply the outlier correction
 dataframe[delta_corrigido_col] = dataframe[delta_col].where(
 (abs(dataframe[delta_col]) <= std_dev) | (dataframe.index == dataframe.index[0]), 0
 )
 
 # Calculate the delta median without the outliers
 medianas[col] = dataframe[delta_corrigido_col].iloc[1:].median()
 
 # Fill zero/empty values with the median
 dataframe[delta_corrigido_col] = dataframe[delta_corrigido_col].where(
 (dataframe[delta_corrigido_col] != 0) | (dataframe.index == dataframe.index[0]), 
 medianas[col]
 )

# Calculate Corrected raw usage
cols_delta = [col for col in dataframe.columns if col.startswith('Delta Corrigido')]
for col in cols_delta:
 _, _, site, device_class = col.split()
 uso_col = f"Uso {site} {device_class}"
 corrigido_col = f"Uso Corrigido {site} {device_class}"
 
 # Initialize the 'Corrected Usage' column with the first value from the 'Usage' column
 dataframe[corrigido_col] = dataframe[uso_col].iloc[0]
 
 # Calculate the cumulative sum of the corrected delta, starting from the second row
 dataframe.loc[1:, corrigido_col] += dataframe.loc[1:, col].cumsum()

# Drop unecessary columns
cols_drop = [col for col in dataframe.columns if col.startswith('Delta')]
dataframe = dataframe.drop(columns=cols_drop)

# Save to .csv
# output_file = 'device_data.csv'
# dataframe.to_csv(os.path.join(output_dir, output_file), index=False)


## 4. Calculation of the Linear Regression
logger.info("4. Cálculo da Regressão Linear")
# Gerar o Horizonte da Previsão
ultima_coleta = dataframe['Time'].max()
data_inicial = ultima_coleta.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(days=1)
forecast_horizonte = 365
forecast_datas = pandas.date_range(start=data_inicial, periods=forecast_horizonte)
forecast_df = pandas.DataFrame({'Time': forecast_datas})

# Generate Forecasts
for col in cols_uso:
 _, site, device_class = col.split(' ')
 forecast = [dataframe[col].iloc[-1]]
 for _ in range(1, forecast_horizonte):
 forecast.append(forecast[-1] + medianas[col]) 
 column_name = f"{site}_{device_class}"
 forecast_df[column_name] = forecast

forecast_df.tail()

## 5. Generate the Summary
logger.info(f"5. Geração do Resumo")
results = {}
summary_text = ""
current_date = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)

for col in cols_uso:
 _, site, device_class = col.split(' ')
 forecast_col = f"{site}_{device_class}"
 
 # Obtain the Capacity and calculate the Rate
 capacidade = dataframe[f'Capacidade {site} {device_class}'].iloc[-1] * 0.75 / TiB
 uso_atual = dataframe[col].iloc[-1] / TiB
 uso_final = forecast_df[forecast_col].iloc[-1] / TiB
 taxa = (uso_final - uso_atual) / forecast_horizonte 
 
 # Check the Threshold
 threshold_mask = (forecast_df[forecast_col] / TiB) >= capacidade
 if threshold_mask.any():
 threshold_date = forecast_df.loc[threshold_mask, 'Time'].min().strftime('%d-%m-%Y')
 else:
 threshold_date = "Limite não atingido no período"
 
 # Calculate remaining Days
 days_remaining = abs((capacidade - uso_atual) / taxa)
 years, months, days = int(days_remaining // 365), int((days_remaining % 365) // 30.42), round((days_remaining % 365) % 30.42)
 doomsday = (dataframe['Time'].iloc[-1] + timedelta(days=days_remaining)).strftime('%d-%m-%Y')
 doomsday_f = (f"{years} anos, {months} meses e {days} dias")
 
 # Generate the Summary
 summary_text += f"""
Resumo {site} {device_class}:
Capacidade (75%): {capacidade:.2f} TiB
Uso em 31/12/2025: {uso_final:.2f} TiB
Taxa de crescimento diária: {taxa:.2f} TiB/dia
Quando atinge 75%: {doomsday}
Tempo até atingir 75%: {doomsday_f}
"""

## 6. Generate the Graphs
logger.info("6. Geração dos Gráficos")
# Obter Sites e Classes
sites = set()
device_classes = set()

for col in dataframe.columns:
 if col.startswith('Uso'):
 parts = col.split()
 sites.add(parts[1])
 device_classes.add(parts[2])

# Generate the Plots
plots = {}
for site in sites:
 for device_class in device_classes:
 try:
 # Verify the columns
 uso_col = f'Uso {site} {device_class}'
 uso_corrigido_col = f'Uso Corrigido {site} {device_class}'
 capacidade_col = f'Capacidade {site} {device_class}'
 forecast_col = f'{site}_{device_class}'
 
 if not all(col in dataframe.columns for col in [uso_col, uso_corrigido_col, capacidade_col]):
 continue

 if forecast_col not in forecast_df.columns:
 continue

 plt.figure(figsize=(12, 6))

 # Plot Lines
 sns.lineplot(x=dataframe['Time'], y=dataframe[uso_col]/TiB,
 label='Uso Bruto', linewidth=1.5, color='purple')
 sns.lineplot(x=forecast_df['Time'], y=forecast_df[forecast_col]/TiB,
 label='Previsão', linestyle='--', linewidth=2, color='navy')

 # Plot Thresholds
 capacity_tib = dataframe[capacidade_col].iloc[-1]/TiB
 limits = [
 (capacity_tib * 0.75, '75%', 'gold'),
 (capacity_tib * 0.85, '85%', 'darkorange'),
 (capacity_tib, '100%', 'crimson')
 ] 
 for value, label, color in limits:
 plt.axhline(y=value, color=color, linestyle='-.',
 linewidth=1.5, alpha=0.8, label=label)

 # Format
 plt.gca().xaxis.set_major_locator(mdates.MonthLocator())
 plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m'))
 plt.xticks(rotation=45, ha='right')
 plt.gca().yaxis.set_major_formatter(ticker.FuncFormatter(lambda x, _: f'{x:,.1f} TiB'))
 plt.gca().yaxis.set_major_locator(ticker.MaxNLocator(10))
 plt.title(f'Previsão Ceph {site} - Classe {device_class.upper()}', fontsize=16, pad=20)
 plt.xlabel('Data', fontsize=12, labelpad=10)
 plt.ylabel(f'Uso {device_class.upper()} (TiB)', fontsize=12, labelpad=10)
 plt.grid(True, linestyle='--', alpha=0.6)

 # Configure Subtitles
 handles, labels = plt.gca().get_legend_handles_labels()
 order = [0, 1, 2, 3, 4]
 plt.legend([handles[idx] for idx in order], [labels[idx] for idx in order], 
 loc='upper left', frameon=True, shadow=True, fontsize=12)
 plt.tight_layout()
 
 # Save Graphs
 buffer = io.BytesIO()
 plt.savefig(buffer, format='png', dpi=100, bbox_inches='tight')
 buffer.seek(0)
 plots[f'{site}_{device_class}'] = base64.b64encode(buffer.getvalue()).decode('utf-8')
 buffer.close()
 plt.close()
 
 except Exception as e:
 print(f"Error generating plot for {site} {device_class}: {str(e)}")
 plt.close()

## 7. Send to Teams
logger.info(f"7. Envio para o Teams")
# Create the message
def create_teams_message(summary_data, plots):
 cards = []
 sections = parse_summary_text(summary_data)
 
 for site in ['SPO', 'BSA']:
 card = {
 "type": "message",
 "attachments": [{
 "contentType": "application/vnd.microsoft.card.adaptive",
 "content": {
 "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
 "type": "AdaptiveCard",
 "version": "1.2",
 "body": [
 {
 "type": "TextBlock",
 "text": "São Paulo" if site == "SPO" else "Brasília",
 "weight": "Bolder",
 "size": "Large",
 "color": "Accent"
 },
 *create_device_sections(sections.get(site, {}))
 ]
 }
 }]
 }
 
 # Add site-specific plots
 site_plots = {k: v for k, v in plots.items() if k.lower().startswith(site.lower())}
 for plot_name, plot_data in site_plots.items():
 image_block = {
 "type": "Image",
 "url": f"data:image/png;base64,{plot_data}",
 "size": "auto"
 }
 card['attachments'][0]['content']['body'].append(image_block)
 
 cards.append(card)
 
 return [json.dumps(card) for card in cards]

# Convert the Summary Text
def parse_summary_text(text):
 sections = {}
 current_section = None
 for line in text.split('\n'):
 line = line.strip()
 if not line:
 continue
 if line.startswith('Resumo'):
 parts = line.replace('Resumo ', '').replace(':', '').split()
 site = parts[0]
 device_class = parts[1]
 if site not in sections:
 sections[site] = {}
 current_section = sections[site].setdefault(device_class, {}) 
 elif line and current_section is not None:
 key_value = line.split(':')
 if len(key_value) == 2:
 key = key_value[0].strip()
 value = key_value[1].strip()
 current_section[key] = value 
 return sections

# Create the Sections
def create_device_sections(devices):
 sections = []
 for device_class, data in devices.items():
 section = {
 "type": "Container",
 "style": "emphasis",
 "items": [
 {
 "type": "TextBlock",
 "text": f"Classe {device_class.upper()}",
 "weight": "Bolder",
 "size": "Small"
 },
 {
 "type": "FactSet",
 "facts": [
 {"title": "Capacidade (75%):", "value": data.get('Capacidade (75%)', 'N/A')},
 {"title": "Uso em 31/12/2025:", "value": data.get('Uso em 31/12/2025', 'N/A')},
 {"title": "Taxa de crescimento:", "value": data.get('Taxa de crescimento diária', 'N/A')},
 {"title": "Quando atinge 75%:", "value": data.get('Quando atinge 75%', 'N/A')},
 {"title": "Tempo até atingir 75%:", "value": data.get('Tempo até atingir 75%', 'N/A')}
 ]
 }
 ]
 }
 sections.append(section)
 return sections
 
# Send to Teams
def send_to_teams(payloads):
 # Chat Capacidade
 WEBHOOK_URL = "https://corp.webhook.office.com/webhookb2/id"

 PROXIES = {'https': 'http://proxy:3128'}
 headers = {'Content-Type': 'application/json'}
 statuses = []
 for payload in payloads:
 response = requests.post(
 WEBHOOK_URL,
 data=payload,
 headers=headers,
 proxies=PROXIES,
 verify=False
 )
 statuses.append(response.status_code)
 
 return statuses

# Create and send the message
messages = create_teams_message(summary_text, plots)
status = send_to_teams(messages)
logger.info(f"-----Fim-----")

Future Improvements

Advanced Forecasting Models: Instead of a linear extrapolation using median deltas, incorporate time series forecasting methods such as ARIMA, Prophet, or machine learning regression models to capture seasonality, trends, and potential non-linear growth patterns more accurately;
Real-Time or Near Real-Time Processing: Adapt the data collection and forecasting pipeline to support streaming or more frequent updates (e.g., hourly), enabling quicker reaction times to sudden usage changes or outages;
Anomaly Detection and Alerting: Integrate automatic anomaly detection techniques beyond simple standard deviation filtering to identify unexpected spikes or degradations. Pair these with proactive alerts in Teams or other channels to notify teams immediately;
Capacity Thresholds as Configurable Parameters: Allow dynamic adjustment of threshold levels (75%, 85%, 100%) via configuration files or environment variables instead of hardcoded values, improving flexibility for different environments or changing policies;
Improved Visualization: Enhance graphs by including confidence intervals on forecasts, interactive dashboards using tools such as Plotly Dash or Grafana panels, and support for multiple longer-term scenarios (e.g., optimistic, pessimistic);
Authentication & Security Enhancements: Secure API calls with proper authentication (OAuth tokens, certificates) and securely manage webhook URLs and proxy credentials;
Extensibility for Other Metrics: Generalize the pipeline to incorporate additional Ceph health and performance indicators (e.g., latency, IOPS, recovery status) for a more comprehensive capacity and cluster health dashboard.

The Ultimate Guide to the AWS Certified Solutions Architect – Professional (SAP-C02) Exam

Tue, 15 Jul 2025 00:00:00 +0000

This post aims to provide a comprehensive and detailed guide for the AWS Certified Solutions Architect – Professional (SAP-C02) exam. It is structured to go beyond a simple list of topics, offering in-depth explanations, architectural patterns, comparative analysis, and practical scenarios to prepare you for the complex questions you will face when taking this test, which i consider one of the hardest among professional-level cloud certifications. Maybe Networking Specialty is even harder, but I haven’t taken it yet. I divided the topics following the domains from the official description to make things a bit easier (for me to organize). If I wanted to go deep into every subject, this would obviously be a book instead of a blog post. But someone taking this test probably knows enough of everything in AWS and needs preparation specifically for test-taking.

Exam Domains and Weighting

The SAP-C02 exam evaluates your expertise across four key domains:

Domain 1: Design Solutions for Organizational Complexity (26%)
Domain 2: Design for New Solutions (29%)
Domain 3: Continuous Improvement for Existing Solutions (25%)
Domain 4: Accelerate Workload Migration and Modernization (20%)

Domain 1: Design Solutions for Organizational Complexity (26%)

This domain tests your ability to design architectures for large-scale, multi-account environments, focusing on networking, security, and governance.

1.1 Architect Multi-Account and Governance Strategies

Fundamental Concepts: A multi-account strategy is an AWS best practice for achieving security isolation, granular billing, and streamlined governance.

AWS Organizations: Allows you to centrally manage and govern your environment as you grow and scale your AWS resources. It provides consolidated billing and lets you apply policies to groups of accounts.
Organizational Units (OUs): A way to group accounts within an Organization to easily manage them and apply governance controls.
AWS Control Tower: Automates the setup of a secure, multi-account AWS environment called a “landing zone,” based on AWS best practices.
Service Control Policies (SCPs): A type of policy in AWS Organizations that helps you centrally manage the maximum available permissions for all accounts in your organization. SCPs act as “guardrails”; they don’t grant permissions but define the boundaries for what IAM principals can do.

Architectural Decisions & Scenarios:

Scenario: A company wants to ensure that no user, including administrators, in any of its development accounts can create resources outside of the eu-west-1 region.
Solution: Create an OU for development accounts. Attach an SCP to this OU with a Deny statement for all actions ("*") where the requested region is not eu-west-1. This provides a centralized and foolproof way to enforce the policy.

1.2 Architect Advanced Identity and Security Strategies

Fundamental Concepts:

Advanced IAM: Beyond basic users and roles, you must know Permission Boundaries, Attribute-Based Access Control (ABAC), and Cross-Account Roles.
- Permission Boundaries: An advanced feature where you use a managed policy to set the maximum permissions that an identity-based policy can grant to an IAM entity. It’s used to safely delegate permissions management.
- ABAC: An authorization strategy that defines permissions based on attributes (tags). This is highly scalable, as you can create a few generic policies that apply based on the tags attached to users and resources.
Secrets Management:
- AWS Secrets Manager: Enables you to automatically rotate, manage, and retrieve database credentials, API keys, and other secrets. It’s the preferred choice for rotating secrets automatically.
- AWS Systems Manager Parameter Store: Provides secure, hierarchical storage for configuration data and secrets management. The Standard tier is free, while the Advanced tier offers more features and incurs costs.

Architectural Decisions & Scenarios:

Scenario: A fast-growing company has hundreds of projects. Managing IAM policies for each project’s resources is becoming complex. They want a scalable permissions model.
Solution: Implement an Attribute-Based Access Control (ABAC) strategy. Tag all resources (e.g., EC2 instances, S3 buckets) with a Project tag. Then, create IAM policies that allow access to resources only if the principal’s Project tag matches the resource’s Project tag. This significantly reduces the number of policies needed.

1.3 Architect Complex Network Connectivity

Fundamental Concepts:

AWS Transit Gateway (TGW): A network transit hub that you can use to interconnect your VPCs and on-premises networks. It simplifies your network architecture, acting as a cloud router.
AWS Direct Connect (DX): A dedicated, private network connection from your premises to AWS. Use a Direct Connect Gateway to connect a single DX connection to multiple VPCs across different regions.
AWS PrivateLink: Provides private connectivity between VPCs, AWS services, and your on-premises networks without exposing your traffic to the public internet. It’s ideal for securely exposing a service in one VPC to consumers in other VPCs.

Comparisons and When to Use Each:

Transit Gateway vs. VPC Peering: VPC Peering is a 1:1 connection and is not transitive. It creates a complex mesh at scale. Transit Gateway uses a hub-and-spoke model, which is far more scalable and manageable for more than a handful of VPCs.
Direct Connect vs. Site-to-Site VPN: VPN is quick to set up and uses the public internet. Direct Connect provides a private, dedicated, and consistent high-bandwidth connection, ideal for large-scale or latency-sensitive workloads.

Architectural Decisions & Scenarios:

Scenario: A global company has 50 VPCs in various regions and an on-premises data center. They need full interconnectivity between all environments.
Solution: Deploy a Transit Gateway in each region and peer them together to create a global network. Connect the on-premises data center to a Direct Connect Gateway, which then connects to the Transit Gateways. This creates a scalable and manageable hub-and-spoke architecture.

Domain 2: Design for New Solutions (29%)

This domain focuses on your ability to select the right AWS service for the job based on requirements for compute, storage, databases, and more.

2.1 Architect Compute Solutions

Key Services & Use Cases:

Amazon EC2: Virtual servers. Use when you need full control over the OS or for legacy applications. Optimize costs with Spot Instances for fault-tolerant workloads and Savings Plans/Reserved Instances for baseline capacity.
AWS Lambda: Serverless functions. Use for event-driven, short-running tasks (up to 15 minutes). Ideal for API backends, data processing, and automation. Use Provisioned Concurrency to mitigate cold starts for latency-sensitive applications.
AWS Fargate: Serverless compute for containers (ECS/EKS). Use when you want to run containers without managing the underlying EC2 instances. It offers a balance between the simplicity of Lambda and the flexibility of containers.

Comparative Table: EC2 vs. Fargate vs. Lambda

Feature	Amazon EC2	AWS Fargate	AWS Lambda
Abstraction	Virtual Machines (IaaS)	Containers (CaaS)	Functions (FaaS)
Management	High (OS, patching, scaling)	Medium (Container definitions)	Low (Code only)
Use Cases	Legacy apps, full control	Microservices, containerized apps	Event-driven, APIs
Cost Model	Per second/hour for the instance	Per vCPU/GB per second	Per request & duration (ms)

Scenario: A new microservice needs to run a containerized application. The team wants to minimize operational overhead and only pay for the exact resources the container uses.

Solution: Deploy the container on AWS Fargate. It eliminates the need to provision and manage EC2 instances, and its pricing model aligns perfectly with paying only for the compute resources consumed by the task.

2.2 Architect Storage Solutions

Key Services & Use Cases:

Amazon S3: Object storage. The default choice for unstructured data, backups, static websites, and data lakes. Use S3 Lifecycle Policies to automatically transition data to cheaper storage classes (e.g., S3 Glacier Deep Archive) for cost optimization.
Amazon EBS: Block storage for EC2 instances. Think of it as a virtual hard drive. Use gp3 volumes for a general-purpose balance of price and performance. Use io2 Block Express for sub-millisecond latency database workloads.
Amazon EFS: Network file system (NFS) for Linux. Use when you need shared file storage across multiple EC2 instances (e.g., for a web content management system).
Amazon FSx: Managed file systems for specific needs. Use FSx for Windows File Server for Windows applications needing SMB shares, and FSx for Lustre for high-performance computing (HPC) workloads.

Scenario: A company is migrating a high-performance computing (HPC) application to AWS that requires a high-throughput, parallel file system to process large datasets.

Solution: Use Amazon FSx for Lustre. It is specifically designed for HPC and data-intensive workloads, providing the high-speed, parallel access to storage that these applications require.

2.3 Architect Database Solutions

Key Services & Use Cases:

Amazon Aurora: A MySQL and PostgreSQL-compatible relational database built for the cloud. Offers superior performance, availability, and scalability compared to standard RDS. Use Aurora Serverless v2 for unpredictable workloads as it scales capacity instantly and granularly.
Amazon DynamoDB: A fully managed NoSQL key-value and document database. Delivers single-digit millisecond performance at any scale. Use for high-traffic web applications, IoT, and gaming. Use DynamoDB Accelerator (DAX) for an in-memory cache to further reduce read latency.
Amazon Redshift: A petabyte-scale data warehouse. Use for business intelligence and large-scale analytical queries (OLAP).
Amazon ElastiCache: An in-memory caching service (supporting Redis and Memcached). Use to offload read traffic from your primary database and improve application latency.

Scenario: An e-commerce site needs a database for its product catalog and shopping cart. The product catalog has a fixed schema, while the shopping cart needs extreme scalability and low latency during flash sales.

Solution: Use a polyglot persistence architecture. Store the product catalog in Amazon Aurora due to its relational nature. Store the shopping cart data in Amazon DynamoDB to handle the high-throughput writes and reads with low latency required during traffic spikes.

2.4 Architect Messaging and Integration Solutions

Comparisons and When to Use Each:

Amazon SQS (Simple Queue Service): A message queue used to decouple application components. A message is pulled by one consumer and processed. Use for background job processing or to buffer requests.
Amazon SNS (Simple Notification Service): A pub/sub messaging service. A message is published to a topic and pushed to many subscribers simultaneously. Use for fan-out notifications or to trigger multiple parallel processes.
Amazon EventBridge: A serverless event bus that connects application components with data from a variety of sources. It’s more advanced than SNS, with powerful filtering rules and integrations with third-party SaaS applications.

Scenario: After a user uploads an image to S3, the image needs to be resized into multiple formats, and a machine learning service needs to analyze it for content. These processes should run in parallel.

Solution: Configure S3 Events to publish a message to an Amazon SNS topic upon object creation. Create multiple subscribers for this topic: one SQS queue for each resizing job (e.g., thumbnail, medium, large) and another SQS queue for the ML analysis job. This fan-out pattern allows all downstream processes to be triggered simultaneously and reliably.

Domain 3: Continuous Improvement for Existing Solutions (25%)

This domain covers optimizing existing architectures for resilience, performance, security, and cost.

3.1 Architect for High Availability and Resilience

Key Concepts & Strategies: Disaster Recovery (DR) strategies are a trade-off between cost and recovery time/point objectives (RTO/RPO).

Backup and Restore: Lowest cost, highest RTO/RPO.
Pilot Light: A minimal version of the environment is always running in the DR region.
Warm Standby: A scaled-down but fully functional version is running in the DR region.
Multi-site Active-Active: The application runs in multiple regions simultaneously, offering near-zero RTO/RPO but at the highest cost.

Architectural Decisions & Scenarios:

Scenario: A critical financial application requires an RPO of seconds and an RTO of less than a minute.
Solution: Implement a Warm Standby or Multi-site Active-Active strategy. Use Amazon Aurora Global Database, which provides cross-region replication with a latency of under a second and allows for fast failover. Use Amazon Route 53 with health checks and failover routing policies to automatically direct traffic to the healthy region.

3.2 Architect for Performance Optimization

Key Concepts & Strategies:

Caching: A fundamental strategy. Use Amazon CloudFront at the edge to cache static and dynamic content close to users. Use Amazon ElastiCache inside your VPC to cache database queries and session data.
Compute Optimization: Use AWS Compute Optimizer to get machine learning-powered recommendations for right-sizing EC2 instances, Auto Scaling groups, and Lambda functions. Consider AWS Graviton (ARM-based) processors for better price-performance for many workloads.
Network Optimization: Use AWS Global Accelerator to get static IP addresses that act as a fixed entry point to your application, routing traffic over the AWS global network to improve performance for global users.

Scenario: A global application is experiencing high latency for users in Asia, even though the primary infrastructure is in the US. The application is not web-based and relies on TCP connections.

Solution: Use AWS Global Accelerator. It will provide static anycast IP addresses that users connect to. Traffic is then routed over the optimized and congestion-free AWS global network to the application endpoints, significantly reducing latency compared to the public internet.

3.3 Architect for Observability and Monitoring

Key Services & Use Cases:

Amazon CloudWatch: The central hub for metrics, logs, and alarms. Use CloudWatch Logs Insights for powerful log querying and CloudWatch Alarms to trigger automated actions (e.g., scaling or notifications).
AWS X-Ray: A distributed tracing service. Use it to analyze and debug microservices architectures by visualizing the path of a request as it travels through different services, identifying bottlenecks and errors.
AWS Config: A service to assess, audit, and evaluate the configurations of your AWS resources. Use it to ensure compliance with corporate policies (e.g., “all EBS volumes must be encrypted”).

Scenario: A complex microservices application is experiencing intermittent timeouts, but developers can’t pinpoint which downstream service is causing the delay.

Solution: Instrument the application code with the AWS X-Ray SDK. X-Ray will trace requests as they flow through the various services, generating a service map that visualizes the connections and latencies. This will quickly identify the specific service that is the source of the bottleneck.

Domain 4: Accelerate Workload Migration and Modernization (20%)

This domain covers the strategies and tools for moving to and optimizing on AWS.

4.1 Architect Migration and Modernization Strategies (The 7 Rs)

Rehost (Lift-and-Shift): Moving applications without changes. Fastest but least cloud-optimized. (Tool: AWS Application Migration Service - MGN).
Replatform (Lift-and-Tinker): Making a few cloud optimizations, like moving a database to RDS.
Refactor/Re-architect: Reimagining the application to use cloud-native features. Highest effort, highest reward.
Repurchase: Moving to a different product, often a SaaS solution.
Retain: Keeping applications on-premises that aren’t ready to move.
Retire: Decommissioning applications that are no longer needed.
Relocate: Moving infrastructure between hypervisors with minimal disruption (e.g., VMware Cloud on AWS).

Scenario: A company wants to migrate its on-premises VMware-based virtual machines to AWS as quickly as possible with minimal changes to the applications.

Solution: A Rehost strategy is the best approach. Use the AWS Application Migration Service (MGN), which continuously replicates the source machines into a staging area in AWS. This allows for minimal cutover downtime when you are ready to launch the instances in AWS.

4.2 Architect for Cost Optimization

Key Concepts & Strategies:

Pricing Models: Understand when to use On-Demand, Savings Plans, Reserved Instances (RIs), and Spot Instances.
- Savings Plans vs. RIs: Savings Plans offer more flexibility than Standard RIs, applying to EC2, Fargate, and Lambda usage across regions and instance families in exchange for a commitment to a consistent amount of usage (measured in $/hour). RIs are still the best option for services like RDS and Redshift.
Resource Optimization:
- Right-Sizing: Use AWS Compute Optimizer to identify and eliminate over-provisioned resources.
- Elasticity: Use Auto Scaling to match capacity to demand, avoiding payment for idle resources.
- Storage Tiering: Use S3 Intelligent-Tiering to automatically move data to the most cost-effective access tier without performance impact or operational overhead.

Scenario: A company has a predictable baseline of EC2 usage for its web servers but also runs large, interruptible batch processing jobs overnight.

Solution: Cover the predictable baseline usage with a Compute Savings Plan to get a significant discount. Run the overnight batch processing jobs on EC2 Spot Instances, which can provide discounts of up to 90% and are ideal for workloads that can tolerate interruptions.

Key Topic Summaries for Quick Reference

Disaster Recovery (DR) Strategies

Strategy	RTO (Time)	RPO (Data Loss)	Cost	Use Case
Backup & Restore	Hours-Days	Hours	Low	Non-critical data
Pilot Light	10s of Mins-Hours	Minutes	Med	Critical apps w/ tolerance
Warm Standby	Minutes	Seconds-Minutes	High	Business-critical apps
Multi-Site Active-Active	Seconds/Zero	Near-Zero	Very High	Mission-critical systems

Advanced IAM and Multi-Account Governance

AWS Organizations & SCPs: Use OUs to group accounts and SCPs to set permission guardrails (e.g., restrict regions).
AWS Control Tower: Automates the creation of a secure, best-practice landing zone.
Permission Boundaries: Limit the maximum permissions a user can grant to other entities they create.
ABAC (Tag-based permissions): A highly scalable way to manage permissions in complex environments.
IAM Roles Anywhere: Allows workloads running outside AWS (e.g., on-premises servers) to obtain temporary AWS credentials using their own X.509 certificates.

Studying C Sharp from Python

Sat, 08 Mar 2025 00:00:00 +0000

In this post, I’m going to analyze the implementation of the Fibonacci sequence in Python and C#, highlighting the main differences in syntax and programming concepts. I’ll also explore some essential C# principles to ease the transition for Python developers. I have a friend who is passionate about .NET. Whenever I show her a snippet of Python code, she wrinkles her nose and says, “Why use Python? .NET is so much better!” So, in her honor, I decided to convert this Python code to C# and study the process step by step. I used AI to generate the C# code and explain the differences. I found this approach fantastic for learning, and I intend to delve deeper into C# in parallel using this study method.

Python Code:

Here’s the Python implementation of the Fibonacci sequence with memoization, tracking function calls and cache hits:

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


import time

cache = {0:0, 1:1}
calls = 0
cache_hits = 0

def fib(n: int) -> int:
 global calls, cache_hits
 if n in cache:
 cache_hits += 1
 return cache[n]
 else:
 calls += 1
 cache[n] = fib(n-1) + fib(n-2)
 return cache[n]

if __name__ == '__main__':
 n = 500

 start_time = time.perf_counter()
 result = fib(n)
 end_time = time.perf_counter()

 elapsed_time_ms = (end_time - start_time) * 1000

 print(f'{result}')
 print(f'{calls} function calls and {cache_hits} cache hits.')
 print(f"Elapsed time: {elapsed_time_ms:.4f} ms")

Translation to C#:

Below is the C# equivalent, translated using Gemini:

Source codecsharp


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


using System;
using System.Collections.Generic;
using System.Diagnostics;

public class Fibonacci
{
 private static Dictionary<int, long> cache = new Dictionary<int, long> { { 0, 0 }, { 1, 1 } };
 private static long calls = 0;
 private static long cacheHits = 0;

 public static long Fib(int n)
 {
 if (cache.ContainsKey(n))
 {
 cacheHits++;
 return cache[n];
 }
 else
 {
 calls++;
 cache[n] = Fib(n - 1) + Fib(n - 2);
 return cache[n];
 }
 }

 public static void Main(string[] args)
 {
 int n = 500;

 Stopwatch stopwatch = new Stopwatch();
 stopwatch.Start();

 long result = Fib(n);

 stopwatch.Stop();
 double elapsedTimeMs = stopwatch.Elapsed.TotalMilliseconds;

 Console.WriteLine(result);
 Console.WriteLine($"{calls} function calls and {cacheHits} cache hits.");
 Console.WriteLine($"Elapsed time: {elapsedTimeMs:F4} ms");
 }
}

Key Concepts for Python Developers Transitioning to C#

This section highlights the core concepts you’ll encounter when moving from Python to C#, focusing on areas where the differences are most pronounced.

Static Typing: Unlike Python’s dynamic typing, C# requires explicit type declarations (e.g., int age = 30;). This enforces type safety at compile time, catching potential errors early. You’ll need to become familiar with common C# data types like int, double, string, bool, char, and long (used for larger integer values).
Object-Oriented Programming (OOP) Paradigm: C# is fundamentally object-oriented. Code is organized around classes, which serve as blueprints for creating objects. Key OOP concepts to master include:
- Classes and Objects: Defining classes with properties (data) and methods (behavior).
- Encapsulation: Protecting data within a class and controlling access using access modifiers (public, private, protected).
- Inheritance: Creating new classes based on existing ones, inheriting their characteristics.
- Polymorphism: Enabling objects of different classes to respond to the same method call in their own way.
Namespaces: Code Organization: Namespaces are like Python modules, but on steroids. They provide a hierarchical way to organize code, preventing naming conflicts and grouping related classes. You’ll use the using keyword to import namespaces and access their contents (e.g., using System;).
Methods: Class-Bound Functions: Methods are functions that belong to a class. They define the actions an object can perform. Pay attention to method signatures, including return types, parameters, and access modifiers. The static keyword allows you to define methods that belong to the class itself, rather than to an instance.
Control Flow with Braces: C# uses curly braces {} to define code blocks for control flow statements like if, else, for, while, and switch. This is a key syntactic difference from Python’s indentation-based approach.
Collections: Data Structures: C# offers a rich set of collection classes for storing and managing data, including arrays, lists (List<T>), and dictionaries (Dictionary<TKey, TValue>). Choosing the right collection type for your needs is crucial for performance and efficiency.
The .NET Ecosystem: C# code runs on the .NET runtime. Understanding the basics of the .NET framework, including the Common Language Runtime (CLR) and the Base Class Library (BCL), will provide a deeper understanding of how C# code is executed and the vast library of pre-built functionality available to you.

Lessons Learned: Key Takeaways for Pythonistas

After converting the Fibonacci sequence and exploring these core concepts, here are the most significant lessons I learned:

Embrace Static Typing: While it might feel verbose at first, static typing leads to more robust and maintainable code. I need to get comfortable with declaring variable types and understanding how the compiler enforces type safety.
OOP is Fundamental: C# is an object-oriented language through and through.
Leverage the .NET Framework: The .NET framework provides a wealth of pre-built classes and methods that can significantly speed up development.
Syntax Matters: Paying close attention to the syntax, especially the use of curly braces for code blocks, is important. Small syntactic errors can prevent the code from compiling.
AI as a Learning Accelerator: Using AI tools to generate code and explain concepts can be a powerful way to learn C#. Focusing on understanding the why behind the code, rather than just the how, is what makes the difference between learning and just letting AI doing everything for you.

Ending Notes

Converting Python code to C# can seem daunting at first, but by understanding the core concepts and syntax differences, the transition becomes much smoother. This exercise with the Fibonacci sequence highlights the importance of static typing, object-oriented programming, and the structure that namespaces provide in C#. What made this learning experience truly powerful was the use of AI to generate the initial C# code and provide explanations. This approach allowed me to focus on understanding the why behind the code, rather than getting bogged down in syntax errors. While Python offers flexibility and rapid prototyping, C# brings robustness and performance, especially beneficial for larger applications. It’s a matter of chosing the right tool for the job. In my opinion, Python is great for quick and small tasks, as well as situations where performance is not a big factor. And, crucially, by using AI as a learning companion, we can accelerate the process of mastering new languages and paradigms, making complex tasks like cross-language translation and concept understanding significantly more accessible.

Dynamic Programming(DP)

Fri, 07 Mar 2025 00:00:00 +0000

Dynamic Programming (DP). Simply put, it’s a powerful algorithmic technique used mainly to solve optimization problems, especiallly those that exhibit two key properties: optimal substructure and overlapping subproblems. Understanding these two properties is crucial to grasping the essence of DP.

Optimal Substructure: A problem exhibits optimal substructure if an optimal solution to the overall problem can be constructed from optimal solutions to its subproblems. In plain English, if the best way to solve the big problem involves using the best solutions to smaller, related problems, then you have optimal substructure. A classic example? Shortest path algorithms. The shortest path from point A to point B necessarily includes the shortest paths to the intermediate points along the way.
Overlapping Subproblems: This means that the problem can be broken down into subproblems which are reused multiple times. A naive recursive approach would repeatedly solve the same subproblems, leading to exponential time complexity. DP avoids this by solving each subproblem only once and storing the result, saving time and improving efficiency.

Let’s consider a well-known example: the Fibonacci sequence. The nth Fibonacci number is defined as the sum of the (n-1)th and (n-2)th Fibonacci numbers.

Naive Recursive Approach (Inefficient):

Source codepython


1
2
3
4
5
6


def fibonacci(n):
 if not isinstance(n, int) or isinstance(n, bool) or n < 0:
 raise ValueError("n must be a non-negative integer")
 if n <= 1:
 return n
 return fibonacci(n-1) + fibonacci(n-2)

As n grows, the number of redundant calculations explodes.

The examples accept non-negative integers, including 0 and 1. Invalid inputs raise ValueError.

Now, let’s look at how Dynamic Programming can solve the same problem using two common approaches:

1. Top-Down with Memoization:

Memoization is essentially caching the results of expensive function calls and reusing them when the same inputs occur again. We start from the top (the original problem) and recursively break it down, storing the results in a dictionary (or similar data structure) to avoid redundant calculations.

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


def fibonacci_memoization(n, memo=None):
 if not isinstance(n, int) or isinstance(n, bool) or n < 0:
 raise ValueError("n must be a non-negative integer")
 if memo is None:
 memo = {}
 if n in memo:
 return memo[n]
 if n <= 1:
 return n
 memo[n] = fibonacci_memoization(n-1, memo) + fibonacci_memoization(n-2, memo)
 return memo[n]

print(fibonacci_memoization(6))

2. Bottom-Up (Tabulation):

Tabulation involves building a table (usually an array or matrix) from the bottom up, starting with the base cases and iteratively computing the solutions to larger subproblems based on the solutions to smaller ones.

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


def fibonacci_tabulation(n):
 if not isinstance(n, int) or isinstance(n, bool) or n < 0:
 raise ValueError("n must be a non-negative integer")
 if n <= 1:
 return n
 table = [0] * (n + 1)
 table[1] = 1
 for i in range(2, n + 1):
 table[i] = table[i-1] + table[i-2]
 return table[n]

print(fibonacci_tabulation(0)) # 0
print(fibonacci_tabulation(1)) # 1
print(fibonacci_tabulation(6)) # 8

Key differences between Memoization and Tabulation:

Memoization (Top-Down): Recursive approach, starts from the original problem, and caches results as needed. Can be more intuitive for some problems.
Tabulation (Bottom-Up): Iterative approach, builds the solution from the base cases up. Can be more efficient in terms of space complexity in some cases, as it may not need to store the entire call stack.

When to Use Dynamic Programming?

The problem can be divided into overlapping subproblems.
The problem exhibits optimal substructure.
You want to optimize the performance of a recursive solution that is repeatedly solving the same subproblems.

Common Applications of Dynamic Programming:

Knapsack Problem: Optimizing the selection of items to maximize value while respecting a weight constraint.
Longest Common Subsequence (LCS): Finding the longest sequence of characters common to two strings.
Edit Distance: Calculating the minimum number of operations (insertions, deletions, substitutions) required to transform one string into another.
Shortest Path Algorithms (e.g., Floyd-Warshall): Finding the shortest paths between all pairs of vertices in a graph.

I wrote two python programs, one using top-down and another using bottom-up. They calculate fib(500) and print the number of iterations, cache hits, function calls and the elapsed time. It is a good way to compare how the different approaches can influence the performance. When i tried to run the top-down program for fib(1000), i reached the recursion limit. Using functools.lru_cache can replace the dictionary, but it does not remove recursive call depth. The iterative version avoids that recursion limit. The global counters in the instrumentation examples below describe a fresh process run; reset them and the cache before comparing repeated runs.

Fibonacci Top-Down Example (Click to Show/Hide)

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


import time

cache = {0:0, 1:1}
calls = 0
cache_hits = 0

def fib(n: int) -> int:
 if not isinstance(n, int) or isinstance(n, bool) or n < 0:
 raise ValueError("n must be a non-negative integer")
 global calls, cache_hits
 if n in cache:
 cache_hits += 1
 return cache[n]
 else:
 calls += 1
 cache[n] = fib(n-1) + fib(n-2)
 return cache[n]

if __name__ == '__main__':
 n = 500

 start_time = time.perf_counter()
 result = fib(n)
 end_time = time.perf_counter()

 elapsed_time_ms = (end_time - start_time) * 1000

 print(f'{result}')
 print(f'{calls} function calls and {cache_hits} cache hits.')
 print(f"Elapsed time: {elapsed_time_ms:.4f} ms")

Fibonacci Bottom-Up Example (Click to Show/Hide)

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


import time

iterations = 0

def fib(n: int) -> int:
 if not isinstance(n, int) or isinstance(n, bool) or n < 0:
 raise ValueError("n must be a non-negative integer")
 global iterations

 if n == 0:
 return 0
 elif n == 1:
 return 1

 fibonacci = [0] * (n + 1)
 fibonacci[0] = 0
 fibonacci[1] = 1

 for i in range(2, n + 1):
 iterations += 1
 fibonacci[i] = fibonacci[i - 1] + fibonacci[i - 2]

 return fibonacci[n]

if __name__ == '__main__':
 n = 500
 start_time = time.perf_counter()
 result = fib(n)
 end_time = time.perf_counter()
 elapsed_time_ms = (end_time - start_time) * 1000

 print(f'{result}')
 print(f'{iterations} iterations.')
 print(f"Elapsed time: {elapsed_time_ms:.4f} ms")

Revision note

October 2026: fixed the tabulation base case for zero, removed the shared mutable default argument from memoization, and added input validation. Memoization still consumes call-stack space; it does not make recursion depth unlimited.

Breadth-First Search (BFS) and Depth-First Search (DFS): A Deep Dive

Wed, 05 Mar 2025 00:00:00 +0000

Breadth-First Search (BFS)

Breadth-First Search (BFS) is a fundamental graph traversal and search algorithm. It systematically explores a graph level by level, starting from a designated start node. BFS will explore all paths one step away from the starting point, then all paths two steps away, and so on, ensuring that the shortest path is found first.

How BFS Works

Initialization:
- Start with a queue data structure, typically implemented using collections.deque in Python for efficient FIFO (First-In, First-Out) operations. The queue initially contains only the start node.
- Maintain a visited set (or list) to keep track of nodes that have already been explored to avoid cycles.
- Optionally, a parent dictionary is used to store the predecessor of each visited node, allowing for path reconstruction later.
Iteration:
- While the queue is not empty:
  - Dequeue a node from the front of the queue. This is the node currently being explored.
  - For each neighbor of the current node:
    - If the neighbor has not been visited:
      - Mark the neighbor as visited.
      - Enqueue the neighbor.
      - Record the current node as the parent of the neighbor (if path reconstruction is needed).
Termination:
- The algorithm terminates when the queue is empty, indicating that all reachable nodes from the start node have been explored.
- If a target node is specified, the search can terminate as soon as the target node is dequeued. The parent dictionary can then be used to reconstruct the shortest path from the start node to the target.

Pros of BFS

Completeness: BFS is complete, meaning that if a path exists between the start node and the target node, BFS will find it.
Optimality: BFS guarantees finding the shortest path (in terms of the number of edges) between the start node and the target node in an unweighted graph.
Simplicity: The algorithm is relatively straightforward to understand and implement.

Cons of BFS

Memory Consumption: BFS can consume a significant amount of memory, especially for large graphs. This is because it stores all the nodes at the current level in the queue. In the worst-case scenario, where the entire graph needs to be traversed, the queue can grow to hold a large fraction of the total number of nodes.
Not Suitable for Weighted Graphs: BFS is designed for unweighted graphs. It finds the shortest path based on the number of edges, not the total weight of the edges. For weighted graphs, algorithms like Dijkstra’s algorithm are more appropriate.

Where BFS Excels

Finding Shortest Paths (Unweighted Graphs): Its primary strength is finding the shortest path in graphs where all edges have the same weight (or no weight).
Connectivity Testing: Determining if two nodes are connected in a graph.
Web Crawlers: BFS can be used to crawl web pages, exploring links level by level.
Social Network Analysis: Finding the shortest chain of connections between two people in a social network.
GPS Navigation: In scenarios where minimizing the number of “hops” is more important than the total distance (e.g., minimizing the number of transfers in public transportation).

Where BFS Fails

Weighted Graphs: As mentioned earlier, BFS is not suitable for finding the shortest path in graphs where edges have different weights.
Very Large Graphs: The memory requirements of BFS can become prohibitive for very large graphs.
Graphs with High Branching Factors: Graphs where each node has many neighbors can lead to a large queue and high memory consumption.

Common Use Cases

Pathfinding in Games: Finding the shortest path for a character to move in a game environment (where all moves have the same cost).
Network Routing: Discovering network topology and finding the shortest paths between devices.
Garbage Collection: Tracing reachable objects in memory to identify those that can be safely deallocated.
Peer-to-Peer Networks: Finding the closest peers in a peer-to-peer network.

Depth-First Search (DFS)

Depth-First Search (DFS) is another fundamental graph traversal and search algorithm. Instead of exploring a graph level by level like BFS, DFS explores as deeply as possible along each branch before backtracking. DFS will pick a path and follow it to the end, then backtrack to the last branching point and try a different path.

How DFS Works

Initialization:
- Start with a stack data structure, typically implemented using a Python list. The stack initially contains only the start node.
- Maintain a visited set (or list) to keep track of nodes that have already been explored to avoid cycles.
- Optionally, a parent dictionary is used to store the predecessor of each visited node, allowing for path reconstruction later.
Iteration:
- While the stack is not empty:
  - Pop a node from the top of the stack. This is the node currently being explored.
  - If the node has not been visited:
    - Mark the node as visited.
    - For each neighbor of the current node:
      - If the neighbor has not been visited:
        
        Push the neighbor onto the top of the stack.
        
        Record the current node as the parent of the neighbor (if path reconstruction is needed).
Termination:
- The algorithm terminates when the stack is empty, indicating that all reachable nodes from the start node have been explored.
- If a target node is specified, the search can terminate as soon as the target node is popped from the stack and found to be the target. The parent dictionary can then be used to reconstruct a path (not necessarily the shortest) from the start node to the target.

Pros of DFS

Memory Efficiency: DFS generally requires less memory than BFS, especially for deep graphs. It only needs to store the nodes on the current path being explored, rather than all the nodes at the current level (as BFS does).
Simple Implementation: DFS can be implemented recursively, making the code very concise (though the iterative version, using a stack, is also straightforward).
Good for Finding Any Path: If you just need to find a path between two nodes (not necessarily the shortest), DFS can be faster than BFS.

Cons of DFS

Not Guaranteed to Find Shortest Path: DFS does not guarantee finding the shortest path. It finds a path, but that path might be much longer than the shortest path.
Can Get Stuck in Infinite Loops: If the graph contains cycles and the visited set is not used correctly, DFS can get stuck in an infinite loop, exploring the same nodes over and over.
May Explore Irrelevant Paths: DFS can explore paths that are far away from the target node before finding the target.

Where DFS Excels

Detecting Cycles: DFS is well-suited for detecting cycles in a graph. If, during the traversal, you encounter a node that has already been visited (and is still on the stack), it indicates the presence of a cycle.
Topological Sorting: DFS is used in algorithms for topological sorting of directed acyclic graphs (DAGs).
Path Existence Checking: Determining if a path exists between two nodes, even if the shortest path is not required.
Solving Puzzles: DFS can be used to explore possible solutions in puzzles and games (e.g., solving a maze, finding a solution to a Sudoku puzzle).

Where DFS Fails

Finding Shortest Paths: When the shortest path is required, BFS or other algorithms like Dijkstra’s algorithm are better choices.
Graphs with Long Paths: In graphs with very long paths, DFS might take a long time to find the target node or explore the entire graph.
Infinite Graphs: DFS can get lost in infinite graphs if there is no clear termination condition.

Common Use Cases

Cycle Detection: As mentioned earlier, DFS is commonly used to detect cycles in graphs.
Topological Sort: Ordering tasks in a project based on dependencies.
Maze Solving: Finding a path from the entrance to the exit.
Backtracking Algorithms: Solving problems where you need to explore different possibilities and backtrack when a dead end is reached (e.g., solving constraint satisfaction problems).
Game AI: Implementing AI for games where the AI needs to explore possible moves and evaluate their consequences.

I added below two examples, one for BFS and another for DFS. The graph used was the one on the image below:

This is the output of both, where we can notice the difference:

bfs.py:

Source codebash


1
2
3


Sequence of visited nodes: ['A', 'B', 'G', 'C', 'D', 'E', 'H', 'J', 'F', 'I']
Visited a total of 10 nodes.
Found path from A to F: ['A', 'B', 'E', 'F']

Visitation Order:

Level 0: [‘A’]
Level 1: [‘B’, ‘G’]
Level 2: [‘C’, ‘D’, ‘E’, ‘H’, ‘J’]
Level 3: [‘F’, ‘I’]

It visits all nodes on the current level before going to the next level.

dfs.py:

Source codebash


1
2
3


Sequence of visited nodes: ['A', 'G', 'J', 'H', 'I', 'B', 'E', 'F']
Visited a total of 8 nodes.
Found Path from A to F: ['A', 'B', 'E', 'F']

Visitation Order:

‘A’: Start node, popped first.
‘G’: From ‘A’, neighbors are [‘B’, ‘G’]. Added to stack as [‘B’, ‘G’], so ‘G’ is popped next (LIFO).
‘J’: From ‘G’, neighbors are [‘H’, ‘J’]. Added as [‘B’, ‘H’, ‘J’], ‘J’ popped.
‘H’: From ‘J’, no new neighbors. Stack now [‘B’, ‘H’], ‘H’ popped.
‘I’: From ‘H’, neighbor ‘I’ added, popped next.
‘B’: Stack now [‘B’], popped after ‘G’’s subtree is exhausted.
‘E’: From ‘B’, neighbors [‘C’, ‘D’, ‘E’]. Added as [‘C’, ‘D’, ‘E’], ‘E’ popped.
‘F’: From ‘E’, neighbors [‘F’, ‘I’]. ‘I’ already visited, so ‘F’ added and popped. Target found, stops.

It goes deep into ‘G’’s branch (‘G’ → ‘J’ → ‘H’ → ‘I’) before backtracking to ‘B’ and then ‘E’ → ‘F’.

BFS Example (Click to Show/Hide)

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


from collections import deque

def breadth_first_search(graph, start_node, target_node=None):
 visited_set, visited_order, queue, parent = {start_node}, [start_node], deque([start_node]), {}
 while queue:
 node = queue.popleft()
 if target_node == node:
 path = [node]
 while node != start_node: node = parent[node]; path += [node]
 return visited_order, path[::-1]
 for neighbor in graph.get(node, []):
 if neighbor not in visited_set:
 visited_set.add(neighbor)
 visited_order += [neighbor]
 queue += [neighbor]
 parent[neighbor] = node
 return visited_order, None

if __name__ == '__main__':
 graph = {
 'A': ['B', 'G'],
 'B': ['C', 'D', 'E'],
 'C': [],
 'D': [],
 'E': ['F', 'I'],
 'F': [],
 'G': ['H', 'J'],
 'H': ['I'],
 'I': [],
 'J': [],
 }

 start_node = 'A'
 target_node = 'F'
 visited, path = breadth_first_search(graph, start_node, target_node)

 print(f"Sequence of visited nodes: {visited}")
 print(f"Visited a total of {len(visited)} nodes.")
 if path:
 print(f"Found path from {start_node} to {target_node}: {path}")
 elif not target_node:
 print(f"Target was not specified.")
 else:
 print(f"No path found from {start_node} to {target_node}.")

DFS Example (Click to Show/Hide)

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


def depth_first_search(graph, start_node, target_node=None):
 visited, stack, parent = [], [start_node], {}
 if start_node not in graph: return visited, None
 while stack:
 node = stack.pop()
 if node not in visited:
 visited += [node]
 if target_node == node:
 path = []
 current = node
 while current in parent or current == start_node:
 path += [current]
 if current == start_node: break
 current = parent[current]
 return visited, path[::-1]
 for neighbor in graph.get(node, []):
 if neighbor not in visited:
 stack += [neighbor]
 parent[neighbor] = node
 return visited, None

if __name__ == '__main__':
 graph = {
 'A': ['B', 'G'],
 'B': ['C', 'D', 'E'],
 'C': [],
 'D': [],
 'E': ['F', 'I'],
 'F': [],
 'G': ['H', 'J'],
 'H': ['I'],
 'I': [],
 'J': [],
 }

 start_node = 'A'
 target_node = 'F'
 visited, path = depth_first_search(graph, start_node, target_node)

 print(f"Sequence of visited nodes: {visited}")
 print(f"Visited a total of {len(visited)} nodes.")
 if path:
 print(f"Found Path from {start_node} to {target_node}: {path}")
 elif not target_node:
 print(f"Target was not specified.")
 else:
 print(f"No path found from {start_node} to {target_node}.")

Comparison of Sorting Algorithms using Python

Thu, 27 Feb 2025 00:00:00 +0000

This blog post documents my journey of revisiting fundamental computer science concepts and sharpening my Python coding skills. After reviewing the classics, I wrote a script to analyze and compare the performance of common sorting algorithms, including QuickSort, MergeSort, SelectionSort, and InsertionSort. Here’s an account of the analysis, the code, and the insights gained.

Background and Approach

This analysis builds upon a script initially developed during the MITx course “Introduction to Computer Science and Programming Using Python,” even incorporating the old post about learning to use decorators. I have since refined the script to remove BubbleSort in favor of InsertionSort, improved code documentation through inline comments and PEP 257-compliant docstrings, and incorporated feedback from a Grok review, including the implementation of randomized pivots in QuickSort to improve its average-case performance.

The aim of the script is to sort lists of 5000 random integers (0 to 4999). I wanted to get a better sense of how they performed, so each algorithm is timed using a decorator. It outputs the time for each run of a list copy and gives a graph at the end.

Before choosing a sorting algorithm, it’s essential to understand how the data is structured. Real-world data often falls into one of several common categories: Random, Nearly Sorted, Reversed, or Few Unique. The choice of sorting algorithm can significantly impact performance depending on the data’s characteristics. For example:

Random: A list of customer IDs in a database, where the order is arbitrary.
Nearly Sorted: A log file where entries are mostly in chronological order but occasionally have timestamps slightly out of sequence.
Reversed: A list of scores in a competition sorted from highest to lowest.
Few Unique: A list of product ratings (1-5 stars), where the number of distinct values is limited.

This consideration is crucial when working with large datasets, where even a small improvement in efficiency (e.g., 10%) can lead to significant savings.

The timing decorator is implemented using time() from the time library:

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


def timing(f):
 """Decorator to measure and print the execution time of a function."""
 def wrap(*args, **kwargs):
 starttime = time.time()
 ret = f(*args, **kwargs)
 finishtime = time.time()
 elapsed = (finishtime - starttime) * 1000.0 # Convert to milliseconds
 print(f'function [{f.__name__}] finished in {elapsed:.3f} ms')
 wrap.elapsed_time = elapsed # Store the elapsed time
 return ret, elapsed
 wrap.elapsed_time = 0 # Initialize elapsed_time attribute
 return wrap

This code defines a timing decorator that runs a start and end timer for a function to determine runtime and measure performance. The result is then stored.

Quick Sort: Figuring Out the Pivot

I’ve read that Quick Sort is one of the most popular sorting algorithms because it’s efficient on average. Quick Sort is based on a concept called divide and conquer. It works by recursively partitioning the list around a chosen element, called the pivot. All elements smaller than the pivot are moved to its left, and all elements larger than the pivot are moved to its right. This process is repeated recursively for the left and right partitions until the entire list is sorted.

A good choice of pivot is crucial. If a bad pivot is chosen, like always picking the smallest or largest element, Quick Sort can degrade to O(n²). That’s why picking a good pivot, such as using the middle element or a randomized pivot, is crucial. One technique to pick a good pivot is to use “median of three”, where we pick the first, middle, and last item, and choose the median of these values. In the worst-case scenario, such as a reversed list and always picking the first element as the pivot, the algorithm makes many comparisons, O(n^2). But if everything is chosen to be perfect, the scenario makes only N log N comparisons

The first time I used Quick Sort I picked the first element as a pivot, which might not be ideal. This time I changed it to pick it randomly, as suggested during a review using Grok.

Despite this worst-case scenario, quickSort is often faster in practice than mergeSort because it doesn’t require extra memory for merging. That’s why it is commonly used in real-world applications, including in many programming languages’ built-in sorting functions. Depending on the size of the list, memory can become an important factor.

Merge Sort: Divide, Conquer, and Combine

Like Quick Sort, Merge Sort employs a divide-and-conquer strategy to sort a list. Instead of sorting the entire list at once, it breaks the list down into smaller, more manageable sublists, sorts them individually, and then merges these sorted sublists back together in a controlled manner.

Here’s a breakdown of the process:

Divide: The list is recursively divided into two halves until you’re left with sublists containing only single elements. Consider the list [5, 2, 4, 6, 1, 3]. It’s divided into [5, 2, 4] and [6, 1, 3], then further into [5], [2, 4], [6], [1, 3], and finally, [5], [2], [4], [6], [1], [3].
Conquer: Each single-element sublist is inherently sorted (a list of one element is always sorted).
Merge: The magic happens during the merging step. The sublists are merged back together in sorted order. For instance, [5] and [2] merge into [2, 5], then [2, 5] and [4] merge into [2, 4, 5]. This continues until the full list is rebuilt, resulting in [1, 2, 3, 4, 5, 6]. The merging is efficient because we’re always combining already sorted sublists, which allows for a linear-time (O(n)) merging process.

Merge Sort has a time complexity of O(n log n) in all cases (best, average, worst), making it a reliable choice when consistent performance is crucial. However, this efficiency comes at a cost: Merge Sort requires additional memory space to store the temporary sublists during the merging process, making it a bit less space-efficient than algorithms like quickSort or insertionSort (in their in-place implementations).

Selection Sort: Repeated Minimum Finding

Selection Sort offers a straightforward, albeit inefficient, approach to sorting. It works by repeatedly finding the smallest element in the unsorted portion of the list and swapping it with the element at the beginning of the unsorted portion. While coding Selection Sort is relatively simple, its performance, with a time complexity of O(n²), is significantly slower than more advanced algorithms, especially for larger datasets.

Consider the list [5, 2, 4, 6, 1, 3]. Selection Sort would:

Find the smallest element (1) and swap it with the first element (5): [1, 2, 4, 6, 5, 3]
Find the smallest element in the remaining unsorted portion (2) (skipping already sorted ones) and swap it with the second element: [1, 2, 4, 6, 5, 3]
Continue this process, repeatedly finding the minimum and swapping it into its correct position.

The primary disadvantage of Selection Sort is its inefficiency. It always performs a complete scan of the unsorted portion of the list, regardless of whether the list is already partially sorted or not. Also, if you have a situation where reading and writing operations are very costly (for example, if elements are stored on slow hard drives or the network), the many swaps that are involved can affect performance.

Insertion Sort: Sorting Like Playing Cards

Insertion Sort is another relatively simple sorting algorithm. One way to think about it is imagining how you sort playing cards in your hand: As you’re dealt a new card, you look at the cards you already have and insert the new card into its correct position by shifting the other cards over if needed.

The algorithm works as follows:

Start with the second element in the list.
Compare it with the element to its left. If the element is smaller, swap them.
Keep moving backward, comparing and swapping until you find the correct position for the current element.
Move on to the next element and repeat the process.
By the time you reach the last element, the entire list is sorted.

To illustrate, consider the list [5, 2, 4, 6, 1, 3]:

i=1, current=2. Compare 2 with 5. Swap: [2, 5, 4, 6, 1, 3]
i=2, current=4. Compare 4 with 5. Swap: [2, 4, 5, 6, 1, 3]
i=3, current=6. Nothing to swap.
i=4, current=1. Compare 1 with 6, 5, 4, 2. Swap till front: [1, 2, 4, 5, 6, 3]
i=5, current =3 Compare 3 with 6, 5, 4, 2, 1, and stop at [1, 2, 3, 4, 5, 6]

While Insertion Sort is easy to understand and implement, its time complexity is O(n²) in the worst case (e.g., a reversed list), making it slow for large, unsorted datasets. However, Insertion Sort shines when dealing with small lists or lists that are nearly sorted. In these scenarios, it can be surprisingly efficient and may even outperform more complex algorithms like QuickSort or MergeSort. The is because it is an adaptive sort where its performance depends on the nature of the already sorted data, versus the nonadaptive methods (such as mergeSort and selectionSort) where the method will run until the data is sorted.

Methodology: Analyzing Sorting Performance

To conduct the analysis, the program follows these steps:

Data Generation: Four lists, each representing one of the scenarios described above, are generated. The list size is set to 1000. We did this in order to not reach maximum recursion limits of the sorting methods.
Sorting and Timing: For each scenario, each sorting algorithm is applied to a copy of the generated list. This ensures that each algorithm operates on the same initial data and that previous sorting operations do not influence subsequent timing results.
Execution Time Measurement: A timing decorator is used to measure the execution time of each sorting operation in milliseconds.
Visualization: The collected timing results are then used to generate a bar chart using the matplotlib library. This chart provides a visual comparison of the performance of the sorting algorithms for each scenario, allowing for easy identification of their relative strengths and weaknesses.

Results

The following results were obtained on my machine using lists of 5000 integers. Remember that performance can vary depending on hardware and specific data characteristics.

Thoughts

The key observations from this analysis are:

QuickSort: Demonstrated the best overall performance, consistently exhibiting the fastest execution times across most scenarios. The iterative implementation and randomized pivot selection likely contributed to its efficiency.
MergeSort: Performed reasonably well, consistently with the other scenarios. Due to linear-time process, it’s more costly to do it and becomes less fast.
SelectionSort: Significantly slower than the O(n log n) algorithms, highlighting the practical limitations of its O(n²) time complexity.
InsertionSort: Showed competitive performance when compared to Selection Sort.

Discussion and Next Steps

This analysis provided a valuable reminder of the performance trade-offs inherent in different sorting algorithms. The results clearly illustrate the advantages of O(n log n) algorithms (QuickSort and MergeSort) for larger datasets compared to O(n²) algorithms (SelectionSort and InsertionSort).

While these findings provide a general overview, several avenues remain for further exploration:

Larger Datasets: Running the analysis with significantly larger list sizes (e.g., 10,000, 50,000, 100,000) would further emphasize the scaling behavior of the algorithms.
Varying Data Characteristics: Experimenting with different data distributions (e.g., different levels of “nearly sorted,” varying the number of unique values) could reveal subtle performance nuances.
Implement standardSort Python can call this function to provide another time measurement.
Compare the Algorithms in Various Scenarios: Talk about the advantages and the disadvantages from each algorithm in every scenario from the graph.

Full Code (Click to Show/Hide)

Source codepython


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221


import time
import random
import functools
from prettytable import PrettyTable
import matplotlib.pyplot as plt

def timing(f):
 """Decorator to measure and print the execution time of a function."""
 def wrap(*args, **kwargs):
 starttime = time.time()
 ret = f(*args, **kwargs)
 finishtime = time.time()
 elapsed = (finishtime - starttime) * 1000.0 # Convert to milliseconds
 print(f'function [{f.__name__}] finished in {elapsed:.3f} ms')
 wrap.elapsed_time = elapsed # Store the elapsed time
 return ret, elapsed
 wrap.elapsed_time = 0 # Initialize elapsed_time attribute
 return wrap

@timing
def quickSort(L):
 """Sorts a list in ascending order using QuickSort with randomized pivot."""
 stack = [(0, len(L) - 1)] # Initial subarray: entire list

 while stack:
 first, last = stack.pop() # Get next subarray to sort

 if first < last:
 pivot_idx = random.randint(first, last)
 L[first], L[pivot_idx] = L[pivot_idx], L[first] # Move pivot to start
 pivotvalue = L[first]
 leftmark = first + 1
 rightmark = last
 done = False

 while not done:
 # Move leftmark until we find an element > pivot
 while leftmark <= rightmark and L[leftmark] <= pivotvalue:
 leftmark += 1
 # Move rightmark until we find an element < pivot
 while L[rightmark] >= pivotvalue and rightmark >= leftmark:
 rightmark -= 1
 # If pointers cross, partitioning is done
 if rightmark < leftmark:
 done = True
 else:
 # Swap elements that are on the wrong side
 L[leftmark], L[rightmark] = L[rightmark], L[leftmark]
 # Place pivot in its final position
 L[first], L[rightmark] = L[rightmark], L[first]

 # Push subarrays onto the stack for later sorting
 stack.append((first, rightmark - 1)) # Sort left partition
 stack.append((rightmark + 1, last)) # Sort right partition

@timing
def mergeSort(L):
 """Sorts a list in ascending order using MergeSort.
 
 Args:
 L: List to be sorted (modified in-place).
 
 Returns:
 None (sorts in-place), plus timing via decorator.
 """
 if len(L) <= 1: # Base case: already sorted
 return
 mergeSortHelper(L)

def mergeSortHelper(L):
 """Recursive helper for MergeSort to divide and merge subarrays.
 
 Args:
 L: List being sorted.
 """
 if len(L) > 1:
 mid = len(L) // 2
 left = L[:mid] # Split into left half
 right = L[mid:] # Split into right half
 mergeSortHelper(left)
 mergeSortHelper(right)
 # Merge the sorted halves back into L
 i = j = k = 0
 while i < len(left) and j < len(right):
 if left[i] < right[j]:
 L[k] = left[i]
 i += 1
 else:
 L[k] = right[j]
 j += 1
 k += 1
 # Copy remaining elements from left, if any
 while i < len(left):
 L[k] = left[i]
 i += 1
 k += 1
 # Copy remaining elements from right, if any
 while j < len(right):
 L[k] = right[j]
 j += 1
 k += 1

@timing
def selectionSort(L):
 """Sorts a list in ascending order using SelectionSort.
 
 Args:
 L: List to be sorted (modified in-place).
 
 Returns:
 None (sorts in-place), plus timing via decorator.
 """
 for i in range(len(L)):
 min_i = i # Assume current position has minimum
 for right in range(i + 1, len(L)):
 if L[right] < L[min_i]:
 min_i = right # Update minimum index
 if min_i != i:
 L[i], L[min_i] = L[min_i], L[i] # Swap if needed

@timing
def insertionSort(L):
 """Sorts a list in ascending order using InsertionSort.
 
 Args:
 L: List to be sorted (modified in-place).
 
 Returns:
 None (sorts in-place), plus timing via decorator.
 """
 for i in range(1, len(L)):
 current_value = L[i]
 position = i
 # Shift larger elements right
 while position > 0 and L[position - 1] > current_value:
 L[position] = L[position - 1]
 position -= 1
 L[position] = current_value # Insert in correct position

def generate_random_list(size):
 """Generates a list of random integers of a given size."""
 return random.sample(range(size), size)

def generate_nearly_sorted_list(size, inversions=5):
 """Generates a nearly sorted list with a specified number of inversions."""
 L = list(range(size))
 for _ in range(inversions):
 i, j = random.sample(range(size), 2)
 L[i], L[j] = L[j], L[i]
 return L

def generate_reversed_list(size):
 """Generates a list of integers in reverse order."""
 return list(range(size - 1, -1, -1))

def generate_few_unique_list(size, num_unique=10):
 """Generates a list with a limited number of unique values."""
 unique_values = random.sample(range(num_unique), num_unique)
 return [random.choice(unique_values) for _ in range(size)]

list_size = 1000

# Dictionary to hold the lists for each scenario
test_lists = {
 "Random": generate_random_list(list_size),
 "Nearly Sorted": generate_nearly_sorted_list(list_size),
 "Reversed": generate_reversed_list(list_size),
 "Few Unique": generate_few_unique_list(list_size)
}

algorithm_names = ["quickSort", "mergeSort", "selectionSort", "insertionSort"]
scenario_names = list(test_lists.keys())

# Store plot for results
results = {algorithm: {scenario: 0 for scenario in scenario_names}
 for algorithm in algorithm_names}

for scenario, data_list in test_lists.items():
 print(f"--- Scenario: {scenario} ---")
 quickSort(data_list.copy())
 mergeSort(data_list.copy())
 selectionSort(data_list.copy())
 insertionSort(data_list.copy())

 # Store the timing result
 results["quickSort"][scenario] = quickSort.elapsed_time
 results["mergeSort"][scenario] = mergeSort.elapsed_time
 results["selectionSort"][scenario] = selectionSort.elapsed_time
 results["insertionSort"][scenario] = insertionSort.elapsed_time

 print() # Add an empty line to separate scenarios

# Plotting the Results
x = range(len(scenario_names)) # x-axis values (scenario indices)
width = 0.2 # Width of the bars

fig, ax = plt.subplots(figsize=(12, 8)) # Adjust figure size for better readability

for i, algorithm in enumerate(algorithm_names):
 times = [results[algorithm][scenario] for scenario in scenario_names]
 offset = width * (i - (len(algorithm_names) - 1) / 2)
 rects = ax.bar([pos + offset for pos in x], times, width, label=algorithm)

 # Add text annotations on top of each bar
 for rect in rects:
 height = rect.get_height()
 ax.annotate(f'{height:.1f}', # Format the time to 1 decimal place
 xy=(rect.get_x() + rect.get_width() / 2, height),
 xytext=(0, 3), # 3 points vertical offset
 textcoords="offset points",
 ha='center', va='bottom')

# Add some text for labels, title and custom x-axis tick labels, etc.
ax.set_xlabel('Scenarios')
ax.set_ylabel('Execution Time (ms)')
ax.set_title('Sorting Algorithm Performance by Scenario')
ax.set_xticks(x)
ax.set_xticklabels(scenario_names) # Use scenario names as x-axis labels
ax.legend()
fig.tight_layout() # Adjust layout to not have any of the plot cut off
plt.show()

Preparation for Google Cloud: Professional Data Engineer (PDE) Exam

Tue, 04 Feb 2025 00:00:00 +0000

I recently passed the Google Cloud Professional Cloud Architect (PCA) exam. With multiple cloud certifications under my belt, I have decided to shift my career towards data engineering and machine learning. This transition aligns with my growing interest in moving away from infrastructure and focusing more on Data and AI. Most of the time, I worked on provisioning, architecting, and maintaining solutions for other teams. Now, I want to be on the other side a bit more. As part of this journey, I’m pursuing the Google Cloud Professional Data Engineer (PDE) certification, and in this article, I’ll share my preparation journey, useful resources, and key takeaways to help you navigate your own study path. All referenced links are listed at the end of the post.

TL;DR

Followed Google Cloud Skills Boost Data Engineer Path

Focused on BigQuery, Dataflow, Dataproc, Pub/Sub

Practiced with ExamPrepper mock tests

Did some Practice Exams I found on Udemy

1. Google Cloud Skills Boost

I started with Google Cloud Skills Boost, specifically the Data Engineer Learning Path, which consists of various hands-on labs. I focused on:

BigQuery: Optimizing queries, partitioning, and clustering.
Dataflow: Streaming and batch processing.
Dataproc: Running Hadoop/Spark jobs on GCP.
Pub/Sub: Event-driven architectures and message queues.
ML/AI: Using Vertex AI, AI Platform, and AutoML.

While Skills Boost provided a solid foundation, in my opinion it lacks in-depth content. The videos are too short, so if you want to follow that format I recommend finding a good course at Udemy or Coursera. I tried to make the best of the labs, tinkering outside the tasks to leverage the free credits. Each module has a Quiz at the end, which also serves as a practice test.

2. Practice Tests

ExamPrepper: There are currently 319 questions available for free and with comments from members. Extremely useful.
Google’s Official Sample Questions: 21 questions, straight from the source. Not enough to prepare, but good to get a feel for the exam.

3. Key Skills for a Google Professional Data Engineer

Designing Data Processing Systems: Architecting batch and streaming data processing solutions using GCP tools such as Dataflow, Dataproc, and Pub/Sub.
Operationalizing Machine Learning Models: Deploying and managing ML models using Vertex AI, AI Platform, and AutoML.
Building and Maintaining Data Pipelines: Implementing reliable and efficient ETL workflows with Cloud Composer and Dataflow.
Ensuring Solution Quality: Applying best practices for data governance, security, and compliance using IAM, Data Catalog, and encryption techniques.
Optimizing Data Solutions: Performance tuning and cost optimization for BigQuery, Cloud Storage, and other GCP data services.
Monitoring and Troubleshooting Data Pipelines: Implementing observability with Cloud Logging, Cloud Monitoring, and error handling strategies.

3. Minimum Services to Know

Data Storage

Data Processing

ML & AI

Security & Governance

4. Understanding open source versus Google Cloud managed services

Many powerful open-source tools support modern data engineering workflows. To streamline operations at scale, Google Cloud often provides fully managed counterparts to these tools. The table below maps essential open-source technologies to their Google Cloud equivalents, highlighting key services.

Open source	Google Cloud managed service
Hadoop, Spark, Hive	Dataproc
Beam	Dataflow
Airflow	Cloud Composer
Kafka, RabbitMQ	Pub/Sub
Cassandra	Cloud Bigtable

5. Notes on Services

5.1. Data Storage & Management

a. BigQuery:
- Schema Design:
  - Star Schema: Fact table (metrics) surrounded by dimension tables (attributes). Exam Question: “A retail company needs to analyze sales data by region, product category, and time. Which schema is most suitable for this purpose?” (Answer: Star Schema). Understand the benefits of denormalization for analytical queries.
  - Snowflake Schema: Dimension tables are further normalized. Exam Question: “In which scenario is Snowflake Schema more appropriate than Star Schema?” (Answer: When dimension tables have complex relationships and data redundancy needs to be minimized). Understand trade-offs between query performance and data consistency.
  - Partitioning: Dividing tables into smaller segments based on a column (e.g., date). Exam Question: “How can you optimize query performance on a large table that is frequently queried by date range?” (Answer: Partition the table by the date column). Understand partitioning key selection (cardinality, data skew).
  - Clustering: Ordering data within partitions based on one or more columns. Exam Question: “A table partitioned by date is frequently queried based on product ID. How can you further improve query performance?” (Answer: Cluster the table by product ID). Understand the interaction between partitioning and clustering.
  - Best Practices: Consider data types (use appropriate types for storage and performance), avoid overly wide tables, and choose partitioning and clustering keys carefully. Exam Question: “Which data type is most efficient for storing boolean values in BigQuery?” (Answer: BOOL).
- Query Optimization:
  - EXPLAIN PLAN: Use EXPLAIN to understand query execution plans and identify bottlenecks (e.g., full table scans). Exam Question: “How can you identify whether a query is performing a full table scan?” (Answer: Use EXPLAIN and look for “Scan” stage with no index).
  - WHERE Clause: Filter data as early as possible in the query. Exam Question: “Which of the following WHERE clauses is most efficient for querying a partitioned table?” (Answer: The one that includes the partitioning column in the filter).
  - JOIN Optimization: Understand different join types (INNER, LEFT, RIGHT, FULL). Use appropriate join conditions and order tables in the JOIN clause. Exam Question: “What is the potential impact of joining two very large tables without appropriate filtering?” (Answer: Can lead to significant query processing costs and slow performance due to Cartesian product).
  - Materialized Views: Pre-compute and store the results of frequently executed queries. Exam Question: “A dashboard relies on a complex aggregation query that takes several minutes to run. How can you improve the dashboard’s performance?” (Answer: Create a materialized view for the aggregation query). Understand when to use materialized views (data freshness, cost).
- Cost Management:
  - On-Demand vs. Flat-Rate Pricing: Understand the difference and when each model is most cost-effective. Exam Question: “When is flat-rate pricing more suitable than on-demand pricing?” (Answer: When you have predictable and consistently high BigQuery usage).
  - Query Quotas: Set quotas to limit query costs. Exam Question: “How can you prevent runaway queries from consuming excessive BigQuery resources?” (Answer: Set query quotas).
  - Data Compression: Use columnar formats like Parquet and ORC to reduce storage costs and improve query performance. Exam Question: “Which data format is most efficient for storing large analytical datasets in BigQuery?” (Answer: Parquet).
  - APPROX_COUNT_DISTINCT, APPROX_TOP_COUNT: Use approximate functions for faster and cheaper cardinality estimation. Exam Question: “You need to estimate the number of unique users in a large table. Which function provides a cost-effective approximation?” (Answer: APPROX_COUNT_DISTINCT).
- Security:
  - IAM Roles: Understand roles like roles/bigquery.dataViewer, roles/bigquery.dataEditor, roles/bigquery.admin. Exam Question: “Which IAM role grants a user the ability to read data from BigQuery tables but not modify them?” (Answer: roles/bigquery.dataViewer).
  - Authorized Views: Grant users access to specific columns or rows in a table without granting access to the entire table. Exam Question: “How can you grant a user access to only a subset of columns in a BigQuery table?” (Answer: Create an authorized view that selects only the desired columns).
  - Data Masking: Mask sensitive data in BigQuery tables to protect privacy. Exam Question: “How can you protect sensitive data like credit card numbers in BigQuery tables while still allowing analysts to perform aggregations?” (Answer: Use data masking).
  - Encryption: BigQuery encrypts data at rest and in transit. Understand CMEK (Customer Managed Encryption Keys) for greater control. Exam Question: “What is the benefit of using CMEK in BigQuery?” (Answer: It gives you control over the encryption keys used to protect your data).
- Data Loading:
  - Batch Loading: Using bq load command or the BigQuery Data Transfer Service. Exam Question: “Which method is most suitable for loading a large CSV file from Cloud Storage into BigQuery?” (Answer: Batch loading using bq load).
  - Streaming Inserts: Inserting data in real-time using the BigQuery Storage Write API. Exam Question: “Which method is most suitable for ingesting real-time data from a streaming source into BigQuery?” (Answer: Streaming inserts using the Storage Write API). Understand the trade-offs between cost and latency.
  - Data Transfer Service: Scheduling automated data transfers from various sources. Exam Question: “How can you automate the transfer of data from YouTube Analytics to BigQuery?” (Answer: Use the BigQuery Data Transfer Service).
- Data Formats:
  - CSV/JSON: Row-oriented, less efficient for analytical queries.
  - Avro/Parquet/ORC: Columnar formats, more efficient for analytical queries, support schema evolution. Exam Question: “Which data format is most efficient for storing analytical data in BigQuery and supports schema evolution?” (Answer: Avro or Parquet). Understand the benefits of columnar storage (reduced I/O, better compression).
- BigQuery ML:
  - Understand creating and evaluating models directly within BigQuery using SQL. Exam Question: “How can you build and deploy a linear regression model directly within BigQuery without using external tools?” (Answer: Use BigQuery ML with the CREATE MODEL statement).
- BigQuery Omni:
  - Know the functionality of querying data residing on other clouds (AWS, Azure) directly from BigQuery. Exam Question: “Your organization has data stored in both BigQuery and AWS S3. How can you analyze this data together using a single SQL interface?” (Answer: Use BigQuery Omni).
b. Cloud Storage:
- Storage Classes:
  - Standard: Frequently accessed data. Exam Question: “Which storage class is best suited for data that needs to be accessed frequently with low latency?” (Answer: Standard).
  - Nearline: Infrequently accessed data with low storage costs but higher access costs. Exam Question: “Which storage class is best suited for data that is accessed less than once a month?” (Answer: Nearline).
  - Coldline: Data accessed very infrequently (e.g., archives). Exam Question: “Which storage class is best suited for data that is accessed less than once a quarter?” (Answer: Coldline).
  - Archive: Data accessed extremely rarely (e.g., disaster recovery backups). Exam Question: “Which storage class is best suited for data that is accessed less than once a year?” (Answer: Archive). Understand the cost and retrieval time trade-offs.
- Lifecycle Management:
  - Automatically transition objects between storage classes based on age or other criteria. Exam Question: “How can you automatically move objects from Standard storage to Nearline storage after 30 days?” (Answer: Configure a lifecycle rule). Understand conditions and actions within lifecycle rules.
- Access Control:
  - IAM Roles: Understand roles like roles/storage.objectViewer, roles/storage.objectCreator, roles/storage.admin. Exam Question: “Which IAM role allows a user to upload objects to a Cloud Storage bucket but not delete them?” (Answer: roles/storage.objectCreator).
  - Signed URLs: Grant temporary access to objects. Exam Question: “How can you grant a user temporary access to download a specific object from Cloud Storage without requiring them to have a Google account?” (Answer: Generate a signed URL). Understand the expiration time and security implications.
- Data Transfer:
  - gsutil: Command-line tool for interacting with Cloud Storage. Exam Question: “What command-line tool is used to copy files between your local machine and Cloud Storage?” (Answer: gsutil).
  - Storage Transfer Service: Migrate large datasets from other cloud providers or on-premises storage to Cloud Storage. Exam Question: “How can you migrate a large dataset from AWS S3 to Cloud Storage?” (Answer: Use the Storage Transfer Service).
- Object Versioning:
  - Keep multiple versions of an object. Exam Question: “How can you recover a previous version of an object that has been accidentally overwritten in Cloud Storage?” (Answer: Enable object versioning). Understand cost implications and how to manage versions.
c. Cloud SQL:
- Database Options:
  - MySQL: Popular open-source relational database.
  - PostgreSQL: Another popular open-source relational database known for its advanced features.
  - SQL Server: Microsoft’s relational database. Exam Question: “Your application requires support for stored procedures written in T-SQL. Which Cloud SQL database option is most suitable?” (Answer: SQL Server). Understand use cases for each database type.
- High Availability:
  - Failover replicas to provide automatic failover in case of a primary instance failure. Exam Question: “How can you ensure that your Cloud SQL database remains available even if the primary instance fails?” (Answer: Configure a failover replica). Understand the trade-offs between cost and availability.
- Backups and Recovery:
  - Automated backups and point-in-time recovery. Exam Question: “How can you restore your Cloud SQL database to a specific point in time before a data corruption incident?” (Answer: Use point-in-time recovery from a backup).
- Performance Tuning:
  - Query optimization, indexing, connection pooling. Exam Question: “What is the purpose of indexing in a relational database like Cloud SQL?” (Answer: To improve the performance of queries that filter or sort data on the indexed column).
- Security:
  - Network configuration, firewall rules, IAM roles. Exam Question: “How can you restrict access to your Cloud SQL instance to only specific IP addresses?” (Answer: Configure firewall rules).
- Cloud SQL Proxy:
  - Provides secure access to Cloud SQL instances without exposing them to the public internet. Exam Question: “What is the primary benefit of using the Cloud SQL Proxy?” (Answer: Provides secure access to Cloud SQL instances). Understand how it works (encrypting traffic, authenticating connections).
d. Cloud Spanner:
- Schema Design:
  - Interleaved Tables: Store related data together to improve query performance. Exam Question: “How can you optimize queries that frequently join data from two tables in Cloud Spanner?” (Answer: Interleave the child table within the parent table).
  - Indexes: Create indexes to speed up queries that filter or sort data. Exam Question: “How can you improve the performance of queries that filter data based on a non-key column in Cloud Spanner?” (Answer: Create an index on that column).
  - Primary Key Selection: The primary key determines how data is distributed across Spanner nodes. Exam Question: “What is the potential impact of choosing a monotonically increasing primary key in Cloud Spanner?” (Answer: Can lead to hotspots and uneven data distribution).
- Data Modeling:
  - Consider data access patterns and relationships when designing the schema.
- Instance Configuration:
  - Choosing the appropriate number of nodes based on workload.
- Transactions:
  - Ensure data consistency and integrity.
- Performance Monitoring:
  - Identify potential bottlenecks.
- Use Cases:
  - When Spanner is a good fit (global scale, strong consistency requirements) and when it’s not (smaller applications, simpler data models).
e. Datastore (Firestore in Datastore mode):
- NoSQL concepts: Understand the differences between relational and NoSQL databases. Exam Question: “What are the key characteristics of a NoSQL database like Datastore?” (Answer: Schema-less, horizontal scalability, eventual consistency).
- Schema-less design: Understand that data can have varying attributes. Exam Question: “What does schema-less mean in the context of Datastore?” (Answer: Documents in a collection can have different sets of attributes).
- Querying limitations: Understand the limitations of querying without indexes. Exam Question: “What happens if you attempt to run a query in Datastore that is not supported by an index?” (Answer: The query will fail).
- Scalability: Datastore is designed to scale horizontally. Exam Question: “How does Datastore handle increasing data volumes and user traffic?” (Answer: It scales horizontally by distributing data across multiple servers).
- Transactions: ACID properties for data consistency. Exam Question: “What are the ACID properties of transactions in Datastore?” (Answer: Atomicity, Consistency, Isolation, Durability).
- Indexes: Creating indexes for efficient queries. Exam Question: “When is it necessary to create an index in Datastore?” (Answer: When you need to query data based on specific properties).

5.2. Data Processing & Pipelines

a. Dataflow:
- Apache Beam:
  - PCollections: Represent distributed datasets. Exam Question: “What is a PCollection in Apache Beam?” (Answer: An immutable, distributed collection of data).
  - PTransforms: Represent data processing operations. Exam Question: “What is a PTransform in Apache Beam?” (Answer: An operation that transforms one or more PCollections into one or more PCollections).
  - Pipelines: Represent the overall data processing workflow. Exam Question: “What is the purpose of a Pipeline object in Apache Beam?” (Answer: To define the overall data processing workflow). Understand the building blocks of Beam pipelines.
- Batch vs. Stream Processing:
  - Batch: Processing data in bulk. Exam Question: “Which processing mode is most suitable for analyzing historical data?” (Answer: Batch processing).
  - Stream: Processing data in real-time. Exam Question: “Which processing mode is most suitable for processing real-time sensor data?” (Answer: Stream processing).
- Windowing:
  - Fixed Windows: Divide data into fixed-size time intervals. Exam Question: “How do fixed windows divide data in stream processing?” (Answer: Into non-overlapping time intervals of a fixed duration).
  - Sliding Windows: Overlapping time intervals. Exam Question: “What is the key difference between fixed windows and sliding windows?” (Answer: Sliding windows overlap, while fixed windows do not).
  - Session Windows: Group data based on activity. Exam Question: “Which type of window is best suited for analyzing user sessions based on periods of inactivity?” (Answer: Session windows). Understand the use cases for each window type.
- Triggers:
  - Control when results are emitted from a window. Exam Question: “What is the purpose of a trigger in stream processing?” (Answer: To control when results are emitted from a window). Understand early firing, late data handling.
- Fault Tolerance:
  - Dataflow automatically handles failures. Exam Question: “How does Dataflow handle worker failures during pipeline execution?” (Answer: It automatically restarts the failed workers and reprocesses the data).
- Autoscaling:
  - Dataflow automatically scales resources based on workload. Exam Question: “How does Dataflow automatically adjust resources based on workload?” (Answer: By dynamically adding or removing workers).
- Templates:
  - Reusable pipelines. Exam Question: “What is the purpose of Dataflow templates?” (Answer: To create reusable pipelines that can be launched with different parameters).
- Monitoring:
  - Using Cloud Monitoring and Dataflow’s UI to track pipeline performance.
b. Dataproc:
- Spark and Hadoop:
  - Understand the basics of Spark and Hadoop:
    - Hadoop: Understand the core components: HDFS (Hadoop Distributed File System) for storage and MapReduce for processing. Exam Question: “What is the primary function of HDFS in a Hadoop cluster?” (Answer: To provide distributed, fault-tolerant storage for large datasets). Know the limitations of MapReduce (e.g., iterative processing).
    - Spark: Understand Spark’s core concepts: RDDs (Resilient Distributed Datasets), DataFrames, Spark SQL, Spark Streaming, MLlib (machine learning library), GraphX (graph processing). Exam Question: “What is an RDD in Spark?” (Answer: An immutable, distributed collection of data). Understand the benefits of Spark’s in-memory processing. Know when to use Spark SQL for structured data processing.
    - Comparison: Understand the differences between Hadoop and Spark and when to choose each. Exam Question: “In what scenario is Spark generally more performant than MapReduce?” (Answer: When processing data iteratively, such as in machine learning algorithms).
- Cluster Configuration:
  - Choosing the appropriate machine types and number of workers:
    - Machine Types: Understand the different machine types available on GCP (e.g., Compute Engine instance types) and their characteristics (CPU, memory, disk). Exam Question: “Which machine type is best suited for CPU-intensive Spark jobs?” (Answer: A machine type with a high number of vCPUs). Consider general-purpose, compute-optimized, and memory-optimized machine types.
    - Number of Workers: Understand how the number of workers affects performance and cost. Consider the size of the dataset, the complexity of the job, and the desired processing time. Exam Question: “Increasing the number of workers in a Dataproc cluster typically leads to which of the following?” (Answer: Increased parallelism and potentially faster job completion time, but also increased cost). Understand the concept of over-provisioning and under-provisioning.
    - Preemptible VMs: Know how to use preemptible VMs to reduce costs. Exam Question: “How can you reduce the cost of a Dataproc cluster if the job is fault-tolerant and can tolerate interruptions?” (Answer: Use preemptible VMs for worker nodes). Understand the limitations of preemptible VMs (they can be terminated at any time).
- Autoscaling:
  - Automatically scaling the number of workers based on workload:
    - Autoscaling Policies: Understand how to configure autoscaling policies (minimum and maximum number of workers, scaling triggers). Exam Question: “How can you configure a Dataproc cluster to automatically scale up when CPU utilization exceeds a certain threshold?” (Answer: Configure an autoscaling policy with a CPU utilization metric trigger).
    - Scaling Triggers: Understand different scaling triggers (CPU utilization, memory utilization, YARN pending memory). Exam Question: “Which metric is most appropriate for triggering autoscaling in a Dataproc cluster that is running memory-intensive Spark jobs?” (Answer: Memory utilization).
    - Graceful Decommissioning: Understand how Dataproc handles worker decommissioning during autoscaling. Exam Question: “What happens to running Spark executors when a worker node is decommissioned during autoscaling?” (Answer: Dataproc attempts to gracefully decommission the executors to minimize data loss).
- Job Submission:
  - Submitting Spark and Hadoop jobs to Dataproc:
    - gcloud dataproc jobs submit command: Understand how to use the gcloud command-line tool to submit jobs. Exam Question: “Which command is used to submit a Spark job to a Dataproc cluster from the command line?” (Answer: gcloud dataproc jobs submit spark).
    - Spark Submit Arguments: Understand common Spark submit arguments (e.g., --class, --jars, --driver-memory, --executor-memory). Exam Question: “Which Spark submit argument specifies the main class of the Spark application?” (Answer: --class).
    - Hadoop JAR Jobs: Understand how to submit Hadoop JAR jobs to Dataproc. Exam Question: “How can you submit a MapReduce job to a Dataproc cluster?” (Answer: Submit a Hadoop JAR job).
    - Jupyter Notebooks: Understand how to use Jupyter notebooks with Dataproc.
- Integration with other GCP services:
  - Cloud Storage, BigQuery, and other GCP services:
    - Cloud Storage: Understand how to use Cloud Storage as the input and output for Dataproc jobs. Exam Question: “How can you configure a Dataproc job to read data from a Cloud Storage bucket?” (Answer: Specify the Cloud Storage bucket path as the input path for the job). Understand the gs:// URI scheme.
    - BigQuery: Understand how to use Spark SQL to query data in BigQuery. Exam Question: “How can you use Spark to read data from a BigQuery table?” (Answer: Use the BigQuery connector for Spark). Understand how to write data from Spark to BigQuery.
    - Cloud Logging: Understand how to integrate Dataproc with Cloud Logging for monitoring and troubleshooting. Exam Question: “How can you collect logs from a Dataproc cluster for analysis?” (Answer: Integrate Dataproc with Cloud Logging).
- Initialization Actions:
  - Customizing Dataproc clusters:
    - Shell Scripts: Understand how to use initialization actions to run shell scripts when creating a Dataproc cluster. Exam Question: “How can you install custom software on all nodes in a Dataproc cluster?” (Answer: Use an initialization action to run a shell script that installs the software).
    - Python Scripts: Understand how to use initialization actions to run Python scripts.
    - Use Cases: Common use cases for initialization actions (e.g., installing dependencies, configuring environment variables, setting up security). Exam Question: “You need to install a specific Python library on all nodes in a Dataproc cluster. How can you automate this process?” (Answer: Use an initialization action to run a pip install command).
- Versioning:
  - Managing different versions of Spark and Hadoop:
    - Dataproc Image Versions: Understand how to select the Dataproc image version when creating a cluster. Exam Question: “How can you ensure that a Dataproc cluster is running a specific version of Spark?” (Answer: Select the appropriate Dataproc image version when creating the cluster). Understand the components included in each image version.
    - Component Gateway: Understand how to access the web UIs for different components (e.g., Spark UI, Hadoop YARN UI) through the Component Gateway.
c. Pub/Sub:
- Publishers and Subscribers:
  - Publishers send messages to topics. Subscribers receive messages from subscriptions.
    - Publisher Responsibilities: Creating and sending messages to a Pub/Sub topic. Error handling for message publishing failures. Exam Question: “What is the primary responsibility of a publisher in a Pub/Sub system?” (Answer: To send messages to a topic).
    - Subscriber Responsibilities: Receiving and processing messages from a subscription. Acknowledging messages to prevent redelivery. Error handling for message processing failures. Exam Question: “What must a subscriber do after receiving a message to prevent it from being redelivered?” (Answer: Acknowledge the message).
    - Decoupling: Understand how Pub/Sub decouples publishers and subscribers. Exam Question: “What is the main benefit of using Pub/Sub to decouple applications?” (Answer: It allows publishers and subscribers to operate independently and scale separately).
- Topics and Subscriptions:
  - Topics are named channels for messages. Subscriptions are named configurations that attach to a topic.
    - Topic Creation: Creating topics with appropriate settings (e.g., message retention duration). Exam Question: “How long are messages retained in a Pub/Sub topic by default?” (Answer: 7 days).
    - Subscription Types: Understanding different subscription types (push, pull).
    - Subscription Configuration: Configuring subscriptions with appropriate settings (e.g., acknowledgement deadline, message filtering).
    - Message Filtering: Using filters to receive only a subset of messages based on attributes. Exam Question: “How can a subscriber receive only messages with a specific attribute value?” (Answer: Configure a filter on the subscription).
- Message Ordering:
  - Pub/Sub provides at-least-once delivery and best-effort ordering.
    - At-Least-Once Delivery: Understand that Pub/Sub guarantees that each message will be delivered at least once, but may be delivered more than once. Exam Question: “What is the delivery guarantee provided by Pub/Sub?” (Answer: At-least-once delivery).
    - Best-Effort Ordering: Understand that Pub/Sub attempts to deliver messages in the order they were published, but this is not guaranteed. Exam Question: “Does Pub/Sub guarantee strict message ordering?” (Answer: No, it provides best-effort ordering).
    - Ordering Keys: Understand how to use ordering keys to ensure that messages with the same key are delivered in order. Exam Question: “How can you ensure that messages with the same key are delivered in the order they were published?” (Answer: Use ordering keys).
- Exactly-Once Delivery:
  - Achieving exactly-once delivery with Pub/Sub.
    - Idempotency: Understand the concept of idempotency and how to design subscribers to be idempotent. Exam Question: “What is idempotency and why is it important when processing messages from Pub/Sub?” (Answer: Idempotency means that processing the same message multiple times has the same effect as processing it once. This is important because Pub/Sub provides at-least-once delivery).
    - Deduplication: Using a mechanism to deduplicate messages at the subscriber.
- Push and Pull Subscriptions:
  - Push subscriptions: Pub/Sub pushes messages to a subscriber endpoint. Pull subscriptions: Subscribers pull messages from Pub/Sub.
    - Push Subscription Configuration: Configuring a push subscription with the correct endpoint URL and authentication settings. Exam Question: “What type of authentication is typically required for a push subscription?” (Answer: The subscriber endpoint must authenticate with Pub/Sub).
    - Pull Subscription Configuration: Understanding how subscribers pull messages from a pull subscription.
    - Use Cases: Know when to use push subscriptions (e.g., when the subscriber is a web server) and when to use pull subscriptions (e.g., when the subscriber needs more control over message processing). Exam Question: “In what scenario is a pull subscription generally preferred over a push subscription?” (Answer: When the subscriber needs to control the rate at which it receives messages).
- Dead Letter Queues:
  - Handling undeliverable messages.
    - Dead Letter Topic: Configuring a dead letter topic for a subscription. Exam Question: “How can you configure a subscription to send undeliverable messages to a separate topic?” (Answer: Configure a dead letter topic).
    - Reprocessing Messages: Understanding how to reprocess messages from a dead letter topic. Exam Question: “What is the purpose of a dead letter queue?” (Answer: To store messages that could not be delivered to the subscriber, allowing them to be investigated and potentially reprocessed).
- Monitoring:
  - Using Cloud Monitoring to track Pub/Sub performance.
    - Key Metrics: Monitoring key metrics such as message publish rate, message delivery rate, and message acknowledgement latency. Exam Question: “Which Cloud Monitoring metric is most useful for tracking the rate at which messages are being published to a Pub/Sub topic?” (Answer: pubsub.googleapis.com/topic/send_message_operation_count).
    - Alerting: Setting up alerts based on these metrics to detect issues.
d. Cloud Composer:
- Apache Airflow:
  - Understand the basics of Apache Airflow (DAGs, tasks, operators).
    - DAGs (Directed Acyclic Graphs): Understand how DAGs define workflows as a series of tasks. Exam Question: “What is a DAG in Apache Airflow?” (Answer: A Directed Acyclic Graph that represents a workflow). Know how to define dependencies between tasks.
    - Tasks: Understand that tasks are the basic units of work in a DAG. Exam Question: “What is a task in Apache Airflow?” (Answer: A single unit of work to be performed in a DAG).
    - Operators: Understand how operators define the type of work that a task performs. Exam Question: “What is the purpose of an operator in Apache Airflow?” (Answer: To define the type of work that a task performs). Be familiar with common operators (e.g., BigQueryOperator, DataflowOperator, BashOperator).
- DAG Design:
  - Designing DAGs to represent complex workflows.
    - Task Dependencies: Defining task dependencies to control the order of execution. Exam Question: “How can you ensure that a task only runs after another task has completed successfully?” (Answer: Define a dependency between the two tasks).
    - Error Handling: Implementing error handling mechanisms in DAGs (e.g., retries, failure callbacks). Exam Question: “How can you configure a task to automatically retry if it fails?” (Answer: Set the retries parameter for the task).
    - Idempotency: Designing tasks to be idempotent.
- Operators:
  - Using Airflow operators.
    - Common Operators: Be familiar with common operators such as BigQueryOperator, DataflowOperator, PubSubPublishOperator, BashOperator, PythonOperator. Exam Question: “Which operator is used to execute a SQL query in BigQuery from an Airflow DAG?” (Answer: BigQueryOperator).
    - Custom Operators: Understand how to create custom operators.
    - Operator Parameters: Understand the parameters that are available for each operator.
- Variables and Connections:
  - Managing configuration and credentials.
    - Variables: Using Airflow variables to store configuration values. Exam Question: “How can you store a configuration value that can be accessed by multiple tasks in an Airflow DAG?” (Answer: Use an Airflow variable).
    - Connections: Using Airflow connections to store credentials for accessing external systems. Exam Question: “How can you securely store the credentials for accessing a BigQuery dataset from an Airflow DAG?” (Answer: Create an Airflow connection with the appropriate credentials).
- Scheduling:
  - Scheduling DAGs to run automatically.
    - Cron Expressions: Using cron expressions to define the schedule for a DAG. Exam Question: “How can you schedule a DAG to run every day at midnight?” (Answer: Use the cron expression 0 0 * * *).
    - Timetables: Using timetables to define more complex schedules.
    - Catchup: Understanding the catchup parameter and its impact on DAG execution.
- Monitoring:
  - Using the Airflow UI and Cloud Monitoring to track DAGs.
    - Airflow UI: Using the Airflow UI to monitor DAG runs, task states, and logs. Exam Question: “How can you view the logs for a specific task in an Airflow DAG?” (Answer: Use the Airflow UI to navigate to the task and view its logs).
    - Cloud Monitoring: Using Cloud Monitoring to track metrics such as DAG run duration and task success rate.
- Environments:
  - Creating and managing Cloud Composer environments.
    - Environment Configuration: Configuring the environment with the appropriate settings (e.g., machine type, number of workers).
    - PyPI Packages: Installing custom PyPI packages in the environment. Exam Question: “How can you install a custom Python library in a Cloud Composer environment?” (Answer: Specify the library in the PyPI packages configuration).
    - Airflow Version: Selecting the appropriate Airflow version for the environment.
e. Cloud Functions:
- Event Triggers:
  - Understand the different event triggers (e.g., Cloud Storage events, Pub/Sub messages, HTTP requests).
    - Cloud Storage Triggers: Triggering a Cloud Function when a file is uploaded to Cloud Storage. Exam Question: “How can you automatically trigger a function when a new image is uploaded to a Cloud Storage bucket?” (Answer: Use a Cloud Storage trigger). Know different event types (e.g., google.storage.object.finalize).
    - Pub/Sub Triggers: Triggering a Cloud Function when a message is published to a Pub/Sub topic. Exam Question: “How can you automatically process messages published to a Pub/Sub topic?” (Answer: Use a Pub/Sub trigger).
    - HTTP Triggers: Triggering a Cloud Function via an HTTP request. Exam Question: “How can you create a serverless API endpoint using Cloud Functions?” (Answer: Use an HTTP trigger).
    - Cloud Scheduler: Triggering a Cloud Function on a schedule.
    - Firebase Triggers: Triggering a Cloud Function based on Firebase events (e.g., user creation, database updates).
- Languages:
  - Supported languages (Python, Node.js, Go, Java, .NET, Ruby, PHP).
    - Language Selection: Choosing the appropriate language based on the requirements of the function. Exam Question: “Your function requires access to a specific Python library. Which language should you use for your Cloud Function?” (Answer: Python).
    - Runtime Environment: Understanding the runtime environment for each language.
- Scaling:
  - Cloud Functions automatically scale to handle incoming requests.
    - Automatic Scaling: Understand that Cloud Functions automatically scale based on the number of incoming requests. Exam Question: “How does Cloud Functions handle increasing traffic to an HTTP-triggered function?” (Answer: It automatically scales the number of function instances).
    - Concurrency: Understanding the concept of concurrency and how it affects scaling.
- Execution Time Limits:
  - Be aware of the execution time limits.
    - Function Timeout: Understand that Cloud Functions have a maximum execution time. Exam Question: “What happens if a Cloud Function exceeds its maximum execution time?” (Answer: The function is terminated). Know the default and maximum timeout values.
    - Background Functions: Understanding how to use background functions for long-running tasks.
- Cold Starts:
  - Understand the concept of cold starts.
    - Cold Start Latency: Understand that cold starts can introduce latency. Exam Question: “What is a cold start in Cloud Functions?” (Answer: The time it takes to initialize a new function instance when it is invoked for the first time or after a period of inactivity).
    - Minimizing Cold Starts: Techniques for minimizing cold starts (e.g., keeping functions warm, using languages with faster startup times).
f. IAM Roles:
- Granting Cloud Functions access to other GCP services.
  - Service Account: Understand that Cloud Functions run under a service account. Exam Question: “Under what identity does a Cloud Function typically run?” (Answer: A service account).
  - IAM Permissions: Granting the service account the necessary IAM permissions to access other GCP services. Exam Question: “Your Cloud Function needs to write data to a BigQuery table. What IAM permission is required?” (Answer: roles/bigquery.dataEditor on the BigQuery dataset).
- Use Cases:
  - Know appropriate use cases (e.g., image resizing, data validation, event-driven processing).
    - Image Resizing: Resizing images when they are uploaded to Cloud Storage.
    - Data Validation: Validating data when it is published to Pub/Sub.
    - Event-Driven Processing: Responding to events in real-time.
5.3. Machine Learning
- a. Vertex AI:
  - Model Training:
    - Understand different options for training models (e.g., using pre-built algorithms, custom training with TensorFlow or PyTorch).
      - Pre-built Algorithms: Understand the available pre-built algorithms in Vertex AI (e.g., tabular regression, tabular classification, image classification, object detection). Exam Question: “You need to train a model to predict customer churn based on tabular data. Which type of pre-built algorithm in Vertex AI is most suitable?” (Answer: Tabular classification). Know the input data requirements for each algorithm.
      - Custom Training: Using TensorFlow, PyTorch, or other frameworks to train models. Exam Question: “When is it necessary to use custom training instead of a pre-built algorithm in Vertex AI?” (Answer: When you need to use a specific model architecture or training procedure that is not supported by the pre-built algorithms). Understand how to package and submit custom training jobs to Vertex AI.
      - Managed Datasets: Understand how to create and manage datasets in Vertex AI for training.
      - Training Pipelines: Orchestrating multiple training steps using Vertex AI Pipelines.
      - Distributed Training: Leveraging distributed training to accelerate model training.
  - Hyperparameter Tuning:
    - Using Vertex AI’s hyperparameter tuning service.
      - Hyperparameters: Understand what hyperparameters are and how they affect model performance. Exam Question: “What are hyperparameters in machine learning?” (Answer: Parameters that control the training process of a model).
      - Search Algorithms: Understand the different search algorithms available in Vertex AI’s hyperparameter tuning service (e.g., grid search, random search, Bayesian optimization). Exam Question: “Which search algorithm is generally most effective for finding optimal hyperparameter values?” (Answer: Bayesian optimization).
      - Optimization Objective: Defining the objective metric to optimize (e.g., accuracy, precision, recall).
      - Trial Budget: Setting a budget for the hyperparameter tuning process.
  - Model Deployment:
    - Deploying models to Vertex AI’s prediction service.
      - Endpoint Creation: Creating an endpoint for serving predictions.
      - Model Deployment: Deploying a model to an endpoint. Exam Question: “What is an endpoint in Vertex AI’s prediction service?” (Answer: A serving environment where models are deployed to receive prediction requests).
      - Traffic Splitting: Splitting traffic between different model versions.
      - Online Prediction: Serving predictions in real-time.
      - Batch Prediction: Generating predictions in batch mode.
  - Model Monitoring:
    - Monitoring model performance and detecting drift.
      - Performance Metrics: Monitoring performance metrics such as accuracy, precision, recall, and AUC.
      - Data Drift: Detecting data drift, which occurs when the input data distribution changes over time.
      - Prediction Drift: Detecting prediction drift, which occurs when the model’s predictions become less accurate over time.
      - Alerting: Setting up alerts to notify you when model performance degrades or data drift is detected.
      - Explainable AI: Using Explainable AI to understand why a model is making certain predictions.
  - Feature Store:
    - Managing and serving features for ML models.
      - Feature Definition: Defining features and their data types.
      - Feature Storage: Storing features in a scalable and reliable feature store.
      - Feature Serving: Serving features to models for training and prediction. Exam Question: “What is the purpose of a feature store in Vertex AI?” (Answer: To provide a centralized repository for managing and serving features for ML models).
      - Online and Offline Serving: Serving features in both online (real-time) and offline (batch) modes.
  - AutoML:
    - Automatically training and deploying machine learning models.
      - Data Preparation: Preparing data for AutoML training.
      - Model Selection: AutoML automatically selects the best model architecture and hyperparameters for the given data.
      - Model Deployment: AutoML automatically deploys the trained model to Vertex AI’s prediction service.
      - Use Cases: Understand appropriate use cases for AutoML (e.g., when you need to train a model quickly and easily without extensive machine learning expertise). Exam Question: “When is AutoML a good choice for training a machine learning model?” (Answer: When you want to quickly train a model with minimal manual effort and machine learning expertise).
- b. TensorFlow and PyTorch on GCP:
  - Framework fundamentals.
    - TensorFlow: Understand the basics of TensorFlow, including tensors, variables, layers, models, and training loops.
    - PyTorch: Understand the basics of PyTorch, including tensors, autograd, models, and training loops.
  - Training on Vertex AI.
    - Custom Training Jobs: Packaging and submitting TensorFlow and PyTorch training jobs to Vertex AI.
    - Distributed Training: Leveraging distributed training with TensorFlow and PyTorch on Vertex AI.
    - GPU and TPU Support: Using GPUs and TPUs to accelerate model training. Exam Question: “What is the primary benefit of using TPUs for training TensorFlow models on GCP?” (Answer: TPUs are designed specifically for machine learning workloads and can provide significant performance improvements over GPUs).
  - TPUs.
    - TPU Architecture: Understand the architecture of TPUs and how they differ from CPUs and GPUs.
    - TPU Pods: Leveraging TPU Pods for large-scale distributed training.
    - TPU Usage: How to configure TensorFlow and PyTorch to use TPUs.

5.4. Data Security & Governance

a. IAM (Identity and Access Management):
- Roles:
  - Predefined roles and custom roles.
    - Predefined Roles: Understand common predefined roles (e.g., roles/viewer, roles/editor, roles/owner, roles/storage.objectViewer, roles/bigquery.dataViewer). Exam Question: “Which predefined role grants a user read-only access to all resources in a project?” (Answer: roles/viewer).
    - Custom Roles: Creating custom roles to grant specific permissions. Exam Question: “When is it necessary to create a custom role?” (Answer: When the predefined roles do not provide the required level of granularity or access control). Understand how to define permissions in a custom role.
- Principals:
  - Users, service accounts, groups.
    - Users: Google accounts that represent individual users.
    - Service Accounts: Non-human accounts used by applications and services. Exam Question: “What type of principal should be used for a Cloud Function that needs to access a BigQuery dataset?” (Answer: A service account). Understand how to create and manage service accounts.
    - Groups: Collections of users and/or service accounts.
- Resource Hierarchy:
  - Organization, folder, project.
    - IAM Policy Inheritance: Understand how IAM policies are inherited through the resource hierarchy. Exam Question: “If an IAM policy is set at the organization level, will it apply to all projects within that organization?” (Answer: Yes, unless overridden at a lower level in the hierarchy).
    - Policy Overriding: Understand how policies can be overridden at lower levels in the hierarchy.
- Service Accounts:
  - Best practices for using service accounts.
    - Least Privilege: Granting service accounts only the necessary permissions.
    - Key Management: Managing service account keys securely. Exam Question: “What is the recommended way to manage service account keys?” (Answer: Use the IAM service account key management features to rotate and disable keys).
    - Auditing: Auditing service account usage.
- Least Privilege:
  - Applying the principle of least privilege.
    - Granting Only Necessary Permissions: Granting users and service accounts only the permissions they need to perform their tasks.
    - Avoiding Overly Broad Roles: Avoiding the use of overly broad roles such as roles/owner.
b. Data Loss Prevention (DLP):
- InfoTypes:
  - Different InfoTypes that DLP can detect (e.g., credit card numbers, social security numbers).
    - Predefined InfoTypes: Understand common predefined InfoTypes such as CREDIT_CARD_NUMBER, US_SOCIAL_SECURITY_NUMBER, EMAIL_ADDRESS.
    - Custom InfoTypes: Creating custom InfoTypes to detect specific types of sensitive data. Exam Question: “You need to detect a specific type of internal employee ID that is not covered by the predefined InfoTypes. How can you achieve this?” (Answer: Create a custom InfoType).
- Inspection Templates:
  - Creating inspection templates.
    - Defining Inspection Configuration: Defining the InfoTypes to detect, the data transformations to apply, and the actions to take. Exam Question: “What is the purpose of an inspection template in DLP?” (Answer: To define a reusable configuration for inspecting data for sensitive information).
    - Reusing Inspection Configurations: Reusing inspection configurations across multiple jobs.
- Actions:
  - Different actions that DLP can take (e.g., redaction, masking, tokenization).
    - Redaction: Removing sensitive data from the output.
    - Masking: Replacing sensitive data with a mask character.
    - Tokenization: Replacing sensitive data with a surrogate value (token). Exam Question: “Which DLP action is most suitable for protecting sensitive data while still allowing it to be used for analysis?” (Answer: Tokenization).
    - De-identification: Transforming data to remove identifying information.
- Integration:
  - Integrating with other GCP services (e.g., Cloud Storage, BigQuery).
    - Scanning Cloud Storage: Scanning Cloud Storage buckets for sensitive data. Exam Question: “How can you use DLP to scan a Cloud Storage bucket for credit card numbers?” (Answer: Create a DLP job that targets the Cloud Storage bucket and specifies the CREDIT_CARD_NUMBER InfoType).
    - Scanning BigQuery: Scanning BigQuery tables for sensitive data.
    - Real-time and Batch Processing: Using DLP for both real-time and batch processing of data.
c. Cloud KMS (Key Management Service):
- Key Rings and Keys:
  - Key Rings and Keys
    - Key Rings: Understand that key rings are logical groupings of keys.
    - Keys: Understand that keys are used for encryption and decryption. Different key types (Symmetric, Asymmetric).
- Key Rotation:
  - Rotating encryption keys.
    - Automatic Key Rotation: Configuring automatic key rotation. Exam Question: “What is the best practice for managing encryption keys in Cloud KMS?” (Answer: Enable automatic key rotation).
    - Manual Key Rotation: Manually rotating keys.
- Encryption and Decryption:
  - Using KMS to encrypt and decrypt data.
    - Using the KMS API: Using the KMS API to encrypt and decrypt data.
    - Envelope Encryption: Understanding envelope encryption.
- IAM Roles:
  - IAM roles required to manage and use encryption keys.
    - roles/cloudkms.cryptoKeyEncrypterDecrypter: Role for encrypting and decrypting data with a key.
    - roles/cloudkms.cryptoKeyAdmin: Role for managing keys and key rings.
- Customer-Managed Encryption Keys (CMEK):
  - Using CMEK.
    - Controlling Encryption Keys: Using CMEK to control the encryption keys used to protect your data. Exam Question: “What is the main benefit of using Customer-Managed Encryption Keys (CMEK)?” (Answer: It gives you control over the encryption keys used to protect your data).
    - Compliance: Meeting compliance requirements.

5.5. Monitoring & Logging

a. Cloud Monitoring:
- Metrics and Logs
  - Metrics: Numerical data that describes the performance and health of your systems. Exam Question: “What are metrics used for in Cloud Monitoring?” (Answer: To track the performance and health of your systems).
  - Logs: Textual data that provides information about events that have occurred in your systems. Exam Question: “What are logs used for in Cloud Logging?” (Answer: To provide information about events that have occurred in your systems).
- Alerting
  - Creating Alerts: Creating alerts to notify you when metrics exceed certain thresholds. Exam Question: “How can you be notified when CPU utilization on a Compute Engine instance exceeds 80%?” (Answer: Create an alert in Cloud Monitoring).
  - Alert Channels: Configuring alert channels to receive notifications (e.g., email, SMS, PagerDuty).
- Dashboards
  - Creating Dashboards: Creating dashboards to visualize metrics and logs. Exam Question: “How can you visualize the CPU utilization of all Compute Engine instances in a project?” (Answer: Create a dashboard in Cloud Monitoring).
  - Custom Dashboards: Creating custom dashboards with specific metrics and visualizations.
- Log-based Metrics
  - Creating Metrics from Logs: Creating metrics based on log data. Exam Question: “How can you create a metric to track the number of errors in your application logs?” (Answer: Create a log-based metric in Cloud Logging).
b. Cloud Logging:
- Metrics and Logs
  - Structured Logging: Understand the benefits of structured logging (e.g., JSON format).
  - Log Fields: Understanding the different fields in a log entry (e.g., timestamp, severity, resource).
- Alerting
  - Creating Alerts: Creating alerts based on log data.
  - Log Filters: Using log filters to define the criteria for triggering alerts.
- Dashboards
  - Creating Dashboards: Creating dashboards to visualize log data.
  - Log Explorer: Using the Log Explorer to search and analyze logs.
- Sinks
  - Routing logs to different destinations (e.g., Cloud Storage, BigQuery, Pub/Sub).
    - Cloud Storage: Exporting logs to Cloud Storage for long-term storage. Exam Question: “How can you archive your application logs for compliance purposes?” (Answer: Export the logs to Cloud Storage using a sink).
    - BigQuery: Exporting logs to BigQuery for analysis. Exam Question: “How can you analyze your application logs using SQL?” (Answer: Export the logs to BigQuery using a sink).
    - Pub/Sub: Exporting logs to Pub/Sub for real-time processing.
    - Log Router Filters: Using log router filters to route specific logs to different destinations.
- Filtering
  - Using filters to find specific log entries.
    - Basic Filters: Using basic filters to search for log entries based on keywords.
    - Advanced Filters: Using advanced filters to search for log entries based on complex criteria.
- Log-based Metrics
  - Creating Metrics from Logs: Creating metrics based on log data.

6. Summary

My preparation involved structured learning through Google Cloud Skills Boost, deep dive with labs and extensive practice exams. Understanding data storage, processing, security, architecture and use cases is critical. If you plan to take the PDE exam, ensure hands-on experience with BigQuery, Dataflow, and Dataproc, and be ready to evaluate real-world scenarios involving data engineering.

7. Resources

Google Cloud Skills Boost - Data Engineer Learning Path: https://www.cloudskillsboost.google/paths/16/
Exam Guide: https://services.google.com/fh/files/misc/professional_data_engineer_exam_guide_english.pdf
Official Sample Questions: https://docs.google.com/forms/d/e/1FAIpQLSfkWEzBCP0wQ09ZuFm7G2_4qtkYbfmk_0getojdnPdCYmq37Q/viewform
Google Cloud Community Guide: https://www.googlecloudcommunity.com/gc/Community-Blogs/Your-guide-to-preparing-for-the-Google-Cloud-Professional-Data/ba-p/543105
Practice Tests on Udemy: https://www.udemy.com/course/gcp-professional-data-engineer-exam-practice-tests-2024/?couponCode=ST5MT020225
Practice Tests on ExamPrepper: https://www.examprepper.co/exam/10/1

Good luck with your PDE journey!

Google Professional Cloud Architect - EHR Healthcare Case Study

Mon, 23 Dec 2024 00:00:00 +0000

1. Scenario Overview

EHR Healthcare is migrating from colocation data centers to Google Cloud to improve scalability, availability, and cost efficiency while maintaining legacy systems. Their requirements span modern application management, hybrid connectivity, analytics, compliance, and automation.

Link: https://services.google.com/fh/files/blogs/master_case_study_ehr_healthcare.pdf

2. Summary of Core Solutions

Requirement	GCP Solution
Container Management	Anthos / GKE
Hybrid Connectivity	Cloud Interconnect, VPN
Monitoring & Logging	Cloud Operations Suite
Availability and Latency	Load Balancer, Multi-Region GKE
Managed Databases	Cloud SQL, Memorystore, MongoDB
Analytics and Trends	BigQuery, Dataflow, Looker
Legacy Integration	Apigee API Management, Pub/Sub
Security and Compliance	IAM, SCC, DLP, Workload Identity
CI/CD	Cloud Build, Artifact Registry
AD FS	Google Cloud Directory Sync

3. Question Breakdown by Subject

A. Application Modernization (Container-Based Apps)

Likely Exam Question: “Which GCP service provides a consistent way to manage container-based applications?”
Answer: Google Kubernetes Engine (GKE)
Why: It’s fully managed, scales automatically, and is designed for containerized workloads. Choose Anthos if on-premises or hybrid environment is a required.

B. Hybrid Connectivity

Likely Exam Question: “What’s the most secure and high-performance connection between on-premises systems and Google Cloud?”
Answer: Cloud Interconnect (if they need high bandwidth) or Cloud VPN (if security over lower bandwidth is enough).
Elimination Tip: If an option mentions “public internet” or generic connectivity, it’s likely incorrect.

C. Logging and Monitoring

Likely Exam Question: “How can EHR centrally monitor and take proactive action on system performance?”
Answer: Google Cloud Operations Suite.
Why: It provides consistent monitoring, logging, and alerting across cloud and hybrid environments.

Key Tools to Recognize:

Cloud Logging: Centralized log collection.
Cloud Monitoring: Performance metrics and uptime monitoring.
Cloud Alerting: Configurable alerts.

D. Scalability and Availability

Likely Exam Question: “How can EHR ensure 99.9% availability for customer-facing systems?”
Answer: Global HTTP(S) Load Balancer + Multi-region GKE deployment.
Why: Load Balancer reduces latency and routes traffic globally, while regional deployments ensure redundancy.

E. Managed Databases

Likely Exam Question: “Which service can EHR use to migrate their MySQL and SQL Server databases to a managed solution?”
Answer: Cloud SQL
Why: It’s a managed database service supporting MySQL and SQL Server.

For Redis: Memorystore.
For MongoDB: Likely MongoDB Atlas (third-party but hosted on GCP).

F. Analytics and Insights

Likely Exam Question: “Which GCP service can analyze large datasets and provide insights into healthcare trends?”
Answer: BigQuery
Why: It’s a serverless data warehouse built for analyzing massive datasets.

If real-time ingestion is involved: Dataflow.
If reporting is mentioned: Looker.

G. CI/CD and Deployment Automation

Likely Exam Question: “Which services enable automated builds and deployments for EHR’s containerized applications?”
Answer: Cloud Build and Artifact Registry
Why: Cloud Build automates CI/CD pipelines, and Artifact Registry manages container images securely.

H. Security and Compliance

Likely Exam Question: “How can EHR meet regulatory compliance while securing sensitive healthcare data?”
Answer:
- Cloud IAM: Centralized access control.
- Workload Identity: Integrates GKE workloads with Active Directory.
- Security Command Center (SCC): Visibility into security vulnerabilities.
- Data Loss Prevention (DLP): Identifies and protects sensitive data.

I. Active Directory Federation Services

Likely Exam Question: “How can EHR sync local AD users with the cloud?”
Answer:
- Google Cloud Directory Sync: Synchronize users and group from local AD to Cloud Identity.

Google Professional Cloud Architect - Helicopter Racing League Case Study

Sun, 22 Dec 2024 00:00:00 +0000

1. Scenario Overview

Helicopter Racing League (HRL) is an emerging esports organization focused on competitive helicopter racing simulations. HRL streams live events globally and offers fans real-time stats, telemetry, and replays. The league is planning to move to Google Cloud to enhance scalability, improve user experience, and optimize costs. Their requirements span real-time data processing, low latency streaming, and robust analytics capabilities.

Link: Helicopter Racing League Case Study

2. Summary of Core Solutions

Requirement	GCP Solution
Real-Time Data Processing	Dataflow, Pub/Sub
Low Latency Streaming	Cloud CDN, Global Load Balancer
User Analytics and Insights	BigQuery, Looker
Scalable Infrastructure for Events	Compute Engine, GKE
AI/ML for Predictions and Highlights	Vertex AI, BigQuery ML
Security and Compliance	IAM, Cloud Armor, SCC
Continuous Integration and Deployment	Cloud Build, Artifact Registry
Cost Optimization	Active Assist, Autoscaler

3. Question Breakdown by Subject

A. Real-Time Data Processing

Likely Exam Question: “Which GCP service processes real-time telemetry data from helicopters during live events?”
Answer: Dataflow and Pub/Sub
Why: Pub/Sub handles message ingestion and distribution, while Dataflow processes streaming data in real time for stats, leaderboards, and replays.

B. Low Latency Streaming

Likely Exam Question: “How can HRL deliver low-latency streams to fans globally?”
Answer: Cloud CDN and Global HTTP(S) Load Balancer
Why: These services ensure fast content delivery and efficient routing of streaming traffic to minimize latency for global audiences.

C. User Analytics and Insights

Likely Exam Question: “Which GCP service enables HRL to analyze fan engagement and event metrics?”
Answer: BigQuery and Looker
Why: BigQuery provides a scalable data warehouse for analyzing massive datasets, and Looker generates user-friendly reports and dashboards.

D. Scalable Infrastructure for Events

Likely Exam Question: “What is the best way to handle traffic spikes during HRL’s live-streamed events?”
Answer: GKE or Compute Engine
Why: GKE is ideal for containerized workloads and scaling services automatically, while Compute Engine offers flexibility for VM-based workloads.

E. AI/ML for Predictions and Highlights

Likely Exam Question: “Which GCP service can HRL use to generate AI-powered predictions and video highlights?”
Answer: Vertex AI and BigQuery ML
Why: Vertex AI trains and deploys custom models for predictions, while BigQuery ML enables lightweight machine learning directly within the data warehouse.

F. Security and Compliance

Likely Exam Question: “How can HRL secure its live-streaming platform and user data?”
Answer:
- IAM: Centralized access control.
- Cloud Armor: Protects against DDoS attacks.
- Security Command Center (SCC): Identifies vulnerabilities and ensures compliance.

G. Continuous Integration and Deployment

Likely Exam Question: “Which GCP tools support automated build and deployment pipelines for HRL’s platform?”
Answer: Cloud Build and Artifact Registry
Why: Cloud Build automates CI/CD pipelines, and Artifact Registry securely stores container images and build artifacts.

H. Cost Optimization

Likely Exam Question: “Which GCP features can HRL use to optimize costs during non-peak times?”
Answer:
- Active Assist: Provides cost-saving recommendations.
- Autoscaler: Dynamically adjusts resources based on demand to avoid over-provisioning.

Google Professional Cloud Architect - Mountkirk Games Case Study

Sat, 21 Dec 2024 00:00:00 +0000

1. Scenario Overview

Mountkirk Games is a gaming company focused on building highly engaging multiplayer mobile games. They aim to migrate their backend infrastructure to Google Cloud to scale with demand, improve deployment agility, and enhance game performance. Their requirements focus on real-time data processing, scalable backends, global availability, and analytics.

Link: Mountkirk Games Case Study

2. Summary of Core Solutions

Requirement	GCP Solution
Real-Time Data Processing	Pub/Sub, Dataflow
Scalable Backend	GKE, App Engine
Global Availability	Global HTTP(S) Load Balancer
Game Analytics and Insights	BigQuery, Dataflow
CI/CD for Game Updates	Cloud Build, Artifact Registry
Security	IAM, Cloud Armor, SCC

3. Question Breakdown by Subject

A. Real-Time Data Processing

Likely Exam Question: “Which GCP services can Mountkirk Games use to handle in-game event streams in real time?”
Answer: Pub/Sub and Dataflow
Why: Pub/Sub ingests and delivers events reliably, while Dataflow processes and transforms streaming data in real time.

B. Scalable Backend

Likely Exam Question: “What is the best way to handle a rapidly growing player base for Mountkirk’s games?”
Answer: GKE or App Engine
Why: GKE supports containerized microservices, while App Engine provides a fully managed PaaS solution for scaling backend services automatically.

C. Global Availability

Likely Exam Question: “How can Mountkirk Games ensure low-latency access for players worldwide?”
Answer: Global HTTP(S) Load Balancer
Why: It routes player traffic to the nearest healthy backend, minimizing latency and ensuring high availability.

D. Game Analytics and Insights

Likely Exam Question: “Which GCP services enable Mountkirk Games to analyze player data and generate insights?”
Answer: BigQuery and Dataflow
Why: BigQuery handles large-scale analytics, while Dataflow processes real-time player telemetry data.

E. CI/CD for Game Updates

Likely Exam Question: “Which GCP tools can automate Mountkirk’s game deployment pipeline?”
Answer: Cloud Build and Artifact Registry
Why: Cloud Build automates the CI/CD process, while Artifact Registry stores container images and build artifacts securely.

F. Security

Likely Exam Question: “How can Mountkirk Games secure their platform against attacks and unauthorized access?”
Answer:
- IAM: Centralized access control.
- Cloud Armor: Protects against DDoS attacks.
- Security Command Center (SCC): Identifies vulnerabilities and ensures compliance.

Google Professional Cloud Architect - TerramEarth Case Study

Fri, 20 Dec 2024 00:00:00 +0000

1. Scenario Overview

TerramEarth is a global manufacturer of heavy machinery for mining and agricultural industries. They aim to improve fleet management and equipment maintenance by migrating to Google Cloud. Their requirements include real-time telemetry processing, predictive analytics, and high availability to support global operations.

Link: TerramEarth Case Study

2. Summary of Core Solutions

Requirement	GCP Solution
Real-Time Telemetry Ingestion	Pub/Sub, Dataflow
Predictive Maintenance	BigQuery, BigQuery ML
High Availability for APIs	Global HTTP(S) Load Balancer
Scalable Processing of Batch Data	Dataflow, Cloud Storage
User Analytics and Reporting	Looker, BigQuery
Secure Fleet Data	IAM, Cloud Armor, SCC
Cost Optimization	Active Assist, Autoscaler

3. Question Breakdown by Subject

A. Real-Time Telemetry Ingestion

Likely Exam Question: “Which GCP services should TerramEarth use to ingest and process real-time equipment telemetry data?”
Answer: Pub/Sub and Dataflow
Why: Pub/Sub handles reliable ingestion of telemetry events, while Dataflow processes and transforms the data for downstream use in real time.

B. Predictive Maintenance

Likely Exam Question: “How can TerramEarth predict when machinery will require maintenance?”
Answer: BigQuery and BigQuery ML
Why: BigQuery stores historical and real-time data, while BigQuery ML trains and runs predictive maintenance models directly on the data.

C. High Availability for APIs

Likely Exam Question: “What GCP service ensures global availability of TerramEarth’s APIs for fleet management?”
Answer: Global HTTP(S) Load Balancer
Why: It distributes traffic globally, ensuring low latency and high availability for APIs.

D. Scalable Processing of Batch Data

Likely Exam Question: “Which services can process large-scale historical telemetry data for analysis?”
Answer: Dataflow and Cloud Storage
Why: Cloud Storage stores batch data, and Dataflow processes it efficiently for analysis.

E. User Analytics and Reporting

Likely Exam Question: “Which GCP services provide actionable insights and reporting for TerramEarth’s operations?”
Answer: BigQuery and Looker
Why: BigQuery performs data analysis at scale, and Looker provides user-friendly dashboards and reports.

F. Security

Likely Exam Question: “How can TerramEarth secure sensitive telemetry data and APIs?”
Answer:
- IAM: Manages access control.
- Cloud Armor: Protects APIs against DDoS attacks.
- Security Command Center (SCC): Identifies vulnerabilities and ensures compliance.

G. Cost Optimization

Likely Exam Question: “How can TerramEarth optimize costs while scaling resources?”
Answer:
- Active Assist: Recommends cost-saving measures.
- Autoscaler: Adjusts resources dynamically based on demand.

Preparation for Google Cloud: Professional Cloud Architect (PCA) Exam

Wed, 11 Dec 2024 00:00:00 +0000

If you’re considering a career in cloud computing, particularly with Google Cloud Platform (GCP), pursuing the Google Cloud Professional Cloud Architect (PCA) certification is a demanding yet rewarding path. By adopting the right learning approach and training plan, you can master key elements of GCP, including enterprise cloud strategies, solution design, and architectural best practices to achieve impactful business results. In this article, I’ll share my journey, insights, and practical tips to help you succeed. All the links mentioned are referred at the end of the post.

Required Experience and Skills

While there are no formal prerequisites for the Google Professional Cloud Architect exam, Google recommends significant experience. Specifically, candidates should possess:

Extensive Cloud and Networking Experience: At least three years of industry experience designing and developing networks, coupled with at least one year specifically designing and managing solutions using Google Cloud Platform (GCP).
Cloud Architecture Design Proficiency: Experience designing and planning cloud architectures from the ground up, either through real-world projects or extensive practice. This includes understanding the technical implementation of managing and provisioning cloud infrastructure.
Security and Governance Expertise: While not requiring a dedicated cybersecurity certification, candidates must demonstrate strong familiarity with designing secure and governance-compliant architectures.
Solution Analysis and Optimization: The ability to analyze and optimize technical and business solutions, considering both technical and business needs.
Real-World Implementation and Reliability: Successful candidates will have managed the implementation of cloud architectures in a real-world setting and demonstrate a strong understanding of how to ensure solution and operations reliability.

Is the Certification Worth It?

For most administrators and network architects, the Google Professional Cloud Architect certification offers significant value. Its benefits extend beyond career advancement:

Advanced Skill Validation: The certification validates critical skills such as creative problem-solving, critical thinking regarding network operations, and the ability to quickly adapt and find solutions.
Career Advancement: It’s particularly valuable for IT professionals aiming for senior roles, demonstrating a mastery of GCP. Even early-career professionals can benefit, using it as a roadmap for skill development.

My Starting Point

Experience: 20 years of commercial IT experience — network, sysadmin, ops, architecture, project management; 2 years with AWS and Azure as an architect; 2 months of GCP experience.

Certifications: AWS Certified Solutions Architect Professional (1 year ago), Microsoft Certified Azure Solutions Architect Expert (2 years ago), Certified Kubernetes Administator (3 years ago).

Learning Process

TL;DR

Used Skills Boost Cloud Architect Path

Prioritized Labs and Quizzes

Familiarized with the 4 case studies

Complemented with Ranga Karanam’s Udemy Course

Familiarized with Google Service Comparison Chart

Reviewed with Sebastian Hooker’s Exam Guide

Did many practice exams

I studied for about three months, although not every day, with some gaps that lasted more than a week. I started with Google Cloud Skills Boost, since it was a prerequisite for an exam voucher, provided due to a partnership between my company and Google. I did the Cloud Architect Learning Path, comprised of 18 courses. My study style is very practical, so I skipped most of the videos from the courses and focused on the labs and quizzes. I also have previous experience with AWS and Azure, so many of the concepts are the same between those cloud providers. What I noticed was that, between the 18 required courses for completing the learning path, there is a lot of overlapping. Many of the labs were almost the same, or even too basic. The evaluation system was also a hassle, I had problems on some of them, wasting time trying to figure out what was different to make it recognize the task as completed instead of effectively learning anything. From what I read, that is a common issue that many people encounter with the platform.

One crucial takeaway from the exam guide is the importance of the case studies. Before the test, I thoroughly reviewed all four case studies to save time during the exam itself. With only 2 hours to complete the test and potentially lengthy questions, it’s essential to be well-prepared. I strongly recommend leveraging the information in the case studies to optimize your efficiency and make the most of your time during the exam.

After that, I went on to Ranga Karanam’s Udemy Course. It had a good rating on the platform, and I personally like Udemy as it is easy to use. Since this was already my second take on all of the content, I listened to the material while doing other stuff, without focusing completely on it. To help me trace a parallel between GCP’s services and the equivalent ones on AWS and Azure, I spent some time studying the comparison chart, provided by Google itself.

Next, I started to review the content and found an excellent summary from Sebastian Hooker, which I read completely and looked deeply on subjects I felt I didn’t know very well. I also went through some excellent prep notes I found online, provided freely by Ammett Williams.

Then it was time to do some practice tests. I started with Ditectrev’s ACE Practice Test, since PCA and ACE have a great deal of overlap and it was free. Then i went to ExamPrepper, which also has some free content, and on the last stretch I used Skillcertpro. I felt very confident after doing so many practice questions, but that strategy always worked for me on my previous certifications. PCA will be my 20th so far. The exam is scheduled for next month, but I wanted to write this guide before, since it also helped me organize everything I needed to cover for the exam. The only paid resources were the Udemy course and Skillcertpro.

Minimum Services to Know

Infra & Ops: Cloud Operations, Google Kubernetes Engine, Cloud Logging, Persistent Disks, Managed Instance Group, Cloud KMS, Organization Policies, Cloud Storage, Google Compute Engine, Cloud IAM
Network: VPC, Shared VPC, VPC Peering, VPC Flow Logging, Serverless VPC Access, Cloud NAT, Cloud VPN, Cloud Interconnect
Apps/Dev: App Engine, Cloud Build, Cloud Scheduler, Cloud Functions, Cloud Run, Firebase, Cloud Pub/Sub
Data: BigQuery, Cloud Bigtable, Cloud Dataproc, Cloud Firestore, Cloud SQL, Cloud Spanner
Processes: Organization Best practices, GKE Best Practices, IAM Best practices, Troubleshooting GKE, Cloud Storage Best practices

Case Studies

The exam will probably have two case studies that might take a lot of time and comprise most of the questions, so it is crucial to study them before the exam, and the official guide provides the description of each scenario:

Opinion About the Exam

The exam is challenging, requiring both a deep understanding of low-level details—such as command-line parameters for tools like gcloud and gsutil—and a solid grasp of high-level concepts across many GCP products.

Having previously passed the Kubernetes CKA exam, which is an “open book” format, I can draw some comparisons. For the CKA, you’re allowed to refer to documentation on kubernetes.io, which makes the exam focused more on practical problem-solving than rote memorization. Although the CKA was difficult, I was well-prepared and finished it significantly faster than the allotted time. There’s little opportunity to rely heavily on documentation during the test, so it’s crucial to have a strong foundational knowledge. The documentation is best used for quick lookups or copying manifest templates to save time. Overall, the CKA was an enjoyable experience because it emphasized real-world tasks. Working directly in the command line to solve problems provided a sense of accomplishment, and it avoided requiring you to memorize information that’s readily available online.

In contrast, the GCP PCA exam is very different. That is also true for all other AWS and Azure exams I took. It requires memorization of details you’d typically copy/paste in your daily work rather than hands-on problem-solving. Without practical, scenario-based tasks, it’s harder to gauge whether you’re fully equipped to handle real challenges on GCP. This difference makes you feel less confident about how well the exam aligns with practical skills needed for day-to-day work in the cloud.

Summary

In short, my journey to prepare for the Google Cloud Professional Cloud Architect certification involved a focused, three-month study plan leveraging a blend of free and paid resources. I prioritized hands-on labs and quizzes from Google Cloud Skills Boost, supplemented by Ranga Karanam’s Udemy course for broader conceptual understanding and Sebastian Hooker’s concise exam guide for focused review. Mastering the four official case studies proved invaluable, as did extensive practice exams. While the exam itself demands significant memorization, my prior cloud experience and targeted approach helped me feel confident going in. I hope this detailed account of my preparation, including the resources and strategies I used, provides a helpful roadmap for your own PCA journey. Remember to prioritize hands-on experience and a deep understanding of the case studies – they are key to success.

Resources

Google Cloud Skills Boost - Cloud Architect Learning Path: https://www.cloudskillsboost.google/paths/12/

Exam Guide: https://cloud.google.com/learn/certification/guides/professional-cloud-architect/

Ranga Karanam’s Udemy Google PCA Course: https://www.udemy.com/course/google-cloud-professional-cloud-architect-certification

Service Comparison Chart: https://cloud.google.com/docs/get-started/aws-azure-gcp-service-comparison

Sebastian Hooker’s Full Exam Guide: https://www.sebhook.com/2024/08/28/google-cloud-professional-cloud-architect-pca-full-exam-guide/

Ditectrev’s ACE Practice Test: https://github.com/Ditectrev/Google-Cloud-Platform-GCP-Associate-Cloud-Engineer-Practice-Tests-Exams-Questions-Answers

ExamPrepper: https://www.examprepper.co/exam/3/1

Skillcertpro practice test: https://skillcertpro.com/product/google-cloud-certified-professional-cloud-architect-practice-exam-test/

Prep Notes by Ammett Williams: https://drive.google.com/file/d/1_UfKnzxodTTk5CuwT0ScSmXegtmgpV5v/view

Installing a Ceph Cluster

Sat, 07 Dec 2024 00:00:00 +0000

In this example I used the Red Hat Ceph, but the procedure is the same for the community version. The official guide for RHCS 7 can be found here: RedHat guide

Preparation:

Configure a Workbench
Populate the hosts inventory
Run the preflight playbook
Create the initial-config.yaml
Run the Bootstrap Script

SSH Key Distribution:

Create an admin user on all servers, with a password and sudo privileges.
Generate an SSH key pair for the admin user on the workbench.
Use ssh-copy-id to copy the public key to all servers for the admin user.
Copy the public and private keys to /home/admin/.ssh on MON 1.

Bootstrap: Execute from MON 1, using sudo as the admin user.

initial-config.yaml:

This initial-config.yaml file defines the configuration for a Ceph cluster, specifying the roles and placement of various Ceph services across multiple hosts. The configuration uses a declarative approach, listing each host and assigning it one or more roles.

The file is structured as a YAML document containing several sections, each representing a different Ceph service. Each section defines the service_type (e.g., host, mon, mgr, osd, rgw, grafana), and then specifies the placement and configuration details for that service.

Host definitions: Multiple sections define individual hosts (ceph-workbench, ceph-1 through ceph-20). Each host has an addr (IP address) and hostname, and is assigned one or more labels (ceph, mons, mgrs, osds, rgws, grafana). These labels determine which Ceph services will be deployed on that host. For example, hosts with the mons label will run the Ceph monitors.

Service placement: Sections for mon, mgr, osd, rgw, and grafana define the placement of these services using the placement directive, which references the labels defined in the host sections. This ensures that the monitors are deployed on hosts labeled mons, OSDs (Object Storage Daemons) on hosts labeled osds, etc.

OSD Configuration: The osd service section includes data_devices: all: true, indicating that all available data devices on the OSD hosts should be used for storage.

RGW Configuration: The rgw (RADOS Gateway) service is configured with a service_id of ceph-cluster and placed on hosts labeled rgws.

Grafana Configuration: The grafana service is defined with a specific configuration, including enabling anonymous access (anonymous_access: true), specifying the protocol (protocol: https), and setting an initial admin password (initial_admin_password: ‘super_difficult_pwd’). This indicates that Grafana will be deployed for monitoring the Ceph cluster.

In summary, this configuration file describes a distributed Ceph cluster with a designated workbench host and multiple hosts serving as monitors, managers, OSDs, RADOS Gateways, and a Grafana instance for monitoring and management. The use of labels and placement directives simplifies the deployment process and ensures that the services are distributed according to the defined roles. The password for grafana should be changed after deployment.

Source codejson


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181


service_type: host
addr: 10.0.0.19
hostname: ceph-workbench
labels:
 - _admin
---
service_type: host
addr: 10.0.0.20
hostname: ceph-1
labels:
 - ceph
 - mons
 - mgrs
 - osds
---
service_type: host
addr: 10.0.0.21
hostname: ceph-2
labels:
 - ceph
 - mons
 - mgrs
 - osds
---
service_type: host
addr: 10.0.0.22
hostname: ceph-3
labels:
 - ceph
 - mons
 - mgrs
 - osds
---
service_type: host
addr: 10.0.0.23
hostname: ceph-4
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.24
hostname: ceph-5
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.25
hostname: ceph-6
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.26
hostname: ceph-7
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.27
hostname: ceph-8
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.28
hostname: ceph-9
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.29
hostname: ceph-10
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.30
hostname: ceph-11
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.31
hostname: ceph-12
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.32
hostname: ceph-13
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.33
hostname: ceph-14
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.34
hostname: ceph-15
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.35
hostname: ceph-16
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.36
hostname: ceph-17
labels:
 - ceph
 - osds
---
service_type: host
addr: 10.0.0.37
hostname: ceph-18
labels:
 - ceph
 - osds
 - grafana
---
service_type: host
addr: 10.0.0.38
hostname: ceph-19
labels:
 - ceph
 - osds
 - rgws
---
service_type: host
addr: 10.0.0.39
hostname: ceph-20
labels:
 - ceph
 - osds
 - rgws
---
service_type: mon
placement:
 label: mons
---
service_type: mgr
placement:
 label: mgrs
---
service_type: osd
placement:
 label: osds
data_devices:
 all: true
---
service_type: rgw
service_id: ceph-cluster
placement:
 label: rgws
---
service_type: grafana
placement:
 label: grafana
spec:
 anonymous_access: true
 protocol: https
 initial_admin_password: 'super_difficult_pwd'

The initial-config.yaml file defines a Ceph cluster topology consisting of 20 hosts in total, distributed as follows:

1 Workbench Host: ceph-workbench acts as the administrative workstation for managing the cluster. It does not run any Ceph daemons.

3 Monitor (MON) Nodes: Three hosts (ceph-1, ceph-2, ceph-3) are designated as monitor nodes. These are crucial for cluster management and coordination.

1 Manager (MGR) Nodes: Three hosts (ceph-1, ceph-2, ceph-3) are designated as manager nodes. These nodes handle higher-level cluster management tasks.

16 Object Storage Daemon (OSD) Nodes: Sixteen hosts (ceph-1, ceph-2, ceph-3, ceph-4 through ceph-16) are configured as OSD nodes. These nodes store the actual data within the Ceph cluster. All available data devices on these nodes will be used for storage.

2 RADOS Gateway (RGW) Nodes: Two hosts (ceph-19, ceph-20) are designated as RGW nodes. These provide object storage access via the S3 protocol.

1 Grafana Node: One host (ceph-18) is dedicated to running Grafana, a monitoring and visualization tool, providing a user interface for cluster monitoring. It’s configured for anonymous access, which should be changed for production use.

Therefore, the topology is a distributed cluster with replication and high availability provided by the multiple monitor nodes, and scalability and storage capacity provided by the numerous OSD nodes. The managers provide high-level management, and the RGW provides external access to the object storage. Grafana enables monitoring of the entire system.

initial-ceph.conf:

The initial-ceph.conf gfile was used to set the images to a local registry, instead of the public ones. This is useful for airgapped installations and also for extra security.

Source codePlain text


1
2
3
4
5


[mgr]
mgr/cephadm/container_image_prometheus = internal-registry.org.local/cdnas/ose-prometheus:v4.15
mgr/cephadm/container_image_node_exporter = internal-registry.org.local/cdnas/ose-prometheus-node-exporter:v4.15
mgr/cephadm/container_image_grafana = internal-registry.org.local/cdnas/grafana-rhel9:latest --remove-signatures
mgr/cephadm/container_image_alertmanager = internal-registry.org.local/cdnas/ose-prometheus-alertmanager:v4.15 --remove-signatures

Bootstrap:

According to the documentation, you must run the bootstrap from MON 1 with user admin:

Source codebash

1

sudo cephadm --image internal-registry.org.local/cdnas/rhceph-7-rhel9:7-424 bootstrap --cluster-network 10.0.0.0/24 --mon-ip 10.0.0.20 --ssh-user admin --apply-spec initial-config.yaml --config initial-ceph.conf

At this point, if everything went well, the cluster will be already deployed. Now you can set up the specifics on your environment, such as CRUSH rules, radosgw pool, adjustments on the osd tree and so on:

OSD Tree:

Check the OSD Tree:

Source codebash

1

ceph osd tree

Create the new buckets:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


ceph osd crush add-bucket custom root
ceph osd crush add-bucket curitiba datacenter
ceph osd crush move curitiba root=custom
ceph osd crush add-bucket room room
ceph osd crush move room datacenter=curitiba
ceph osd crush add-bucket AT45 rack
ceph osd crush add-bucket AT46 rack
ceph osd crush add-bucket AT47 rack
ceph osd crush move AT45 room=room-3b
ceph osd crush move AT46 room=room-3b
ceph osd crush move AT47 room=room-3b

Move the nodes:

Source codebash

1
2

ceph osd crush move ceph-1 rack=AT45
ceph osd crush move ceph-2 rack=AT45

Since it is a new cluster, no data movement is expected. However, in a production environment, moving nodes will definitely cause data to be moved, so you should proceed with caution.

CRUSH Rule:

In this example I created Erasure Code profiles and rules, and also a new replicated rule in order to avoid using the default one. This is for extra safety, since newly added OSDs get added to the default root on the OSD tree. By creating a new root and moving all OSDs under it, you prevent data movement if someone inadvertently adds an OSD to the cluster.

Erasure Code profile:

Source codePlain text

1

ceph osd erasure-code-profile set 8_4 k=8 m=4 crush-root=custom

Erasure Code CRUSH Rule:

Source codePlain text

1

ceph osd crush rule create-erasure 8_4_erasure_rule 8_4

Replicated Rule for root custom:

Source codePlain text

1

ceph osd crush rule create-replicated replicated_rule_custom custom host

After moving all the OSDs from root default to root custom, the crush rules need to be adjusted:

New rules:

Source codejson


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126


[
 {
 "rule_id": 0,
 "rule_name": "replicated_rule",
 "type": 1,
 "steps": [
 {
 "op": "take",
 "item": -1,
 "item_name": "default"
 },
 {
 "op": "chooseleaf_firstn",
 "num": 0,
 "type": "host"
 },
 {
 "op": "emit"
 }
 ]
 },
 {
 "rule_id": 1,
 "rule_name": "2_2_erasure_rule",
 "type": 3,
 "steps": [
 {
 "op": "set_chooseleaf_tries",
 "num": 5
 },
 {
 "op": "set_choose_tries",
 "num": 100
 },
 {
 "op": "take",
 "item": -43,
 "item_name": "custom"
 },
 {
 "op": "chooseleaf_indep",
 "num": 0,
 "type": "host"
 },
 {
 "op": "emit"
 }
 ]
 },
 {
 "rule_id": 2,
 "rule_name": "4_2_erasure_rule",
 "type": 3,
 "steps": [
 {
 "op": "set_chooseleaf_tries",
 "num": 5
 },
 {
 "op": "set_choose_tries",
 "num": 100
 },
 {
 "op": "take",
 "item": -43,
 "item_name": "custom"
 },
 {
 "op": "chooseleaf_indep",
 "num": 0,
 "type": "host"
 },
 {
 "op": "emit"
 }
 ]
 },
 {
 "rule_id": 3,
 "rule_name": "8_4_erasure_rule",
 "type": 3,
 "steps": [
 {
 "op": "set_chooseleaf_tries",
 "num": 5
 },
 {
 "op": "set_choose_tries",
 "num": 100
 },
 {
 "op": "take",
 "item": -43,
 "item_name": "custom"
 },
 {
 "op": "chooseleaf_indep",
 "num": 0,
 "type": "host"
 },
 {
 "op": "emit"
 }
 ]
 },
 {
 "rule_id": 4,
 "rule_name": "replicated_rule_custom",
 "type": 1,
 "steps": [
 {
 "op": "take",
 "item": -43,
 "item_name": "serpro"
 },
 {
 "op": "chooseleaf_firstn",
 "num": 0,
 "type": "host"
 },
 {
 "op": "emit"
 }
 ]
 }
]

Modify the pool rules:

Source codebash


1
2
3
4
5


ceph osd pool set .mgr crush_rule replicated_rule_custom
ceph osd pool set .rgw.root crush_rule replicated_rule_custom
ceph osd pool set default.rgw.log crush_rule replicated_rule_custom
ceph osd pool set default.rgw.control crush_rule replicated_rule_custom
ceph osd pool set default.rgw.meta crush_rule replicated_rule_custom

Creation of a Pool for radosgw:

Source codePlain text


1
2
3
4
5
6


ceph osd pool create default.rgw.buckets.data 128 128 erasure 8_4 8_4_erasure_rule
ceph osd pool create default.rgw.buckets.index 128 128 replicated replicated_rule_custom
ceph osd pool create default.rgw.buckets.non-ec 128 128 replicated replicated_rule_custom
ceph osd pool application enable default.rgw.buckets.data rgw
ceph osd pool application enable default.rgw.buckets.index rgw
ceph osd pool application enable default.rgw.buckets.non-ec rgw

Test User for radosgw:

Source codePlain text

1

radosgw-admin user create --uid="poseidon" --display-name="poseidon"

Usually these pools get automatically created the first time you access the radosgw gateway and create a bucket. But since it is desirable to choose specific CRUSH rules, I prefer to create them previously according to my needs.

Deploying a Local EKS Cluster on AWS Outposts

Fri, 06 Dec 2024 00:00:00 +0000

The Terraform code provisions an EKS cluster on an AWS Outpost, with the control plane running locally on the Outpost. This differs significantly from deploying an EKS cluster in a standard AWS region, primarily in how worker nodes are managed.

The Amazon EKS update history demonstrates the platform’s continuous evolution, focusing on enhanced security, scalability, and integration with other AWS services. Since its initial release in June 2018, there’s been a significant increase in available features and functionalities, including support for newer Kubernetes versions, expansion to new AWS regions, and the introduction of new deployment options like Fargate and local clusters on AWS Outposts, which is the focus of this work. The introduction of features like managed node groups significantly simplified infrastructure management, automating tasks such as provisioning and lifecycle management of nodes. The availability of EKS-optimized AMIs, including options with GPU support and the Bottlerocket operating system, offers greater flexibility and performance for diverse workloads. Furthermore, integration with services like Amazon EBS, EFS, and FSx for Lustre, via CSI drivers, expanded storage options for the clusters. However, for a local Amazon EKS cluster on AWS Outposts, these resources are significantly more limited; managed node groups are unavailable.

We also observed that the Control Plane AMIs are automatically provisioned with Bottlerocket, where we only specify the major Kubernetes release, and the latest minor version is selected. The challenge lies in provisioning worker nodes, which requires selecting an EKS-optimized AMI from the available options in the AWS catalog; only versions with Amazon Linux 2023 are found. While not inherently problematic, ensuring both control plane and worker nodes use the same Kubernetes version proved difficult, requiring testing three or four AMIs to find the closest match.

Finally, we lacked the usual EKS add-ons, which in this scenario require manual provisioning, such as CSI, CNI, Cluster Load Balancer Controller, Metrics Server, or Kube-prometheus-stack, to name a few but not necessarily all.

Important Considerations for Provisioning EKS on Outposts: AMI: Ensure you use an Amazon Machine Image (AMI) compatible with your Outpost and the chosen Kubernetes version. Review the use of data.aws_ami; manually specifying the AMI is strongly recommended to guarantee compatibility. Not all region-available versions are available on the Outpost; there’s usually a two-version lag.

Security: Carefully review security rules in the Security Groups. Limit access only to necessary ports and IPs.

Scalability: Adjust min_size, desired_capacity, and max_size values in the ASG to meet scalability needs. Arbitrary values were configured only to enable code construction and simplified usage tests for the proof of concept.

High Availability: For high availability, consider creating the cluster across multiple subnets associated with the Outpost. Due to limitations in the proof-of-concept environment, only available subnets were used.

Monitoring: Implement a monitoring system for the EKS cluster and worker nodes. CloudWatch is the most straightforward option, but due to associated costs and integration with pre-existing tools, other options might be more suitable, such as a Grafana stack (Loki, Mimir, Grafana, Tempo, Prometheus).

The following is a commented version of the code, highlighting key configurations and differences compared to an EKS cluster in a region:

Part 1: EKS Control Plane

This section describes the infrastructure required for the EKS control plane on the Outpost. Note that the control plane configuration is similar to a region cluster, but with the crucial addition of the outpost_config.

providers.tf:

Source codeyaml


1
2
3


provider "aws" {
 region = local.region
}

terraform.tf:

Source codeyaml


1
2
3
4
5
6
7


terraform {
 backend "s3" {
 bucket = "outposts-bucket"
 key = "eks-local-terraform-state"
 region = "us-east-1"
 }
}

variables.tf:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


variable "eks_cluster_name" {
 type = string
 default = "eks_local"
}
variable "eks_cluster_version" {
 type = string
 default = "1.30" # available versions differ between regional and local cluster
}
variable "eks_vpc" {
 type = string
 default = "vpc-id0000" # replace with your vpc id 
}
variable "eks_subnet" {
 type = string
 default = "subnet-id0000" # replace with your subnet id 
}
variable "bastion_subnet" {
 type = string
 default = "subnet-id0000" # replace with your subnet id 
}
variable "control_plane_instance_type" {
 type = string
 default = "m5.large" # Choose an instance type that is available in your outpost
}
variable "tags" {
 type = map(any)
 description = "(Optional) Tags to apply to all tag-able resources."
 default = {
 terraform = "true"
 }
}

outputs.tf:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


output "eks_cluster_endpoint" {
 description = "EKS Cluster Endpoint"
 value = aws_eks_cluster.eks_cluster.endpoint
}

output "eks_cluster_ca" {
 value = base64decode(aws_eks_cluster.eks_cluster.certificate_authority[0].data)
}

output "eks_cluster_arn" {
 description = "EKS Cluster ARN"
 value = aws_eks_cluster.eks_cluster.arn
}

output "kubeconfig" {
 value = <<EOF
apiVersion: v1
kind: Config
clusters:
- cluster:
 server: ${aws_eks_cluster.eks_cluster.endpoint}
 certificate-authority-data: ${aws_eks_cluster.eks_cluster.certificate_authority.0.data}
 name: ${aws_eks_cluster.eks_cluster.name}
contexts:
- context:
 cluster: ${aws_eks_cluster.eks_cluster.name}
 user: ${aws_eks_cluster.eks_cluster.name}
 name: ${aws_eks_cluster.eks_cluster.name}
current-context: ${aws_eks_cluster.eks_cluster.name}
users:
- name: ${aws_eks_cluster.eks_cluster.name}
 user:
 exec:
 apiVersion: "client.authentication.k8s.io/v1beta1"
 command: "aws"
 args:
 - "eks"
 - "get-token"
 - "--cluster-name"
 - "${aws_eks_cluster.eks_cluster.name}"
 - "--region"
 - "${local.region}"
EOF
 description = "Kubeconfig for the EKS Cluster"
 sensitive = true
}

main.tf:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152


locals {
 region = "us-east-1"
}

data "aws_outposts_outpost" "my_outpost" {
 id = "op-06d4362ae79bc4247"  # Outpost id
}

data "aws_vpc" "vpc_eks" {
 id = var.eks_vpc
}
data "aws_subnet" "vpc_eks" {
 id = var.eks_subnet
}

data "aws_subnet" "vpc_bastion" {
 id = var.bastion_subnet
}

# Role for the Control Plane
resource "aws_iam_role" "eks_cluster_role" {
 name = "eks-cluster-role"

 assume_role_policy = jsonencode({
 Version = "2012-10-17"
 Statement = [
 {
 Effect = "Allow"
 Principal = {
 Service = "eks.amazonaws.com"
 }
 Action = "sts:AssumeRole"
 },
 {
 Effect = "Allow"
 Principal = {
 Service = "ec2.amazonaws.com"
 }
 Action = "sts:AssumeRole"
 }
 ]
 })
}

# Control Plane Policies
resource "aws_iam_role_policy_attachment" "eks_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
 role = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_LocalOutpost_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonEKSLocalOutpostClusterPolicy"
 role = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
 role = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_VPCResourceController_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
 role = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_ssm_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
 role = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cloudwatch_logs" {
 policy_arn = "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
 role = aws_iam_role.eks_cluster_role.name
}

# CloudWatch Log Group
resource "aws_cloudwatch_log_group" "eks_log_group" {
 name = "/aws/eks/${var.eks_cluster_name}/cluster-logs"
 retention_in_days = 7
}

# Security group for EKS control plane in Outposts
resource "aws_security_group" "control_plane_sg" {
 name = "${var.eks_cluster_name}-control-plane-sg"
 vpc_id = data.aws_vpc.vpc_eks.id
 description = "Security group for EKS control plane in Outposts"

 ingress {
 description = "Allow inbound HTTPS for EKS API server"
 from_port = 443
 to_port = 443
 protocol = "tcp"
 cidr_blocks = [data.aws_subnet.subnet_eks.cidr_block, data.aws_subnet.subnet_bastion.cidr_block]
 }

 egress {
 description = "Allow all outbound traffic"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# EKS Cluster
resource "aws_eks_cluster" "eks_cluster" {
 name = var.eks_cluster_name
 version = var.eks_cluster_version
 role_arn = aws_iam_role.eks_cluster_role.arn

 enabled_cluster_log_types = ["audit"] # choose accordingly

 vpc_config {
 endpoint_private_access = true
 endpoint_public_access = false
 security_group_ids = [aws_security_group.control_plane_sg.id]
 subnet_ids = [data.aws_subnet.subnet_eks.id]
 }

 outpost_config {
 outpost_arns = [data.aws_outposts_outpost.my_outpost.arn]
 control_plane_instance_type = var.control_plane_instance_type
 }

 access_config {
 authentication_mode = "CONFIG_MAP"  # Local Outpost clusters only support config_map
 bootstrap_cluster_creator_admin_permissions = true
 }

 upgrade_policy {
 support_type = "STANDARD" # you can set it to anything but it will end up as EXTENDED.
 }

 tags = var.tags

 depends_on = [
 aws_iam_role_policy_attachment.eks_policy,
 aws_iam_role_policy_attachment.eks_cni_policy,
 aws_iam_role_policy_attachment.eks_VPCResourceController_policy,
 aws_iam_role_policy_attachment.eks_LocalOutpost_policy
 ]

}

# writes a kubeconfig file locally
resource "null_resource" "write_kubeconfig" {
 provisioner "local-exec" {
 command = <<EOT
 aws eks update-kubeconfig --name ${aws_eks_cluster.eks_cluster.name} --region ${local.region}
 EOT
 }
}

After provisioning the Control Plane, it’s necessary to configure the aws-auth ConfigMap with the role that will be associated with the worker nodes. In this example it was also associated the instance profile associated with a bastion host, used to access the EKS cluster via kubectl:

Config Map aws-auth:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apiVersion: v1
kind: ConfigMap
metadata:
 name: aws-auth
 namespace: kube-system
data:
 mapRoles: |
 - rolearn: arn:aws:iam::597088017848:role/eks-nodegroup-role
 username: system:node:{{EC2PrivateDNSName}}
 groups:
 - system:bootstrappers
 - system:nodes
 - rolearn: arn:aws:iam::597088017848:role/myrole-ec2-ssm-enhanced
 username: bastion
 groups:
 - system:masters

Part 2: EKS Nodes

When provisioning EKS on Outpost as a Local Cluster, it is not possible to use Node Groups. The solution is to create a Launch Template, associate it with an Autoscaling Group and use a user-data script to configure the nodes in the EKS cluster.

providers.tf:

Source codeyaml


1
2
3


provider "aws" {
 region = local.region
}

terraform.tf:

Source codeyaml


1
2
3
4
5
6
7


terraform {
 backend "s3" {
 bucket = "outposts-bucket"
 key = "eks-local-terraform-state"
 region = "us-east-1"
 }
}

variables.tf:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


variable "eks_cluster_name" {
 type = string
 default = "eks_local"
}
variable "eks_cluster_version" {
 type = string
 default = "1.30" # available versions differ between regional and local cluster
}
variable "eks_vpc" {
 type = string
 default = "vpc-id0000" # replace with your vpc id 
}
variable "eks_subnet" {
 type = string
 default = "subnet-id0000" # replace with your subnet id 
}
variable "bastion_subnet" {
 type = string
 default = "subnet-id0000" # replace with your subnet id 
}
variable "control_plane_instance_type" {
 type = string
 default = "m5.large"
}
variable "control_plane_instance_type" {
 type = string
 default = "m5.large" # Choose an instance type that is available in your outpost
}
variable "node_group_min_size" {
 type = string
 default = "1"
}
variable "node_group_desired_size" {
 type = string
 default = "2"
}
variable "node_group_max_size" {
 type = string
 default = "3"
}
variable "node_group_name" {
 type = string
 description = "(Optional) The name to be used for the self-managed node group. By default, the module will generate a unique name."
 default = "node_group"
}
variable "tags" {
 type = map(any)
 description = "(Optional) Tags to apply to all tag-able resources."
 default = {
 terraform = "true"
 }
}

main.tf:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191


locals {
 region = "us-east-1"
}

data "aws_outposts_outpost" "my_outpost" {
 id = "op-id0000" # your Outpost ID
}

data "aws_vpc" "vpc_eks" {
 id = var.eks_vpc
}
data "aws_subnet" "subnet_eks" {
 id = var.eks_subnet
}

data "aws_subnet" "subnet_bastion" {
 id = var.bastion_subnet
}

data "aws_eks_cluster" "eks_cluster" {
 name = var.eks_cluster_name
}

data "aws_security_group" "control_plane_sg" {
 name = "${var.eks_cluster_name}-control-plane-sg"
}

# Instance Type
data "aws_ec2_instance_type" "selected_instance" {
 instance_type = var.node_group_instance_type
}

# AMI
data "aws_ami" "eks_ami" {
 owners = ["amazon"]
 most_recent = true
 filter {
 name = "name"
 values = ["amazon-eks-node-${data.aws_eks_cluster.eks_cluster.version}*"]
 }
 filter {
 name = "architecture"
 values = data.aws_ec2_instance_type.selected_instance.supported_architectures
 }
 filter {
 name = "virtualization-type"
 values = data.aws_ec2_instance_type.selected_instance.supported_virtualization_types
 }
 filter {
 name = "root-device-type"
 values = data.aws_ec2_instance_type.selected_instance.supported_root_device_types
 }
}

# User Data for the Worker Nodes - Script to join the nodes to the EKS cluster. `--enable-local-outpost true` is crucial.
data "template_file" "user_data" {
 template = <<-EOT
 #!/bin/bash
 set -o xtrace
 /etc/eks/bootstrap.sh ${var.eks_cluster_name} \
 --enable-local-outpost true \
 --cluster-id '${data.aws_eks_cluster.eks_cluster.id}'
 EOT
}

# Worker Nodes Role
resource "aws_iam_role" "eks_nodegroup_role" {
 name = "eks-nodegroup-role"

 assume_role_policy = jsonencode({
 Version = "2012-10-17"
 Statement = [
 {
 Effect = "Allow"
 Principal = {
 Service = "eks.amazonaws.com"
 }
 Action = "sts:AssumeRole"
 },
 {
 Effect = "Allow"
 Principal = {
 Service = "ec2.amazonaws.com"
 }
 Action = "sts:AssumeRole"
 }
 ]
 })
}

# Node Policies - These policies ensure that nodes have correct access to the necessary resources.
resource "aws_iam_role_policy_attachment" "nodegroup_cni_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
 role = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_iam_role_policy_attachment" "nodegroup_worker_node_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
 role = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_iam_role_policy_attachment" "nodegroup_ecr_read_only_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
 role = aws_iam_role.eks_nodegroup_role.name
}

resource "aws_iam_role_policy_attachment" "nodegroup_ssm_policy" {
 policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
 role = aws_iam_role.eks_nodegroup_role.name
}

# IAM Instance Profile
resource "aws_iam_instance_profile" "eks_node_instance_profile" {
 name = "${var.eks_cluster_name}-node-instance-profile"
 role = aws_iam_role.eks_nodegroup_role.name
}

# Worker Nodes Security group
resource "aws_security_group" "nodes_sg" {
 name = "${var.eks_cluster_name}-nodes-sg"
 vpc_id = data.aws_vpc.vpc_eks.id
 description = "Security group for EKS nodes in Outposts"

 ingress {
 description = "Allow all node-to-node communication"
 from_port = 0
 to_port = 0
 protocol = "-1"
 self = true
 }

 ingress {
 description = "Allow nodes to communicate with EKS control plane"
 from_port = 443
 to_port = 443
 protocol = "tcp"
 security_groups = [data.aws_security_group.control_plane_sg.id]
 }

 egress {
 description = "Allow all outbound traffic"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# Launch Template for the Worker Nodes
resource "aws_launch_template" "outposts_node_template" {
 name_prefix = "${var.eks_cluster_name}-node-template-"
 # image_id = data.aws_ami.eks_ami.id
 image_id = "ami-023745e977279d33c" # Either use the filters set previously or pick manually an ami from the catalog to ensure version matching with the control plane nodes
 instance_type = var.node_group_instance_type
 user_data = base64encode(data.template_file.user_data.rendered)

 vpc_security_group_ids = [aws_security_group.nodes_sg.id]
 key_name = "my-precious-key" # SSH key

 iam_instance_profile {
 name = aws_iam_instance_profile.eks_node_instance_profile.name
 }
}

# ASG for Worker Nodes
resource "aws_autoscaling_group" "outposts_asg" {
 name = "${var.eks_cluster_name}-asg"
 desired_capacity = 2
 max_size = 4
 min_size = 1
 vpc_zone_identifier = [data.aws_subnet.subnet_eks.id]
 launch_template {
 id = aws_launch_template.outposts_node_template.id
 version = "$Latest"
 }

 health_check_type = "EC2"
 health_check_grace_period = 300
 force_delete = true

 tag {
 key = "Name"
 value = "${var.eks_cluster_name}-${var.node_group_name}"
 propagate_at_launch = true
 }
 tag {
 key = "kubernetes.io/cluster/${var.eks_cluster_name}"
 value = "owned"
 propagate_at_launch = true
 }
}

After a couple of minutes, the worker nodes will get registered in the kubernetes cluster. The master nodes remain as NotReady due to the absence of the CNI.

Deploying Grafana Mimir on Kubernetes

Tue, 19 Nov 2024 00:00:00 +0000

I deployed Mimir alongside Loki and Grafana on a seven-node AlmaLinux Kubernetes cluster, using Rook Ceph as S3-compatible object storage. Mimir provided a central metrics backend for multiple Prometheus servers. Loki handled logs through a separate ingestion and storage path.

Technical revision — October 2026: this revision corrects the Helm chart and argument order, the Ceph Secret name, and bucket commands that omitted the Ceph endpoint. It also separates a historical configuration example from claims that require runtime and failure testing.

Architecture and versions

Prometheus sends metrics to Mimir using remote write. Grafana queries Mimir using PromQL. In the classic architecture, ingesters receive samples and produce blocks that are stored in object storage; store gateways make those blocks available to the query path. This is not a log-storage backend for Loki.

The downloadable values.yaml targets grafana/mimir-distributed chart 5.6.0, Mimir 2.15.0, rendered for Kubernetes 1.30 with Helm 3.17.3. These versions define a reproducible historical reference for this revision; they are not asserted to be the exact versions from the original deployment or a current production baseline.

Current Mimir charts use a newer ingest-storage architecture by default. Follow the current deployment guide for a new installation, and review Kafka and migration requirements before moving from classic architecture.

values.yamlyaml · 71 linesDownload
Expand sourceCollapse source

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71


# Historical classic architecture: chart 5.6.0 (Mimir 2.15.0).
# Rendered locally; this is not a current Mimir 3.x production configuration.
global:
 dnsService: kube-dns
 dnsNamespace: kube-system
 clusterDomain: cluster.local.
 extraEnvFrom:
 - secretRef:
 name: mimir-s3
mimir:
 structuredConfig:
 multitenancy_enabled: true
 common:
 storage:
 backend: s3
 s3:
 endpoint: s3.example.com:443
 region: us-east-1
 access_key_id: ${AWS_ACCESS_KEY_ID}
 secret_access_key: ${AWS_SECRET_ACCESS_KEY}
 insecure: false
 blocks_storage:
 s3:
 bucket_name: mimir-blocks
 alertmanager_storage:
 s3:
 bucket_name: mimir-alertmanager
 ruler_storage:
 s3:
 bucket_name: mimir-ruler
ingester:
 replicas: 3
 zoneAwareReplication:
 enabled: false
 persistentVolume:
 storageClass: rook-ceph-block
 size: 20Gi
distributor:
 replicas: 2
querier:
 replicas: 2
query_frontend:
 replicas: 2
query_scheduler:
 replicas: 2
compactor:
 replicas: 1
 persistentVolume:
 storageClass: rook-ceph-block
 size: 20Gi
store_gateway:
 replicas: 3
 zoneAwareReplication:
 enabled: false
 persistentVolume:
 storageClass: rook-ceph-block
 size: 10Gi
alertmanager:
 replicas: 2
 persistentVolume:
 storageClass: rook-ceph-block
nginx:
 replicas: 2
minio:
 enabled: false
metaMonitoring:
 serviceMonitor:
 enabled: false
 grafanaAgent:
 enabled: false
 installOperator: false

The sample enables tenant isolation, uses separate buckets for blocks, ruler and Alertmanager state, and injects credentials through a Kubernetes Secret. It disables the bundled MinIO and the historical Grafana Agent metamonitoring integration.

Zone-aware replication is disabled explicitly. The three ingesters and store gateways, two query components and routing replicas are illustrative topology choices, not evidence of high availability. Verify anti-affinity, replication, available nodes, PVC placement and disruption behavior. Resource requests, limits, caches and disk sizing still need to match measured series cardinality, ingestion rate, retention and query load.

Ceph identity and buckets

Use a dedicated Ceph user with restricted access to the three Mimir buckets. If Rook manages a CephObjectStoreUser named mimir for object-store, its Secret is:

Source codebash

1

kubectl -n rook-ceph describe secret rook-ceph-object-user-object-store-mimir

The old example incorrectly read the Loki user’s Secret. Keep these identities separate. Through your secret-management workflow, provide a mimir-s3 Secret in namespace mimir with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The chart injects these values and enables environment expansion. The source values file contains variable references rather than real key material.

Create the buckets against the actual Ceph endpoint. This Bash example assumes an AWS CLI profile ceph-mimir with that user’s credentials:

Source codebash


1
2
3
4
5
6
7


export AWS_PROFILE=ceph-mimir
export CEPH_S3_ENDPOINT=https://s3.example.com
# For an internal CA:
# export AWS_CA_BUNDLE=/path/to/internal-ca.pem
for bucket in mimir-blocks mimir-ruler mimir-alertmanager; do
 aws --endpoint-url "$CEPH_S3_ENDPOINT" --region us-east-1 s3 mb "s3://$bucket"
done

Unlike the AWS CLI URL, the Mimir S3 endpoint field in this example uses host:port, without a scheme; insecure: false selects TLS. Configure trusted certificates for the Mimir containers as well as the CLI. Do not bypass certificate validation to work around a missing internal CA.

Render before installing

Download values.yaml, adapt the endpoint, DNS and StorageClass, and render the exact chart:

Source codebash


1
2
3


helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm template mimir grafana/mimir-distributed --version 5.6.0 --namespace mimir --kube-version 1.30.0 -f values.yaml > rendered.yaml

After reviewing the manifests and target environment, the historical installation command is:

Source codebash

1

helm upgrade --install mimir grafana/mimir-distributed --version 5.6.0 --namespace mimir --create-namespace -f values.yaml --wait --timeout 10m

The release is mimir and the chart is grafana/mimir-distributed. Local lint and rendering passed during this revision. No cluster installation, object-store connection or failure test was performed.

Secure the write and query paths

Mimir uses X-Scope-OrgID to identify a tenant. An authenticating reverse proxy must authorize clients and control the tenant header. Enabling multi-tenancy does not by itself authenticate callers. See Mimir authentication and authorization.

Configure Prometheus remote write to the authenticated Mimir endpoint at /api/v1/push; configure the Grafana Prometheus data source to the authenticated query endpoint with the /prometheus prefix. Match credentials and tenant identity at the proxy. The old metamonitoring example pointed to a Prometheus service as if it were Mimir; that configuration has been removed.

Before using the service, verify:

Pod readiness, ring membership, PVCs and placement across the intended failure domains.
A known metric written through Prometheus and queried through Grafana, including the correct tenant.
Bucket access, successful block uploads and queries for data older than the ingesters’ recent window.
Rejection of unauthorized requests and cross-tenant access attempts.
Restarts, node failure, recovery, retention and resource usage under representative load.

The storage cluster’s total disk capacity does not determine Mimir’s replica count. Ingestion rate, active series, retention and query behavior are the inputs to sizing. A successful Helm render is a useful first check; operational claims require the runtime evidence above.

Deploying Rook Ceph on Kubernetes for Grafana Loki and Mimir

Fri, 15 Nov 2024 00:00:00 +0000

Deploying Rook Ceph on Kubernetes: A Detailed Walkthrough

This blog post documents my experience deploying Rook, a Ceph orchestrator for Kubernetes, within our existing cluster. This setup provides a highly available, scalable object storage solution integrated directly into our Kubernetes environment. The process involved careful disk preparation, Rook deployment, and subsequent configuration and testing.

Rook, Ceph, and the Why

Rook simplifies the deployment and management of Ceph, a distributed storage system, within Kubernetes. This means we get a robust, self-managing, and scalable storage solution without the complexities of manual Ceph installation. For this deployment, we used Rook v1.13.3 and Ceph v18.2. The base directory for all operations was /srv/cluster-logs/rook.

Preparing the Storage

Before deploying Rook, I had to prepare the storage devices that would serve as OSDs (Object Storage Devices). This involved wiping any existing filesystem information on the chosen disks. The initial disk listing (lsblk) confirmed the available disks and partitions:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 3.6T 0 disk
sdb 8:16 0 3.6T 0 disk
sdc 8:32 0 3.6T 0 disk
sdd 8:48 0 3.6T 0 disk
sde 8:64 0 3.6T 0 disk
sdf 8:80 0 3.6T 0 disk
sdg 8:96 0 3.6T 0 disk
sdh 8:112 0 3.6T 0 disk
sdi 8:128 0 3.6T 0 disk
sdj 8:144 0 3.6T 0 disk
sdk 8:160 0 3.6T 0 disk
sdl 8:176 0 3.6T 0 disk
sdm 8:192 0 223.5G 0 disk
├─sdm1 8:193 0 600M 0 part /boot/efi
├─sdm2 8:194 0 1G 0 part /boot
└─sdm3 8:195 0 221.9G 0 part
 ├─almalinux_logsrv1233-root 253:0 0 70G 0 lvm /
 └─almalinux_logsrv1233-swap 253:1 0 4G 0 lvm
nvme2n1 259:4 0 1.5T 0 disk
nvme1n1 259:5 0 1.5T 0 disk
nvme0n1 259:6 0 1.5T 0 disk
nvme3n1 259:7 0 1.5T 0 disk

The wipefs command was used remotely via SSH to securely wipe the selected disks:

Source codebash


1
2
3
4
5
6


# Wipe selected disks across multiple nodes
for j in {247..251}; do ssh admin@10.0.0.$j "for i in {a..l}; do sudo wipefs -a /dev/sd\$i; done"; done
for j in {246..251}; do ssh admin@10.0.0.$j "sudo wipefs -a /dev/nvme0n1"; done
for j in {246..251}; do ssh admin@10.0.0.$j "sudo wipefs -a /dev/nvme1n1"; done
for j in {246..251}; do ssh admin@10.0.0.$j "sudo wipefs -a /dev/nvme2n1"; done
for j in {246..251}; do ssh admin@10.0.0.$j "sudo wipefs -a /dev/nvme3n1"; done

This script iterates through a range of IP addresses (10.0.0.247 to 10.0.0.251) representing our worker nodes and executes the wipefs command on each specified device. This ensures clean disks for our Ceph OSDs.

Rook Deployment

1. Cloning the Repository:

The Rook repository was cloned to the specified base directory:

Source codebash


1
2
3


cd /srv/cluster-logs/
git clone --single-branch --branch v1.13.3 https://github.com/rook/rook.git
cd rook/deploy/examples

Cloning a specific branch ensures consistency and avoids potential issues with newer features.

2. Creating Core Components:

Basic Rook components were created using kubectl:

Source codebash

1

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

These YAML files define the Custom Resource Definitions (CRDs), common resources, and the operator itself, forming the foundation for the Rook deployment.

3. Deploying the Ceph Cluster:

The core of the deployment is the cluster.yaml file:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
 name: rook-ceph
 namespace: rook-ceph
spec:
 cephVersion:
 image: registry.internal/ceph:v18.2.1 # Custom Ceph Image
 allowUnsupported: false # Strict Ceph version check
 # ... other specifications such as data directory, monitoring, network settings ...
 storage:
 useAllNodes: true # Utilize all nodes for OSD placement
 useAllDevices: true # Utilize all available devices
 # ... advanced settings such as placement, cleanup policies, resource allocation, etc.

This YAML file defines the Ceph cluster configuration, including the Ceph version (using a custom image from registry.internal), data directory location, monitoring settings, network parameters, OSD placement strategy and more. The useAllNodes and useAllDevices options simplified OSD deployment by utilizing all available resources.

The cluster was created with:

Source codebash

1
2

cd /srv/cluster-logs/rook/deploy
kubectl create -f cluster.yaml

4. Deploying the Toolbox:

The Rook toolbox provides useful CLI tools for managing the Ceph cluster:

Source codebash

1

kubectl create -f examples/toolbox.yaml

This deploys a container providing access to Ceph commands within the Kubernetes environment.

5. Verification:

The status of the Rook deployment was verified using:

Source codebash

1

kubectl -n rook-ceph get pod

This command lists all pods in the rook-ceph namespace, showing the status of various components like MONs, OSDs, and managers.

6. Accessing the Ceph CLI:

The Ceph CLI was accessed using the toolbox:

Source codebash

1

k -n rook-ceph exec -ti rook-ceph-tools-6cfb855d85-p2kfg -- bash

This command provides a summary of the Ceph cluster status:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


bash-4.4$ ceph -s
 cluster:
 id: aca5b61e-a4fa-4e2e-b9d2-e6c1f59f7563
 health: HEALTH_OK

 services:
 mon: 3 daemons, quorum a,b,c (age 2d)
 mgr: a(active, since 5h), standbys: b
 osd: 112 osds: 112 up (since 2d), 112 in (since 2d)
 rgw: 1 daemon active (1 hosts, 1 zones)

 data:
 pools: 9 pools, 89 pgs
 objects: 379 objects, 16 MiB
 usage: 11 GiB used, 346 TiB / 346 TiB avail
 pgs: 89 active+clean

7. Object Storage (RADOS Gateway):

Object storage was deployed using:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
 name: object-store
 namespace: rook-ceph
spec:
 metadataPool:
 failureDomain: host
 replicated:
 size: 3
 dataPool:
 failureDomain: host
 # For production it is recommended to use more chunks, such as 4+2 or 8+4
 erasureCoded:
 dataChunks: 4
 codingChunks: 2
 preservePoolsOnDelete: true
 gateway:
 # sslCertificateRef:
 port: 80
 # securePort: 443
 instances: 1

This YAML file configures the RADOS Gateway (RGW), providing object storage compatible with S3. The configuration includes the selection of metadata and data pools, as well as gateway settings like port and number of instances.

An Ingress resource was then created to expose the RGW externally, allowing access to the object storage over HTTP.

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: rook-ceph-rgw
 namespace: rook-ceph
spec:
 ingressClassName: haproxy
 rules:
 - host: s3-k8s-logs.internal
 http:
 paths:
 - backend:
 service:
 name: rook-ceph-rgw-object-store
 port:
 number: 80
 path: /
 pathType: ImplementationSpecific
 - host: s3-in-k8s-logs.internal
 http:
 paths:
 - backend:
 service:
 name: rook-ceph-rgw-object-store
 port:
 number: 80
 path: /
 pathType: ImplementationSpecific
 tls:
 - hosts:
 - s3-k8s-logs.internal
 - s3-in-k8s-logs.internal

8. RBD Pool and StorageClass:

A block storage pool (rbd-pool) and a corresponding StorageClass (rook-ceph-block) were created to allow Kubernetes pods to provision Persistent Volumes using Ceph RBD. The StorageClass definition includes parameters for cluster ID, pool name, image format, and other RBD specific settings:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
 name: rbd-pool
 namespace: rook-ceph
spec:
 failureDomain: host
 replicated:
 size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
 clusterID: rook-ceph
 pool: rbd-pool
 imageFormat: "2"
 imageFeatures: layering
 csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
 csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
 csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
 csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
 csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
 csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true

9. Dashboard Ingress:

An Ingress resource was configured to expose the Ceph dashboard. This allows accessing the dashboard through a web browser.

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: rook-ceph-dashboard
 namespace: rook-ceph
spec:
 ingressClassName: haproxy
 rules:
 - host: ceph-k8s-logs.internal
 http:
 paths:
 - backend:
 service:
 name: rook-ceph-mgr-dashboard
 port:
 number: 8443
 path: /
 pathType: ImplementationSpecific
 tls:
 - hosts:
 - ceph-k8s-logs.internal

10. Dashboard SSL:

The process involved disabling SSL on the dashboard temporarily, due to initial compatibility issues with the existing HAProxy setup.

Source codebash


1
2
3
4


k -n rook-ceph exec -ti rook-ceph-tools-6cfb855d85-p2kfg -- bash
bash-4.4$ ceph config set mgr mgr/dashboard/ssl false
bash-4.4$ ceph mgr module disable dashboard
bash-4.4$ ceph mgr module enable dashboard

11. Upgrades:

Rook and Ceph were upgraded incrementally by cloning the updated repository and applying the necessary YAML manifests, in order to test the procedure before any workload were running in the cluster. The incremental upgrade strategy ensures minimal downtime and risk during the upgrade process.

Source codebash


1
2
3


cd deploy/examples
kubectl apply -f common.yaml -f crds.yaml
kubectl -n $ROOK_OPERATOR_NAMESPACE set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.15.2

This detailed walkthrough provides a comprehensive guide to deploying and managing Rook Ceph within a Kubernetes cluster. The use of Rook simplifies the process considerably, allowing for a highly available and scalable storage solution integrated directly into our Kubernetes infrastructure. The careful configuration ensures our storage system is tuned for performance and resilience.

Building a Kubernetes Cluster with Kubespray: A Personal Journey

Sat, 09 Nov 2024 00:00:00 +0000

Building a Kubernetes Cluster with Kubespray and Monitoring Stack: A Personal Journey

This blog post details my experience setting up a Kubernetes cluster on 7 bare-metal nodes using Kubespray, followed by the deployment of a comprehensive monitoring stack comprised of Rook, Mimir, Grafana, and Loki, which will be described in separated posts. This setup provides robust storage, metrics, and logging capabilities for our internal use.

Kubespray: The Orchestrator

Kubespray is a powerful Ansible-based tool for deploying and managing Kubernetes clusters. It automates the complex process of setting up a Kubernetes control plane and worker nodes, handling tasks like installing Kubernetes components, configuring networking, and managing etcd. It allows for flexible customization through inventory files and group variables. I chose version 2.26.0 for this deployment at the time.

Prerequisites

Since I used bare metal servers that became available from another project, I previously installed the operating system and configured everything, from packages to network and authentication.

Step-by-Step Setup

1. Git Cloning and Docker Image:

First, we clone the Kubespray repository and pull the necessary Docker image:

Source codebash


1
2
3
4
5
6


# Clone the Kubespray repository
git clone https://github.com/kubernetes-sigs/kubespray.git
git checkout v2.26.0

# Pull the Kubespray Docker image
docker pull quay.io/kubespray/kubespray:v2.26.0

This ensures we have the latest version (or the specific version we need) of Kubespray. The Docker image contains all necessary Ansible components and avoids potential conflicts with local installations. Never use the latest version commited to the repository, always use a stable release.

2. Inventory Configuration:

The heart of Kubespray’s configuration lies in its inventory files. These define the nodes in your cluster, their roles, and relevant network information. My inventory file (inventory/k8s-logs/hosts.yml) looks like this:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


all:
 hosts:
 # Worker Nodes
 logsrv1234:
 ansible_host: 10.0.0.245 # Ansible connection IP
 ip: 10.1.0.45 # Internal IP address
 access_ip: 10.1.0.45 # IP for external access (if needed)
 node_labels:
 node-role.kubernetes.io/node: "" # Label indicating it's a worker node

 # ... (similar entries for other worker nodes logsrv1233 - logsrv1228) ...

 # Ingress Node
 ingress-1:
 ansible_host: 10.0.0.241
 ip: 10.1.0.41
 access_ip: 10.1.0.41
 node_labels:
 node-role.kubernetes.io/node: "" # Ingress node also needs node role

 # Master Nodes
 master-1:
 ansible_host: 10.0.0.242
 ip: 10.1.0.42
 access_ip: 10.1.0.42
 node_labels:
 node-role.kubernetes.io/master: "" # Label designating a master node

 # ... (similar entries for master-2 and master-3) ...

 children:
 # Grouping hosts based on their roles for simplified management
 kube_control_plane:
 hosts:
 master-1:
 master-2:
 master-3:
 kube_node:
 hosts:
 logsrv1234:
 logsrv1233:
 # ... other worker and ingress node ...
 etcd:
 hosts:
 master-1:
 master-2:
 master-3:
 k8s_cluster:
 children:
 kube_control_plane:
 kube_node:
 calico_rr:
 hosts: {} # Calico configuration (empty for now)

This meticulously defines each node, its IP addresses (ansible_host for Ansible connection, ip for cluster internal communication, and access_ip for external access), and its role within the Kubernetes cluster using node labels. The children section groups hosts for easier management in Ansible playbooks.

3. Group Variables:

The group_vars directory contains YAML files that set global and group-specific variables for the Kubespray deployment. The all.yml file sets general settings:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


---
bin_dir: /usr/local/bin
loadbalancer_apiserver_port: 6443
loadbalancer_apiserver_healthcheck_port: 8081
upstream_dns_servers:
 - 10.0.0.15
 - 10.0.0.16
http_proxy: "http://proxy.internal:3128" # environment without direct internet access
https_proxy: "http://proxy.internal:3128" # environment without direct internet access
no_proxy: "10.0.0.0/8,localhost,127.0.0.0/8,172.16.0.0/16,*.internal" # environment without direct internet access
no_proxy_exclude_workers: false
kube_webhook_token_auth: false
kube_webhook_token_auth_url_skip_tls_verify: false
ntp_enabled: false
ntp_manage_config: false
ntp_servers:
 - "0.pool.ntp.org iburst"
 - "1.pool.ntp.org iburst"
 - "2.pool.ntp.org iburst"
 - "3.pool.ntp.org iburst"
unsafe_show_logs: false
allow_unsupported_distribution_setup: false

And k8s-cluster.yml configures Kubernetes-specific settings:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


---
kube_version: v1.30.4  # Kubernetes version
kube_network_plugin: calico # CNI plugin to be used
kube_config_dir: /etc/kubernetes
kube_script_dir: "{{ bin_dir }}/kubernetes-scripts"
kube_manifest_dir: "{{ kube_config_dir }}/manifests"
kube_cert_dir: "{{ kube_config_dir }}/ssl"
kube_token_dir: "{{ kube_config_dir }}/tokens"
kube_api_anonymous_auth: true
local_release_dir: "/tmp/releases"
retry_stagger: 5
kube_owner: kube
kube_cert_group: kube-cert
kube_log_level: 2
credentials_dir: "{{ inventory_dir }}/credentials"
kube_network_plugin_multus: false
kube_service_addresses: 10.36.192.0/18
kube_pods_subnet: 172.16.0.0/16
kube_network_node_prefix: 24
enable_dual_stack_networks: false
kube_service_addresses_ipv6: fd85:ee78:d8a6:8607::1000/116
kube_pods_subnet_ipv6: fd85:ee78:d8a6:8607::1:0000/112
kube_network_node_prefix_ipv6: 120
kube_apiserver_ip: "{{ kube_service_addresses | ipaddr('net') | ipaddr(1) | ipaddr('address') }}"
kube_apiserver_port: 6443
kube_proxy_mode: ipvs
kube_proxy_strict_arp: true
kube_proxy_nodeport_addresses: >-
 {%- if kube_proxy_nodeport_addresses_cidr is defined -%}
 [{{ kube_proxy_nodeport_addresses_cidr }}]
 {%- else -%}
 []
 {%- endif -%} 
kube_encrypt_secret_data: false
cluster_name: k8s-logs
ndots: 2
dns_mode: coredns
enable_nodelocaldns: true
enable_nodelocaldns_secondary: false
nodelocaldns_ip: 169.254.25.10
nodelocaldns_health_port: 9254
nodelocaldns_second_health_port: 9256
nodelocaldns_bind_metrics_host_ip: false
nodelocaldns_secondary_skew_seconds: 5
enable_coredns_k8s_external: false
coredns_k8s_external_zone: k8s_external.local
enable_coredns_k8s_endpoint_pod_names: false
resolvconf_mode: host_resolvconf
deploy_netchecker: false
skydns_server: "{{ kube_service_addresses | ipaddr('net') | ipaddr(3) | ipaddr('address') }}"
skydns_server_secondary: "{{ kube_service_addresses | ipaddr('net') | ipaddr(4) | ipaddr('address') }}"
dns_domain: "{{ cluster_name }}"
container_manager: containerd
kata_containers_enabled: false
kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}"
k8s_image_pull_policy: IfNotPresent
kubernetes_audit: false
default_kubelet_config_dir: "{{ kube_config_dir }}/dynamic_kubelet_dir"
podsecuritypolicy_enabled: false
volume_cross_zone_attachment: false
persistent_volumes_enabled: false
event_ttl_duration: "1h0m0s"
auto_renew_certificates: false
kubeadm_patches:
 enabled: false
 source_dir: "{{ inventory_dir }}/patches"
 dest_dir: "{{ kube_config_dir }}/patches"
containerd_limit_open_file_num: 1048576
kube_proxy_metrics_bind_address: 0.0.0.0:10249
kube_vip_enabled: true
kube_vip_controlplane_enabled: true
kube_vip_address: 10.46.0.40
apiserver_loadbalancer_domain_name: master-k8s-logs.local
loadbalancer_apiserver:
 address: 10.0.0.40
 port: 6443
kube_vip_interface: ens224
kube_vip_services_enabled: false
kube_vip_arp_enabled: true

These variables define crucial parameters like the Kubernetes version, networking configuration, proxy settings, and more. The comments within the YAML files offer further detail on the purpose of these variables.

4. SSH Key Generation:

Next, an SSH key pair is generated for passwordless authentication to the bare metal nodes:

Source codebash

1

ssh-keygen -t rsa -f id_rsa_kubespray

This key will be used by Ansible to connect to and manage the cluster nodes without requiring manual password entry. Remember to copy the public key to all the nodes’ authorized_keys file.

5. Running Kubespray within a Docker Container:

To run Kubespray, we use a Docker container to isolate the environment:

Source codebash


1
2
3
4


docker run --rm -it \
 --mount type=bind,source="$(pwd)"/inventory/k8s-logs,dst=/inventory \
 --mount type=bind,source="${HOME}"/cluster-logs/kubespray/id_rsa_kubespray,dst=/root/.ssh/id_rsa \
 quay.io/kubespray/kubespray:v2.26.0 bash

This command runs the Kubespray image, mounting the inventory directory and the SSH private key into the container. The --rm flag removes the container after execution. The bash command starts a bash shell within the container.

6. Running the Playbook:

Once inside the container, the main Kubespray playbook is run:

Source codebash

1

ansible-playbook -i inventory/hosts.yml cluster.yml -b

-i inventory/hosts.yml specifies the inventory file, cluster.yml is the main playbook, and -b enables become (sudo) execution. This command initiates the automated deployment of the entire Kubernetes cluster.

7. Obtaining the kubeconfig:

After successful playbook execution, the Kubernetes admin.conf file is located at /etc/kubernetes/admin.conf on any master node. This file is copied and used to authenticate with the cluster using kubectl.

8. kubectl Adjustments:

For access from outside the local network (e.g., via VPN), the kubeconfig file may need adjustments to use IP addresses instead of FQDNs for proper connection.

9. Rook, Mimir, Grafana, and Loki Deployment (Not detailed here):

Following the Kubernetes cluster setup, I installed Rook for persistent storage, Mimir for long-term metrics storage, Grafana for visualizing the metrics, and Loki for log aggregation. These steps are beyond the scope of this post but were crucial for creating our monitoring stack. I will describe each installation on dedicated posts.

This entire process provides a solid foundation for a robust and scalable Kubernetes cluster. The use of Kubespray significantly streamlined the deployment, and the monitoring stack ensures we can effectively manage and troubleshoot our infrastructure. The detailed inventory and group variable configurations provide a solid template for future cluster deployments.

Deploying Grafana Loki on Kubernetes

Tue, 22 Oct 2024 00:00:00 +0000

I deployed Loki as part of an observability platform on a seven-node, AlmaLinux bare-metal Kubernetes cluster, using Rook Ceph for object storage. The goal was to centralize logs from infrastructure services, including Ceph and HAProxy, and query them through Grafana.

Technical revision — October 2026: the original article incorrectly described Mimir as Loki’s log-storage backend and reversed the release/chart arguments in the installation command. The architecture and commands below correct those errors. The versioned example is a historical reference, not a recommendation to start a new production deployment on these releases.

Where logs and metrics go

Loki ingests and queries logs. Mimir ingests and queries Prometheus metrics. Each uses its own buckets in the S3-compatible Ceph object store; Grafana queries both as separate data sources. Mimir is not in Loki’s log-storage path.

Source codetext


1
2
3


Log files → collector → authenticating gateway → Loki → Ceph S3 log buckets
Prometheus → authenticating gateway → Mimir → Ceph S3 metric buckets
Grafana → Loki (LogQL) and Mimir (PromQL)

Loki’s distributors accept writes and route them to ingesters. Ingesters maintain recent data and write chunks to object storage. Queriers retrieve log data from ingesters and storage; query frontends and schedulers coordinate query work. The compactor handles index compaction and, when explicitly configured, retention.

Labels identify streams. High-cardinality labels can greatly increase stream count and index overhead; they should not be described as creating one index shard per label. Keep labels bounded and use log content or structured metadata for values such as request IDs.

Versioned configuration

The downloadable values.yaml targets grafana/loki chart 6.24.0, Loki 3.3.2, with a Kubernetes 1.30 rendering target. These are reference versions selected for this revision; the original deployment’s exact chart version was not recorded in the article. The example was checked with Helm 3.17.3 using lint and template rendering. It has not been installed into a cluster during this revision.

For a new deployment, select supported versions and follow the current Loki installation guide. The chart has since moved to the Grafana Community repository; review its migration instructions instead of applying these historical values to a current chart.

values.yamlyaml · 102 linesDownload
Expand sourceCollapse source

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102


# Historical reference: grafana/loki chart 6.24.0 (Loki 3.3.2).
# Rendered locally; adapt endpoint, Secret, PVCs and authentication before use.
deploymentMode: Distributed
global:
 clusterDomain: cluster.local
 dnsService: kube-dns
 dnsNamespace: kube-system
loki:
 auth_enabled: true
 commonConfig:
 replication_factor: 3
 schemaConfig:
 configs:
 - from: "2024-04-01"
 store: tsdb
 object_store: s3
 schema: v13
 index:
 prefix: loki_index_
 period: 24h
 storage:
 type: s3
 bucketNames:
 chunks: loki-chunks
 ruler: loki-ruler
 admin: loki-admin
 s3:
 endpoint: https://s3.example.com
 region: us-east-1
 accessKeyId: ${AWS_ACCESS_KEY_ID}
 secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
 s3ForcePathStyle: true
 insecure: false
ingester:
 replicas: 3
 zoneAwareReplication:
 enabled: false
 persistence:
 enabled: true
 claims:
 - name: data
 size: 10Gi
 storageClass: rook-ceph-block
 extraArgs: &expandEnv
 - -config.expand-env=true
 extraEnvFrom: &storageEnv
 - secretRef:
 name: loki-s3
querier:
 replicas: 2
 maxUnavailable: 1
 extraArgs: *expandEnv
 extraEnvFrom: *storageEnv
queryFrontend:
 replicas: 2
 maxUnavailable: 1
 extraArgs: *expandEnv
 extraEnvFrom: *storageEnv
queryScheduler:
 replicas: 2
 extraArgs: *expandEnv
 extraEnvFrom: *storageEnv
distributor:
 replicas: 2
 maxUnavailable: 1
 extraArgs: *expandEnv
 extraEnvFrom: *storageEnv
compactor:
 replicas: 1
 persistence:
 enabled: true
 claims:
 - name: data
 size: 10Gi
 storageClass: rook-ceph-block
 extraArgs: *expandEnv
 extraEnvFrom: *storageEnv
indexGateway:
 replicas: 2
 maxUnavailable: 1
 persistence:
 enabled: true
 size: 10Gi
 storageClass: rook-ceph-block
 extraArgs: *expandEnv
 extraEnvFrom: *storageEnv
gateway:
 replicas: 2
read:
 replicas: 0
write:
 replicas: 0
backend:
 replicas: 0
singleBinary:
 replicas: 0
minio:
 enabled: false
lokiCanary:
 enabled: false
test:
 enabled: false

Important choices in this example:

Three ingesters and replication factor three require enough schedulable nodes. The chart’s anti-affinity and your storage topology must be checked; replicas alone do not prove fault tolerance.
Zone-aware replication is explicitly disabled. This is a host-level reference, not a claim of resilience across availability zones.
The compactor runs as a singleton. Its responsibilities do not justify one replica per cluster node. Retention is not enabled by this example; define and test a retention policy before relying on automatic deletion. See Loki retention.
PVCs reference rook-ceph-block. Confirm that StorageClass, free space, placement and reclaim policy before use.
Replica counts, cache defaults, resources and disruption budgets require load testing. They are not a sizing recommendation for a 300 TB storage cluster.
Canary and Helm tests are disabled until an authenticated collection/query path is configured. Enable equivalent end-to-end monitoring before production use.

Object storage and credentials

Provision a dedicated Ceph user with access limited to Loki’s buckets. With a Rook CephObjectStoreUser named loki in object store object-store, inspect the generated Secret’s metadata with:

Source codebash

1

kubectl -n rook-ceph describe secret rook-ceph-object-user-object-store-loki

That Secret belongs to the rook-ceph namespace. A Secret reference cannot cross namespaces: use your secret-management workflow to supply a separate loki-s3 Secret in namespace loki, containing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The chart injects these variables and expands them in the Loki configuration. Do not commit key values or rendered configuration containing real credentials.

Configure a trusted HTTPS endpoint in the values file. For an internal CA, install or mount the required CA and configure the client trust settings for the chosen version. insecure: false assumes a certificate chain the client trusts; it does not install your CA.

Bucket commands must point explicitly to Ceph, not the default AWS S3 endpoint. The following Bash example assumes an AWS CLI profile ceph-loki that already contains the restricted Ceph user’s credentials:

Source codebash


1
2
3
4
5
6
7


export AWS_PROFILE=ceph-loki
export CEPH_S3_ENDPOINT=https://s3.example.com
# If the endpoint uses an internal CA:
# export AWS_CA_BUNDLE=/path/to/internal-ca.pem
for bucket in loki-chunks loki-ruler loki-admin; do
 aws --endpoint-url "$CEPH_S3_ENDPOINT" --region us-east-1 s3 mb "s3://$bucket"
done

Use the same endpoint, bucket names and region/signature settings in Loki. Create buckets once, and verify existing buckets rather than assuming a failed creation command means the storage is unusable. TLS verification remains enabled.

Render and install deliberately

In an empty working directory, download values.yaml, adapt its endpoint and DNS settings, and review the manifests:

Source codebash


1
2
3


helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm template loki grafana/loki --version 6.24.0 --namespace loki --kube-version 1.30.0 -f values.yaml > rendered.yaml

The correct command order is release name, chart name, followed by flags. After verifying the target cluster, namespace, Secrets, storage, resources and authentication design, the corresponding historical installation command is:

Source codebash

1

helm upgrade --install loki grafana/loki --version 6.24.0 --namespace loki --create-namespace -f values.yaml --wait --timeout 10m

helm template verifies chart rendering; it does not validate storage connectivity, application configuration at runtime or the cluster’s capacity to schedule the pods.

Authentication, collection and verification

auth_enabled: true enables Loki’s multi-tenant mode. Clients must supply the appropriate X-Scope-OrgID, but that header alone is not proof of identity. Put an authenticating and authorizing proxy in front of the API, and have it set or validate tenant identity. The chart’s routing gateway does not automatically establish this security boundary. Keep direct service access restricted. See Loki authentication.

The original collection example used Promtail. It reached end of life on March 2, 2026. Use a supported collector such as Grafana Alloy for a new setup, following the Promtail migration guidance. Give the collector only the filesystem permissions it needs; running as root is not a universal prerequisite.

Add Loki’s authenticated query endpoint as a Grafana data source. Validate ingestion and querying for a known log line, tenant separation, pod placement, PVC binding and object-store access. Then test restarts and node failure under representative load. Those checks are necessary before claiming high availability or choosing the final replica counts.

Hugo website on GitHub Pages

Tue, 16 May 2023 00:00:00 +0000

I’m gonna describe how this website is hosted and deployed using GitHub and Hugo, with the help of the embedded CI/CD tools to automate the site generation and deployment.

The first step is to create a new empty repository on GitHub, with the name following the format mysite.github.io. That’s how GitHub knows that this repository is going to be a static website:

Then, visit the repository you just created. From the main menu choose Settings > Pages. In the center of your screen you will see this:

The default option is “Deploy from a branch”. Change it to “GitHub Actions”.

You can also choose a custom domain if you don’t want to use the default github.io domain:

Now, clone the repository locally:

Source codebash

1

git clone git@github.com:biofalopes/biofalopes.github.io.git # change according to your username and repository name.

Now is time to populate your repository with the workflow yaml. Create a folder called .github and a subfolder called workflows:

Source codebash

1

mkdir -p .github/workflows

Then, create this yaml file inside. It can have any name, let’s use hugo.yaml:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


# Sample workflow for building and deploying a Hugo site to GitHub Pages
name: Deploy Hugo site to Pages

on:
 # Runs on pushes targeting the default branch
 push:
 branches:
 - main

 # Allows you to run this workflow manually from the Actions tab
 workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
 contents: read
 pages: write
 id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
 group: "pages"
 cancel-in-progress: false

# Default to bash
defaults:
 run:
 shell: bash

jobs:
 # Build job
 build:
 runs-on: ubuntu-latest
 env:
 HUGO_VERSION: 0.137.1
 steps:
 - name: Install Hugo CLI
 run: |
 wget -O ${{ runner.temp }}/hugo.deb https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb \
 && sudo dpkg -i ${{ runner.temp }}/hugo.deb 
 - name: Install Dart Sass
 run: sudo snap install dart-sass
 - name: Checkout
 uses: actions/checkout@v4
 with:
 submodules: recursive
 fetch-depth: 0
 - name: Setup Pages
 id: pages
 uses: actions/configure-pages@v5
 - name: Install Node.js dependencies
 run: "[[ -f package-lock.json || -f npm-shrinkwrap.json ]] && npm ci || true"
 - name: Build with Hugo
 env:
 HUGO_CACHEDIR: ${{ runner.temp }}/hugo_cache
 HUGO_ENVIRONMENT: production
 TZ: America/Los_Angeles
 run: |
 hugo \
 --gc \
 --minify \
 --baseURL "${{ steps.pages.outputs.base_url }}/" 
 - name: Upload artifact
 uses: actions/upload-pages-artifact@v3
 with:
 path: ./public

 # Deployment job
 deploy:
 environment:
 name: github-pages
 url: ${{ steps.deployment.outputs.page_url }}
 runs-on: ubuntu-latest
 needs: build
 steps:
 - name: Deploy to GitHub Pages
 id: deployment
 uses: actions/deploy-pages@v4
 ```

This yaml is a template provided by the Hugo team, and can be found here. What is does is configure the required permissions and settings, download and install hugo’s binaries on the runner, run hugo to generate the website and then deploy it to GitHub Pages. This saves you all the hassle of having to do it locally, and you can simply push your markdown files to the repository and everything else will be handled by the Workflow.

The next step is to copy the hugo files. There are many different ways to do that, and I prefer to use a simple method that doesn’t involve git submodules or having to install go dependencies locally. That might be necessary depending on the theme you choose, but for my case the basic is enough. The theme I use currently is the Stack, that can be found here. First, clone the theme repository:

Source codebash

1

git clone git@github.com:CaiJimmy/hugo-theme-stack.git

Almost every theme will have a exampleSite folder inside, which you should copy first:

Source codebash

1

cp -r ~/hugo-theme-stack/exampleSite/* ~/mysite.github.io/

Next, copy the entire theme to the themes folder:

Source codebash

1
2

mkdir ~/mysite.github.io/themes
cp -r ~/hugo-theme-stack ~/mysite.github.io/themes/

It might look a bit counterintuitive or confusing, but the theme must be inside the themes folder. There can be multiple different themes, and hugo will know which one to use through the hugo.yaml file on the root folder. The problem is that each theme uses its own set of parameters within hugo’s capabilities, and that makes it a bit difficult to seamlessly change themes. I changed the theme of my website a couple of times already, and I had to start fresh every time, bringing just the content and even then having to adjust some things.

The last step is to push everything to GitHub, after configuring and testing your website locally, which I will not get into details here. But basically you have to set all the parameters and then test your pages to verify syntax, image fitment and so on. To push it:

Source codebash


1
2
3


git add .
git commit -a -m "Initial Commit"
git push

If everything worked correctly, you can check the workflow by clicking in the Actions menu:

Then it’s just a matter of opening the URL and verifying if everything went on correctly. Most of the times when the site is correctly rendered locally but not after publishing, is due to a misconfigured hugo.yaml file, usually at the baseurl parameter.

HashiCorp Certified: Terraform Associate Exam

Sun, 07 Aug 2022 00:00:00 +0000

In order to pass the Terraform Associate Exam I followed the Official guide as a baseline for my studies, as I usually do for certification exams and it always works out well for me.

The HashiCorp Terraform Associate is considered a foundational level certification, so you can expect to have essential knowledge and skills on the concepts and not necessarily lots of hands-on practice. If you already work with Terraform you probably be able to go very quicky over the topics but it is important to know some details and also some commands that might not be very common for you. You must also know the features that exist on Terraform Enterprise packages and Terraform Cloud.

Important notes about the test: it is 57 questions long and is a proctored exam. You get 60 minutes to complete and you can mark questions for later review. the certification is valid for two years. Since Terraform is quickly and constantly evolving, it makes sense that it has a shorter validity than most certifications, which is three years.

My personal tips for this and any other certification exam: do lots of practice tests, go over every subject at least once even if you know it already from previous experience, make notes and don’t just study for the test, apply the knowledge in labs or personal projects. I would say that practicing is at least 2x more efficient in making the knowledge permanent than reading or watching videos.

Official Exam Information from Hashicorp can be found here: HashiCorp Certified: Terraform Associate (002)

Objective	Description
1	Understand Infrastructure as Code (IaC) concepts
2	Understand Terraform’s purpose (vs other IaC)
3	Understand Terraform basics
4	Use the Terraform CLI (outside of core workflow)
5	Interact with Terraform modules
6	Navigate Terraform workflow
7	Implement and maintain state
8	Read, generate, and modify configuration
9	Understand Terraform Cloud and Enterprise capabilities

What I’m gonna do here is go over each of the objectives and cover (almost) everything required to know in order to pass the exam. Most of the contents are provided on the official study guide and the exam review, so I’ll be using it as a reference. It’s probably be a long read, but I rather have it all in one place than making several posts and split the content.

1: Understand Infrastructure as Code (IaC) concepts

1a: Explain what IaC is

Infrastructure as code (IaC) tools allow you to manage infrastructure with configuration files rather than through a graphical user interface. IaC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share.

1b Describe advantages of IaC patterns

IaC makes it easy to provision and apply infrastructure configurations by standardizing the workflow. This is accomplished by using a common syntax across a number of different infrastructure providers (e.g. AWS, GCP). Some key advantages of using IaC patterns are:

Reusability: you can write or use modules to reuse code that are common between different projects;
Consistency: managing infrastructure manually are prone to errors;
Idempotency: no matter how many times you run your IaC and, what your starting state is, you will end up with the same end state. This simplifies the provisioning of Infrastructure and reduces the chances of inconsistent results;
Versioning and Source Control: by using a VCS like Git you can get visibility and security on your infrastructure.

2 Understand Terraform’s purpose (vs other IaC)

2a Explain multi-cloud and provider-agnostic benefits

Terraform is cloud-agnostic and allows a single configuration to be used to manage multiple providers, and to even handle cross-cloud dependencies. With Terraform, users can manage a heterogeneous environment with the same workflow by creating a configuration file to fit the needs of that platform or project. Terraform plugins called providers let Terraform interact with cloud platforms and other services via their application programming interfaces (APIs). HashiCorp and the Terraform community have written over 1,000 providers to manage resources on Amazon Web Services (AWS), Azure, Google Cloud Platform (GCP), Kubernetes, Helm, GitHub, Splunk, and DataDog, just to name a few.

2b Explain the benefits of state

Terraform keeps track of your real infrastructure in a state file, which acts as a source of truth for your environment. Terraform uses the state file to determine the changes to make to your infrastructure so that it will match your configuration.

3 Understand Terraform basics

3a Handle Terraform and provider installation and versioning

Terraform can be installed using the user’s terminal:
https://learn.hashicorp.com/tutorials/terraform/install-cli

Source codePlain text


1
2
3
4
5
6
7
8


## Mac OS
brew install terraform

## Windows
choco install terraform

## Linux
sudo apt install terraform

Alternatively, Terraform can be manually installed by downloading the binary to your computer.

3b Describe plug-in based architecture

Terraform uses a plugin-based architecture to support hundreds of infrastructure and service providers. Initializing a configuration directory downloads and installs providers used in the configuration. Terraform plugins are compiled for a specific operating system and architecture, and any plugins in the root of the user’s plugins directory must be compiled for the current system. A provider is a plugin that Terraform uses to translate the API interactions with that platform or service.

Terraform must initialize a provider before it can be used. The initialization process downloads and installs the provider’s plugin so that it can later be executed. Terraform knows which provider(s) to download based on what is declared in the configuration files. For example:

Source codePlain text


1
2
3


provider "aws" {
 region = "us-west-2"
}

The provider block can contain the following meta-arguments:

version - constrains which provider versions are allowed
- Note: HashiCorp recommends using provider requirements instead
alias - enables using the same provider with different configurations (e.g. provisioning resources in multiple AWS regions)

By default, a plugin is downloaded into a subdirectory of the working directory so that each working directory is self-contained. As a consequence, if there are multiple configurations that use the same provider then a separate copy of its plugin will be downloaded for each configuration. To manually install a provider, move it to:

Source codePlain text


1
2
3
4
5


## MacOS / Linux
~/.terraform.d/plugins

## Windows
%APPDATA%\terraform.d\plugins

Given that provider plugins can be quite large, users can optionally use a local directory as a shared plugin cache. This is enabled through using the plugin_cache_dir setting in the CLI configuration file.

Source codePlain text

1

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

This configuration ensures each plugin binary is downloaded only once.

3c Demonstrate using multiple providers

To instantiate the same provider for multiple configurations, use the alias argument. For example, the AWS provider requires specifying the region argument. The following code block demonstrates how alias can be used to provision resources across multiple regions using the same configuration files.

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


provider "aws" {
	region = "us-east-1"
}

provider "aws" {
	region = "us-west-1"
	alias = "pncda"
}

resource "aws_vpc" "vpc-pncda" {
	cidr_block = "10.0.0.0/16"
	provider = "aws.pncda"
}

3d Describe how Terraform finds and fetches providers

Providers are released on a separate rhythm from Terraform itself, and so each provider has its own version number. For production use, consider constraining the acceptable provider versions in the configuration to ensure that new versions with breaking changes will not be automatically installed by terraform init in future.

Any non-certified or third-party providers must be manually installed, since terraform init cannot automatically download them.

The required_version setting can be used to constrain which versions of Terraform can be used with the configuration.

Source codePlain text


1
2
3


terraform {
 required_version = ">= 0.14.3"
}

The value for required_version is a string containing a comma-delimited list of constraints. Each constraint is an operator followed by a version number. The following operators are allowed:

Operator	Usage	Example
`=` (or no operator)	Use exact version	`"= 0.14.3"` Must use v0.14.3
`!=`	Version not equal	`"!=0.14.3"` Must not use v0.14.3
`>` or `>=` or `<` or `<=`	Version comparison	`">= 0.14.3"` Must use a version greater than or equal to v0.14.3
`~>`	Pessimistic constraint operator that both both the oldest and newest version allowed	`"~>= 0.14"` Must use a version greater than or equal to v0.14 but less than v0.15 (which includes v0.14.3)

Similarly, a provider version requirement can be specified. The following is an example limited the version of AWS provider:

Source codePlain text


1
2
3
4


provider "aws" {
	region = "us-west-2"
	version = ">=3.1.0"
}

It is recommended to use these operators in production to ensure the correct version is being used and avoid accidental upgrades that might have breaking changes.

3e Explain when to use and not use provisioners and when to use local-exec or remote-exec

Provisioners can be used to model specific actions on the local machine or on a remote machine. For example, a provisioner can enable uploading files, running shell scripts, or installing or triggering other software (e.g. configuration management) to conduct initial setup on an instance. Provisioners are defined within a resource block:

Source codePlain text


1
2
3
4
5
6
7
8


resource "aws_instance" "example" {
 ami = "ami-b374d5a5"
 instance_type = "t2.micro"

 provisioner "local-exec" {
 command = "echo hello > hello.txt"
 }
}

Multiple provisioner blocks can be used to define multiple provisioning steps.

Source codePlain text

1

Note: HashiCorp recommends that provisioners should only be used as a last resort.

Types of Provisioners

This section will cover the various types of generic provisioners. There are also vendor specific provisioners for configuration management tools (e.g. Salt, Puppet).

1. File

The file provisioner is used to copy files or directories from the machine executing Terraform to the newly created resource.

Source codePlain text


1
2
3
4
5
6
7
8


resource "aws_instance" "web" {
 # ...

 provisioner "file" {
 source = "conf/myapp.conf"
 destination = "/etc/myapp.conf"
 }
}

The file provisioner supports both ssh and winrm type connections.

2. `local-exec`

The local-exec provisioner runs by invoking a process local to the user’s machine running Terraform. This is used to do something on the machine running Terraform, not the resource provisioned. For example, a user may want to create an SSH key on the local machine.

Source codePlain text


1
2
3
4
5


resource "null_resource" "generate-sshkey" {
 provisioner "local-exec" {
 command = "yes y | ssh-keygen -b 4096 -t rsa -C 'terraform-kubernetes' -N '' -f ${var.kubernetes_controller.["private_key"]}"
 }
}

3. `remote-exec`

Comparatively, remote-exec which invokes a script or process on a remote resource after it is created. For example, this may be used to bootstrap a newly provisioned cluster or to run a script.

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


resource "aws_instance" "example" {
 key_name = aws_key_pair.example.key_name
 ami = "ami-04590e7389a6e577c"
 instance_type = "t2.micro"

connection {
 type = "ssh"
 user = "ec2-user"
 private_key = file("~/.ssh/terraform")
 host = self.public_ip
 }

provisioner "remote-exec" {
 inline = [
 "sudo amazon-linux-extras enable nginx1.12",
 "sudo yum -y install nginx",
 "sudo systemctl start nginx"
 ]
 }
}

Both SSH and winrm connections are supported.

By default, provisioners are executed when the defined resource is created and during updates or other parts of the lifecycle. It is intended to be used for bootstrapping a system. If a provisioner fails at creation time, the resource is marked as tainted. Terraform will plan to destroy and recreate the tainted resource at the next terraform apply command.

By default, when a provisioner fails, it will also cause the terraform apply command to fail. The on_failure parameter can be used to specify different behavior.

Source codePlain text


1
2
3
4
5
6
7
8


resource "aws_instance" "web" {
 # ...

 provisioner "local-exec" {
 command = "echo The server's IP address is ${self.private_ip}"
 on_failure = "continue"
 }
}

Note: Expressions in provisioner blocks cannot refer to the parent resource by name. Use the self object to represent the provisioner’s parent resource (see previous example).

Additionally, provisioners can also be configured to run when the defined resource is destroyed. This is configured by specifying when = “destroy” within the provisioner block.

Source codePlain text


1
2
3
4
5
6
7
8


resource "aws_instance" "web" {
 # ...

 provisioner "local-exec" {
 when = "destroy"
 command = "echo 'Destroy-time provisioner'"
 }
}

By default, a provisioner only runs at creation. To run a provisioned at deletion, it must be explicitly defined.

4 Use the Terraform CLI (outside of core workflow)

4a Given a scenario: choose when to use `terraform fmt` to format code

4b Given a scenario: choose when to use `terraform taint` to taint Terraform resources

4c Given a scenario: choose when to use `terraform import` to import existing infrastructure into your Terraform state

4d Given a scenario: choose when to use `terraform workspace` to create workspaces

4e Given a scenario: choose when to use `terraform state` to view Terraform state

4f Given a scenario: choose when to enable verbose logging and what the outcome/value is

5 Interact with Terraform modules

5a Contrast module source options

5b Interact with module inputs and outputs

5c Describe variable scope within modules/child modules

5d Discover modules from the public Terraform Module Registry

5e Defining module version

6 Navigate Terraform workflow

6a Describe Terraform workflow ( Write -> Plan -> Create )

6b Initialize a Terraform working directory (`terraform init`)

6c Validate a Terraform configuration (`terraform validate`)

6d Generate and review an execution plan for Terraform (`terraform plan`)

6e Execute changes to infrastructure with Terraform (`terraform apply`)

6f Destroy Terraform managed infrastructure (`terraform destroy`)

7 Implement and maintain state

7a Describe default local backend

7b Outline state locking

7c Handle backend authentication methods

7d Describe remote state storage mechanisms and supported standard backends

7e Describe effect of Terraform refresh on state

7f Describe backend block in configuration and best practices for partial configurations

7g Understand secret management in state files

8 Read, generate, and modify configuration

8a Demonstrate use of variables and outputs

8b Describe secure secret injection best practice

8c Understand the use of collection and structural types

8d Create and differentiate resource and data configuration

8e Use resource addressing and resource parameters to connect resources together

8f Use Terraform built-in functions to write configuration

8g Configure resource using a dynamic block

8h Describe built-in dependency management (order of execution based)

9 Understand Terraform Cloud and Enterprise capabilities

9a Describe the benefits of Sentinel, registry, and workspaces

9b Differentiate OSS and Terraform Cloud workspaces

9c Summarize features of Terraform Cloud

Managing AWS Autoscaling Groups and Load Balancers using Terraform

Mon, 01 Aug 2022 00:00:00 +0000

This lab builds on my first AWS/Terraform example: two subnets, an Application Load Balancer (ALB), and an Auto Scaling Group (ASG) that launches web servers in two Availability Zones.

October 2026 revision: the original subnets were outside the declared VPC CIDR. This version derives them from the VPC, uses a Launch Template instead of a Launch Configuration, and restricts instance HTTP ingress to the ALB security group. AWS recommends migrating to Launch Templates; new accounts cannot create launch configurations.

Scope and prerequisites

This is an HTTP laboratory serving a static, non-sensitive page. It creates billable EC2, ALB and public IPv4 resources. The instances use public subnets for outbound package installation without a NAT gateway, but accept HTTP only from the ALB. There is no SSH ingress or embedded key pair.

For production, design private application subnets, outbound connectivity, HTTPS with an ACM certificate, logging, access management and an application-specific scaling signal. Two Availability Zones provide placement options; actual availability still depends on healthy capacity, application behavior and dependencies.

Use Terraform 1.9 or newer and the AWS provider 6.70.0. Supply an Amazon Linux 2023 x86_64 AMI ID for your selected region. The region must expose at least two usable Availability Zones. Do not substitute an Ubuntu AMI: the bootstrap uses dnf and httpd.

Authenticate using the AWS provider’s normal credential chain, such as an AWS SSO profile. Keep credentials outside the configuration:

Source codebash

1
2

aws sso login --profile sandbox
export AWS_PROFILE=sandbox

The environment-variable syntax above is for Bash. In PowerShell, use $env:AWS_PROFILE = "sandbox".

Complete configuration

Download main.tf into an empty lab directory. terraform init installs the pinned provider and writes .terraform.lock.hcl; commit that lockfile in your lab repository to retain provider checksums.

main.tfhcl · 197 linesDownload
Expand sourceCollapse source

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197


terraform {
 required_version = ">= 1.9, < 2.0"
 required_providers {
 aws = {
 source = "hashicorp/aws"
 version = "6.70.0"
 }
 }
}

provider "aws" {
 region = var.region
}

variable "region" {
 type = string
 default = "us-east-1"
}

variable "ami_id" {
 description = "An Amazon Linux 2023 x86_64 AMI available in the selected region."
 type = string
 validation {
 condition = can(regex("^ami-[0-9a-f]+$", var.ami_id))
 error_message = "Provide an AMI ID for the selected region."
 }
}

data "aws_availability_zones" "available" {
 state = "available"
}

resource "aws_vpc" "lab" {
 cidr_block = "10.10.0.0/16"
 enable_dns_support = true
 enable_dns_hostnames = true
 tags = { Name = "asg-lab" }
}

resource "aws_subnet" "public" {
 count = 2
 vpc_id = aws_vpc.lab.id
 cidr_block = cidrsubnet(aws_vpc.lab.cidr_block, 8, count.index + 1)
 availability_zone = data.aws_availability_zones.available.names[count.index]
 map_public_ip_on_launch = true
}

resource "aws_internet_gateway" "lab" {
 vpc_id = aws_vpc.lab.id
}

resource "aws_route_table" "public" {
 vpc_id = aws_vpc.lab.id
 route {
 cidr_block = "0.0.0.0/0"
 gateway_id = aws_internet_gateway.lab.id
 }
}

resource "aws_route_table_association" "public" {
 count = 2
 subnet_id = aws_subnet.public[count.index].id
 route_table_id = aws_route_table.public.id
}

resource "aws_security_group" "alb" {
 name_prefix = "asg-lab-alb-"
 vpc_id = aws_vpc.lab.id
 ingress {
 from_port = 80
 to_port = 80
 protocol = "tcp"
 cidr_blocks = ["0.0.0.0/0"]
 }
 egress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

resource "aws_security_group" "web" {
 name_prefix = "asg-lab-web-"
 vpc_id = aws_vpc.lab.id
 ingress {
 from_port = 80
 to_port = 80
 protocol = "tcp"
 security_groups = [aws_security_group.alb.id]
 }
 egress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

resource "aws_lb" "web" {
 name = "asg-lab-web"
 load_balancer_type = "application"
 internal = false
 security_groups = [aws_security_group.alb.id]
 subnets = aws_subnet.public[*].id
}

resource "aws_lb_target_group" "web" {
 name = "asg-lab-web"
 port = 80
 protocol = "HTTP"
 vpc_id = aws_vpc.lab.id
 health_check {
 path = "/"
 matcher = "200"
 }
}

resource "aws_lb_listener" "http" {
 load_balancer_arn = aws_lb.web.arn
 port = 80
 protocol = "HTTP"
 default_action {
 type = "forward"
 target_group_arn = aws_lb_target_group.web.arn
 }
}

resource "aws_launch_template" "web" {
 name_prefix = "asg-lab-web-"
 image_id = var.ami_id
 instance_type = "t3.micro"
 vpc_security_group_ids = [aws_security_group.web.id]
 metadata_options {
 http_endpoint = "enabled"
 http_tokens = "required"
 }
 user_data = base64encode(<<-USERDATA
 #!/bin/bash
 set -euo pipefail
 dnf install -y httpd
 printf 'ASG lab: %s\n' "$(hostname)" > /var/www/html/index.html
 systemctl enable --now httpd
 USERDATA
 )
}

resource "aws_autoscaling_group" "web" {
 name = "asg-lab-web"
 min_size = 2
 desired_capacity = 2
 max_size = 4
 health_check_type = "ELB"
 health_check_grace_period = 300
 default_instance_warmup = 120
 vpc_zone_identifier = aws_subnet.public[*].id
 target_group_arns = [aws_lb_target_group.web.arn]

 launch_template {
 id = aws_launch_template.web.id
 version = tostring(aws_launch_template.web.latest_version)
 }

 instance_refresh {
 strategy = "Rolling"
 preferences {
 min_healthy_percentage = 100
 max_healthy_percentage = 200
 }
 }

 lifecycle {
 ignore_changes = [desired_capacity]
 }
 tag {
 key = "Name"
 value = "asg-lab-web"
 propagate_at_launch = true
 }
 depends_on = [aws_route_table_association.public, aws_lb_listener.http]
}

resource "aws_autoscaling_policy" "cpu" {
 name = "asg-lab-cpu"
 policy_type = "TargetTrackingScaling"
 autoscaling_group_name = aws_autoscaling_group.web.name
 target_tracking_configuration {
 predefined_metric_specification {
 predefined_metric_type = "ASGAverageCPUUtilization"
 }
 target_value = 50
 }
}

output "lab_url" {
 value = "http://${aws_lb.web.dns_name}"
}

How the pieces fit

The VPC is 10.10.0.0/16. cidrsubnet derives 10.10.1.0/24 and 10.10.2.0/24, keeping both subnets inside it. A default route through the internet gateway supplies the lab’s outbound connectivity.

The ALB accepts HTTP and forwards to its target group. The ASG registers its instances with that target group; the instance security group accepts port 80 only from the ALB security group. User data installs Apache and writes the hostname into the page. IMDSv2 is required, although this bootstrap does not need to query instance metadata.

The ASG starts with two instances and a range of two to four. Target tracking adjusts capacity toward 50% average CPU utilization and manages its own alarms. ignore_changes = [desired_capacity] assigns running capacity to autoscaling after initial creation; Terraform still manages the minimum and maximum bounds.

The ASG references a numeric Launch Template version, so changing the template is visible to Terraform and can trigger an instance refresh. Refresh preferences retain healthy capacity while replacements start. This requires spare capacity and sufficient quotas; it is not a guarantee of uninterrupted application service. Check refresh status and application traffic during any rollout.

Validate, plan and verify

Source codebash


1
2
3
4


terraform init
terraform fmt -check
terraform validate
terraform plan -var="ami_id=ami-REPLACE_WITH_YOUR_AL2023_AMI" -out=lab.tfplan

Replace the illustrative AMI ID before planning. Review account, region, subnet placement, instance counts and costs in the saved plan. To provision this lab after that review:

Source codebash

1
2

terraform apply lab.tfplan
terraform output -raw lab_url

Check target-group health, fetch the page through the ALB, and observe ASG activity and CloudWatch metrics before assessing scaling. A format/schema check cannot prove that an AMI boots or that your account has sufficient quota. The configuration passed terraform validate with Terraform 1.9.8 and AWS provider 6.70.0. It was not deployed into an AWS account.

Clean up when finished:

Source codebash

1
2

terraform plan -destroy -var="ami_id=ami-REPLACE_WITH_YOUR_AL2023_AMI" -out=destroy.tfplan
terraform apply destroy.tfplan

Keep state and plan files private: they may contain sensitive infrastructure information. Review the destroy plan just as carefully as the creation plan.

References

Basic Dev Environment on AWS using Terraform

Sat, 30 Jul 2022 00:00:00 +0000

It’s good to have a basic terraform code that deploys a basic environment on AWS whenever you need to run some quick tests on a free tier account.

We’ll have the following resources deployed with this procedure:

1 VPC
1 Subnet
1 Internet Gateway
1 Route Table (with a default route and an association)
1 Security group
1 Key Pair
1 EC2 Instance

It’s a very simple and you can build on top of it according to your needs, like adding a RDS instance or more EC2 instances, changing the instance type and so on. The code will be organized in the following structure:

main.tf: resource declaration
providers.tf: aws provider configuration
outputs.tf: output declaration
datasources.tf: we’ll get the AMI information here
variables.tf: only to inform our OS
terraform.tfvars: same as previous
userdata.tpl: script to be executed on first boot
linux-ssh-config.tpl: SSH config for Linux
windows-ssh-config.tpl: SSH config for Windows

I added a local-exec provisioner that adds the host information in the ssh configuration, and with that we can then use it with VS Code and Remote SSH extension to connect directly to it and run our code.

I’ll put the code first and then explain the most important parts. Let’s start with providers.tf:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


terraform {
 required_providers {
 aws = {
 source = "hashicorp/aws"
 version = "~> 3.0"
 }
 }
}

provider "aws" {
 shared_credentials_file = "/home/mando/.aws/credentials"
 profile = "mando"
 region = "us-east-1"
}

Terraform requires credentials to access your account on AWS. You can choose different approaches for that, like putting the keys directly instad of pointing to a credentials file. Although that works and might even be easier, you’ll probably have your code pushed into a git repository which would then expose the keys. More information on how to configure the provider can be found here -> Docs overview | hashicorp/aws.

Now for the main.tf:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104


resource "aws_vpc" "mando_vpc" {
 cidr_block = "10.10.0.0/16"
 enable_dns_hostnames = true
 enable_dns_support = true

 tags = {
 Name = "dev"
 }

}

resource "aws_subnet" "mando_public_subnet" {
 vpc_id = aws_vpc.mando_vpc.id
 cidr_block = "10.10.1.0/24"
 map_public_ip_on_launch = true
 availability_zone = "us-east-1b"

 tags = {
 Name = "dev-public"
 }
}

resource "aws_internet_gateway" "mando_internet_gw" {
 vpc_id = aws_vpc.mando_vpc.id

 tags = {
 Name = "dev-igw"
 }
}

resource "aws_route_table" "mando_public_rt" {
 vpc_id = aws_vpc.mando_vpc.id

 tags = {
 Name = "dev-public-rt"
 }
}

resource "aws_route" "mando_default_route" {
 route_table_id = aws_route_table.mando_public_rt.id
 destination_cidr_block = "0.0.0.0/0"
 gateway_id = aws_internet_gateway.mando_internet_gw.id

}

resource "aws_main_route_table_association" "mando_public_assoc" {
 vpc_id = aws_vpc.mando_vpc.id
 route_table_id = aws_route_table.mando_public_rt.id
}

resource "aws_security_group" "mando_sg" {
 name = "mando-sg"
 description = "mando Security Group"
 vpc_id = aws_vpc.mando_vpc.id

 ingress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["200.199.198.197/32"] # Replace with your public IP
 }

 egress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
 tags = {
 Name = "mando-sg"
 }
}

resource "aws_key_pair" "mando_key" {
 key_name = "mando-key"
 public_key = "ssh-rsa AAAABACH3L0Lc2EAAAADAQAPNCDAgQDELie/jIMM8uno12enId2YTmTjK1OGZJtTJFoSPdXIwn79qpZYQ3WXL8PlI/8dqFyGXvQj5bGJbgEydjSYVHFXFhPr4sdKcjguWbu895EjK2DgalcYuC1+6jBbFxiodoObsc+84m81+BACH3L0LQU3cm/rNKufrh6d21jIe4sQVul+WzJ9E8aPk34rPmRPgjYvh1T/P2hdgiUyJmKqOtDYwpokDRad+3W+iwGfoBACH3L0LoCWJ2rYzz6j80FKoiHm9cnSXvErezT7aAdenVzY3nEE4ylnHWVUdmzXN7IbCSLsDV3sdn0+c5E6oDX2/k1VwtSQ8TrUblM7AdpuB4ADniUSYvLqjd/NBIiHODzV6qZxXqoltVTsrTpbCWf1A063PBACH3L0L/F3mxBihWRAKfD1iqqfMXmYvAPosOkJ3u1yuwy/eCi6Q3SmA5n0vBSVKmYdUB9yQdAimWcUqabRzXLz+g8BrUxCBHwOf4+IZAp2AseJeoDQs0aqMwybr/k= mando" # replace with your key
}

resource "aws_instance" "mando_node" {
 ami = data.aws_ami.server_ami.id
 instance_type = "t2.micro"
 key_name = aws_key_pair.mando_key.id
 vpc_security_group_ids = [aws_security_group.mando_sg.id]
 subnet_id = aws_subnet.mando_public_subnet.id
 user_data = file("userdata.tpl")

 root_block_device {
 volume_size = 10
 }

 provisioner "local-exec" {
 command = templatefile("${var.host_os}-ssh-config.tpl", {
 hostname = self.public_ip,
 user = "ubuntu",
 identityfile = "~/.ssh/id_rsa"
 })
 interpreter = var.host_os == "windows" ? ["powershell", "-Command"] : ["bash", "-c"]

 }

 tags = {
 Name = "mando-node"
 }
}

We’ll not be using the default VPC or any other that was previously deployed, so we start by creating a VPC, a subnet and an Internet Gateway. We then have to create the route table, the default route and associate it with our VPC.

After that, we create a security group and configure it to allow TCP access from our public IP. This of course can be adjusted according to specific needs, so I used a simple example that is not restrictive regarding what can be accessed, but very restrictive in the source IP. This prevents any access from outside, but might stop you from working with more people or from different places, and you’ll have to adjust the SG whenever your public ip changes.

Then we create a key pair, and here you can use an existing one or generate a specific for this case, and you can also point to a file instead of pasting the key on the code.

And last, but not least important, we specify an EC2 instance. The instance type is one that falls into free tier, and the root_block_device block is used to change the default size to 10. This can also be changed accordingly.

datasources.tf:

Source codePlain text


1
2
3
4
5
6
7
8
9


data "aws_ami" "server_ami" {
 most_recent = true
 owners = ["099720109477"]

 filter {
 name = "name"
 values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
 }
}

We use a data block to determine the AMI ID, since it changes between regions. We also use a filter block with a * at the end of the name, so we always pick the latest version. This can be changed when you want to lock in a specific version, which would be a best practice in production environments. Since we’re just testing, we might use the updated image with the latest security fixes.

variables.tf:

Source codePlain text


1
2
3


variable "host_os" {
 type = string
}

terraform.tfvars:

Source codePlain text

1

host_os = "linux"

A single variable is declared and are used just for informing which OS we use (Windows or Linux).

outputs.tf:

Source codePlain text


1
2
3


output "dev_ip" {
 value = aws_instance.mando_node.public_ip
}

The only output we have is the public IP address of the EC2 instance, which we’ll need to access it.

userdata.tpl:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


#!/bin/bash

sudo apt -y update &&
sudo apt -y install \
 apt-transport-https \
 software-properties-common \
 ca-certificates \
 curl \
 gnupg \
 lsb-release &&

sudo mkdir -p /etc/apt/keyrings &&
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg &&

echo \
 "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null &&

sudo apt -y update &&
sudo apt -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin &&
sudo groupadd docker &&
newgrp docker &&
sudo usermod -aG docker ubuntu

This userdata will install docker on the EC2 instance.

windows-ssh-config.tpl:

Source codePlain text


1
2
3
4
5
6
7


add-content -path c:/users/fabio/.ssh/config -value @'

Host ${hostname}
 HostName ${hostname}
 User ${user}
 IdentityFile ${identityfile}
'@

linux-ssh-config.tpl:

Source codePlain text


1
2
3
4
5
6
7


cat << EOF >> ~/.ssh/config

Host ${hostname}
 HostName ${hostname}
 User ${user}
 IdentityFile ${identityfile}
EOF

These two template files - windows-ssh-config.tpl and linux-ssh-config.tpl - are used by the local-exec provisioner to add the host information in our ssh configuration. This will make it easier to access it from VS Code.

After you apply this code, you can then proceed to configure Remote SSH on VS Code and use it as a remote dev node. I’ll not cover here how to do this since the focus is the Terraform itself, and because it is easily done.

Getting started with Jenkins Declarative Pipelines

Mon, 18 Oct 2021 00:00:00 +0000

After hours and hours of training, videos, documentation, and tutorials, it was time to put into practice the concepts I learned about Jenkins, one of the most used automation tools for CI/CD.

It is known that there are many other more modern solutions today, like GitHub Actions or Gitlab CI, for instance, which is a common approach since you can leverage the SCM directly to automate your tasks instead of depending on an external tool. However, Jenkins is still widely adopted and you can perform many automation tasks without much effort using it. It’s all going to depend on the situation, the current project, and the team. Traditionally, Jenkins jobs were created using the Jenkins UI and were called FreeStyle jobs. In Jenkins 2.0, a new way was introduced using a technique called pipeline as code, where jobs are created using a script file containing the steps to be executed. That scripted file is called Jenkinsfile. That’s just a glimpse on Jenkins pipelines, there’s much more depth to it and you can always dive deeper using the official documentation at https://www.jenkins.io/doc/book/pipeline/.

Before we start to get into the real stuff, let me just mention a reference that inevitably came into my mind when I first read the name “Jenkins”, which I’m sure that most gamers from early 2000s are gonna remember:

If you’d like to watch the video, here’s the YouTube link to it: Leeroy Jenkins HD 1080p

Alright, let’s start with the basics.

1. What’s a CI/CD Pipeline?

A CI/CD pipeline is usually described as the processes and stages of automation on software delivery. There’s usually a specific tool used to achieve this, that orchestrates with SCM and the other tools. It can be used to build code, run unit tests, smoke tests, call external tools like SonarQube, Anchore, Trivy, build Docker images and deploy to environments like Kubernetes or wherever the application is delivered. By performing standardized operations and tests, a CI/CD pipeline can help reduce manual errors, provide feedback to developers, and allow fast iterations.

2. What’s a Jenkinsfile?

A Jenkinsfile is just a text file, usually checked in along with the project’s source code in an SCM. Ideally, each application will have its Jenkinsfile. A Jenkinsfile can be written in two ways - “scripted pipeline syntax” or “declarative pipeline syntax”.

3. What’s a Jenkins Scripted Pipeline?

Scripted pipelines run on the Jenkins master with the help of a lightweight executor. It uses very few resources to translate the pipeline into atomic commands. Both declarative and scripted syntax are different from each other and are defined differently. Scripted pipelines are written in Groovy.

It requires knowledge of the Groovy language;
The Jenkinsfile starts with the word ’node';
Advanced cabapilities;
Groovy engine;
Can contain standard programming constructs like if-else block, try-catch block, etc.

An example of a Scripted Pipeline:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


node {
 stage('Build') {
 if (env.BRANCH_NAME == 'master') {
 echo "I'm on master brand, building application"
 bat "msbuild ${C:\\Jenkins\\my_project\\workspace\\test\\my_project.sln}"
 } else {
 echo 'Towards the center of the Cloud Computer.'
 }
 }
 stage('Selenium') {
 echo 'Running Selenium Smoke Tests'
 dir(bachelol) { //changes the path to “bachelol”
 bat "mvn clean test -Dsuite=SMOKE_TEST -Denvironment=QA"
 } 
 }
}

4. What’s a Jenkins Declarative Pipeline?

The Declarative Pipeline is relatively new and provides a simplified, opinionated syntax on top of the Pipeline subsystems. Its syntax offers an easy way to create pipelines. It contains a predefined hierarchy and gives you the ability to control all aspects of a pipeline execution in a simple, straightforward manner.

The latest addition in Jenkins pipeline job creation technique;
Needs to use the predefined constructs to create pipelines; Hence, it is not flexible as a scripted pipeline;
The Jenkinsfile starts with the word ‘pipeline’, and there is a predefined structure;
Usually, the preferred way to start, as they offer a rich set of features, come with less learning curve & no prerequisite to learn a programming language like Groovy just for the sake of writing pipeline code;
We can also validate the syntax of the Declarative pipeline code before running the job. It helps to avoid a lot of runtime issues with the build script.

5. Why run Jenkins on Docker?

Despite the inherent advantages (and caveats) of running anything in Docker, the main reasons I could think of are:

Ability to keep the server configuration under version control;
Run multiple copies of the server anywhere;
Integrate with Kubernetes and other orchestration platforms;
Jenkins’ official Docker image is widely adopted and maintained;
Simple implementation = simple administration.

The main (Drawback|Caveat|Problem|Issue|Flaw) is when you want to use agents with docker too. To do that you’re gonna need some implementation of Docker in Docker. There are a few ways to do that, but the two easiest ways are connecting the Jenkins container directly to the host’s Docker socket using -v /var/run/docker.sock:/var/run/docker.sock and changing the permissions, or enabling TCP on the Docker server-side.
If you can run your Jenkins server on Kubernetes, you can enable it to launch new agent pods inside the cluster which gives you even more flexibility. I’ll try that in the future since I’m out of credits on AWS and my dev cluster is behind a proxy, which is a real pain to deal with.

6. Building a Jenkins Image

The Dockerfile below creates an image based on the latest Jenkins LTS from Dockerhub, and preinstalls the plugins needed:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


FROM jenkins/jenkins:lts

# Skip setup wizard
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false"

# Set docker group
ARG DOCKER_GID=998

# Get plugins
RUN /usr/local/bin/install-plugins.sh \
 workflow-multibranch:latest \
 pipeline-model-definition:latest \
 pipeline-stage-view:latest \
 git:latest \
 credentials:latest \
 slack:latest

# Install Docker & docker-compose
USER root

RUN apt-get update && \
 apt-get -y install apt-transport-https \
 ca-certificates \
 curl \
 gnupg2 \
 software-properties-common && \
 curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && \
 add-apt-repository \
 "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
 $(lsb_release -cs) \
 stable" && \
 apt-get update && \
 groupadd -g ${DOCKER_GID} docker && \
 apt-get -y install docker-ce docker-ce-cli containerd.io

RUN curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose

RUN usermod -aG docker jenkins 

USER jenkins

I referred to the official documentation:
https://docs.docker.com/compose/install/
https://docs.docker.com/engine/install/ubuntu/

To spin up a Jenkins instance on my local machine, I used a docker-compose.yml file and put everything on the same folder. The jenkins service points to a dockerfile on the same path, and the container will be accessible on localhost:9080. I mapped the docker.sock to the container and matched the docker GID with the host, and also created and mapped a jenkins_home folder locally to persist data:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


version: "3.8"
services:
 jenkins:
 build:
 context: .
 dockerfile: Dockerfile
 ports:
 - "127.0.0.1:9080:8080"
 volumes:
 - ./jenkins_home:/var/jenkins_home
 - /var/run/docker.sock:/var/run/docker.sock
 restart: unless-stopped

Now that we have a running Jenkins instance with the plugins installed and with access to the host’s dockerd, let’s start using it to automate a build.

7. The Spring PetClinic Sample Application

To test everything, I downloaded the “Spring PetClinic Sample Application” (can be found here: https://github.com/spring-projects/spring-petclinic), which is written in Java using Spring Boot and built with Maven. If you clone the official repository, you’ll get a Maven Wrapper bundled together that can be used to build the application. So before trying to automate anything with Jenkins, let’s first build, test, and run the application manually:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Clone the repo
git clone https://github.com/spring-projects/spring-petclinic.git

# Enter the directory
cd spring-petclinic

#Build the application
./mvnw package

#Run the application
java -jar target/*.jar

The first time I ran the mvnw package command, it took a while to finish since it downloaded all the dependencies. After that, subsequent runs took around a minute to finish. That’s pretty much all we have to do to test if it builds and runs properly, I just followed the instructions from the repository without changing anything. If it runs, we know that the code should work inside the pipeline too. The changes will come when we use the Maven from Jenkins and try to run JUnit tests.

Output from the build:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10


[INFO] Building jar: ./github/spring-petclinic/target/spring-petclinic-2.5.0-SNAPSHOT.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.5.4:repackage (repackage) @ spring-petclinic ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:15 min
[INFO] Finished at: 2021-10-19T11:35:43-03:00
[INFO] ------------------------------------------------------------------------

Output from java -jar target/*.jar:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


java -jar target/*.jar


 |\ _,,,--,,_
 /,`.-'`' ._ \-;;,_
 _______ __|,4- ) )_ .;.(__`'-'__ ___ __ _ ___ _______
 | | '---''(_/._)-'(_\_) | | | | | | | | |
 | _ | ___|_ _| | | | | |_| | | | __ _ _
 | |_| | |___ | | | | | | | | | | \ \ \ \
 | ___| ___| | | | _| |___| | _ | | _| \ \ \ \
 | | | |___ | | | |_| | | | | | | |_ ) ) ) )
 |___| |_______| |___| |_______|_______|___|_| |__|___|_______| / / / /
 ==================================================================/_/_/_/

:: Built with Spring Boot :: 2.5.4


2021-10-19 11:38:31.201 INFO 39806 --- [ main] o.s.s.petclinic.PetClinicApplication : Starting PetClinicApplication v2.5.0-SNAPSHOT using Java 11.0.11 on my-machine with PID 39806 (./github/spring-petclinic/target/spring-petclinic-2.5.0-SNAPSHOT.jar started by biofa in ./github/spring-petclinic)
2021-10-19 11:38:31.204 INFO 39806 --- [ main] o.s.s.petclinic.PetClinicApplication : No active profile set, falling back to default profiles: default
2021-10-19 11:38:32.230 INFO 39806 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2021-10-19 11:38:32.288 INFO 39806 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 50 ms. Found 4 JPA repository interfaces.
2021-10-19 11:38:33.020 INFO 39806 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-10-19 11:38:33.036 INFO 39806 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2021-10-19 11:38:33.037 INFO 39806 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.52]
2021-10-19 11:38:33.128 INFO 39806 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2021-10-19 11:38:33.128 INFO 39806 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1865 ms
2021-10-19 11:38:33.377 INFO 39806 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-10-19 11:38:33.590 INFO 39806 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-10-19 11:38:33.919 INFO 39806 --- [ main] org.ehcache.core.EhcacheManager : Cache 'vets' created in EhcacheManager.
2021-10-19 11:38:33.934 INFO 39806 --- [ main] org.ehcache.jsr107.Eh107CacheManager : Registering Ehcache MBean javax.cache:type=CacheStatistics,CacheManager=urn.X-ehcache.jsr107-default-config,Cache=vets
2021-10-19 11:38:33.941 INFO 39806 --- [ main] org.ehcache.jsr107.Eh107CacheManager : Registering Ehcache MBean javax.cache:type=CacheStatistics,CacheManager=urn.X-ehcache.jsr107-default-config,Cache=vets
2021-10-19 11:38:34.021 INFO 39806 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
2021-10-19 11:38:34.081 INFO 39806 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.32.Final
2021-10-19 11:38:34.204 INFO 39806 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2021-10-19 11:38:34.332 INFO 39806 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2021-10-19 11:38:34.955 INFO 39806 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2021-10-19 11:38:34.962 INFO 39806 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2021-10-19 11:38:36.371 INFO 39806 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 13 endpoint(s) beneath base path '/actuator'
2021-10-19 11:38:36.441 INFO 39806 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2021-10-19 11:38:36.453 INFO 39806 --- [ main] o.s.s.petclinic.PetClinicApplication : Started PetClinicApplication in 5.753 seconds (JVM running for 6.136)

8. Using Maven on Jenkins

There are a couple of different ways to use Maven with Jenkins. There is a Maven plugin for Jenkins, and we can install Maven on the Jenkins server and set it up at “Global Tool Configuration”. Since we want to use docker containers as executors or agents, that would not be an option. We could also use the wrapper that comes with the project, but a better and simple route would be to use the official Maven docker image, and that way we can have an agent always updated and completely separate Maven from our Jenkins server. This approach makes it easier to manage the environment and also allows for a single Jenkins server to work with different types of builds with a single installation, without having to embed every tool used on the server.

I built a pipeline with six stages: Init, Build, Test and Build Image, Publish Image and Prod. Some of the main aspects are:

It mounts the m2 directory, which is used by maven, locally to have cache between builds, and also settings.xml to enable setting custom parameters to Maven;
Maven 3.8.3 with AdoptOpenJDK 11 Docker Image is used as an agent only for the stages that require it;
Dockerhub credentials were added previously on Jenkins.

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86


pipeline {
 agent none
 environment {
 DEPLOY_TO = 'staging'
 imageName = "fabioctba/spring-petclinic"
 registryCredential = 'dockerhub'
 dockerImage = ''
 }
 stages {
 stage('Init'){
 agent any
 steps {
 slackSend channel: '#jenkins', message: "${env.BUILD_ID} on ${env.JENKINS_URL} - Starting"
 sh 'git --version'
 echo "Deploying to ${DEPLOY_TO}"
 }
 }
 stage('Build') {
 agent {
 docker {
 image 'maven:3.8.3-adoptopenjdk-11'
 args '-v m2:/root/.m2 -v settings.xml:/root/.m2/settings.xml'
 }
 }
 steps {
 slackSend channel: '#jenkins', message: "${env.BUILD_ID} on ${env.JENKINS_URL} - Building"
 sh 'mvn -B -DskipTests clean package'

 }
 }
 stage('Test') {
 agent {
 docker {
 image 'maven:3.8.3-adoptopenjdk-11'
 args '-v m2:/root/.m2 -v settings.xml:/root/.m2/settings.xml'
 }
 }
 steps {
 slackSend channel: '#jenkins', message: "${env.BUILD_ID} on ${env.JENKINS_URL} - Running Tests"
 sh 'mvn test' 
 }
 post {
 always {
 archiveArtifacts artifacts: 'target/*.jar'
 junit 'target/surefire-reports/*.xml' 
 }
 }
 }
 stage('Build Image') { 
 agent any
 steps {
 slackSend channel: '#jenkins', message: "${env.BUILD_ID} on ${env.JENKINS_URL} - Building Image"

 script {
 dockerImage = docker.build imageName
 }
 }
 }
 stage('Publish Image') { 
 agent any
 steps {
 slackSend channel: '#jenkins', message: "${env.BUILD_ID} on ${env.JENKINS_URL} - Building Image"

 script {
 docker.withRegistry( '', registryCredential ) {
 dockerImage.push("$BUILD_NUMBER")
 dockerImage.push('latest')
 }
 }
 }
 }
 stage('Prod') {
 agent any
 when { 
 allOf { 
 branch 'master'; 
 environment name: 'DEPLOY_TO', value: 'production'
 } 
 }
 steps {
 slackSend channel: '#jenkins', message: "${env.BUILD_ID} on ${env.JENKINS_URL} - Running Production stage"
 echo 'Some extra step when on production release'
 }
 }
 }
}

I used the Slack Plugin to integrate with Jenkins, following this documentation. I used only two variables just to illustrate what can be sent as a message, but a full list of environment variables can be found directly on Jenkins accessing http://localhost:9080/env-vars.html. Of course, you have to change localhost:9080 to your server’s address.

The Init stage prints the versions of the main tools used, useful to debug problems;

The Build stage runs mvn package, which creates the jar files;

The Test stage run the Unit tests. Spring Pet Clinic has a total of 40 tests configured, and that would depend on how the dev team created their tests;

The Build Image stage creates a Docker image;

The Publish Image stage publishes the image to Dockerhub;

The Prod stage doesn’t do anything, it’s just an example of a conditional step that can be executed depending on a variable.

Notice that there isn’t a stage to clone the repository at the beginning; this stage is implicit when you configure Jenkins to pull the pipeline from SCM, which is the plan here.

Here’s the Dockerfile I used to publish the application:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


FROM openjdk:11-jre-slim

RUN groupadd --gid 1000 java \
 && useradd --uid 1000 --gid java --shell /bin/bash --create-home java

USER java

VOLUME /tmp

WORKDIR /app

COPY --chown=java:java ./target/*.jar /app/

CMD ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app/*.jar"]

That is a working pipeline, and I wanted to include most of the concepts I saw in the courses. There are other features I’d like to try in the future like shared libraries and multi-branch pipelines, but this might go into a different post.

a. Jenkins, From Zero To Hero: Become a DevOps Jenkins Master

Udemy, by Ricardo Andre Gonzalez Gomez, 2018
https://www.udemy.com/course/jenkins-from-zero-to-hero/

I found this course a little outdated and looks like the author stopped maintaining it in 2020. However, if you never used Jenkins before and want a cheap course to start, I strongly recommend this one. It’s also well recommended in many other sources and has a high rate at Udemy. Ricardo focuses a lot on Docker, so it might be better targeted to someone without any experience in Docker also.

b. Continuous Integration with Jenkins

Pluralsight, by different authors, updated constantly
https://app.pluralsight.com/paths/skill/continuous-integration-with-jenkins

Pluralsight has a total of 8 courses, each with around 2 hours of duration, which makes the Jenkins Skill Path. I took four of them during the last Pluralsight Free Week and like other courses I took from Pluralsight, the quality was much superior. Unfortunately, Pluralsight costs a great amount of money, so I usually stay on the lookout for their periodically free weeks and free weekends. The good thing is that you can download the course material and most of the authors have GitHub repositories where you can check the code later.

10. References

https://stackoverflow.com/questions/44440164/what-are-the-advantages-of-running-jenkins-in-a-docker-container

https://www.cinqict.nl/blog/building-a-jenkins-development-docker-image

https://tomgregory.com/building-a-spring-boot-application-in-jenkins/

If you read it to the end, I’d like to hear your comments, and hope this helped you in some way. Cheers.

.
.
.
.

And sorry for the long post... here's a potato.

Source: @truth.potato

Using Python and Docker to monitor Ceph S3 endpoints externally

Mon, 20 Sep 2021 00:00:00 +0000

I needed to perform external monitoring on our S3 endpoints to better observe how the service was performing, thus getting a better representation of user experience. I use basically Python and Docker to accomplish the task.

We have two relatively large Ceph clusters (2.5PB, 600 OSDs) at the company that provides object storage using implementing the S3 API, so we can leverage the AWS SDK and use it to perform operations like creating buckets, putting objects and so on, and measuring the success and the time consumed to perform each operation. With those metrics we can then create graphics, panels and alerts using Grafana for instance. Using some kind of KISS methodology (keep it simple, stupid), running everything inside a docker container would give the flexibility to throw it anywhere and get the same results. The initial goal was to run two containers on AWS pointing to both sites (where the two main Ceph Clusters live) and two others on each site, one pointing to the other.

For the task I used Python as the programming language with the Prometheus Client for Python, found at https://github.com/prometheus/client_python. I divided the code in sections to talk about each part.

Libraries

prometheus_client for metrics, boto3 for s3 connection, random

Source codePlain text


1
2
3
4
5
6
7


from prometheus_client import Histogram, start_http_server, Summary, Gauge, Counter
import boto3
import random
from io import BytesIO
import os
import time
import logging

Variables

Here we define our endpoint address, credentials, name of the bucket and objects, the interval between tests, the histogram buckets and how many objects we want to put simultaneously (for histogram metrics):

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10


CLUSTER_ENDPOINT = ''
ACCESS_KEY = ''
SECRET_ACCESS_KEY = ''
BUCKET_NAME = 'the_lord_of_buckets'
OBJECT_NAME = 'the_ruler'
PORT=9102
TIME_TO_SLEEP = 120
LOG_LEVEL=logging.INFO
BUCKETS=[0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95 ,1]
OBJCOUNT=15

Metrics

Here we define what metrics we want to generate, or collect: ALL_TESTSm S3_FAIL, CREATE, PUT, DELETE, GET_OBJECT, LIST_OBJECTS, REMOVE_BUCKET, DELETE_OBJECTS. For each operation there are three metrics defined: Summary, Gauge and Histogram. Labels were used for each operation.

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


ALL_TESTS_SUMMARY = Summary('s3_request_duration_seconds', 'Time spent running the test',['task'])
ALL_TESTS_GAUGE = Gauge('s3_request_last_duration_seconds','Time spent on the last test',['task'])
ALL_TESTS_HIST = Histogram('s3_request_duration_seconds_histogram','Time spent on the last test',['task'],buckets=BUCKETS)
S3_FAIL = Counter('s3_requests_fails', 'Number of fails in this process')
CREATE_BUCKET_SUMMARY = ALL_TESTS_SUMMARY.labels('create')
CREATE_BUCKET_GAUGE = ALL_TESTS_GAUGE.labels('create')
CREATE_BUCKET_HIST = ALL_TESTS_HIST.labels('create')
PUT_OBJECT_SUMMARY = ALL_TESTS_SUMMARY.labels('put')
PUT_OBJECT_GAUGE = ALL_TESTS_GAUGE.labels('put')
PUT_OBJECT_HIST = ALL_TESTS_HIST.labels('put')
DELETE_OBJECT_SUMMARY = ALL_TESTS_SUMMARY.labels('delete')
DELETE_OBJECT_GAUGE = ALL_TESTS_GAUGE.labels('delete')
DELETE_OBJECT_HIST = ALL_TESTS_HIST.labels('delete')
GET_OBJECT_SUMMARY = ALL_TESTS_SUMMARY.labels('get')
GET_OBJECT_GAUGE = ALL_TESTS_GAUGE.labels('get')
GET_OBJECT_HIST = ALL_TESTS_HIST.labels('get')
LIST_OBJECTS_SUMMARY = ALL_TESTS_SUMMARY.labels('list_objects')
LIST_OBJECTS_GAUGE = ALL_TESTS_GAUGE.labels('list_objects')
LIST_OBJECTS_HIST = ALL_TESTS_HIST.labels('list_objects')
LIST_BUCKETS_SUMMARY = ALL_TESTS_SUMMARY.labels('list_buckets')
LIST_BUCKETS_GAUGE = ALL_TESTS_GAUGE.labels('list_buckets')
LIST_BUCKETS_HIST = ALL_TESTS_HIST.labels('list_buckets')
REMOVE_BUCKET_SUMMARY = ALL_TESTS_SUMMARY.labels('remove')
REMOVE_BUCKET_GAUGE = ALL_TESTS_GAUGE.labels('remove')
REMOVE_BUCKET_HIST = ALL_TESTS_HIST.labels('remove')
DELETE_OBJECTS_SUMMARY = ALL_TESTS_SUMMARY.labels('delete_many')
DELETE_OBJECTS_GAUGE = ALL_TESTS_GAUGE.labels('delete_many')
DELETE_OBJECTS_HIST = ALL_TESTS_HIST.labels('delete_many')

Class S3Metrics

This is where stuff gets done! There is an init function to set the parameters from the variables, start logging and create an a s3 resource from the boto3 library. Decorators were used to measure the time spend in each function and the run_all_tests funcion calls all the tests, which is the one we use as we’ll see next.

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139


class S3Metrics:
 s3 = None
 bucket = None
 file_size = 512
 data = None
 key_name = None
 data_out = None

 def __init__(self, bucket_name, key_name):
 cluster_endpoint = os.getenv('CLUSTER_ENDPOINT',default=CLUSTER_ENDPOINT)
 access_key = os.getenv('ACCESS_KEY',default=ACCESS_KEY)
 secret_access_key = os.getenv('SECRET_ACCESS_KEY',default=SECRET_ACCESS_KEY)
 log_level = os.getenv('LOG_LEVEL',default=LOG_LEVEL)
 self.bucket = bucket_name
 self.key_name = key_name
 self.generate_data()
 logging.basicConfig(level=int(log_level))
 self.s3 = boto3.resource('s3',
 endpoint_url=cluster_endpoint,
 aws_access_key_id=access_key,
 aws_secret_access_key=secret_access_key)

 @CREATE_BUCKET_HIST.time()
 @CREATE_BUCKET_GAUGE.time()
 @CREATE_BUCKET_SUMMARY.time()
 def create_bucket(self):
 try:
 bucket = self.s3.Bucket(self.bucket)
 bucket.create()
 except:
 S3_FAIL.inc()

 @PUT_OBJECT_HIST.time()
 @PUT_OBJECT_GAUGE.time()
 @PUT_OBJECT_SUMMARY.time()
 def put_object(self,key_name=OBJECT_NAME):
 try:
 bucket = self.s3.Bucket(self.bucket)
 bucket.put_object(
 Key=key_name,
 Body=self.data)
 except:
 S3_FAIL.inc()
 
 def put_objects(self):
 for i in range(0,OBJCOUNT):
 self.put_object(self.key_name+str(i))

 @DELETE_OBJECT_HIST.time()
 @DELETE_OBJECT_GAUGE.time()
 @DELETE_OBJECT_SUMMARY.time()
 def del_object(self):
 try:
 object = self.s3.Object(self.bucket,self.key_name)
 object.delete()
 except:
 S3_FAIL.inc()

 @DELETE_OBJECTS_HIST.time()
 @DELETE_OBJECTS_GAUGE.time()
 @DELETE_OBJECTS_SUMMARY.time()
 def del_objects(self):
 try:
 bucket = self.s3.Bucket(self.bucket)
 objects = bucket.objects.all()
 for object in objects:
 object.delete()
 except:
 S3_FAIL.inc()

 @GET_OBJECT_HIST.time()
 @GET_OBJECT_GAUGE.time()
 @GET_OBJECT_SUMMARY.time()
 def get_object(self):
 self.data_out = None
 try:
 bucket = self.s3.Bucket(self.bucket)
 data = BytesIO()
 bucket.download_fileobj(Fileobj=data, Key=self.key_name)
 self.data_out = data.getvalue().decode()
 except:
 S3_FAIL.inc()

 def compare_object(self):
 if self.data != self.data_out:
 S3_FAIL.inc()

 @LIST_BUCKETS_HIST.time()
 @LIST_BUCKETS_GAUGE.time()
 @LIST_BUCKETS_SUMMARY.time()
 def list_buckets(self):
 try:
 all_buckets = self.s3.buckets.all()
 for bucket in all_buckets:
 logging.debug("%s %s", bucket.creation_date, bucket.name)
 except:
 S3_FAIL.inc()

 @LIST_OBJECTS_HIST.time()
 @LIST_OBJECTS_GAUGE.time()
 @LIST_OBJECTS_SUMMARY.time()
 def list_objects(self):
 try:
 bucket = self.s3.Bucket(self.bucket)
 objects = bucket.objects.all()
 for object in objects:
 logging.debug("%s", object.key)
 except:
 S3_FAIL.inc()

 @REMOVE_BUCKET_HIST.time()
 @REMOVE_BUCKET_GAUGE.time()
 @REMOVE_BUCKET_SUMMARY.time()
 def remove_bucket(self):
 try:
 bucket = self.s3.Bucket(self.bucket)
 bucket.delete()
 except:
 S3_FAIL.inc()

 def run_all_tests(self):
 self.create_bucket()
 self.put_object()
 self.list_buckets()
 self.list_objects()
 self.get_object()
 self.compare_object()
 self.del_object()
 self.put_objects()
 self.del_objects()
 self.remove_bucket()

 def generate_data(self):
 MAX_LIMIT = 126
 data = ''
 for _ in range(self.file_size):
 random_integer = random.randint(32, MAX_LIMIT)
 data += (chr(random_integer))
 self.data = data

Main

Here we start the prometheus exporter and call the method run_all_tests() from class S3Metrics inside an infinite while loop, so the script will run continuously until manually stopped.

Source codePlain text


1
2
3
4
5
6
7
8
9


def main():
 start_http_server(PORT)
 s3metrics = S3Metrics(BUCKET_NAME, OBJECT_NAME)
 while True:
 s3metrics.run_all_tests()
 time.sleep(TIME_TO_SLEEP)

if __name__ == '__main__':
 main()

So now that we have our s3metrics.py, we have to build a container to run it. I created a requirements.txt file to make it easier to adjust the package versions in the future, and used in the Dockerfile the python:3 base image since it already have most of what I need.

Requirements

Source codePlain text


1
2
3
4
5
6
7
8


boto3==1.17.54
botocore==1.20.54
jmespath==0.10.0
prometheus-client==0.10.1
python-dateutil==2.8.1
s3transfer==0.4.0
six==1.15.0
urllib3==1.26.4

Dockerfile

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


FROM python:3

WORKDIR /app

COPY requirements.txt .

COPY s3-metrics.py .

RUN python3 -m venv .venv; bash -c 'source .venv/bin/activate'; pip install -r requirements.txt

CMD [ "python", "./s3-metrics.py" ]

Build and Run

Now we can run the container inside an EC2 instance or anywhere capable of running a docker container (you don’t say, lol). I’ll later post my terraform script used to deploy it on AWS with more details. Now for the Grafana part:

Grafana

Here’s how I configured a Bar gauge graph to see the PUT Histogram:

We can see that the majority of the operations take less than 0.5 seconds to finish, so if we start seeing an increase in the other histogram buckets, it means that we probably have a problem.

I also created a panel to watch for errors and one to see all the operations combined:

Conclusion

That kind of monitoring can give a good observability to a service like Ceph S3 Gateway, but this could easily be applied to any other service with some changes. Before that, we used UpTimeRobot to achieve a similar goal, but from my perspective you can have much more control and flexibility with a container like that performing connectivity tests, and in this case we go much deeper by mimicking user operations on our tests.

Using Rook to leverage Ceph storage on a Kubernetes cluster

Sat, 17 Jul 2021 00:00:00 +0000

I recently got 10 bare-metal servers to play with, they used to be part of our first Ceph cluster and got replaced with more powerful hardware. So i built two k8s clusters and decided to give Rook a try.

As the cluster grew bigger, we purchased not only more servers but with different configuration. But those Dell R530 servers still have pretty decent power to run many internal demands we have, so i built a Ceph cluster with four of them using CentOS 8, Ceph Octopus and deployed everything in containers. But i want to talk about what i did with the remaining six servers, that became two kubernetes clusters - one with Flatcar and the other with Centos, to replicate some production scenarios we have. Since the servers have 8 2TB HDD each and we use just one for the OS, that would be a perfect scenario to experiment with Rook. I tinkered with it previously with some labs but now i could really test its usability.

Getting Rook to run is pretty straightforward, it might not even be necessary to have great knowledge about ceph for the initial setup. However it would be mandatory to know about Ceph administration for a production environment. There are many administrative tasks that gets taken care of by Rook, but after working with Ceph for the past 3 years i can assure you that there will be many situations where the sysadmin will have to step up.

I used the quickstart guide as a reference, it can be found on the Rook GitHub page: https://rook.github.io/docs/rook/v1.6/ceph-quickstart.html.

The only parameter i changed from the defaults was the ROOK_ENABLE_DISCOVERY_DAEMON, since i wanted it to automatically detect and provision the OSDs on every empty device. It’s important to erase the disks prior to the deployment, because it will skip any device with partitions. For that you can use wipefs or even dd. Be aware that if the device has LVM partitions and you want to avoid a reboot you might want to delete the LVs and VGs first. If you wipe the device the kernel might not release the mapped lvm volumes and you will not be able to delete after that. Needless to say that i made that mistake a few times before learning the lesson.

Commands i used:

Source codePlain text


1
2
3
4
5
6


git clone --single-branch --branch master https://github.com/rook/rook.git
git checkout release-1.6
cd rook/cluster/examples/kubernetes/ceph
sed -i s/"ROOK_ENABLE_DISCOVERY_DAEMON: \"false\""/"ROOK_ENABLE_DISCOVERY_DAEMON: \"true\""/ operator.yaml
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
kubectl create -f cluster.yaml

Checking if the pods were created:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


$ kubectl -n rook-ceph get pod
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-7tkgk 3/3 Running 25 95d
csi-cephfsplugin-cv2kq 3/3 Running 6 95d
csi-cephfsplugin-provisioner-bc5cff84-8h5b4 6/6 Running 19 95d
csi-cephfsplugin-provisioner-bc5cff84-cb477 6/6 Running 6 95d
csi-cephfsplugin-qvqjc 3/3 Running 3 95d
csi-rbdplugin-2rqwk 3/3 Running 25 95d
csi-rbdplugin-g5tkj 3/3 Running 3 95d
csi-rbdplugin-provisioner-97957587f-gvn6r 6/6 Running 0 22d
csi-rbdplugin-provisioner-97957587f-lflc7 6/6 Running 8 95d
csi-rbdplugin-vcsdj 3/3 Running 6 95d
rook-ceph-crashcollector-node1-7fcbcd7dc6-6fbq6 1/1 Running 2 73d
rook-ceph-crashcollector-node2-7f699c88c-78gvw 1/1 Running 0 22d
rook-ceph-crashcollector-node3-77777995d8-dtsrh 1/1 Running 1 95d
rook-ceph-mgr-a-8486cbdf64-dsdn2 1/1 Running 0 24d
rook-ceph-mon-a-7758d4d54c-crq4s 1/1 Running 2 95d
rook-ceph-mon-d-77db79f9b9-fcd5r 1/1 Running 0 41d
rook-ceph-mon-e-7d44dcff6b-zjpfl 1/1 Running 0 22d
rook-ceph-operator-66f7668857-sr8bw 1/1 Running 2 95d
rook-ceph-osd-0-57b7d8b47b-k5ps8 1/1 Running 2 95d
rook-ceph-osd-1-548c7bc54d-zsp8r 1/1 Running 1 95d
rook-ceph-osd-10-7875d885d8-h2wdt 1/1 Running 0 22d
rook-ceph-osd-11-5585fb7856-gxjfx 1/1 Running 2 95d
rook-ceph-osd-12-fd5bcb8f8-z9v4c 1/1 Running 1 95d
rook-ceph-osd-13-784c797d46-qmxj4 1/1 Running 0 22d
rook-ceph-osd-14-6b689bfb4-sfffq 1/1 Running 2 95d
rook-ceph-osd-15-64984bf7db-h99q2 1/1 Running 1 95d
rook-ceph-osd-16-6f66f68868-ft8d8 1/1 Running 0 22d
rook-ceph-osd-17-66fcdfc8c8-rg76t 1/1 Running 2 95d
rook-ceph-osd-18-85f9d5567b-6jjph 1/1 Running 1 95d
rook-ceph-osd-19-969b58fb-ztzxx 1/1 Running 0 22d
rook-ceph-osd-2-ffc69957b-846mj 1/1 Running 2 95d
rook-ceph-osd-20-5c86c5d556-26kkp 1/1 Running 0 22d
rook-ceph-osd-3-79cb4b8f78-wxczm 1/1 Running 1 95d
rook-ceph-osd-4-9587d6994-z94l2 1/1 Running 0 22d
rook-ceph-osd-5-74cb556886-sqp6g 1/1 Running 2 95d
rook-ceph-osd-6-5fb96bcb-2lgqr 1/1 Running 1 95d
rook-ceph-osd-7-85f8659c9d-7z5tn 1/1 Running 0 22d
rook-ceph-osd-8-6765676bfb-kjxr5 1/1 Running 2 95d
rook-ceph-osd-9-67b4f98cdb-h9bd8 1/1 Running 1 95d
rook-ceph-osd-prepare-node1-6x7dr 0/1 Completed 0 133m
rook-ceph-osd-prepare-node2-z6jl8 0/1 Completed 0 133m
rook-ceph-osd-prepare-node3-rvhth 0/1 Completed 0 133m
rook-ceph-tools-6f58686b5d-znnsg 1/1 Running 0 24d
rook-discover-2684s 1/1 Running 7 95d
rook-discover-2nlkj 1/1 Running 1 95d
rook-discover-4rhxm 1/1 Running 2 95d

The output is long but i wanted to show that Rook actually created a OSD por for every device in each node, and i configured kubernetes to allow that on the master node either.

We can see that we now have a bunch of new resource definitions, enabling us to interact with Ceph in a declarative way, instead of recurring to the CLI to manage users, filesystems, pools and many other resources:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


$ kubectl get crd | grep 'rook\|objectbucket'
cephblockpools.ceph.rook.io 2021-04-01T19:30:54Z
cephclients.ceph.rook.io 2021-04-01T19:30:53Z
cephclusters.ceph.rook.io 2021-04-01T19:30:53Z
cephfilesystems.ceph.rook.io 2021-04-01T19:30:53Z
cephnfses.ceph.rook.io 2021-04-01T19:30:53Z
cephobjectrealms.ceph.rook.io 2021-04-01T19:30:54Z
cephobjectstores.ceph.rook.io 2021-04-01T19:30:53Z
cephobjectstoreusers.ceph.rook.io 2021-04-01T19:30:54Z
cephobjectzonegroups.ceph.rook.io 2021-04-01T19:30:54Z
cephobjectzones.ceph.rook.io 2021-04-01T19:30:54Z
cephrbdmirrors.ceph.rook.io 2021-04-01T19:30:53Z
objectbucketclaims.objectbucket.io 2021-04-01T19:30:54Z
objectbuckets.objectbucket.io 2021-04-01T19:30:54Z
volumes.rook.io 2021-04-01T19:30:54Z

From that point we can access the Ceph CLI by interacting with the rook-ceph-tools pod:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


$ kubectl exec -it -n rook-ceph rook-ceph-tools-6f58686b5d-znnsg -- ceph status
cluster:
 id: 72e9c4a9-4315-45e0-ad43-cd79d22fb2be
 health: HEALTH_OK

services:
 mon: 3 daemons, quorum a,d,e (age 7d)
 mgr: a(active, since 3w)
 osd: 21 osds: 21 up (since 3w), 21 in (since 3w)
 rgw: 1 daemon active (my.store.a)

task status:

data:
 pools: 30 pools, 401 pgs
 objects: 2.12k objects, 2.4 GiB
 usage: 29 GiB used, 38 TiB / 38 TiB avail
 pgs: 401 active+clean

io:
 client: 7.0 KiB/s rd, 72 KiB/s wr, 6 op/s rd, 16 op/s wr

Now we have two new StorageClasses to work with:

Source codePlain text


1
2
3
4


$ kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate false 64d
rook-ceph-bucket rook-ceph.ceph.rook.io/bucket Delete Immediate false 84d

And then we provisioned some volumes to a few applications running on the cluster:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


$ kubectl get pv -A
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-1c9151ae-3e9e-49ae-adbf-f365de4eb718 8Gi RWO Delete Bound cluster-spd/data-etcd-1 rook-ceph-block 18d
pvc-39c5a513-1c50-48a9-9e4d-354ff84ea8b4 8Gi RWO Delete Bound cluster-spb/data-etcd-0 rook-ceph-block 24d
pvc-53f2816a-e4c2-434d-8600-20206d3f6f75 8Gi RWO Delete Bound cluster-spc/data-etcd-0 rook-ceph-block 18d
pvc-553c1a1b-0131-44a9-8a67-293dea40c161 5Gi RWO Delete Bound gitlab/redis-data-gitlab-redis-master-0 rook-ceph-block 56d
pvc-6c38a9b7-8e23-4478-a431-34fd4f9ec158 8Gi RWO Delete Bound cluster-spd/data-etcd-2 rook-ceph-block 18d
pvc-76bfab8e-6ae6-4b11-8223-ea9e97bbccf4 8Gi RWO Delete Bound cluster-spc/data-etcd-2 rook-ceph-block 18d
pvc-836c1ff1-c92d-467c-b45b-b970824e2b04 8Gi RWO Delete Bound cluster-spc/data-etcd-1 rook-ceph-block 18d
pvc-87e7bd8e-250e-4bd4-8268-3fb49597d0ab 10Gi RWO Delete Bound gitlab/gitlab-minio rook-ceph-block 39d
pvc-bac3dc77-73ae-4800-821a-04faa15e627d 8Gi RWO Delete Bound cluster-spb/data-etcd-1 rook-ceph-block 24d
pvc-ca087b82-038b-418a-a187-8e28617fbfae 8Gi RWO Delete Bound cluster-spd/data-etcd-0 rook-ceph-block 18d
pvc-d255ca76-b62f-413b-82b8-184ff3cb0026 8Gi RWO Delete Bound cluster-spb/data-etcd-2 rook-ceph-block 24d
pvc-d331d766-72d7-46c4-9cba-c0e8fea7a2b8 50Gi RWO Delete Bound gitlab/repo-data-gitlab-gitaly-0 rook-ceph-block 56d
pvc-e76d375b-03b5-4bc0-8725-bc406cb7ee94 8Gi RWO Delete Bound gitlab/data-gitlab-postgresql-0 rook-ceph-block 56d

This is how the pvc manifest looks like:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


$ kubectl get pvc -n gitlab gitlab-minio -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gitlab-minio
 namespace: gitlab
 labels:
 app: minio
spec:
 storageClassName: rook-ceph-block
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 10Gi

We can also provision Buckets using S3 via Rook. We used this to provide a backend for a Hashicorp Vault we have on the same environment:

First we have to create the StorageClass:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: rook-ceph-bucket
provisioner: rook-ceph.ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
 objectStoreName: my-store
 objectStoreNamespace: rook-ceph
 region: us-east-1

Then we create a ObjectBucketClaim, which is a new resource type defined by the Rook CRD:

Source codePlain text


1
2
3
4
5
6
7
8


apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
 name: ceph-bucket
 namespace: vault
spec:
 generateBucketName: ceph-bucket
 storageClassName: rook-ceph-bucket

Using Ansible for Hardening Linux Servers

Sun, 07 Feb 2021 00:00:00 +0000

Server security is paramount in today’s interconnected world. A single vulnerability can expose your entire infrastructure to malicious actors. Manual hardening across multiple servers is time-consuming, error-prone, and inconsistent. This is where Ansible shines. Ansible’s automation capabilities allow us to implement robust security measures efficiently and reliably, ensuring consistent security posture across our entire fleet.

This blog post demonstrates how to leverage Ansible to harden Linux servers, focusing on key security best practices. We’ll build upon a previous example, enhancing it with crucial security features. The example includes hardening Apache web servers, but the principles apply broadly to other server types.

The Ansible Playbook:

This Ansible playbook will implement the following security measures:

Disable Password Authentication (SSH): Eliminates the risk of brute-force attacks targeting password-based SSH logins.
Create and Secure an Admin User: Establishes a dedicated administrative account with strong authentication using SSH keys.
Configure sudo Permissions: Grants the admin user appropriate sudo privileges.
Install and Configure Fail2ban: Implements a security tool to detect and ban malicious IP addresses attempting unauthorized logins.
Limit SSH Authentication Attempts: Reduces the window of opportunity for brute-force attacks by setting a maximum number of authentication retries.

Here’s the Ansible playbook (hardening.yml):

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194


---
- hosts: webservers
 become: true
 tasks:
 # Security Hardening Section
 - name: Disable password authentication (SSH)
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^PasswordAuthentication'
 line: 'PasswordAuthentication no'
 state: present

 - name: Add admin user
 user:
 name: adminuser
 createhome: yes
 shell: /bin/bash
 state: present

 - name: Add admin user to sudoers
 lineinfile:
 path: /etc/sudoers
 line: "adminuser ALL=(ALL:ALL) ALL"
 create: yes
 state: present
 mode: '0440'

 - name: Copy authorized_keys
 copy:
 src: authorized_keys
 dest: /home/adminuser/.ssh/authorized_keys
 owner: adminuser
 group: adminuser
 mode: '0600'
 notify: restart sshd

 - name: Ensure .ssh directory exists
 file:
 path: /home/adminuser/.ssh
 state: directory
 mode: '0700'
 owner: adminuser
 group: adminuser

 - name: Install fail2ban
 apt:
 name: fail2ban
 state: present
 when: ansible_distribution == 'Ubuntu'

 - name: Install fail2ban (CentOS/RHEL)
 yum:
 name: fail2ban
 state: present
 when: ansible_distribution == 'CentOS' or ansible_distribution == 'RedHat'

 - name: Enable fail2ban service
 service:
 name: fail2ban
 state: started
 enabled: yes

 # SSH Hardening Tasks
 - name: Set SSH Protocol to 2
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^Protocol'
 line: 'Protocol 2'
 state: present

 - name: Disable SSH Protocol 1
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^Protocol 1'
 line: 'Protocol 1'
 state: absent

 - name: Disable RSA Authentication
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^RSAAuthentication'
 line: 'RSAAuthentication no'
 state: present

 - name: Set SSH Port
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^Port\ '
 line: "Port {{ ssh_port | default(22) }}" #Default to port 22 if ssh_port variable not defined
 state: present

 - name: Disable Root Login
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^PermitRootLogin\ '
 line: 'PermitRootLogin no'
 state: present

 - name: Disable Password Authentication
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^PasswordAuthentication\ '
 line: 'PasswordAuthentication no'
 state: present

 - name: Disable Empty Passwords
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^PermitEmptyPasswords\ '
 line: 'PermitEmptyPasswords no'
 state: present

 - name: Enable Strict Modes
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^StrictModes\ '
 line: 'StrictModes yes'
 state: present

 - name: Ignore Rhosts
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^IgnoreRhosts\ '
 line: 'IgnoreRhosts yes'
 state: present

 - name: Disable Rhosts Authentication
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^RhostsAuthentication\ '
 line: 'RhostsAuthentication no'
 state: present

 - name: Disable Rhosts RSA Authentication
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^RhostsRSAAuthentication\ '
 line: 'RhostsRSAAuthentication no'
 state: present

 - name: Set Client Alive Interval
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^ClientAliveInterval\ '
 line: 'ClientAliveInterval 300'
 state: present

 - name: Set Client Alive Count Max
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^ClientAliveCountMax\ '
 line: 'ClientAliveCountMax 0'
 state: present

 - name: Disable TCP Forwarding
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^AllowTcpForwarding\ '
 line: 'AllowTcpForwarding no'
 state: present

 - name: Disable X11 Forwarding
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^X11Forwarding\ '
 line: 'X11Forwarding no'
 state: present

 - name: Set KexAlgorithms
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^KexAlgorithms\ '
 line: "KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256"
 state: present

 - name: Set Ciphers
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^Ciphers\ '
 line: "Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr"
 state: present

 - name: Set MACs
 lineinfile:
 path: /etc/ssh/sshd_config
 regexp: '^MACs\ '
 line: "MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com"
 state: present

 handlers:
 - name: restart sshd
 service:
 name: sshd
 state: restarted

Before Running:

Prepare authorized_keys: Create a file named authorized_keys containing the public key for the adminuser account.
Inventory File: Configure your Ansible inventory file to point to your target servers.
Caution: The sudoers file is critical. Errors can lock you out. Review this section carefully.

Running the Playbook:

Execute the playbook using: ansible-playbook hardening.yml

Beyond the Basics:

This playbook provides a solid foundation. Further enhancements could include:

Regular Security Updates: Integrate Ansible with your update management system.
Security Auditing: Implement regular security audits using Ansible to monitor system configurations.
Vulnerability Scanning: Integrate Ansible with vulnerability scanning tools.
Intrusion Detection Systems (IDS): Configure and manage IDS using Ansible.

While not necessarily considered as hardening settings, it is common and desirable to use Ansible to configure system updates, logrotate, log shipping, time synchronization and basic system tuning. The playbook below performs those tasks:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112


---
- hosts: webservers
 become: true
 tasks:

 # 1. System Updates and Security Patches
 - name: Update apt cache (Ubuntu/Debian)
 apt:
 update_cache: yes
 when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'

 - name: Update yum cache (CentOS/RHEL)
 yum:
 update_cache: yes
 when: ansible_distribution == 'CentOS' or ansible_distribution == 'RedHat'

 - name: Upgrade system packages (Ubuntu/Debian)
 apt:
 upgrade: dist
 state: present
 when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'

 - name: Upgrade system packages (CentOS/RHEL)
 yum:
 update: minimal
 state: present
 when: ansible_distribution == 'CentOS' or ansible_distribution == 'RedHat'

 # 2. Time Synchronization
 - name: Install ntp
 apt:
 name: ntp
 state: present
 when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'

 - name: Install ntp (CentOS/RHEL)
 yum:
 name: ntp
 state: present
 when: ansible_distribution == 'CentOS' or ansible_distribution == 'RedHat'

 - name: Start NTP service
 service:
 name: ntpd
 state: started
 enabled: yes

 # 3. Logging and Monitoring: Logrotate and Loki/Promtail

 - name: Install logrotate
 apt:
 name: logrotate
 state: present
 when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'

 - name: Install logrotate (CentOS/RHEL)
 yum:
 name: logrotate
 state: present
 when: ansible_distribution == 'CentOS' or ansible_distribution == 'RedHat'


 - name: Configure logrotate for apache
 template:
 src: logrotate.conf.j2
 dest: /etc/logrotate.d/apache2
 when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'

 - name: Configure logrotate for httpd
 template:
 src: logrotate.conf.j2
 dest: /etc/logrotate.d/httpd
 when: ansible_distribution == 'CentOS' or ansible_distribution == 'RedHat'

 - name: Install Promtail
 apt:
 name: promtail
 state: present
 when: ansible_distribution == 'Ubuntu' or ansible_distribution == 'Debian'

 - name: Install Promtail (CentOS/RHEL)
 yum: 
 name: promtail 
 state: present
 when: ansible_distribution == 'CentOS' or ansible_distribution == 'RedHat'


 - name: Configure Promtail
 template:
 src: promtail.yml.j2
 dest: /etc/promtail/config.yml

 - name: Start Promtail service
 service:
 name: promtail
 state: started
 enabled: yes


 # 7. Basic System Tuning
 - name: Increase net.core.so_max_conn
 sysctl:
 name: net.core.so_max_conn
 value: "65535"
 state: present
 sysctl_file: /etc/sysctl.conf
 - name: Increase net.ipv4.tcp_max_syn_backlog
 sysctl:
 name: net.ipv4.tcp_max_syn_backlog
 value: "8192"
 state: present
 sysctl_file: /etc/sysctl.conf

Template files:

logrotate.conf.j2:

Source codejinja2


1
2
3
4
5
6
7
8
9


/var/log/apache2/*log {
 daily
 rotate 7
 compress
 delaycompress
 missingok
 notifempty
 copytruncate
}

promtail.yml.j2:

Source codejinja2


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


scrape_configs:
 - job_name: system
 static_configs:
 - targets:
 - localhost
 labels:
 job: system
 __path__: /var/log/apache2/*.log
 - job_name: custom_logs
 static_configs:
 - targets:
 - localhost
 labels:
 job: custom_logs
 __path__: /path/to/your/custom/logs/*.log


client:
 url: http://loki:3100/loki/api/v1/push

By automating server hardening with Ansible, you gain significant advantages in efficiency, consistency, and overall security posture. This approach allows you to manage security effectively, even across large server deployments. Remember to always test your playbooks thoroughly in a non-production environment before deploying them to production servers.

Web scrapper using Python

Fri, 20 Nov 2020 00:00:00 +0000

So I needed to learn how to scrape a web page using Python.

My teammate suggested that I could learn a few tricks in python by scraping FIIs, which is the brazilian equivalent to american REITs. I did some research and found some good examples using a module called pandas (https://realpython.com/python-csv/) , a tool designed for data analisys and manipulation and very popular among people working with data science.

My goal was much less ambitious but i thought i might as well use what everyone else is using, since it would provide a good learning opportunity.

I based my code on Renata Magner’s (https://github.com/RenataMagner/web_scraping_fii/blob/master/web_scraping-fiiv2.ipynb). I still have to implement error handling and many other improvements on the code, that’s just the first version. Without further ado, that’s what i came up with:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


from bs4 import BeautifulSoup
import requests
import pandas as pd

url = 'https://www.fundsexplorer.com.br/ranking'
html = requests.get(url).content
soup = BeautifulSoup(html, 'html.parser')

# stores scraped data on a list
results = soup.find_all("td")

# counts how many entries we got. each entry has 26 'td' lines each.
fii_count=int(len(results)/26)

# initialize the variables
fii_info={}
fii_list=[]

'''
index - field
[0] - COD FUNDO
[1] - SETOR
[2] - PRECO_ATUAL
[3] - LIQUIDEZ DIARIA
[4] - DIVIDENDO
[5] - dividend_yield
[6] - dy_3m
[7] - dy_6m
[8] - dy_12m
[9]- dy_3m_media
[10] - dy_6m_media
[11] - dy_12m_media
[12] - dy_ano
[13] - variacao_preco
[14] - rentabilidade_periodo
[15] - rentabilidade_acumulada
[16] - patrimonio_liq
[17] - vpa
[18] - p_vpa
'''

# runs through collected data and extracts the desided info
for fii in range(fii_count):
 fii_info['codigo_fundo']=results[fii*26].get_text()
 fii_info['preco_atual']=results[fii*26+2].get_text()
 fii_info['setor']=results[fii*26+1].get_text()
 fii_info['p/vpa']= results[fii*26+18].get_text()
 fii_info['dividend_yield']=results[fii*26+5].get_text()
 fii_info['dividendo']=results[fii*26+4].get_text()
 fii_info['dy_12m_media']=results[fii*26+11].get_text()
 fii_list.append(fii_info)
 fii_info={}

# generates the CSV output
fii_table = pd.DataFrame(data=fii_list)
fii_table.to_csv('fii_table.csv')

References:

https://github.com/RenataMagner/web_scraping_fii/blob/master/web_scraping-fiiv2.ipynb

https://realpython.com/python-csv/

https://www.fundsexplorer.com.br/ranking

Comparison of Sorting Algorithms using Python Decorators

Tue, 17 Nov 2020 00:00:00 +0000

I wrote this exercise while learning Python decorators and sorting algorithms. The goal was to time an operation without putting timing code inside every algorithm.

Technical revision — October 2026: the original decorator skipped recursive calls in MergeSort, and SelectionSort swapped elements outside its outer loop. Those implementations did not reliably sort the input, so their published timings were not a valid comparison. The corrected example below checks every result against Python’s sorted before reporting a measurement.

Time the outer operation

A decorator wraps a function and can add behavior around its execution. Here, timed returns the function’s result and its elapsed time in seconds, measured with perf_counter. This intentionally changes the return contract to a pair; a timing decorator is not automatically transparent to its caller.

MergeSort uses an undecorated recursive helper. Every recursive call executes normally, while the decorator times the complete outer operation once. SelectionSort swaps inside the outer loop, after finding the minimum of the remaining suffix. All three implementations modify the supplied list in place.

Corrected example

Download sorting.py. It uses only the Python standard library.

sorting.pypython · 80 linesDownload
Expand sourceCollapse source

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80


"""Correctness first: time the outer call, not recursive subproblems."""

from functools import wraps
from random import Random
from statistics import median
from time import perf_counter
import platform


def timed(function):
 @wraps(function)
 def wrapper(*args, **kwargs):
 start = perf_counter()
 result = function(*args, **kwargs)
 return result, perf_counter() - start
 return wrapper


@timed
def merge_sort(values):
 def sort(items):
 if len(items) < 2:
 return
 middle = len(items) // 2
 left, right = items[:middle], items[middle:]
 sort(left)
 sort(right)
 i = j = k = 0
 while i < len(left) and j < len(right):
 if left[i] <= right[j]:
 items[k] = left[i]
 i += 1
 else:
 items[k] = right[j]
 j += 1
 k += 1
 items[k:] = left[i:] + right[j:]
 sort(values)


@timed
def selection_sort(values):
 for i in range(len(values)):
 minimum = i
 for j in range(i + 1, len(values)):
 if values[j] < values[minimum]:
 minimum = j
 values[i], values[minimum] = values[minimum], values[i]


@timed
def bubble_sort(values):
 for end in range(len(values) - 1, 0, -1):
 swapped = False
 for i in range(end):
 if values[i] > values[i + 1]:
 values[i], values[i + 1] = values[i + 1], values[i]
 swapped = True
 if not swapped:
 break


def benchmark(size=1000, repeats=5, seed=42):
 original = Random(seed).sample(range(size * 10), size)
 expected = sorted(original)
 print(f"Python {platform.python_version()} | {platform.system()} {platform.machine()}")
 print(f"{size} distinct integers | seed={seed} | {repeats} repetitions | median ms")
 for algorithm in (merge_sort, selection_sort, bubble_sort):
 samples = []
 for _ in range(repeats):
 values = original.copy() # Copying is outside the timed section.
 _, elapsed = algorithm(values)
 if values != expected:
 raise AssertionError(f"{algorithm.__name__} did not sort correctly")
 samples.append(elapsed * 1000)
 print(f"{algorithm.__name__}: {median(samples):.3f}")


if __name__ == "__main__":
 benchmark()

Run it with:

Source codebash

1

python sorting.py

Reading the measurements

The benchmark uses 1,000 distinct integers generated with seed 42, five repetitions per algorithm, and reports the median elapsed time. Each run starts with a fresh copy of the same list. Copying the input and verifying the result happen outside the timed section. The output records the Python version and operating-system architecture.

These timings describe this implementation and this input on the machine running it. They are not a universal ranking: already sorted inputs, duplicates, interpreter versions and machine load affect measurements. Before drawing performance conclusions, repeat the comparison across those cases and include Python’s built-in sort as a practical baseline.

For these implementations, MergeSort takes O(n log n) time and O(n) auxiliary storage; SelectionSort takes O(n²) time. BubbleSort takes O(n²) in the worst case and stops early when a pass makes no swaps.

The original invalid benchmark results have been removed. The executable example generates new measurements and refuses to report a timing for an incorrect result.

References

Ceph features and its relation to the client kernel version

Mon, 16 Nov 2020 00:00:00 +0000

While working on an upgrade on one of our ceph cluster from Luminous to Nautilus, i needed to come up with a way to detect any client with older versions, and check if we could break anything after the upgrade.

I started checking the documentation as always, but all i found was a command called ceph features, which gives a summarized output:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


[root@mon-1 ~]# ceph features
{
 "mon": {
 "group": {
 "features": "0x3ffddff8eeacfffb",
 "release": "luminous",
 "num": 3
 }
 },
 "mds": {
 "group": {
 "features": "0x3ffddff8eeacfffb",
 "release": "luminous",
 "num": 3
 }
 },
 "osd": {
 "group": {
 "features": "0x3ffddff8eeacfffb",
 "release": "luminous",
 "num": 192
 }
 },
 "client": {
 "group": {
 "features": "0x27018fb86aa42ada",
 "release": "jewel",
 "num": 422
 },
 "group": {
 "features": "0x2f018fb86aa42ada",
 "release": "luminous",
 "num": 95
 },
 "group": {
 "features": "0x3ffddff8eeacfffb",
 "release": "luminous",
 "num": 18
 }
 }
}

So that got me worried, since I was now looking at 422 clients apparently using jewel and we were already using Nautilus for a while. Since this specific cluster was “born” in jewel, I thought that it might be the case that many clients didn’t get any upgrades since.

Another way to see who’s connected to the cluster is the command ceph tell mds.\hostname` client ls`, but it shows only cephfs clients, which solves only a part of the problem. At least we can see the client’s IP address (adresses and names omitted with […]):

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


"inst": "client.1400410 v1:[...]]:0/1769731892",
 "completed_requests": [],
 "prealloc_inos": [],
 "used_inos": [],
 "client_metadata": {
 "features": "0x00000000000001ff",
 "entity_id": [...],
 "hostname": [...],
 "kernel_version": "4.19.[...]",
 "root": [...]
 }

Short note: there’s no documentation explaining those codes and its respective features, so I’m going to look at the source code and try to come up with a table. I’ll post it as soon as I manage to.

Searching further i came across the command ceph daemon mon.hostname sessions:

Source codePlain text


1
2
3
4
5
6
7
8


[root@mon-1 ~]# ceph daemon mon.`hostname` sessions | grep jewel
 "MonSession(client.354037 v1:[...]]:0/3173650400 is open allow profile rbd, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.394132 v1:[...]:0/295867623 is open allow profile rbd, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.785407 v1:[...]:0/507474248 is open allow profile rbd, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.367883 v1:[...]:0/3274665177 is open allow profile rbd, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.812187 v1:[...]:0/733157529 is open allow profile rbd, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.1390811 v1:[...]:0/1470215921 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.1400452 v1:[...]:0/3851374868 is open allow r, features 0x27018fb86aa42ada (jewel))",

Now i started to see some light at the end, since that command gave me the ip address and the version for each client connected. I knew that we didn’t have any clients using ceph-fuse, so it was safe to assume that everyone was using the kernel module to mount either RBD or cephfs. That cleared some confusion I had at the beginning between the ceph-common package version and the kernel version. It is not required to have that package installed in order to use the kernel module, so i then pointed my efforts at checking the clients’ kernel versions.

The documentation also recommended some debugging at mon:

The ceph features command that reports the total number of clients and daemons and their features and releases. If the debugging level for Monitors is set to 10 (debug mon = 10), addresses and features of connecting and disconnecting clients are logged to log file on a local file system.

So i tried that and here’s what we get with debug mon = 10:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


2020-11-10 08:55:59.084 7f31c9015700 0 --1- [v2:[...]:3300/0,v1:[...]:6789/0] >> conn(0x561a94aed180 0x561a94ba5000 :6789 s=ACCE
PTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:[...]:40764/0)
2020-11-10 08:55:59.085 7f31c500d700 10 mon.ceph-1@0(leader) e1 ms_handle_accept con 0x561a94aed180 no session
2020-11-10 08:55:59.085 7f31c500d700 10 mon.ceph-1@0(leader) e1 _ms_dispatch new session 0x561a94adcd80 MonSession(unknown.0 - is open , featu
res 0x27018fb86aa42ada (jewel)) features 0x27018fb86aa42ada
2020-11-10 08:55:59.085 7f31c500d700 10 mon.ceph-1@0(leader).auth v3676 preprocess_query auth(proto 0 34 bytes epoch 0) from unknown.0 -
2020-11-10 08:55:59.085 7f31c500d700 10 mon.ceph-1@0(leader).auth v3676 prep_auth() blob_size=34
2020-11-10 08:55:59.085 7f31c500d700 10 mon.ceph-1@0(leader).auth v3676 _assign_global_id 34110 (max 44096)
2020-11-10 08:55:59.085 7f31c500d700 2 mon.ceph-1@0(leader) e1 send_reply 0x561a958ba780 0x561a95246fc0 auth_reply(proto 2 0 (0) Success) v1
2020-11-10 08:55:59.086 7f31c500d700 10 mon.ceph-1@0(leader).auth v3676 preprocess_query auth(proto 2 32 bytes epoch 0) from unknown.0 -
2020-11-10 08:55:59.086 7f31c500d700 10 mon.ceph-1@0(leader).auth v3676 prep_auth() blob_size=32
2020-11-10 08:55:59.086 7f31c500d700 10 mon.ceph-1@0(leader) e1 ms_handle_authentication session 0x561a94adcd80 con 0x561a94aed180 addr - MonSession(unknown.0 - is open , features 0x27018fb86aa42ada (jewel))

Without debug mon = 10, only the first line is logged.

That is very useful to detect the clients’ features when they are connecting, but not so much when you can’t have the luxury to reset everyone’s connection. Besides, leaving that level of debugging enabled all the time would flag our node as an abuser at the log server in a matter of hours.

What i knew already about our clients was that there was some elastic nodes using RBD, probably with CentOS 7 and consequently using kernel 3.x; Some other stuff also using RBD on either CentOS 7 or CentOS 8; and a kubernetes cluster using CoreOS and Flatcar with kernel 4.x. So after some checking, I compared the output from a client using kernel 4.19.x and another with 5.4.x:

4.19.x:

Source codePlain text


1
2
3
4
5
6
7
8
9


[root@mon-1 ~]# ceph daemon mon.`hostname` sessions | grep [...]
 "MonSession(client.95408384 [...]:0/923650579 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.95446387 [...]:0/421761626 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(client.95427342 [...]:0/1106672011 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(unknown.0 [...]:0/2823489235 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(unknown.0 [...]:0/4141488860 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(unknown.0 [...]:0/662892667 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(unknown.0 [...]:0/940509445 is open allow r, features 0x27018fb86aa42ada (jewel))",
 "MonSession(unknown.0 [...]:0/1147570690 is open allow r, features 0x27018fb86aa42ada (jewel))",

5.4.x:

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


root@mon-1 ~]# ceph daemon mon.`hostname` sessions | grep [...]
 "MonSession(unknown.0 [...]:0/2258512824 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/4186418195 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/232634731 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(client.95587947 [...]:0/3963151684 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/1164059037 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/1228306108 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/2811111809 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/4254695077 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(client.95587926 [...]:0/479729343 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/340770437 is open allow r, features 0x2f018fb86aa42ada (luminous))",
 "MonSession(unknown.0 [...]:0/425381347 is open allow r, features 0x2f018fb86aa42ada (luminous))",

After that I could come with some conclusion. We’ll probably keep seeing jewel features for a long time, since CentOS/RHEL 8 and most of the main distros uses kernel 4.x on their latest LTS versions. I expected that at kernel 3.x i’d get Jewel features and from 4.x on we would get Luminous, but apparently the latest features are only available on kernels 5.x and beyond. I’m not sure if there is any feature that would be marked as nautilus or any newer ceph version, so I still need to dig deeper on that subject.

References:

https://ceph.io/community/new-luminous-upgrade-complete/

https://docs.ceph.com/en/latest/rados/operations/upmap/#enabling

https://docs.ceph.com/en/latest/man/8/ceph/#osd

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3.0/html/release_notes/major_updates

Obtaining a list of Ceph features from the hexadecimal value

Mon, 16 Nov 2020 00:00:00 +0000

After some digging i found the list of possible features, its respective kernel version requirement and/or when it became available.

This can be found in src/include/ceph_features.h, and I got it by cloning the ceph repo at git://github.com/ceph/ceph .

That information might be useful when you are trying to determine if your clients might need a kernel upgrade and what kind of RBD or cephfs features you can enable on your server side without breaking compatibility. It’s always ideal to have every new feature availabe whenever possible, but that is not always the case when you have a medium to large deployment, multiple clients with different workloads and scenarios - in other words, a Real World situation.

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90


import sys

feature_list = [
 "( 0, 1, UID)",
 "( 1, 1, NOSRCADDR) // 2.6.35 req",
 "( 2, 3, SERVER_NAUTILUS)",
 "( 3, 1, FLOCK) // 2.6.36",
 "( 4, 1, SUBSCRIBE2) // 4.6 req",
 "( 5, 1, MONNAMES)",
 "( 6, 1, RECONNECT_SEQ) // 3.10 req",
 "( 7, 1, DIRLAYOUTHASH) // 2.6.38",
 "( 8, 1, OBJECTLOCATOR)",
 "( 9, 1, PGID64) // 3.9 req",
 "(10, 1, INCSUBOSDMAP)",
 "(11, 1, PGPOOL3) // 3.9 req",
 "(12, 1, OSDREPLYMUX)",
 "(13, 1, OSDENC) // 3.9 req",
 "(14, 2, SERVER_KRAKEN)",
 "(15, 1, MONENC)",
 "(16, 3, SERVER_OCTOPUS) | (16, 3, OSD_REPOP_MLCOD)",
 "(17, 3, OS_PERF_STAT_NS)",
 "(18, 1, CRUSH_TUNABLES) // 3.6",
 "(19, 2, OSD_PGLOG_HARDLIMIT)",
 "(20, 3, SERVER_PACIFIC)",
 "(21, 2, SERVER_LUMINOUS) // 4.13 | (21, 2, RESEND_ON_SPLIT) | (21, 2, RADOS_BACKOFF) | (21, 2, OSDMAP_PG_UPMAP) | (21, 2, CRUSH_CHOOSE_ARGS)",
 "(22, 2, OSD_FIXED_COLLECTION_LIST)",
 "(23, 1, MSG_AUTH) // 3.19 req (unless nocephx_require_signatures)",
 "(24, 2, RECOVERY_RESERVATION_2)",
 "(25, 1, CRUSH_TUNABLES2) // 3.9",
 "(26, 1, CREATEPOOLID)",
 "(27, 1, REPLY_CREATE_INODE) // 3.9",
 "(28, 2, SERVER_MIMIC)",
 "(29, 1, MDSENC) // 4.7",
 "(30, 1, OSDHASHPSPOOL) // 3.9",
 "DEPRECATED(31, 1, MON_SINGLE_PAXOS, NAUTILUS)",
 "(32, 3, STRETCH_MODE)",
 "RETIRED(33, 1, MON_SCRUB, JEWEL, LUMINOUS)",
 "RETIRED(34, 1, OSD_PACKED_RECOVERY, JEWEL, LUMINOUS)",
 "(35, 1, OSD_CACHEPOOL) // 3.14",
 "(36, 1, CRUSH_V2) // 3.14",
 "(37, 1, EXPORT_PEER) // 3.14",
 "RETIRED(38, 1, OSD_ERASURE_CODES, MIMIC, OCTOPUS)",
 "(39, 1, OSDMAP_ENC) // 3.15",
 "(40, 1, MDS_INLINE_DATA) // 3.19",
 "(41, 1, CRUSH_TUNABLES3) // 3.15 | (41, 1, OSD_PRIMARY_AFFINITY)",
 "(42, 1, MSGR_KEEPALIVE2) // 4.3 (for consistency)",
 "(43, 1, OSD_POOLRESEND) // 4.13",
 "RETIRED(44, 1, ERASURE_CODE_PLUGINS_V2, MIMIC, OCTOPUS)",
 "RETIRED(45, 1, OSD_SET_ALLOC_HINT, JEWEL, LUMINOUS)",
 "(46, 1, OSD_FADVISE_FLAGS)",
 "(47, 1, MDS_QUOTA) // 4.17",
 "(48, 1, CRUSH_V4) // 4.1",
 "RETIRED(49, 1, OSD_MIN_SIZE_RECOVERY, JEWEL, LUMINOUS)",
 "RETIRED(50, 1, MON_METADATA, MIMIC, OCTOPUS)",
 "RETIRED(51, 1, OSD_BITWISE_HOBJ_SORT, MIMIC, OCTOPUS)",
 "RETIRED(52, 1, OSD_PROXY_WRITE_FEATURES, MIMIC, OCTOPUS)",
 "RETIRED(53, 1, ERASURE_CODE_PLUGINS_V3, MIMIC, OCTOPUS)",
 "RETIRED(54, 1, OSD_HITSET_GMT, MIMIC, OCTOPUS)",
 "RETIRED(55, 1, HAMMER_0_94_4, MIMIC, OCTOPUS)",
 "(56, 1, NEW_OSDOP_ENCODING) // 4.13 (for pg_pool_t >= v25)",
 "(57, 1, MON_STATEFUL_SUB) // 4.13 | (57, 1, SERVER_JEWEL)",
 "(58, 1, CRUSH_TUNABLES5) // 4.5 | (58, 1, NEW_OSDOPREPLY_ENCODING) | (58, 1, FS_FILE_LAYOUT_V2)",
 "(59, 1, FS_BTIME) | (59, 1, FS_CHANGE_ATTR) | (59, 1, MSG_ADDR2)",
 "(60, 1, OSD_RECOVERY_DELETES) // *do not share this bit*",
 "(61, 1, CEPHX_V2) // 4.19, *do not share this bit*",
 "(62, 1, RESERVED) // do not use; used as a sentinel",
 "DEPRECATED(63, 1, RESERVED_BROKEN, LUMINOUS) // client-facing"
]

def settolist(featureset):
 '''Converts a featureset hex to a feature list.
 
 Parameters:
 int:featureset

 Returns:
 list:featureset_list
 '''
 featureset_hex = featureset
 featureset_list = []
 featureset_bin = bin(int(featureset_hex, 16))[2:].zfill(64)
 count = 0
 for bit in (featureset_bin):
 if bit == '1':
 featureset_list.append(feature_list[count])
 count += 1
 return featureset_list

output = settolist(str(sys.argv[1]))
print("\n".join(output))

A test run using an entry from my test cluster:

MonSession(client.95280473 [IP]:0/110518927 is open allow profile rbd, features 0x27018fb86aa42ada (jewel))"

Source codePlain text


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


python features.py 0x27018fb86aa42ada

( 2, 3, SERVER_NAUTILUS)
( 5, 1, MONNAMES)
( 6, 1, RECONNECT_SEQ) // 3.10 req
( 7, 1, DIRLAYOUTHASH) // 2.6.38
(15, 1, MONENC)
(16, 3, SERVER_OCTOPUS) | (16, 3, OSD_REPOP_MLCOD)
(20, 3, SERVER_PACIFIC)
(21, 2, SERVER_LUMINOUS) // 4.13 | (21, 2, RESEND_ON_SPLIT) | (21, 2, RADOS_BACKOFF) | (21, 2, OSDMAP_PG_UPMAP) | (21, 2, CRUSH_CHOOSE_ARGS)
(22, 2, OSD_FIXED_COLLECTION_LIST)
(23, 1, MSG_AUTH) // 3.19 req (unless nocephx_require_signatures)
(24, 2, RECOVERY_RESERVATION_2)
(26, 1, CREATEPOOLID)
(27, 1, REPLY_CREATE_INODE) // 3.9
(28, 2, SERVER_MIMIC)
RETIRED(33, 1, MON_SCRUB, JEWEL, LUMINOUS)
RETIRED(34, 1, OSD_PACKED_RECOVERY, JEWEL, LUMINOUS)
(36, 1, CRUSH_V2) // 3.14
RETIRED(38, 1, OSD_ERASURE_CODES, MIMIC, OCTOPUS)
(40, 1, MDS_INLINE_DATA) // 3.19
(42, 1, MSGR_KEEPALIVE2) // 4.3 (for consistency)
RETIRED(45, 1, OSD_SET_ALLOC_HINT, JEWEL, LUMINOUS)
RETIRED(50, 1, MON_METADATA, MIMIC, OCTOPUS)
RETIRED(52, 1, OSD_PROXY_WRITE_FEATURES, MIMIC, OCTOPUS)
RETIRED(54, 1, OSD_HITSET_GMT, MIMIC, OCTOPUS)
(56, 1, NEW_OSDOP_ENCODING) // 4.13 (for pg_pool_t >= v25)
(57, 1, MON_STATEFUL_SUB) // 4.13 | (57, 1, SERVER_JEWEL)
(59, 1, FS_BTIME) | (59, 1, FS_CHANGE_ATTR) | (59, 1, MSG_ADDR2)
(60, 1, OSD_RECOVERY_DELETES) // *do not share this bit*
(62, 1, RESERVED) // do not use; used as a sentinel

Intro to Ansible

Sat, 03 Oct 2020 00:00:00 +0000

First, some terminology: With Ansible, a master server can configure a slave server. To do this, the master connects to the slave via SSH and then executes tasks. Each task describes a configuration step, for example installing a package using yum. Each task calls a module that implements the current task, for example the yum module. If a file is to be copied, the copy module is used; the systemd module can be used to manage systemd services, etc. Ansible version 2.7 comes with around 2,100 modules. In addition, additional modules can be easily imported.

First steps

The first step in using Ansible is to write an inventory. There, users specify which hosts should be orchestrated and they can also be grouped together. Inventories can be written in either YAML or INI format. In the following example, a web server is to be deployed; to do this, create the following inventory:

Source codeyaml


1
2
3
4
5


---
all:
 hosts:
 webserver1:
 ansible_host: 10.0.0.2

The dict keys all and hosts are specified and reserved by Ansible, so they must not be used for other purposes. A new server is now added to the list of all hosts, which should be known in Ansible as webserver1. Ansible is also informed that it should be available under the IP 10.0.0.2. The ansible_host parameter is used for this. If you want to manage not just one web server, but three and also a database server, you also add these servers to the inventory:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


---
all:
 hosts:
 webserver1:
 ansible_host: 10.0.0.2
 webserver2:
 ansible_host: 10.0.0.3
 webserver3:
 ansible_host: 10.0.0.4
 dbserver:
 ansible_host: 10.0.1.1

In principle, the inventory is finished at this point, but it is recommended to define inventory groups straight away, for example to address all web servers collectively. To do this, a children dict-key is introduced:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


---
all:
 hosts:
 webserver1:
 ansible_host: 10.0.0.2
 webserver2:
 ansible_host: 10.0.0.3
 webserver3:
 ansible_host: 10.0.0.4
 dbserver1:
 ansible_host: 10.0.1.1
 children:
 webservers:
 hosts:
 webserver1:
 webserver2:
 webserver3:
 dbservers:
 dbserver1:

After defining an inventory, Ansible can generally be used in two ways: The ad hoc mode allows you to execute individual tasks, i.e. one-time module calls. For example, you can first test whether all servers in the inventory are accessible to Ansible:

$ ansible -i inventory.yaml all -m ping

The inventory to be used is specified with -i. Since Ansible should address all hosts in the inventory, the keyword all is used. Alternatively, you could also write group names webservers and dbservers or the names of individual hosts (webserver1, …) here. The module for the task is specified with -m. The Ping module does more than a classic ICMP ping. It checks whether an SSH connection between master and slave is possible and whether a compatible version of Python is installed on the slave server.

The output of this command might look like this:

Source codebash


1
2
3
4


webserver1 | SUCCESS => {
 "changed": false,
 "ping": "pong"
}

After this successful check, further ad hoc commands can now be executed, such as installing a package:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


$ ansible -i inventory.yaml webservers -m package -a "name=nginx state=present" -b

webserver1 | SUCCESS => {
 "cache_update_time": 1553085209,
 "cache_updated": false,
 "changed": true,
 "stderr": "",
 "stderr_lines": [],
 "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n libnginx-mod-http-geoip libnginx-mod-http-image-filter\n libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream\n nginx-common nginx-core\nSuggested packages:\n fcgiwrap nginx-doc\nThe following NEW packages will be installed:\n libnginx-mod-http-geoip libnginx-mod-http-image-filter\n libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream nginx\n nginx-comm
 .........
}

The -b flag causes a privilege escalation so that, for example, a new package can be installed. The -a flag is used to pass the arguments that a module call should contain. In this case, this is the name of the package and the desired state, for example that a package should be installed.

Like most other tools of this type, Ansible complies with the idempotence principle. Users do not write specific commands, but rather define which state a system should reach, i.e. that a package should be installed. If - tailored to the specific case - nginx should be present on a web server, Ansible will either reinstall the package or simply do nothing if the package is already present. This behavior can also be observed when the call is made a second time:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


$ ansible -i inventory.yaml webservers -m package -a "name=nginx state=present" -b

webserver1 | SUCCESS => {
 "cache_update_time": 1553085209,
 "cache_updated": false,
 "changed": false
}

webserver2 | SUCCESS => {
 "cache_update_time": 1553085209,
 "cache_updated": false,
 "changed": false
}

webserver3 | SUCCESS => {
 "cache_update_time": 1553085209,
 "cache_updated": false,
 "changed": false
}

Ansible – like any configuration management system – requires information about the host it is supposed to manage:

Operating system
CPU
RAM
Network
Installed packages
…

The setup module is used to collect and display this:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


$ ansible -i inventory.yaml webserver1 -m setup

webserver2 | SUCCESS => {
 "ansible_facts": {
 "ansible_all_ipv4_addresses": [
 "10.0.0.3",
 "10.0.122.1",
 ],
 "ansible_all_ipv6_addresses": [
 "2001:0db8:85a3:0000:0000:8a2e:0370:7334",
 "2001:db8:3333:4444:5555:6666:7777:8888",
 ],
 "ansible_apparmor": {
 "status": "enabled"
 },
 "ansible_architecture": "x86_64",
 "ansible_bios_date": "03/01/2019",
 "ansible_bios_version": "1.5.0",
 ...
 }
}

Installation and configuration

Installing and configuring a web server usually involves more than one task:

Configuring the firewall
Installing the nginx package
Adjusting the nginx configuration
Restarting the nginx service
Setting a message-of-the-day per server

If you want to combine several tasks, you use a playbook. In this, the hosts key is first used to define in YAML syntax which hosts of the inventory the playbook should run on; in the example below, on all members of the webservers group. Then all tasks are defined under the tasks key, which should also run in exactly this order. Below is the finished playbook, which will be explained in its individual steps.

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


- hosts: webservers
 tasks:
 - name: Open port 80
 become: true
 firewalld:
 service: http
 permanent: true
 state: enabled
 register: firewall_setting

 - name: Restart the firewalld service
 become: true
 service:
 name: firewalld
 state: restarted
 when: firewall_setting.changed

 - name: Install nginx
 become: true
 package:
 name: "nginx-{{ nginx_version }}"
 state: present

 - name: Copy stylesheet in place
 become: true
 copy:
 src: stylesheet.css
 dest: "{{ nginx_root }}/stylesheet.css"
 register: stylesheet

 - name: Create default page
 become: true
 template:
 src: index.html.j2
 dest: "{{ nginx_root }}/index.html"
 register: indexfile

 - name: Restart nginx service
 become: true
 service:
 name: nginx
 state: restarted
 when: indexfile.changed or stylesheet.changed

 - name: Setup motd
 become: true
 template:
 src: motd.j2
 dest: /etc/motd

The folder structure now looks like this:

Source codebash


1
2
3
4


.
|
├── inventory.yaml
└── playbook.yaml

The first task uses the firewalld module to allow the http service within the firewall. The task also has a register parameter. If you specify this, Ansible saves the result of the task in a variable. The task also contains the parameter become: true. This means that this task is executed with rights escalation (the default is sudo). This authorization can also be set globally for the entire playbook, but for security and audit reasons it is recommended to work in a fine-grained manner and “authorize” each task individually.

After reconfiguring the firewall, it should be restarted - for this purpose the service module is used, which is intended for systemd-based services, among other things.

However, the restart should only happen if the previous task has resulted in a change: This behavior is controlled by the when parameter and the query of the firewall_setting variable, which was defined in the first task.

In the third task, the package module installs the nginx package on the web servers. To do this, the desired state is passed on - in this case present - and the name of the nginx package. Alternatively, the latest state could also be used, which would reference the latest version that the system gets from its respective package sources. In this case, the name is {{ nginx_version | default(“1.15.5”)}}. This is the first example of how variables can be used in Ansible.

Jinja syntax is used for this in Ansible. Variables are always put in {{ }} blocks. A variable called nginx_version must therefore be defined here. This can be saved in a file ./group_vars/webservers.yaml, for example:

Source codeyaml

1
2

---
nginx_version: 1.1.15

The folder structure now looks like this:

Source codebash


1
2
3
4
5
6


.
|
├── group_vars
│ └── webservers.yaml
├── inventory.yaml
└── playbook.yaml

Variables will be discussed in more detail later. For now, however, it should be mentioned that all variables defined in group_vars/webservers.yaml are available to all hosts that belong to the inventory group webservers. Similarly, you can create a group_vars/all.yaml file whose values then apply to all hosts in all inventory groups.

The next two tasks should now deliver a static website that should display information about the server’s operating system. To do this, first use the copy module to copy a CSS stylesheet into the root folder of the nginx web server, which the static website should later use. Before this can happen, the nginx_root variable must be set:

Source codeyaml


1
2
3
4


#group_vars/all.yaml
---
nginx_version: 1.12.2
nginx_root: "/usr/share/nginx/html"

For the website itself, the host facts that were previously used in connection with the setup module are used. If not explicitly prevented, Ansible collects these facts as the very first “internal” task in every playbook run, before the first user-defined task runs. Therefore, variables such as ansible_distribution, ansible_os_family, ansible_all_ipv4_adresses, etc. are available during the playbook.

The following template is used for the website:

Source codehtml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


<html>
<head>
<title>Information page: {{ ansible_hostname }}</title>
<style type="text/css">@import url('./stylesheet.css') all;</style>
</head>
<body>
<h1>Information about running host</h1>

<table>
 <tr> <th colspan='2'>OS Facts</th> </tr>
 <tr> <td>This system is running on</td> <td>{{ ansible_distribution }}</td> </tr>
 <tr> <td>Version:</td> <td>{{ ansible_distribution_version }}</td> </tr>
 <tr> <td>OS Family:</td> <td>{{ ansible_os_family}}</td> </tr>
 <tr> <td>Used package manager:</td> <td>{{ ansible_pkg_mgr }}</td> </tr>
 <tr> <td>AppArmor</td> <td>{{ ansible_apparmor['status'] }}</td> </tr>
 <tr> <td>Selinux</td> <td>{{ ansible_selinux['status'] }}</td> </tr>
 <tr> <td>Python Version</td> <td>{{ ansible_python_version }}</td> </tr>
</table>

<table style='float:left;margin-right:50px'>
 <tr> <th colspan='2'>Network information</th> </tr>
 <tr> <td>Hostname</td> <td>{{ ansible_nodename }}</td> </tr>
 <tr> <td>IPv4 addresses</td> <td>{{ ", ".join(ansible_all_ipv4_addresses) }}</td> </tr>
 <tr> <td>IPv6 addresses</td> <td>{{ ", ".join(ansible_all_ipv6_addresses) }}</td> </tr>
 <tr> <td>DNS servers</td> <td>{{ ", ".join(ansible_dns['nameservers']) }}</td> </tr>
 <tr> <td>DNS search domain</td> <td>{{ ", ".join(ansible_dns['search']) }}</td> </tr>
 {% for key, value in ansible_default_ipv4.iteritems() %}
 <tr> <td>Default IPv4 interface -- {{ key }}</td> <td>{{ value }}</td> </tr>
 {% endfor %}
 {% for key, value in ansible_default_ipv6.iteritems() %}
 <tr> <td>Default IPv6 interface -- {{ key }}</td> <td>{{ value }}</td> </tr>
 {% endfor %}
 <tr> <td>Hostname</td> <td>{{ ansible_hostname }}</td> </tr>
 <tr> <td>FQDN</td> <td>{{ ansible_fqdn }}</td> </tr>
</table>

<table>
 <tr> <th colspan='2'>Environment variables</th> </tr>
 {% for key, value in ansible_env.iteritems() %}
 <tr> <td>{{ key }}</td><td>{{ value }}</td> </tr>
 {% endfor %}
</table>

</body>
</html>

The variables included in Jinja syntax are then filled in by Ansible when the file is copied to the desired location using the template module. Template files are given the file extension .j2. With the template file and the stylesheet you then get the following folder structure:

Source codebash


1
2
3
4
5
6
7
8


.
|
├── group_vars
│ └── webservers.yaml
├── index.html.j2
├── inventory.yaml
├── playbook.yaml
└── stylesheet.css

The following task is similar to restarting the firewall service: If the index file or the stylesheet changes during deployment, nginx should be restarted. This is not necessarily necessary; the delivered files can also be changed during operation. The last task should now set a message of the day, i.e. a message that is displayed when logging into the server via SSH. Here, too, the template module is used to copy the desired file to etc/motd. A motd.j2 template file is added:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


.
|
├── group_vars
│ └── webservers.yaml
├── index.html.j2
├── inventory.yaml
├── motd.j2
└── playbook.yaml

# motd.j2
Hello and welcome on {{ ansible_hostname }}.
This is a {{ ansible_distribution }}-Server running in Version {{ ansible_distribution_version }}.
This is the {{ webserver_name }} instance.
Please note: This server is deployed and managed with Ansible.

The ansible_* variables come from the host facts, similar to the website template. However, this does not apply to webserver_name. Since this variable should be different for each server, it must also be set accordingly. There are several ways to do this. Two of them will be presented here.

Analogous to the variables defined for inventory groups in the group_vars folder, this can also happen for individual hosts:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


.
|
├── group_vars
│ └── webservers.yaml
├── host_vars
│ ├── webserver1.yaml
│ ├── webserver2.yaml
│ └── webserver3.yaml
├── index.html.j2
├── inventory.yaml
├── motd.j2
└── playbook.yaml

Source codeyaml


1
2
3


# host_vars/webserver1.yaml
---
webserver_name: "Webserver 1"

Source codeyaml


1
2
3


# host_vars/webserver2.yaml
---
webserver_name: "Webserver 2"

Source codeyaml


1
2
3


# host_vars/webserver3.yaml
---
webserver_name: "Webserver 3"

Variables defined in this way only apply to the host with the same name as the file.

Variables at host level can also be defined in the inventory file:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# inventory.yaml
---
all:
 hosts:
 webserver1:
 ansible_host: 10.0.0.2
 webserver_name: "Webserver 1"
 webserver2:
 ansible_host: 10.0.0.3
 webserver_name: "Webserver 2"
 webserver3:
 ansible_host: 10.0.0.4
 webserver_name: "Webserver 3"
 dbserver1:
 ansible_host: 10.0.1.1
 children:
 webservers:
 hosts:
 webserver1:
 webserver2:
 webserver3:
 dbservers:
 dbserver1:

The playbook is now basically finished and ready to run:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


$ ansible-playbook playbook.yaml -i inventory.yaml

PLAY [webservers] *************************************************************

TASK [Gathering Facts] ********************************************************
ok: [webserver1]
ok: [webserver2]
ok: [webserver3]

TASK [Open port 80] *******************************************
changed: [webserver1]
changed: [webserver2]
changed: [webserver3]

TASK [Restart the firewalld service] **********
changed: [webserver1]
changed: [webserver2]
changed: [webserver3]

TASK [Install ginx] *********************************************
changed: [webserver1]
changed: [webserver2]
changed: [webserver3]

TASK [Create default page] ****************************************************
changed: [webserver1]
changed: [webserver2]
changed: [webserver3]

TASK [Restart nginx service] **************************************************
changed: [webserver1]
changed: [webserver2]
changed: [webserver3]

TASK [Setup motd] *************************************************************
changed: [webserver1]
changed: [webserver2]
changed: [webserver3]

PLAY RECAP ********************************************************************
webserver1 : ok=7 changed=6 unreachable=0 failed=0
webserver2 : ok=7 changed=6 unreachable=0 failed=0
webserver3 : ok=7 changed=6 unreachable=0 failed=0

Collecting the facts is considered the first task, and this is the only task that does not cause any changes when it is initially executed. So that you do not always have to specify the inventory when executing a playbook, you can also create a configuration file ansible.cfg:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


.
|
├── ansible.cfg
├── group_vars
│ └── webservers.yaml
├── host_vars
│ ├── webserver1.yaml
│ ├── webserver2.yaml
│ └── webserver3.yaml
├── index.html.j2
├── inventory.yaml
├── motd.j2
└── playbook.yaml

Source codebash


1
2
3


# ansible.cfg
[defaults]
inventory=./inventory

By default, Ansible creates a default configuration file in /etc/ansible.cfg. However, this can be overwritten with a file ~/.ansible.cfg or a local one, as in this case. Ansible prioritizes these files in reverse order:

path specified in ANSIBLE_CONFIG environment variable
local in ./ansible.cfg
~/.ansible.cfg
/etc/ansible.cfg

In the file, you can not only specify one (or any number of additional) inventories, but also configure Ansible in numerous other ways:

SSH parameters, options
additional external modules
connection parameters
…

Many of these parameters can also be set as environmental variables. A complete list can be displayed using the CLI tool ansible-config.

The role concept

The final playbook now contains a number of tasks, but these are not easily reusable. To solve this problem and generally create more modularity, Ansible offers the role concept. If you transfer this to the nginx example, the folder structure changes:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


.
├── ansible.cfg
├── group_vars
│ └── webservers.yaml
├── host_vars
│ └── webserver1.yaml
├── inventory.yaml
├── playbook.yaml
└── roles
 └── nginx
 ├── defaults
 │ └── main.yaml
 ├── files
 │ └── stylesheet.css
 ├── tasks
 │ └── main.yaml
 ├── templates
 │ ├── index.html.j2
 │ └── motd.j2
 └── vars
 └── main.yaml

The playbook also looks different now:

Source codeyaml


1
2
3


- hosts: webservers
 roles:
 - nginx

Tasks are no longer called there, but a list of roles. In this case, this list only contains one element: the role for the nginx installation. This role can be found in the nginx subfolder of the new roles folder. The main entry point for a role is the main.yaml file in the tasks folder. All tasks that were previously in the playbook can now be found there:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# roles/nginx/tasks/main.yaml
---
- name: Open port 80
 become: true
 firewalld:
 service: http
 permanent: true
 state: enabled
 register: firewall_setting

 [...]

In addition, a role has the following folders:

files: This is where you can find files (except templates!) that the role uses. In this case, stylesheet.css.
templates: This is where you can find templates that the role uses. In this case, the website itself and the motd template.
vars/defaults: This is where you can find variables that a role uses. Depending on the purpose, these can be stored in both /vars/main.yaml and defaults/main.yaml.

When placing variables in different places, it is important to understand the order of these definition places. The following list contains (in ascending order) only the places that have been mentioned so far in this article. A complete list can be found in the Ansible documentation here.

variables in /role/defaults/main.yaml,
variables in inventory.yaml,
variables in group_vars/*.yaml,
variables in host_vars/*.yaml,
variables that were automatically determined from the setup module and
variables in /role/vars/main.yaml.

If you imagine a role that is supposed to install an Apache web server on Debian, the following options for variable placement arise:

The standard version number of the Apache installation is set in roles/apache/defaults/main.yaml. This can easily be overwritten later if necessary. The standard path for the root directory of the web server and the ports that are to be used are also set.

Source codeyaml


1
2
3
4
5


--- roles/apache/defaults/main.yaml
apache_root: "/var/www/html"
apache_version: 2.4.29
apache_listen_port: 80
apache_listen_ssl_port: 443

roles/apache/vars/main.yaml contains the name of the packages that are installed during the role. These are fixed, i.e. overwriting is not necessary.

Source codeyaml


1
2
3
4


---
apache_packages:
 - apache2
 - apache2-utils

A good role is reusable and transferable, which means that it is not limited to a single operating system, for example. If we take the Apache example again, there are differences in the package naming for Debian and Redhat, for example. In order to be able to use a role generically for this, we once again make use of Ansible facts.

The following examples can be found here. The directory structure looks simplified as follows:

Source codebash


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


.
├── playbook.yaml
└── roles
 └── apache
 ├── handlers
 │ └── main.yaml
 ├── tasks
 │ └── main.yaml
 └── vars
 ├── Debian.yaml
 └── RedHat.yaml

Depending on the operating system, either the variables from roles/apache/vars/Debian.yaml or roles/apache/vars/RedHat.yaml should be included:

Source codeyaml


1
2
3
4
5
6
7


---
#roles/apache/vars/Debian.yaml
apache_server_root: /etc/apache2
apache_service_name: apache2
apache_packages:
 - apache2
 - apache2-utils

Source codeyaml


1
2
3
4
5
6
7
8


--- 
#roles/apache/vars/RedHat.yaml
apache_server_root: /etc/httpd
apache_service_name: httpd
apache_packages:
 - httpd
 - httpd-devel
 - mod_ssl # Debian already ships the SSL module in `apache2`

In roles/apache/tasks/main.yaml findet sich folgendes:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


---
- name: Include OS-specific parameters
 include_vars: {{ ansible_os_family }}.yaml

- name: Install apache packages
 package:
 name: "{{ apache_packages }}"
 state: present
 notify: restart apache

- name: Activate mod_ssl if Debian
 file:
 src: "{{ apache_server_root }}/mods-available/mod_ssl"
 dest: "{{ apache_server_root }}/mods-enabled/mod_ssl"
 state: link
 notify: restart apache
 when: ansible_os_family == 'Debian'"

The first task uses the include_vars module to include the variable file appropriate for the current distribution. The second task then installs the packages defined in the respective file. The last task is a Debian peculiarity:

The Apache installation already includes various modules by default. To activate them, you can either use the CLI tool a2enmod, or you can create file links “by hand” from

Source codePlain text

1

/etc/apache2/mods-available/<mod_name>

after

Source codePlain text

1

/etc/apache2/mods-enabled/<mod_name>

The latter also happens in the task shown here, with the help of the file module. The when line ensures that a task is only executed if the following condition is evaluated as true. In this case, an operating system from the Debian family.

After activating a module, the server must be restarted. The handler concept is used here for this. Handlers are tasks that can be triggered at any point in the playbook. After the playbook has been completed, all handlers that were triggered during the playbook are processed in the order in which they were defined. In the Apache example, there is exactly one handler task:

Source codeyaml


1
2
3
4
5
6


#/roles/apache/handlers/main.yaml
---
- name: restart apache
 service:
 name: "{{ apache_service_name }}"
 state: restarted

A handler is triggered in the context of a task with the key notify and the task name of the handler. If you look more closely, you will notice that the task that installs the Apache packages also calls the same handler. This is due to the fact that in the RedHat scenario the Apache SSL module is installed as a separate package. In the Debian case, the handler is thus triggered twice. However, this does not mean that it is executed twice - even triggering a handler multiple times only results in it being executed once.

With the tag concept, Ansible can execute individual tasks of a playbook and skip all others. This can be explained well using an example: A company writes Java web applications, which it packages in war files using Maven and then deploys on a TomCat server. The latest test version is to be deployed and tested on a TomCat every night. Both the initial installation of TomCat and the nightly deployment of the latest version should be done with Ansible. The following playbook was written for this purpose:

Source codeyaml


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


---
- name: Install Tomcat Server
 import_tasks: tomcat_setup.yaml
 tags:
 - inital_setup

- name: Deploy war file
 maven_artifact:
 group_id: com.company
 artifact_id: web-app
 version: latest
 extension: war
 repository_url: 'https://repo.company.com/maven'
 dest: /var/lib/tomcat7/webapps/web-app.war
 notify: restart_tomcat
 tags:
 - initial_setup
 - nightly_test

The first task uses import_tasks to import a yaml file that contains all the tasks for installing and configuring a TomCat server. The second task downloads the latest version of the webapp as a war file, puts it in the TomCat server’s webapp folder, and triggers a handler that starts the TomCat server. In addition, both tasks have tags: the first only initial_setup, the second initial_setup and nightly_test.

This definition allows the following to be done:

ansible-playbook tomcat.yaml runs the playbook as usual, all tasks are executed.
ansible-playbook tomcat.yaml –tags <tag_name> executes all tasks that have been tagged with the corresponding tag.
ansible-playbook tomcat.yaml –tags initial_setup would also execute both tasks in this case. On the other hand, with ansible-playbook tomcat.yaml --tags nightly_test, only the new version of the web application can be deployed every night, without Ansible also checking the initial setup again for implementation and changes.

Conclusion

This article is of course not intended to be and cannot be a comprehensive and complete presentation of Ansible: There is much more that can be said about each of the points mentioned here. If you are interested, you can either take a look at the official Ansible documentation here or the community: be it in the form of IRC channels (#ansible, #ansible-dev) or the numerous relevant meet-ups and lectures that now exist.

The ideas and concepts shown here are intended to convey enough to give an initial insight and to dare to get started - it’s worth it.

Articles on Fabio Lopes

PHP2Shell on AWS: Incident Response and the Limits of Boot-Time Patching

The advisory was only the beginning

1. Start wide: identify potential exposure across applications

2. Contain exposure while remediation is prepared

A legacy PHP 7.4 module required a different decision

3. Verify runtimes in ECS workloads, not just on their EC2 hosts

Rebuilding corrected container images

4. A different remediation path: PHP directly on EC2

5. The deeper issue: new EC2 instances depended on a live repository

Why “install latest at boot” is not a patching strategy

6. Follow-up engineering work: identified, not yet completed

Build and validate versioned AMIs

Make remaining bootstrap tasks failure-aware

Strengthen inventory and deployment evidence

Rehearse dependency failures

7. What the different outcomes taught us

Final thoughts

References

Beyond Day-1 Terraform: Rethinking Platform Operations in My Environment

Introduction: The Day-2 Challenge

1. Understanding the Environment and Its Boundaries

Decision 1: Reference Infrastructure Owned by Other Teams

2. Separating Infrastructure Management from Application Delivery

Decision 2: Define Which Workflow Owns Each Attribute

3. Security Groups That Reflect the Actual Traffic Flow

4. Reducing Manual Work in Application Deployments

Decision 3: Use Internal CI/CD as the Deployment Boundary

5. Making Terraform Changes More Predictable

Separating Terraform State by Operational Responsibility

Protecting Stateful Resources

Detecting Unexpected Changes

6. The Trade-Offs I’m Willing to Make

Final Thoughts

Production Engineering for Single-Node Workloads: Controlled Releases, SQLite Backups, and Native ARM Builds

Architectural Steering & AI-Augmented Engineering

System Architecture: Single-Node Edge Topology

Native C++ Bindings Across Environments

Rules for Native Module Isolation

SQLite in Production: WAL Mode, Backup Integrity, and OCI Sync

1. Online Backup API Execution

2. Automated Integrity Validation

3. Keyless Cloud Replication via OCI Instance Principals

4. Controlled Restore and Session Invalidation

Process Management and Controlled Releases

Why I Chose PM2 Fork Mode

Versioned Release Directories

Deployment Lifecycle Execution

Stateful Security & Deterministic Session Invalidation

Session Versioning via auth_state

CI/CD Security Analysis (Semgrep SAST)

Operational Evolution After 10 Months (October 2026 Review)

Key Takeaways for SREs and Platform Engineers

Revision note

References

Automating Ceph Storage Forecasting with Grafana Mimir

Context and decisions at the time

Intro

What was done

Data Treatment and Forecasting Approach

The code

Future Improvements

The Ultimate Guide to the AWS Certified Solutions Architect – Professional (SAP-C02) Exam

Exam Domains and Weighting

Domain 1: Design Solutions for Organizational Complexity (26%)

1.1 Architect Multi-Account and Governance Strategies

1.2 Architect Advanced Identity and Security Strategies

1.3 Architect Complex Network Connectivity

Domain 2: Design for New Solutions (29%)

2.1 Architect Compute Solutions

2.2 Architect Storage Solutions

2.3 Architect Database Solutions

2.4 Architect Messaging and Integration Solutions

Domain 3: Continuous Improvement for Existing Solutions (25%)

3.1 Architect for High Availability and Resilience

3.2 Architect for Performance Optimization

3.3 Architect for Observability and Monitoring

Domain 4: Accelerate Workload Migration and Modernization (20%)

4.1 Architect Migration and Modernization Strategies (The 7 Rs)

4.2 Architect for Cost Optimization

Session Versioning via `auth_state`