Vsasf Tech ICT Academy, Enugu in early 2025 introduced a hybrid learning system that is flexible for all her courses offered to the general public. With E-learn platform powered by Vsasf Nig Ltd, all students can continue learning from far distance irrespective of one's location, hence promoting ODL system of education for Nigerians and the world at large.

Students are encouraged to continue learning online after fully registered through the academy's registration portal. All fully registered students with training fee payment completed can click on the login link Login to continue to access their course materials online

Kubernetes Horizontal Pod Autoscaling is another handy resource of kubernetes which helps you in autoscaling and managing your pods when there are overwhelming loads on your pods and the pods reach the defined limits.

As a DevOps engineer you might have experienced exceeding resources like CPU and Ram on your workload and this is considered pretty normal, but things will be messed up when you are absent to take control of pods. Fortunately, HPA perfectly fits this issue and it auto scales your pods when resource usage meets the limits..

This is how Kubernetes HPA work, the metric server sends metrics of resource consumption to HPA and based on the rules you have defined in HPA manifest file, this object decides to scale up or down the pods. For example, if the CPU usage was more than 80 percentage, the HPA order replica Set and deployment to scale up pods and if the usage came below 10 percentage, the additional pods will be removed.

Let’s get our hands dirty and deploy a simple project with Kubernetes horizontal auto scaling step by step:

1. Deploy Metric Server

Metric server is an additional module of Kubernetes which is part of Kubernetes-sigs repository. Kubernetes metric server acts like a metric exporter on Kubernetes cluster that expose resource usage of nodes, pods … for many kinds of purposes. As mentioned before, HPA use metric server to observe pods resource usage.

To deploy the metric server, get the latest manifest file of metric server from here and add these parameters on the metric server deployment file and run.

spec: hostNetwork: true containers: - args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --kubelet-use-node-status-port - --metric-resolution=15s - --kubelet-insecure-tls

kubectl apply -f metric-server.yaml file

After deploying the manifest file, check the availability of metric server with the below command. Remember the marked API must be visible and True.

Now you can see the pods and nodes metrics with these two commands

kubectl top pods

kubectl top nodes

2. Deploy an application

Deploying a containerized application is a must to test HPA which can be anything you want but you have to determine resource limitation or request parameters on the manifest Yaml file.

resources: limits: memory: "128Mi" cpu: "500m"

In my example I have a simple web server and defined limits on my resources.

3. Creating HPA

After deploying your application, it is time for creating HPA manifest.

In this article I have mentioned both versions of autoscaling/v1 and autoscaling/v2

This is the manifest file for v1

In this approach we have defined the API Version of our deployment object and the name of it.

We have demonstrated the minimum replicas of the pointed deployment and the maximum replicas, although the target CPU usage is 70 percent. It means if the CPU usage was higher than 70% the replicas would be scaled up to 4 replicas and after decreasing CPU usage under 70%, the replicas numbers will be 1 again.

In my example, I have added “stress” package to the base image and increased CPU usage but you can do it with http request or any other way based on your application. I have entered the container environment and ran this command “stress -c 10” which simulates like 10 core CPU.

This is the result after some seconds and the pods scaled up until 4. The average container resource is like 70% and after reaching the threshold, the replicas would scale up to 4 pods.

Autoscaling version 2 is the new and better approach which gives you more accessibility on pods and lets you assign different policy on the HPA.

On the autoscaling/v2 you can still see min and max replicas keys and they have same behavior. But the most important feature is behavior. Behavior is divided into two scale down and scale up section which lets you define policies for both up and down scale.

As it seems in the scale up policy section If the pod`s CPU usage became higher that 50 percentage, after 0 seconds the pods will be scaled up to 4 replicas. But in the scale down part if the CPU usage would be lower than 90% and yet after 15 seconds and staying the usage stable for 300 seconds, the pods would scale down to one replica.

4. Extra points

There are some other points in the v2 API version which I should mention.

Look at this example:

behavior: scaleDown: policies: - type: Percent value: 10 periodSeconds: 60 - type: Pods value: 5 periodSeconds: 60 selectPolicy: Min

We have two policies here, to ensure that no more than 5 Pods are removed per minute, you can add a second scale-down policy with a fixed size of 5, and set selectPolicy to minimum. Setting selectPolicy to Min means that the autoscaler chooses the policy that affects the smallest number of Pods.

selectPolicy: Min

If you set selectPolicy to “Min,” the HPA will select the policy with the minimum value among the matching policies during a scaling decision.

Example: If you have two policies for scaling up based on CPU utilization, one with a percentage increase of 150% and another with a pod count increase of 5, the policy with the minimum increase (5 in this case) will be selected.

selectPolicy: Max

Conversely, if you set selectPolicy to “Max,” the HPA will select the policy with the maximum value among the matching policies during a scaling decision.

Example: Using the same scenario as above, if you have two policies for scaling up based on CPU utilization, one with a percentage increase of 150% and another with a pod count increase of 5, the policy with the maximum increase (150% in this case) will be selected.

The selectPolicy value of Disabled turns off scaling the given direction. So to prevent downscaling policy would be used.

Authors: T. C. Okenna
Register for this course: Enrol Now

Kubernetes on AWS: Step by Step

In this article we provide step-by-step instructions for several common ways to set up a Kubernetes cluster on AWS:

• Creating a cluster with kobs — kops is a production-grade tool used to install, upgrade and manage Kubernetes on AWS.
• Creating a cluster with Amazon Elastic Kubernetes Service (EKS) — the managed Kubernetes service provided by Amazon. You can create a Kubernetes cluster with EKS using the AWS Management Console.
• Creating a cluster with Rancher — Rancher is a Kubernetes management platform that eases the deployment of Kubernetes and containers.

Deploying Kubernetes on AWS Using Kops

Kops is a production-grade tool used to install, upgrade, and operate highly available Kubernetes clusters on AWS and other cloud platforms using the command line.

Installing a Kubernetes Cluster on AWS

Before proceeding, make sure to have installed kubectl , kops , and AWS CLI tools.

Configure AWS Client with Access Credentials

Make sure AWS IAM user has the following permissions for kops to function properly:

– AmazonEC2FullAccess
– AmazonRoute53FullAccess
– AmazonS3FullAccess
– IAMFullAccess
– AmazonVPCFullAccess

Configure AWSCLIi with this user’s credentials by running:

# aws configure

Create S3 Bucket for Cluster State Storage

Create a dedicated S3 bucket that will be used by kops to store the state representing the cluster. We’ll name this bucket my-cluster-state :

# aws s3api create-bucket --bucket my-cluster-state

Make sure to activate bucket versioning to be able to later recover or revert to a previous state:

# aws s3api put-bucket-versioning --bucket my-cluster-state --versioning-configuration Status=Enabled

DNS Setup

On the DNS side, you can go with either public or private DNS. For public DNS, a valid top-level domain or subdomain is required to create the cluster. DNS is required by worker nodes to discover the master and by the master to discover all the etcd servers. A domain whose registrar is not AW creates a Route 53 hosted zone on AWS and changes nameserver records on your registrar accordingly.

In this example,e we’ll be using a simple, private DNS to create a gossip-based cluster . The only requirement to set this up is for our cluster name to end with k8s.local.

Creating the Kubernetes Cluster

The following command will create a 1 master (an m3.medium instance) and 2 nodes (two t2.medium instances) cluster in us-west-2a availability zone:

# kops create cluster  --name my-cluster.k8s.local  --zones us-west-2a  --dns private  --master-size=m3.medium  --master-count=1  --node-size=t2.medium  --node-count=2  --state s3://my-cluster-state  --yes

Some of the command options in the above example have default values: --master-size , --master-count , --node-size , and --node-count . We’ve used the default values so the result would be the same if we hadn’t specified those options. Also,o note that kops will create one master node in each availability zone specified, so this option: --zones us-west-2a,us-west-2b would result in 2 master nodes, one in each of the two zones (even if --master-count was not specified in the command line).

Note that cluster creation may take a while as instances must boot, download the standard Kubernetes components and reach a “ready” state. Kops provides a command to check the state of the cluster and check it’s ready:

# kops validate cluster --state=s3://my-cluster-state Using cluster from kubectl context: my-cluster.k8s.local Validating cluster my-cluster.k8s.local INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master m3.medium 1 1 us-west-2a nodes Node t2.medium 2 2 us-west-2a NODE STATUS NAME ROLE READY ip-172-20-32-203.us-west-2.compute.internal node True ip-172-20-36-109.us-west-2.compute.internal node True ip-172-20-61-137.us-west-2.compute.internal master True Your cluster my-cluster.k8s.local is ready

If you want to make some changes to the cluster, do so by running:

# kops edit cluster my-cluster.k8s.local # kops update cluster my-cluster.k8s.local --yes

Upgrading the Cluster to a Later Kubernetes Release

Kops can upgrade an existing cluster (master and nodes) to the latest recommended release of Kubernetes without having to specify the exact version. Kops supports rolling cluster upgrades where the master and worker nodes are upgraded one by one.

1. Update Kubernetes

# kops upgrade cluster  --name $NAME  --state s3://my-cluster-state  --yes

2. Update the state store to match the cluster state.

# kops update cluster  --name my-cluster.k8s.local  --state s3://my-cluster-state  --yes

3. Perform the rolling update.

# kops rolling-update cluster  --name my-cluster.k8s.local  --state s3://my-cluster-state  --yes

This will perform updates on all instances in the cluster, first master and then workers.

Delete the Cluster

To destroy an existing cluster that we used for experimenting or trials, for example, we can run:

# kops delete cluster my-cluster.k8s.local  --state=s3://my-cluster-state  --yes

Using Kubernetes EKS Managed Service

Amazon Elastic Container Service for Kubernetes (EKS) is a fully managed service that takes care of all the cluster setup and creation, ensuring multi-AZ support on all clusters and automatic replacement of unhealthy instances (master or worker nodes).

By default clusters in EKS consist of 3 masters spread across 3 different availability zones to protect against the failure of a single AWS availability zone:

Standing up a new Kubernetes cluster with EKS can be done simply using the AWS Management Console. After getting access to the cluster, containerized applications can be scheduled in the new cluster in the same fashion as with any other Kubernetes installation:

Launching Kubernetes on EC2 Using Rancher

Rancher is a complete container management platform that eases the deployment of Kubernetes and containers.

Setting Up Rancher in AWS

Rancher (the application) runs on RancherOS, which is available as an Amazon Machine Image (AMI), and thus can be deployed on an EC2 instance.

Create RancherOS Instance on EC2

After installing and configuring the AWS CLI tool, proceed to create an EC2 instance using RancherOS AMI. Check RancherOS documentation for AMI ids for each region. For example this command:

$ aws ec2 run-instances --image-id ami-12db887d --count 1 --instance-type t2.micro --key-name my-key-pair --security-groups my-sg

will create one new t2.micro EC2 instance with RancherOS on ap-south-1 AWS region. Make sure to use the correct key name and security group. Also,o make sure the security group enables traffic to TCP port 8080 to the new instance.

Start Rancher Server

When the new instance is ready, just connect using ssh and start the Rancher server:

$ sudo docker run --name rancher-server -d --restart=unless-stopped  -p 8080:8080 rancher/server:stable

This might take a few minutes. Once done, the UI can be accessed on port 8080 of the EC2 instance . Since by default anyone can access Rancher’s UI and API, it is recommended to set up access control.

Creating a Kubernetes cluster via Rancher in AWS

Configure Kubernetes environment template

An environment in Rancher is a logical entity for sharing deployments and resources with different sets of users. Environments are created from templates.

Create the Kubernetes Cluster (environment)

Adding a Kubernetes environment is just a matter of selecting the adequately configured template for our use case and inputting the cluster name. If access control is turned on, we can add members and select their membership role . Anyone added to the membership list would have access to the environment.

Add Hosts to Kubernetes Cluster

We need to add at least one host to the newly created Kubernetes environment. In this case, the hosts will be previously created AWS EC2 instances.

Once the first host has been added, Rancher will automatically start the deployment of the infrastructure (master) including Kubernetes services (i.e. kubelet, etcd,Kubee-proxy, etc). Hosts that will be used as Kubernetes nodes will require TCP ports 10250 and 10255 to be open for kubectl. Make sure to review the full list of Rancher requirements for the hosts.

It might take a few minutes for the Kubernetes cluster setup/update to complete, after adding hosts to the Kubernetes environment:

Deploying Applications in the Kubernetes Cluster

Once the cluster is ready containerized applications can be deployed using either the Rancher application catalog or kubectl.

Other Options for Deploying Kubernetes in the Cloud

Besides the Kubernetes deployment options already covered, other tools can be used to deploy Kubernetes on public clouds like AWS. Each tool has its unique features and building blocks:

• Heptio — Heptio provides a solution based on CloudFormation and kubeadm to deploy Kubernetes on AWS, and supports multi-AZ. Heptio is suitable for users already familiar with the CloudFormation AWS orchestration tool.
• Kismatic Enterprise Toolkit (KET) — KET is a collection of tools with sensible defaults which are production-ready to create enterprise-tuned clusters of Kubernetes.
• kubeadm — The kubeadm project is focused on a solution to build a simple cluster on AWS using Terraform. It is an adequate tool for tests and proofs-of-concept only as it doesn’t support multi-AZ and other advanced

Authors: Medium, AVM Consulting Blog
Register for this course: Enrol Now

Testing Kubernetes Application

Kubernetes enabled the use of a Cloud Native model to build, deploy and scale applications. But it also introduces some challenges on how to test these application, or rather modules, effectively.

What tests can I use for my microservices?

There are common types of test that you can run with any application, but microservices architecture that run on Kubernetes introduce a new set of scenarios that need to be tested as well, making the list of the types of tests you can run for your microservices architecture longer.

Here's a list of test types:

• Unit Tests: These tests focus on individual components or functions of a microservice. They are usually performed during the development phase and don't involve Kubernetes.
• Integration Tests: Integration tests check if different components of an application work together as expected. They can involve testing communication between microservices or databases and interactions with external APIs.
• End-to-End (E2E) Tests: E2E tests simulate user workflows and verify that the entire application functions as expected from the user's perspective. These tests often require a Kubernetes environment to run the application components.
• Performance or Load Tests: Performance tests assess the application's ability to handle load and maintain responsiveness under various conditions. These tests may involve stress testing, load testing, and benchmarking on a Kubernetes cluster.
• Resilience Tests or Chaos Tests: This type of tests evaluate an application's ability to recover from failures and continue functioning. They can involve chaos testing, failover testing, and disaster recovery testing on a Kubernetes cluster.

Challenges of running tests in Kubernetes

There are amazing tools that can run multiple types of types of tests in Kubernetes, but they all have similar challenges that you might face when running these tests.

These challenges include:

• Managing long pipeline of tests: putting all your test in a single monolith CI/CD pipeline can slow down heavily the Integration and Deployment of the microservices.
• Storing the result of tests: all test tools generate artifacts, these could be text based or it could be screenshot or a video of your app. You will have to think of how to store these artifacts in order to retrieve them and see what went wrong.
• Knowing when to run a test: usually tests are run in a CI/CD pipeline, but not every type of test should be run with every execution of CI/CD, for example, compute-intensive tests or load tests should be run in your CI/CD pipeline.
• Retriggering of tests: if your tests live in a CI/CD pipeline and you want to retrigger a test, you will have to retrigger the entire CI/CD pipeline. This can incur huge wait time specially if your pipelines are long.

These are some of the tests challenges we faced when building out application with a microservice architecture. The right strategy to operationally manage these tests should help scale your testing efforts in your DevOps lifecycle.

Where and when can I run my tests?

Not all the tests are run at the same time and from the same place. For different scenarios and test types there's an optimal time in the DevOps lifecycle where you can test the application.

• Unit tests: These tests usually have a fast feedback loop that would enable developers to run them in their machines in seconds and once deployed, they can run in your Continuous Integration pipeline quickly.
• Integration tests: These tests are usually also executed in the Continuous Integration pipeline as they tend to be relatively fast and if well
• End-to-End (E2E) tests: E2E tests . These tests tend to test out the entire system so you want to be flexible on when to run them. Your CI/CD pipeline can be a place to hold these tests, but they tend to overcomplicate how you orchestrate the execution of the test and the retrieval of the results.
• Performance or Load tests: These tests can be scheduled at specific intervals during the development cycle or triggered by specific events such as a new release candidate (or manually ;)). Performance tests should be executed in an environment that closely resembles the production environment to accurately gauge the application's performance under real-world conditions.

What tools can I use to test my microservices?

As described earlier that are different types of tests that you can run in Kubernetes and at different

1. Artillery: Artillery is a load testing tool that can be used to simulate high traffic on a Kubernetes application. It can help you identify performance issues, bottlenecks, and other issues that may arise under heavy load.
2. Curl: Curl is a command-line tool that can be used to make HTTP requests to Kubernetes resources like pods, services, and deployments. You can use it to test connectivity and troubleshoot issues with Kubernetes resources.
3. Cypress: Cypress is an end-to-end testing tool that can be used to test the functionality of your Kubernetes application. It can help you identify and fix issues with your application's user interface and workflows.
4. Ginkgo: Ginkgo is a testing framework that can be used to write and run automated tests for your Kubernetes application. It's designed to be highly configurable and can be used to test a variety of different scenarios.
5. Gradle: Gradle is a build automation tool that can be used to build and deploy your Kubernetes application. It can help you manage dependencies, run tests, and package your application for deployment.
6. JMeter: JMeter is a load testing tool that can be used to simulate high traffic on your Kubernetes application. It can help you identify performance issues and bottlenecks under different load conditions.
7. K6: K6 is another load testing tool that can be used to test the performance of your Kubernetes application. It's designed to be developer-friendly and can be used to write and run tests in JavaScript.
8. KubePug: KubePug is a tool that can be used to scan your Kubernetes manifests for potential security vulnerabilities. It can help you identify security issues before they become a problem.
9. Maven: Maven is a build automation tool that can be used to build and deploy your Kubernetes application. It's designed to be highly configurable and can be used to manage dependencies, run tests, and package your application for deployment.
10. Playwright: Playwright is an end-to-end testing tool that can be used to test the functionality of your Kubernetes application. It's designed to be highly configurable and can be used to test a variety of different scenarios.
11. Postman: Postman is a tool that can be used to test your Kubernetes API endpoints. It's designed to be highly configurable and can be used to test a variety of different scenarios.
12. SoapUI: SoapUI is a tool that can be used to test SOAP and REST web services. It can help you test the functionality of your Kubernetes application's API endpoints and identify issues with your application's data exchange.

Conclusion

In conclusion, Kubernetes has revolutionized the way we build, deploy, and scale applications using a Cloud Native model. However, it also brings unique challenges in testing microservices effectively.

To overcome these challenges and scale testing efforts in your DevOps lifecycle, it's essential to adopt the right strategy for managing and executing tests. By doing so, you can ensure the reliability, performance, and security of your Kubernetes-based microservices applications.

Authors: T. C. Okenna, Testkube
Register for this course: Enrol Now

Introduction to Kubernetes (K8S)

Authors: T. C. Okenna
Register for this course: Enrol Now

Continuous Integration and Continuous Delivery

Continuous integration (CI) and continuous delivery (CD) are key pillars of modern development. If you’re new to these concepts, here’s a quick rundown:

• Continuous integration (CI): Developers frequently commit their code to a shared repository, triggering automated builds and tests. This practice prevents conflicts and ensures defects are caught early.
• Continuous delivery (CD): With CI in place, organizations can then confidently automate releases. That means shorter release cycles, fewer surprises, and the ability to roll back changes quickly if needed.

Leveraging CI/CD can dramatically improve your team’s velocity and quality. Once you experience the benefits of dependable, streamlined pipelines, there’s no going back.

Why combine Docker and Jenkins for CI/CD?

Docker allows you to containerize your applications, creating consistent environments across development, testing, and production. Jenkins, on the other hand, helps you automate tasks such as building, testing, and deploying your code. I like to think of Jenkins as the tireless “assembly line worker,” while Docker provides identical “containers” to ensure consistency throughout your project’s life cycle.

Here’s why blending these tools is so powerful:

• Consistent environments: Docker containers guarantee uniformity from a developer’s laptop all the way to production. This consistency reduces errors and eliminates the dreaded “works on my machine” excuse.
• Speedy deployments and rollbacks: Docker images are lightweight. You can ship or revert changes at the drop of a hat — perfect for short delivery process cycles where minimal downtime is crucial.
• Scalability: Need to run 1,000 tests in parallel or support multiple teams working on microservices? No problem. Spin up multiple Docker containers whenever you need more build agents, and let Jenkins orchestrate everything with Jenkins pipelines.

For a DevOps junkie like me, this synergy between Jenkins and Docker is a dream come true.

Setting up your CI/CD pipeline with Docker and Jenkins

Before you roll up your sleeves, let’s cover the essentials you’ll need:

• Docker Desktop (or a Docker server environment) installed and running. You can get Docker for various operating systems.
• Jenkins downloaded from Docker Hub or installed on your machine. These days, you’ll want jenkins/jenkins:lts (the long-term support image) rather than the deprecated library/jenkins image.
• Proper permissions for Docker commands and the ability to manage Docker images on your system.
• A GitHub or similar code repository where you can store your Jenkins pipeline configuration (optional, but recommended).

Pro tip: If you’re planning a production setup, consider a container orchestration platform like Kubernetes. This approach simplifies scaling Jenkins, updating Jenkins, and managing additional Docker servers for heavier workloads.

Building a robust CI/CD pipeline with Docker and Jenkins

After prepping your environment, it’s time to create your first Jenkins-Docker pipeline. Below, I’ll walk you through common steps for a typical pipeline — feel free to modify them to fit your stack.

1. Install necessary Jenkins plugins

Jenkins offers countless plugins, so let’s start with a few that make configuring Jenkins with Docker easier:

• Docker Pipeline Plugin
• Docker
• CloudBees Docker Build and Publish

How to install plugins:

1. Open Manage Jenkins > Manage Plugins in Jenkins.
2. Click the Available tab and search for the plugins listed above.
3. Install them (and restart Jenkins if needed).

Code example (plugin installation via CLI):

Pro tip (advanced approach): If you’re aiming for a fully infrastructure-as-code setup, consider using Jenkins configuration as code (JCasC). With JCasC, you can declare all your Jenkins settings — including plugins, credentials, and pipeline definitions — in a YAML file. This means your entire Jenkins configuration is version-controlled and reproducible, making it effortless to spin up fresh Jenkins instances or apply consistent settings across multiple environments. It’s especially handy for large teams looking to manage Jenkins at scale.

2. Set up your Jenkins pipeline

In this step, you’ll define your pipeline. A Jenkins “pipeline” job uses a Jenkinsfile (stored in your code repository) to specify the steps, stages, and environment requirements.

Example Jenkinsfile:

Let’s look at what’s happening here:

1. Checkout: Pulls your repository.
2. Build: Creates a built docker image (your-org/your-app) with the build number as a tag.
3. Test: Runs your test suite inside a fresh container, ensuring Docker containers create consistent environments for every test run.
4. Push: Pushes the image to your Docker registry (e.g., Docker Hub) if the tests pass.

3. Configure Jenkins for automated builds

Now that your pipeline is set up, you’ll want Jenkins to run it automatically:

• Webhook triggers: Configure your source control (e.g., GitHub) to send a webhook whenever code is pushed. Jenkins will kick off a build immediately.
• Poll SCM: Jenkins periodically checks your repo for new commits and starts a build if it detects changes.

Which trigger method should you choose?

• Webhook triggers are ideal if you want near real-time builds. As soon as you push to your repo, Jenkins is notified, and a new build starts almost instantly. This approach is typically more efficient, as Jenkins doesn’t have to continuously check your repository for updates. However, it requires that your source control system and network environment support webhooks.
• Poll SCM is useful if your environment can’t support incoming webhooks — for example, if you’re behind a corporate firewall or your repository isn’t configured for outbound hooks. In that case, Jenkins routinely checks for new commits on a schedule you define (e.g., every five minutes), which can add a small delay and extra overhead but may simplify setup in locked-down environments.

Personal experience: I love webhook triggers because they keep everything as close to real-time as possible. Polling works fine if webhooks aren’t feasible, but you’ll see a slight delay between code pushes and build starts. It can also generate extra network traffic if your polling interval is too frequent.

4. Build, test, and deploy with Docker containers

Here comes the fun part — automating the entire cycle from build to deploy:

1. Build Docker image: After pulling the code, Jenkins calls docker.build to create a new image.
2. Run tests: Automated or automated acceptance testing runs inside a container spun up from that image, ensuring consistency.
3. Push to registry: Assuming tests pass, Jenkins pushes the tagged image to your Docker registry — this could be Docker Hub or a private registry.
4. Deploy: Optionally, Jenkins can then deploy the image to a remote server or a container orchestrator (Kubernetes, etc.).

This streamlined approach ensures every step — build, test, deploy — lives in one cohesive pipeline, preventing those “where’d that step go?” mysteries.

5. Optimize and maintain your pipeline

Once your pipeline is up and running, here are a few maintenance tips and enhancements to keep everything running smoothly:

• Clean up images: Routine cleanup of Docker images can reclaim space and reduce clutter.
• Security updates: Stay on top of updates for Docker, Jenkins, and any plugins. Applying patches promptly helps protect your CI/CD environment from vulnerabilities.
• Resource monitoring: Ensure Jenkins nodes have enough memory, CPU, and disk space for builds. Overworked nodes can slow down your pipeline and cause intermittent failures.

Pro tip: In large projects, consider separating your build agents from your Jenkins controller by running them in ephemeral Docker containers (also known as Jenkins agents). If an agent goes down or becomes stale, you can quickly spin up a fresh one — ensuring a clean, consistent environment for every build and reducing the load on your main Jenkins server.

Why use Declarative Pipelines for CI/CD?

Although Jenkins supports multiple pipeline syntaxes, Declarative Pipelines stand out for their clarity and resource-friendly design. Here’s why:

• Simplified, opinionated syntax: Everything is wrapped in a single pipeline { ... } block, which minimizes “scripting sprawl.” It’s perfect for teams who want a quick path to best practices without diving deeply into Groovy specifics.
• Easier resource allocation: By specifying an agent at either the pipeline level or within each stage, you can offload heavyweight tasks (builds, tests) onto separate worker nodes or Docker containers. This approach helps prevent your main Jenkins controller from becoming overloaded.
• Parallelization and matrix builds: If you need to run multiple test suites or support various OS/browser combinations, Declarative Pipelines make it straightforward to define parallel stages or set up a matrix build. This tactic is incredibly handy for microservices or large test suites requiring different environments in parallel.
• Built-in “escape hatch”: Need advanced Groovy features? Just drop into a script block. This lets you access Scripted Pipeline capabilities for niche cases, while still enjoying Declarative’s streamlined structure most of the time.
• Cleaner parameterization: Want to let users pick which tests to run or which Docker image to use? The parameters directive makes your pipeline more flexible. A single Jenkinsfile can handle multiple scenarios — like unit vs. integration testing — without duplicating stages.

Declarative Pipeline examples

Below are sample pipelines to illustrate how declarative syntax can simplify resource allocation and keep your Jenkins controller healthy.

Example 1: Basic Declarative Pipeline

• Runs on any available Jenkins agent (worker).
• Uses two stages in a simple sequence.

Example 2: Stage-level agents for resource isolation

• Each stage runs in its own container, preventing any single node from being overwhelmed.
• agent none at the top ensures no global agent is allocated unnecessarily.

Example 3: Parallelizing test stages

• Splits tests into two parallel stages.
• Each stage can run on a different node or container, speeding up feedback loops.

Example 4: Parameterized pipeline

• Lets you choose which tests to run (unit, integration, or both).
• Only executes relevant stages based on the chosen parameter, saving resources.

Example 5: Matrix builds

• Defines a matrix of PLATFORM x BROWSER, running each combination in parallel.
• Perfect for testing multiple OS/browser combinations without duplicating pipeline logic.

Using Declarative Pipelines helps ensure your CI/CD setup is easier to maintain, scalable, and secure. By properly configuring agents — whether Docker-based or label-based — you can spread workloads across multiple worker nodes, minimize resource contention, and keep your Jenkins controller humming along happily.

Best practices for CI/CD with Docker and Jenkins

Ready to supercharge your setup? Here are a few tried-and-true habits I’ve cultivated:

• Leverage Docker’s layer caching: Optimize your Dockerfiles so stable (less frequently changing) layers appear early. This drastically reduces build times.
• Run tests in parallel: Jenkins can run multiple containers for different services or microservices, letting you test them side by side. Declarative Pipelines make it easy to define parallel stages, each on its own agent.
• Shift left on security: Integrate security checks early in the pipeline. Tools like Docker Scout let you scan images for vulnerabilities, while Jenkins plugins can enforce compliance policies. Don’t wait until production to discover issues.
• Optimize resource allocation: Properly configure CPU and memory limits for Jenkins and Docker containers to avoid resource hogging. If you’re scaling Jenkins, distribute builds across multiple worker nodes or ephemeral agents for maximum efficiency.
• Configuration management: Store Jenkins jobs, pipeline definitions, and plugin configurations in source control. Tools like Jenkins Configuration as Code simplify versioning and replicating your setup across multiple Docker servers.

With these strategies — plus a healthy dose of Declarative Pipelines — you’ll have a lean, high-octane CI/CD pipeline that’s easier to maintain and evolve.

Troubleshooting Docker and Jenkins Pipelines

Even the best systems hit a snag now and then. Here are a few hurdles I’ve seen (and conquered):

• Handling environment variability: Keep Docker and Jenkins versions synced across different nodes. If multiple Jenkins nodes are in play, standardize Docker versions to avoid random build failures.
• Troubleshooting build failures: Use docker logs -f <container-id> to see exactly what happened inside a container. Often, the logs reveal missing dependencies or misconfigured environment variables.
• Networking challenges: If your containers need to talk to each other — especially across multiple hosts — make sure you configure Docker networks or an orchestration platform properly. Read Docker’s networking documentation for details, and check out the Jenkins diagnosing issues guide for more troubleshooting tips.

Conclusion

Pairing Docker and Jenkins offers a nimble, robust approach to CI/CD. Docker locks down consistent environments and lightning-fast rollouts, while Jenkins automates key tasks like building, testing, and pushing your changes to production. When these two are in harmony, you can expect shorter release cycles, fewer integration headaches, and more time to focus on developing awesome features.

A healthy pipeline also means your team can respond quickly to user feedback and confidently roll out updates — two crucial ingredients for any successful software project. And if you’re concerned about security, there are plenty of tools and best practices to keep your applications safe.

Authors: T. C. Okenna, Docker Docs
Register for this course: Enrol Now