Managing Jenkins Software tips to optimize DevOps maintenance

Disclaimer: we love Jenkins and support hundreds of developers daily with it. The team here at Servana manages hundreds of Jenkins controllers and we wanted to help all the software teams self-hosting Jenkins Software out there with some tips that we consider essential when managing Jenkins software today.

Despite what people think, Jenkins' popularity is still really high. Jenkins Software is the first choice for busy software development teams because with Jenkins pipelines and the enormous plugin ecosystem, they can automate the various stages in the software development lifecycle.

However, to fully harness the potential of Jenkins software, teams need to take on reasonably complicated optimization, maintenance, and update tasks. In this comprehensive guide, we will unveil insider tips and best practices to help you manage Jenkins software effectively. We'll cover installing Jenkins as well as updating and hardening robust security measures to safeguard your pipelines, and efficiently managing plugins and dependencies to enhance functionality. By leveraging these insights, you can elevate your software development processes, gain a competitive edge, and unlock the full potential of Jenkins.

Checkout our Managed Jenkins Service to streamline your CI/CD pipelines and achieve unparalleled efficiency.

How do you optimize performance while ensuring reliability and efficiency?

No one wants slow pipelines, slow CI instances, and a slow Jenkins. What causes slowness? The first culprit is the size of the instance. Jenkins is built with Java and requires enough memory to be able to run garbage collection routines while also being able to operate. Jenkins is a filesystem-based application and this means it relies on storage iops. Your instance needs to be rightsized for the total number of jobs on the Jenkins controller and once Jenkins has started enough memory to read the config files. Then you want some available space for garbage collection. With containers, we have become disconnected from the server and it is now more common to run out of Inodes before you run out of disk space.

1. Use an instance with enough capacity.

2. To maintain inode levels, limit pipeline build history.

3. Tune the JVM to make sure there is adequate space for a garbage collection.

Efficient resource allocation lies at the heart of performance optimization. By closely monitoring resource usage, you gain valuable insights into the demands placed on your Jenkins server. This knowledge empowers you to make informed decisions regarding server sizing, increasing or decreasing the size of the server as needed to maintain peak performance.

What are the most important Jenkins configuration best practices?

Yes, there are thousands of configurations and it can be difficult to know what to do where. To improve how you manage the Jenkins configuration, we recommend using the configuration as a code plugin. It will help to make certain configurations read only if you manage the configuration as code externally to the Jenkins controller.

Setup the Support Core plugin.

When you have issues with a pipeline, for example, errors on the controller or agent, then the Support Core plugin can be used to create a thread dump of the pipeline and enable you to understand what could be causing the issue.

Create Views when you have large amounts of jobs and pipelines. It can seem overwhelming managing hundreds of pipelines, but this process becomes much simpler when you create views.

Use shared libraries and declarative pipelines to manage your application's lifecycle processes.

Under Manage Jenkins -> System -> Job Config History

1. Configuration exclude file pattern

queue\.xml|nodeMonitors\.xml|UpdateCenter\.xml|global-build-stats|LockableResourcesManager\.xml|MilestoneStep\.xml|cloudbees-disk-usage-simple\.xml

1. Check 'Do not save duplicate history'

2. Under 'Show build badges' check 'Always'

3. Check 'Show the change message window in jobs' configure pages'

Under Manage Jenkins -> System -> # of executors

Turn off the executors on the controller by setting this to zero.

Under Manage Jenkins -> Security -> CSRF Protection

Turn on the Crumb Issuer mechanisms to prevent unauthorized actions from being executed for legitimate users.

Under Manage Jenkins -> Security -> API Token -> API Usage Statistics

Turn on API usage statistics to see what api tokens are being used and when the last time a token was used.

Configure Jenkins' JVM Flags on a Server

-XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -Dhudson.model.DirectoryBrowserSupport.CSP= -Djenkins.model.Jenkins.logStartupPerformance=true

Configure Jenkins' JVM Flags on Docker

-XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0 -Dhudson.model.DirectoryBrowserSupport.CSP= -Djenkins.model.Jenkins.logStartupPerformance=true

Joining the Big Leagues: Tuning Jenkins' GC For Responsiveness and Stability (cloudbees.com)

Access Control

1. Don't enable anonymous access.

2. Don't use Jenkins' user database for authentication.

3. Use third-party access control via SAML, OIDC.

4. Setup and configure the Role-based Strategy plugin.

Implement role-based access controls to limit users' actions based on their roles and responsibilities. This prevents unauthorized access, data breaches, and configuration manipulation.

How do we manage plugins and dependencies?

Vulnerability management with Jenkins is managed by regularly updating your Jenkins Controller. The further back your controller is from the latest LTS release, the more complex the update will be. With our Managed Jenkins Service, we update our controllers multiple times per month and this helps to address vulnerabilities but also makes each update more manageable.

Updating Jenkins between LTS releases requires you to take the following steps.

1. Update all installed and compatible plugins with the existing LTS release.

2. Backup the configuration.

3. Update the Jenkins LTS release.

4. Update all installed and compatible plugins with the new updated LTS release.

The big challenge with Jenkins updates is the compatibility of the plugins between releases. If a plugin is not compatible with the release, it will not load into the Jenkins runtime at startup and cause issues including thread dumps and configuration loss. A backup of the current Jenkins configuration will save the agony of a configuration loss in the event that a plugin is not compatible.

What are the security best practices for managing access to Jenkins Software

Securing your Jenkins instance should be a top priority to prevent unauthorized access and data breaches. Access control for your Jenkins controller really is quite simple. It just takes some effort to understand how to implement the controls because it can be difficult to understand what kinds of configuration boundaries exist in Jenkins. On a Jenkins controller, it is possible to restrict access to specific Jobs / Pipelines for certain users and groups. It is also possible to manage credentials at the Folder level.

Why we use RBAC to manage Access Control

We believe RBAC to be an essential component to a self-hosted Jenkins server and as a result Jenkins administrators should invest time in implementing it. Role-based Access enables a Jenkins Administrator to define fine-grained permissions for different users and groups. Grouping users along functional duty lines so that developers can manage their pipelines but not necessarily install or configure the Jenkins software.

We upgrade authentication with 2FA/MFA

You should also enable two-factor authentication (2FA) to add an extra layer of security, requiring users to provide a second form of identification when logging in.

We limit exposure with IP Whitelisting

Restrict access to your Jenkins controller to safe IPs only. Having a Jenkins controller publicly accessible on the internet can be risky without the right security infrastructure like a Web Application Firewall with OWASP Top 10 detection and prevention.

At Servana, our Managed Jenkins Service by default is configured with 3 groups, a ReadOnly, Developer, and Administrator group. It is possible for our customers to go further, for example, creating Folder-level isolation for specific users. We also provide the ability to add factors to your access control policy. We support OTP and WebAuthn.

Have we missed anything? Let us know on our socials. We think managing Jenkins is much easier with these recommendations.