How We Fixed a Critical cPanel Bug Before cPanel Did

If you use any of our Web Hosting or WordPress Hosting services, you’re probably familiar with cPanel. It’s a crucial system for the vast majority of our customers, and in fact it’s used by countless hosting providers like us around the world. The makers of cPanel are usually incredibly careful with new releases and updates, but that doesn't mean that everything is always perfect.

After cPanel released version 120 last year, we noticed something concerning in our DNS (domain name system) infrastructure. As our domain name explainer makes clear, this system needs to work properly for people to visit your website:

Behind the scenes, websites are actually identified through a string of numbers called internet protocol (IP) addresses. These are easy for machines to read, but useless for humans. The domain name system layers easy-to-remember domain names on top. A system of domain name servers translates between the URLs that we all know and use, and the IP addresses that direct internet traffic to the right servers.

The new bug stopped some changes that customers made, like updating IP addresses or email configurations, from getting pushed out to the internet properly. If updates like these don't take proper effect, then DNS breaks. People trying to visit your website would see errors instead, or emails would go undelivered.

We started working on this problem the day we identified it. We couldn't afford to let our clients experience any service disruptions. In fact, we rolled out a solution before cPanel did!

Identifying the bug

Our team discovered a critical bug in the nameserver services that list the IP address that each domain or subdomain points to. Any domain that doesn't point to the right IP address ends up being a dead end.

The issue manifested in two ways. When our customers deleted zone files or domains through cPanel (or a related system called WHM), the records weren't being properly removed. Instead, they were being duplicated in the configuration file that holds this data. That prevented the service from restarting properly and stopped the changes from taking effect. To make matters worse BIND, which is a service that converts domain names to IPs, was also struggling with zone files containing 300 or more entries - a bug recorded by AlmaLinux.

So we knew what was going wrong. The next step was to stop those problems from occurring, and making sure that we'd know if anything like this happened again.

Fixing things fast, and improving our monitoring

First, we fixed the BIND version to a stable release that wasn't affected by these issues. This removed a lot of current issues, but it wasn’t enough to ensure that things would stay stable. So we developed a monitoring system that keeps a constant eye on our DNS services. This system automatically detects and removes any duplicate entries before they can cause problems, ensuring our nameserver service stays healthy and operational. We enhanced our monitoring capabilities across all nameservers as well, which gave us real-time visibility into the health of our DNS service and the ability to catch potential issues before they impact our customers.

This proactive approach not only prevented any service disruptions but also gave us valuable insights into our infrastructure's health and performance.

It wasn't until cPanel release 126.0.5, more than two months later, that the official fix was implemented by the cPanel team. By that time, our infrastructure was already well-protected by our custom solution, and we had the added advantage of better monitoring than before.

Best of all, you didn’t notice anything

This experience reinforced some crucial principles in our operations. It illustrated the importance of good monitoring systems and the value of responding rapidly to potential issues. It’s another example of the benefits of adopting automation, which can be a game-changer for maintaining system health. It was a reminder of the critical nature of protecting your services, even when dealing with issues in third-party systems.

Remember how we said that cPanel is used by countless web hosting providers around the world? We don’t know how many of them did nothing while cPanel spent two months fixing things, but we do know that inaction would have caused problems for customers. That’s not the MyHost way.

When we talk about the benefits of proactive system administration, this is exactly the sort of thing that we mean. We worked to prevent service disruptions caused by a third-party bug, and we also took the opportunity to improve our overall capabilities and make our systems more resilient. As we continue to evolve our infrastructure, this sort of proactivity will keep paying off.

The best part? You never had to worry about any of this. While we were working behind the scenes to keep everything running smoothly, you could focus on what matters to you – running your business. We're here to handle the technical challenges so you don't have to.

We can't always wait for providers to fix issues. A lot of hosting providers won’t have rolled up their sleeves and solved the problem before cPanel did. Even fewer would have come out of this in a better place than they started. This proactive approach is what sets us apart – we don't just wait for solutions, we create them.

Mail image by Сергей from Pixabay