How the CDK Disruption Ground Retail Automotive to Halt

Recently, S&P Global Mobility projected that new light vehicle sales volume in June 2024 would reach 1.4 million units, maintaining pace with the previous month’s result. It also translated out to one of the strongest seasonally adjusted results since 2021.

Then, on Wednesday, June 19, two cyber incidents halted software firm CDK’s core Dealer Management System (DMS)—one of its many dealer-serving components—an event that shut down the operations of nearly 15,000 car dealerships across North America. Because CDK’s solutions covered a wide range of dealership processes, the unexpected disruption brought retail sales to a literal standstill for more than a week, forcing some dealerships back into the world of paper orders.

While CDK’s DMS was back online by July 4th (not all components are up and running as of this writing), Anderson Economic Group estimated that total dealership losses could top $940 million.

An Unstoppable Force

“The key issue was that no one was prepared for this,” notes Scott Spatz, partner and President of Cooperative Systems. “This disruption differed from many security incidents because it was 100% cloud-based. This incident was different because data is stored and processed almost entirely in CDK’s cloud. Our recovery depends almost entirely on the cloud provider’s ability to recover from their own emergency, and there is often limited oversight of the cloud provider’s own disaster recovery planning and capabilities beforehand .So, we can’t really get an understanding day to day of their security posture. Architecting an incident response plan and a business company plan for a purely cloud environment is very different from a hosting environment. It was only when the disruption was made public that everyone became aware. Within an hour of learning about the incident, we initiated fallback procedures for our dealer customers.”

Cooperative Systems counts among its clients some of the largest dealerships in the Northeast, as well as several smaller operations.

Bigger dealerships—those with thousands of employees and multiple locations—that use an upstream vendor like CDK generally have their own incident-response plans. In that case, Cooperative Systems offers advisory committee services through live communication channels with dealership executive teams to support general response procedures. That said, regardless of size, every dealership has some form of backup and incident-response plan that Cooperative Systems helps develop and execute on.

What the Disruption Revealed

“One of the major weaknesses that we saw in almost everybody’s DR planning was that it was not nearly comprehensive enough,” notes Spatz. “It either wasn’t up to date enough, wasn’t planned and trained on, or gaps existed that just didn’t cover all the anticipated scenarios.”

The CDK disruption served to shine a bright light on those gaps.

“You might have processes that typically take place entirely online, like an employee’s workflow for a particular task,” he says. “When the system locked up and dealers’ portals shut down, those tasks had to be done on paper. That brought up questions around where that paperwork could be found. Administrative tasks like that became a nightmare. You don’t have stacks of legal pads sitting around in the closet anymore.”

The Cooperative Systems Response

As with any IT provider, Cooperative Systems moved quickly on behalf of its auto dealer clients. But they did have an advantage other partners didn’t heading into the crisis.

“We have done quite a bit of security work with CDK’s internal team,” Spatz points out. “We’ve actually surfaced a number of bugs or security weaknesses and worked with CDK to get those fixed for all customers, which is just part of the service that we provide for our customers.”

For example, several months back during regular system maintenance, one of our Security Engineers discovered that certain CDK software components still relied on TLS 1.0 for communication (a security protocol that should have been removed long ago). Our team was able to work with CDK’s software development to identify and remove that vulnerable code dependency and improve overall security for all 15,000 businesses that rely on CDK software.

Through that background, Coopsys was able to help their auto dealer clients in a number of ways.

Among the heaviest lifts was the rollout of a complete security information and event management (SIEM) solution for one of Coopsys’s largest customers within 24 hours.

“We had complete visibility into all their networks the moment that CDK went down,” says Spatz. “We knew that any downtime in getting the new system online was minuscule compared to what might have occurred should something undesirable come through CDK’s software management tools. In fact, we had detected an event just like that earlier in the year, so we knew how to keep it out of our environment. Our experience with CDK benefitted all our clients, not just the larger ones.”

The Learnings

The CDK disruption was a seismic event that carried with it several lessons for the longer term.

“First and foremost, you need to have redundant systems,” asserts Spatz. “If one cloud goes down, you need a solid backup.”

Another key in dealing with the unimaginable is by preparing as well as possible.

“Table topping—trying to simulate every possible scenario and response—is something that we work on with many customers and this incident was like a live tabletop,” he says. “The more preparation, the faster your response and the least amount of downtime.”

Another critical aspect of response is having a risk management plan for yourself and that you can also push out to your vendors.

“We call that third party-risk management,” Spatz says. “We have a process for doing that internally and it’s basically a platform that guides clients through an incident like this. It includes performing a business impact analysis that helps you understand how any external software or other product that touches your existing processes can affect them. It can be complicated, but it’s something that needs to be addressed.”

Regardless of the type of business you operate, contact us to learn how Cooperative Systems can help you develop an IT security plan that meets your specific needs.