Editor's Corner: AbacusNext Responds to Cloud9 Ransomware Attack

paper and typewriter
iStock
Share this content

It’s not often we run editorial items such as this, but given that members of our audience were likely impacted by this incident, we thought the responsible thing to do would be to post the client letter from AbacusNext CEO Alessandra Lezama.

For those unaware, over the Labor Day weekend Cloud9 Realtime, a cloud hosting service used by many accounting professionals which AbacusNext acquired earlier this year was hit with a ransomware attack. One of those affected was our own regular blogger, professional bookkeeper and QuickBooks ProAdvisor Jody Linick who wrote about her experience.

In order to balance out some of her comments and better inform our audience, we have agreed to post Lezama's letter. As we are a community site for accounting professionals we welcome all comments from the community as well as those that serve it.

As such, please review the CEO’s letter below and feel free to comment:

(issued 9/11/2017)

Dear Valued Cloud9 Client:

This email is to provide you with greater insight into the security incident that occurred this past Labor Day weekend. As we informed you in our email communication from 09/04/17, Cloud9 was subject to a ransomware attack that started with one of our largest clients and made its way across the network to affect approximately 30% of clients hosted in both of our California and Texas data centers.

Upon the immediate discovery of the attack, the Cloud9 network was shut down as a security measure to prevent any further compromise. Once the threat was contained, the network connectivity was restored and the detailed assessment and recovery of maliciously encrypted files begun. 

As a service provider, Cloud9 takes downtime for our clients very seriously and understands its impact on your organization, your employees and your clients especially given the impending tax deadline of 9/15. As I review the emergency response (“Code-Red”) details, management and staff responded in earnest, following the strictest of emergency protocols having All-Hands-On Deck working around the clock to restore affected clients.

The recovery process was not the same for all affected clients however; clients hosted out of the Cloud9 California datacenter were the first group to recover as our Cloud9 engineers were able to restore encrypted files from our back-ups within the same day of the emergency window. The recovery of the remaining 30% of affected clients hosted out of our Texas datacenter ran progressively as our engineers applied scripts to recover encrypted files from their remote environments whilst minimizing the disruption to files that our clients had been creating since the beginning of this incident. 

To put the magnitude of the restoration process in perspective, the data files that ran a process of recovery for affected clients in Texas alone exceed 300 terabytes. Whilst utilizing advanced technology and human resources, this process inevitably took time to complete.

At this time, we are pleased to report that 100% of files were recovered successfully and that this incident did not cause any permanent data loss to any of Cloud9 clients.

Though we have no evidence that client files or Personal Identifiable Information were taken or exposed as a result of this incident, we take any form of criminal activity extremely seriously. We have reported this event to the FBI, as well as engaged with the cybersecurity experts at RSA to assist in the forensic analysis of this event. We have also sent clients, separately, more information to better understand what triggers obligations under a data breach notice statute.

As with any service-impacting event, we are aware of negative commentary that was made about this event on social media where clients expressed dissatisfaction with communications Cloud9 has provided throughout this process. Mostly however, we are thankful to the hundreds of clients whom through this trying event patiently worked with us through the process of restoring their services back to normal and showered us with emails of encouragement and gratitude.

As way of more information to our clients, I will note that our organization maintains a strictly regimented Standard Operating Procedure (SOP) for when an emergency event (Code-Red) is triggered. This SOP mandates that a sequence of steps be effectuated to ensure all available resources are directed to immediately contain, mitigate, and resolve the issue.

Our top priority during this sequence is security, followed by assessment of the impact and organization of the company’s response activities to the Code Red. After those steps are exhausted and we have accurate information to communicate to our clients, we do so via our web advisory, email and our technical support lines.

At the close of the emergency event, we conduct a thorough investigation and postmortem analysis, and implement any remediation necessary. Unlike many organizations, our Code Red policy immediately escalates to the top level of the company, including me (the Chief Executive Officer), Chief Technology Officer, and Chief Sales and Marketing Officer, who were directly engaged at the forefront, approving action steps to go beyond the company’s contractual obligations to ensure that all available resources are deployed.

Cloud9 stands by its decision to deploy its crisis response strategy, especially given the time, resources and nature of this event. I could not be more proud of the sacrifices our team members (and their families) made in order to ensure that affected clients were restored as quickly as feasibly possible.

Additionally, since the acquisition of Cloud9 earlier this year, our leadership team has developed a plan that will make changes to Cloud9’s overall technology infrastructure to provide better stability and superior security and performance options to our clients. While the competing interest of maintaining our clients’ business continuity during tax season required delaying the implementation of some changes and improvements, certain efforts had begun prior to the Labor Day Weekend event.

Ultimately, our staff’s focus, the time and resources expended, and the recent infrastructure changes that were already implemented, served to greatly mitigate the overall impact of the event and enabled a resolution to be achieved.

Now that all client data has been restored, our primary objective is to work diligently with RSA to complete a forensic analysis of this event and deliver that investigation to you.  As part of the remediation process, we are making available to you a technical task force.

Upon your request, this team will work with you to analyze your current technology stack including hosted and non-hosted applications and provide consultation on the different security options available to you that will better meet your security and hosting needs going forward.  As the Chief Executive Officer of AbacusNext - the parent company of Cloud9 - I can assure you that every level of our organization is committed to your success, now and on an ongoing basis. 

Sincerely,

Alessandra Lezama

Chief Executive Officer

About Seth Fineberg

Replies

Please login or register to join the discussion.

Sep 12th 2017 20:30

Hi Seth, Thank you for posting this. I agree that it is important, and fair and honest reporting, to post Ms. Lezama's internal communication to Cloud9 users. Most of us affected still have a lot of questions, so I look forward to seeing the results of the analysis and consultation AbacusNext offers in the final paragraph.

Thanks (1)
Sep 14th 2017 18:30

the one thing I'm going to suggest here is that the hosting services provider is not in the best position to provide an overall discussion of a client company's security posture. The offer to "analyze your current technology stack including hosted and non-hosted applications and provide consultation on the different security options available to you that will better meet your security and hosting needs going forward" is little more than a sales effort to gain more hosting business. The hosting service provides a level of security that should be communicated to the client, and the client should factor that into their ENTIRE information security approach (of which the hosting solution is only a part, no matter how hard they try). I've been in the business a long time, and this was an ugly situation all the way around. Lack of communication frustrated customers, and certainly a difficult situation made things chaotic internally. However, patting themselves on the back for how well it was handled and then offering consulting intended to sell more hosting service just doesn't seem right. Abacus and C9 are hosting company and a software company, not an IT consulting or cybersecurity consulting firm. There are boundaries... I know.

Thanks (1)
avatar
Sep 15th 2017 20:07

Oy Vey. Poor Cloud9. Poor clients. Under California consumer protection laws, the disclosure requirements for an accounting firm that hosted with them at the time of the breach must certainly go into effect (no?), and they'll need to disclose to their clients--it's a 70's shampoo commercial of disclosure ****show. Add to it that there's no easy way of telling that the data wasn't copied out to another location. What will Abacus' legal clients make of all of this?

Thanks (1)