It’s not often we run editorial items such as this, but given that members of our audience were likely impacted by this incident, we thought the responsible thing to do would be to post the client letter from AbacusNext CEO Alessandra Lezama.
For those unaware, over the Labor Day weekend Cloud9 Realtime, a cloud hosting service used by many accounting professionals which AbacusNext acquired earlier this year was hit with a ransomware attack. One of those affected was our own regular blogger, professional bookkeeper and QuickBooks ProAdvisor Jody Linick who wrote about her experience.
In order to balance out some of her comments and better inform our audience, we have agreed to post Lezama's letter. As we are a community site for accounting professionals we welcome all comments from the community as well as those that serve it.
As such, please review the CEO’s letter below and feel free to comment:
Dear Valued Cloud9 Client:
This email is to provide you with greater insight into the security incident that occurred this past Labor Day weekend. As we informed you in our email communication from 09/04/17, Cloud9 was subject to a ransomware attack that started with one of our largest clients and made its way across the network to affect approximately 30% of clients hosted in both of our California and Texas data centers.
Upon the immediate discovery of the attack, the Cloud9 network was shut down as a security measure to prevent any further compromise. Once the threat was contained, the network connectivity was restored and the detailed assessment and recovery of maliciously encrypted files begun.
As a service provider, Cloud9 takes downtime for our clients very seriously and understands its impact on your organization, your employees and your clients especially given the impending tax deadline of 9/15. As I review the emergency response (“Code-Red”) details, management and staff responded in earnest, following the strictest of emergency protocols having All-Hands-On Deck working around the clock to restore affected clients.
The recovery process was not the same for all affected clients however; clients hosted out of the Cloud9 California datacenter were the first group to recover as our Cloud9 engineers were able to restore encrypted files from our back-ups within the same day of the emergency window. The recovery of the remaining 30% of affected clients hosted out of our Texas datacenter ran progressively as our engineers applied scripts to recover encrypted files from their remote environments whilst minimizing the disruption to files that our clients had been creating since the beginning of this incident.
To put the magnitude of the restoration process in perspective, the data files that ran a process of recovery for affected clients in Texas alone exceed 300 terabytes. Whilst utilizing advanced technology and human resources, this process inevitably took time to complete.
At this time, we are pleased to report that 100% of files were recovered successfully and that this incident did not cause any permanent data loss to any of Cloud9 clients.
Though we have no evidence that client files or Personal Identifiable Information were taken or exposed as a result of this incident, we take any form of criminal activity extremely seriously. We have reported this event to the FBI, as well as engaged with the cybersecurity experts at RSA to assist in the forensic analysis of this event. We have also sent clients, separately, more information to better understand what triggers obligations under a data breach notice statute.
As with any service-impacting event, we are aware of negative commentary that was made about this event on social media where clients expressed dissatisfaction with communications Cloud9 has provided throughout this process. Mostly however, we are thankful to the hundreds of clients whom through this trying event patiently worked with us through the process of restoring their services back to normal and showered us with emails of encouragement and gratitude.
As way of more information to our clients, I will note that our organization maintains a strictly regimented Standard Operating Procedure (SOP) for when an emergency event (Code-Red) is triggered. This SOP mandates that a sequence of steps be effectuated to ensure all available resources are directed to immediately contain, mitigate, and resolve the issue.
Our top priority during this sequence is security, followed by assessment of the impact and organization of the company’s response activities to the Code Red. After those steps are exhausted and we have accurate information to communicate to our clients, we do so via our web advisory, email and our technical support lines.
At the close of the emergency event, we conduct a thorough investigation and postmortem analysis, and implement any remediation necessary. Unlike many organizations, our Code Red policy immediately escalates to the top level of the company, including me (the Chief Executive Officer), Chief Technology Officer, and Chief Sales and Marketing Officer, who were directly engaged at the forefront, approving action steps to go beyond the company’s contractual obligations to ensure that all available resources are deployed.
Cloud9 stands by its decision to deploy its crisis response strategy, especially given the time, resources and nature of this event. I could not be more proud of the sacrifices our team members (and their families) made in order to ensure that affected clients were restored as quickly as feasibly possible.
Additionally, since the acquisition of Cloud9 earlier this year, our leadership team has developed a plan that will make changes to Cloud9’s overall technology infrastructure to provide better stability and superior security and performance options to our clients. While the competing interest of maintaining our clients’ business continuity during tax season required delaying the implementation of some changes and improvements, certain efforts had begun prior to the Labor Day Weekend event.
Ultimately, our staff’s focus, the time and resources expended, and the recent infrastructure changes that were already implemented, served to greatly mitigate the overall impact of the event and enabled a resolution to be achieved.
Now that all client data has been restored, our primary objective is to work diligently with RSA to complete a forensic analysis of this event and deliver that investigation to you. As part of the remediation process, we are making available to you a technical task force.
Upon your request, this team will work with you to analyze your current technology stack including hosted and non-hosted applications and provide consultation on the different security options available to you that will better meet your security and hosting needs going forward. As the Chief Executive Officer of AbacusNext - the parent company of Cloud9 - I can assure you that every level of our organization is committed to your success, now and on an ongoing basis.
Chief Executive Officer