My bookkeeping practice has been using an Intuit Licensed Commercial Hosting service since 2008, when I moved all my client’s QuickBooks Desktop files and their related documents (bank statements, payroll reports, legal docs, etc.) to Cloud9 Realtime.
Using a hosting service makes my local computer basically a ‘dumb terminal,’ meaning I use my laptop to connect to the internet, whereby I access Cloud9 and work off of their server - I launch their installed versions of QuickBooks (QB), Word and Excel. I lease my own Private Virtual Server at Cloud9, instead of being on a shared platform. This gives me better performance in terms of speed, and I have control over how much disk space I lease each year, as well as how many user licenses I purchase.
Some of my clients purchase their own Cloud9 licenses, so they can login and work on their books; they own their own QuickBooks licenses. For other clients, I set up and create QB files using my own Pro Advisor QB licenses, and the clients have no access to their QB data or docs. At the end of each year, I send each client hosted on Cloud9 a thumb drive with all their docs, and their QB file.
The benefits of a hosted service include anytime/anywhere access for both myself and my clients who have Cloud9 licenses, the security of having all my firm’s data backed up in the cloud, and it keeps me out of the IT business as I don’t have to manage my own server. If my laptop dies or is stolen, I’m not out of business, as all my data is stored at Cloud9. And Cloud9 does 30 day rolling backups, so I can ask for a file up to a month old to be restored, if for any reason I need that service. In fact, I pay extra to have quarterly Cloud9 backups, so I can look back up to a year to retrieve a file if need be.
Earlier this year Cloud9 was acquired and became a fully owned subsidiary of AbacusNext, a software and private cloud services provider which is also the maker of AbacusLaw and Amicus Attorney.
Ransomware Attack over Labor Day Weekend
Using C9 has worked well for me for almost 10 years, with the occasional bumps and glitches, support cases, and normal frustrations one encounters with any online service. In fact, I was a member of Cloud9’s Client Advisory Board for over 3 years, and I attended their annual Cloud Computing conference in San Diego.
Everything was running along smoothly until this past Labor Day weekend, when Cloud9/AbacusNext became the victim of a ransomware attack. At first, I didn’t know this is what was happening. I only knew that when I tried to login Saturday morning of the holiday weekend, I either couldn’t get in to my Cloud9 sever, or if I did it was running so slow it was not worth trying to get any work done.
When I downloaded a Word doc to my local computer, I was unable to open the doc, which had this error message: “We’re sorry. We can’t open [Doc Name] because we found a problem with its contents. Details: This file is corrupt or cannot be opened.” I submitted a support ticket to C9, and when I did not get any response after 4 hours, I called the Support Center.
I was unsure if I would reach a live person over a holiday weekend, but I did, and was told they were having issues with one of their data centers, and technicians were working over the weekend to address the issues. Bottom line: I was unable to access my data until Monday morning, Labor Day. I received no notification from C9 that my data was accessible, I just tried to login and was successful, so I started working.
On Monday evening I received an email notification from Cloud9 stating that on September 2, 2017, after multiple clients reported login and latency issues, engineers were dispatched to the data center and concluded an unauthorized, outside party had gained access to the Cloud9 network. The unauthorized access was revoked by 6 p.m. September 4 at which point it was determined that certain client files were maliciously encrypted by ransomware.
Cloud9 triggered backup restore measures for their California datacenter, which they announced was now 100% restored. They also noted a “Root Cause Analysis” was underway, and that they hired a world leading cybersecurity firm to assist with their investigation.
They also reported, “we do not have reason to believe that any of our clients’ files were opened, viewed or copied by the intruder” nor “that any credit card numbers or other personal information was viewed, copied or otherwise acquired.” I was stunned. Happily, they offered a dedicated toll free number to call to address questions related to the incident.
Is Anyone There? Radio Silence by AbacusNext
It turns out I had quite a few questions, so imagine my disappointment when I called the dedicated toll free number, only to learn I had reached the AbacusNext support center, and a Level 1 technician who could not answer a single one of my questions. It was time to spring into action so I emailed Cloud9’s Manager of Technical Services, but have yet to receive a reply.
AbacusNext does not provide email addresses for their management team on their website, so I was unable to email their technical services managers. So I created some posts on the Accountex (formerly Sleeter Group) Forum, and LinkedIn. Subsequently, I learned that other Cloud9 users, also frustrated by the lack of communication from AbacusNext, had created a Facebook (FB) group in an attempt to share information with other Cloud9 users called “Cloud9/AbacusNext Ramsomeware hack.”
Thank goodness for this FB group, otherwise most of us would have no idea of the status of events. This 95-member group shared valuable information, and I also learned that AbacusNext did send some additional email announcements, none of which I received. I also learned that the Cloud9 Texas Data Center might also have been compromised, and one member even suggested that AbacusNext paid the ransom, although that is just speculation.
The Vultures Are Circling
While I did not hear much from AbacusNext, I have received unsolicited contacts from two Cloud9 competitors, one of which even joined the FB group to politely explain why his service is better than Cloud9/AbacusNext. One AbacusNext manager I reached over LinkedIn offered to put me in touch with the AbacusNext migration team, essentially suggesting I abandon my Cloud9 contract and sign up directly with AbacusNext instead. Not a great time to be making a sales pitch.
Lessons I Learned
When choosing a cloud service provider, I realize I now need to find out answers to these questions:
1. What are your emergency procedures in case of an intruder attack?
2. What is your disaster recovery plan?
3. How long does it take to restore services after a crisis?
4. How do you communicate with your customers during a crisis? Is there a designated point of contact?
5. What is your Service Level Agreement (SLA) for uptime?
6. What is your SLA for access?
7. What are my options for downloading files on my own?
8. What options and fees are offered for having your team download data monthly, quarterly or yearly to a physical hard disk drive and ship it to me?
Also, be sure to read and understand the Terms and Conditions and SLA you are signing. And consider adding cyber security coverage to any business liability insurance you have.
To the “Cloud9/AbacusNext Ransomeware hack” FB group: Please add your lessons learned in the Comments below.
Lessons AbacusNext Needs to Learn
I have seen no messages from the AbacusNext CEO, no apology, no mea culpa. I strongly believe that if AbacusNext had got in front of this issue from the start by creating a true dedicated call-in line, sending out daily updates, and creating a server status webpage with regular updates, then their customers would be more forgiving.
In these kinds of situations, people want assurances that they are being heard - they do not want to feel ignored. I suggested in my posts that AbacusNext hold a virtual Town Hall Meeting, but no such thing materialized.
I hold the AbacusNext leadership, particularly the CEO, fully responsible for this failure to communicate and the resulting anger in the Cloud9 community. I understand it’s been an “all hands on deck” situation for the entire last week, but were the marketing and communications people down at the data center too? More communications from them would have prevented the wholesale exodus Cloud9 can now expect.
To Stay or Not to Stay
So should I stay on Cloud9/AbacusNext, or should I move my data? Some people on the FB group plan to return to local in-house servers, with cloud backups. Some are planning to move to a Cloud9 competitor.
I don’t want to become my own IT department, so I will remain with a cloud hosting service, but it won’t be Cloud9. AbacusNext has lost a 10-year old client and advocate, due to their own deplorable lack of transparency and communication.
Let us all take lessons from this for our own practices. Communication is key – we must communicate with our clients in good times, as well as bad, own up to our mistakes, say “I’m sorry” when warranted, and promise to do better in the future. I was lucky, this time. Access to my data was impaired for only 48 hours, and that was over a holiday weekend.
I learned many people were unable to access their data for the entire week. Some missed payroll and tax filing deadlines. Others spent as much time trying to calm their own clients as they spent trying to get information from Cloud9/AbacusNext. And I just read on the FB group that AbacusNext now says clients may request a copy of the Post Incident Report, but they are not freely offering it otherwise.
Jody Linick is an AIPB Certified Bookkeeper and a QuickBooks® Certified Pro Advisor. Her company, FitBooksPro (formerly called Linick Consulting), specializes in remote bookkeeping services using hosted QuickBooks and QuickBooks Online. You can find her series of Blog posts here.