Lessons learned from Code Spaces



Last Tuesday, Code Spaces came under a DDoS Attack. The attacker then obtained access to Code Spaces’ Amazon console and demanded ransom for the safe return of the account. For some unknown or yet-to-be-explained reason, Code Spaces decided to start changing Amazon credentials in an effort to regain control of the account. This is where the story goes from bad to worse. The attacker, noticed Code Spaces’ activity and retaliated by deleting EC2 instances, S3 buckets, and EBS snapshots. This effectively laid waste to all of Code Spaces infrastructure and backups. The devastation was so bad, it not only knocked Code Spaces offline, but offline indefinitely. Game over.

We feel for Code Spaces. Building, deploying, and maintaining an online service is a difficult endeavour. Ne’er-do-wells lurk at every corner trying to knock your service offline or steal data. The sophistication varies from script kiddies to state sponsored attacks. The most potent attacks though, come from the inside. Regardless of the source of the attack, there were many small failures that aligned to create a catastrophic failure.

A primary miss in the Code Spaces’ backup strategy was to backup to a different provider or at least offsite. This is one of the basic disaster recovery tenets. In the fullness of time, Code Spaces should explain why they never got around to it. The cloud has facilitated many difficult to perform tasks, so much so that it’s easy to forget the basics.

It appears that Code Spaces never attempted to escalate to Amazon once their account had been compromised. Working with the vendor would have been prudent. Amazon does have S3 bucket versioning which may have been able to preserve the backups.

A well-rehearsed disaster recovery plan should have also been in place for security breaches and attacks. If you provide an online service, you will be attacked. It’s likely you’ll also be breached. A comprehensive plan includes how to recover the service, issue press releases, and most importantly communicate with customers. It seems that Code Spaces may have acted too hastily without trying to download and preserve what data they could prior to changing account credentials. Following well-planned protocols could have prevented the unfortunate decision.

Finally, when analyzing the full threat matrix a cloud provider faces, it’s important to protect against both internal and external threats. Authentic8’s Silo cloud browser does just that. Administrators can grant access to the important web consoles and portals without revealing the credentials to the end users. When users don't know their credentials, they can't fall victim to phishing attempts or other exploits. When credentials aren't typed on the device, keyloggers and shoulder surfers are rendered useless. This unique ability provides a company to protect itself both from internal and external threats to their most important web apps.

We hope Code Spaces can recover quickly. We also hope that every provider learns from this experience.