news.ycombinator.com• Nov 18, 2021• 1 min read
Ask HN: Cloud provider accidentally deletes instancesAnyone have an experience with a cloud provider deleting your instances, volumes, resources, etc?I work for a small org that utilizes a cloud provider (not top 3, but top 10). Yesterday we noticed that all our instances, attached volumes, sec groups, network resources, from our Test and PROD environments had disappeared. After scrambling we found an audit trail, that showed all the deletes were from an SRE working for the provider. Apparently they were doing "cleanup" and somehow our account number became part of that.We were told we needed to recreate everything from scratch shortly after opening a ticket. Luckily we host our Terraform and config management on-prem, and a few snapshots and volumes they weren't successful in deleting. Still a lot of manual rebuilding as not everything is automated.Two of us have been working to recover things for many hours. Lower environments will likely take a lot longer.Just curious if anyone else has had something like this occur. I have a hard tim