Who, Me? Welcome to another week of work, a moment The Register celebrates with a new installment of Who, Me? It’s the reader-contributed column in which you ‘fess up to follies, false moves, and faux pas – and explain how you escaped.
This week, meet a reader we’ll Regomize as “Chase” who develops free open source software for scientists.
The Register knows that open source development can be thankless, so thanks for your efforts, Chase!
But we digress. Back to Chase’s story.
“We use AWS EC2 to run ephemeral build agents for continuous integration,” he told The Register. “It may spin up a dozen instances per day and shut them down when they’re no longer used.”
Chase and his fellow developers have used this technique for about ten years, and their AWS bills have been predictable – usually between $1,000 and $2,000 each month. They therefore fell out of the habit of regularly scrutinizing their bill.
“Then I made a small mistake that cost us a ton of money,” Chase admitted. While updating one of his crew’s Amazon Machine Images – a template for a virtual appliance that runs in AWS – he accidentally unchecked an option to delete cloud storage volumes when the instance terminated.
The update didn’t alter how the AMIs operated. But leaving that box unticked meant that when instances shut down, the storage volumes they used persisted instead of disappearing.
Remember how Chase told us he and his colleagues used a dozen instances a day?
Two months after his mistaken click, he had therefore unwittingly created hundreds of volumes, each consuming 100 GB and earning AWS a pretty penny.
“I didn’t suspect anything was wrong until I got an email from the organization that sponsors our AWS usage,” Chase told Who, Me? “They wanted to know what changed because our cloud charges for the last two months were $40,000.”
Chase said Amazon did the right thing and forgave 40 percent of those bills, but the entity that sponsored the project’s cloud bills was deeply unhappy and moved it to another cloud.
“The moral is, even if you have steady AWS usage for years, make sure you have an alarm set when cost starts to exceed expected amounts,” Chase told Who, Me?
Have you made a mistake that made your cloud costs soar? If so, click here to send an email to Who, Me? We’d love the chance to share your mistake – sensitively and anonymously as we did for Chase – so your fellow readers can learn what not to do. ®