Looks like the admin stored the wallet.dat and backups on an ec2 instance, and then shut it down while doing a RAM upgrade without realizing he would lose everything on the disk.
Translation of post:
DECLARATION I hereby inform all users of the service www.bitomat.pl of system failure that occurred on 26 July 2011 and its consequences. At the outset I would like to apologize for such a long delay in publication of this statement. I explained that it was dictated only by the good service investigation conducted to determine the causes of failure and the people responsible for it. Unfortunately, to date, despite intensive efforts, could not determine these issues. However, I believe that the longer pause to disclose this communication at this stage would be unreasonable. I am also aware that service users are the appropriate explanation. On 26 July 2011 at about 23:00, I noticed that absorbs all Bitcoin server machine resources, and probably not used for making. So there was need to increase the amount of RAM in the server. As a result of that procedure - suddenly the whole virtual machine has been erased, all data stored on the server has been lost!, Including records concerning bitcoinowego portfolio and its backups (backups). I have taken action have established that the disappearance of the data was the result of the introduction of virtual server settings, which he never would have introduced. Amazon Web Services Company, which is located servers, website says that the machine that has been cleared has been set up in such a way as to be irretrievably destroyed automatically with the data on disks attached to it at the time of her arrest by the shutdown. We are constantly trying to determine who made changes to these settings and whether it will be able to recover lost data. Unfortunately partnership with Amazon Web Services, which was placed servers service is difficult. Once I realized that I deleted the machine have redeemed the biggest package of technical support, I talked to the manager, asked about the security of disk space, I explained, so far unsuccessfully. Still exerts pressure on the Amazon Web Services to accelerate their activities but without concrete results. At the moment I am unable to clearly determine the causes of crashes, I suppose that it is the result of actions of third parties, which are causing the server tried to cancel to hide their illegal activities, or intentionally wanted to website disappeared. If my suppositions are confirmed, the fact will let police and prosecutors. At the same time if possible take action through which it would be possible to recreate lost data. But what I need to interact with the server's owner, and that as I mentioned above is difficult. At this point I wish to inform and assure you that your cash deposited into your bank account service and not converted into BTC and unpaid cash from the sale of BTC remain safe and intact. Any further findings will you keep. At the same time I am counting on your help in solving the problem. I realize that the situation is very difficult, and you fear for the fate of their BTC. We are constantly working on a solution to the crisis, and I'm open to your suggestions. Currently going to: cancel all active orders so far, to restore service to operate to allow the performance of any operation (in particular the payment of PLN). Please your suggestions and ideas. I wish to inform you that I had several conversations with potential investors from home and abroad. Www.bitomat.pl service is on sale for EUR 17,000 BTC. If interested, please contact us at bartek@szabat.com. Best regards Bartek Shabbat Service Administrator www.bitomat.pl
Cynical translation of that: I didn't understand the setup and made a change which deleted everything. I haven't found anyone else to blame yet but I'm desperately trying.
There was need to increase the amount of RAM in the server. As a result of that procedure - suddenly all data stored on the server has been lost! [..] the disappearance of the data was the result of the introduction of virtual server settings, which he never would have introduced.
For Amazon persistent storage, you don't just change a setting, you need to pay extra for EBS storage and design your service to use it, so saying the following:
We are constantly trying to determine who made changes to these settings and whether it will be able to recover lost data.
Suggests a continuing lack of understanding, and that makes it seem more likely that the service was built on quicksand and is now sunk. Even if we assume that they did have EBS volumes available and someone had redesigned their service to not use them, that they have no access audits, no regular setup audits, no long term backups, no revision control history, no change procedure and no email records (and nobody who remembers anything about it) to give them a clue what happened, that's pretty damning in itself for a financial service.
Once I realized that I deleted the machine [..] At the moment I am unable to clearly determine the causes of crashes
Maybe an artefact in the translation, but you can't accept that you caused it while also being unable to determine the cause!
I suppose that it is the result of actions of third parties, which are causing the server tried to cancel to hide their illegal activities
If this is a serious admission that their Amazon account was hacked, the assurance that unpaid cash from the sale of bit coins is safe seems a bad idea, and hiding it at the bottom of another announcement is really poor form. If it's not a serious admission then it reeks of pointing the blame anywhere except 'here' and is also poor form.
1. How do you "increase the amount of RAM" on an EC2 instance other than spinning up a new larger instance?
2. Why didn't they use EBS and take frequent snapshots to S3?
It sounds like they were running an instance store EC2, then fired up a larger instance and stopped the first one? This has to be a textbook case of how not to use EC2.
You can't. Instance types are fixed. He was likely using an EBS-root configured AMI, where the mount point for EBS is /. These are EBS-backed, but since the EBS block is automatically created when you spool up the instance; it is also automatically deleted when you terminate the instance, unless you specify otherwise.
This is not actually true. If you Stop an EBS-root EC2 instance, the EBS Volume can be safely detached and moved along to other instances, or you can restart the instance and your data will be ok.
Additionally, There's no excuse for not snapshotting your EBS volumes of any sort, and these snapshots persist beyond the lifetime of the volume.
I find it just a little sad that relatively few (less than all) EBS customers don't snapshot their volumes.
You do not have to move your volumes: while the instance is in a "stopped" state you can change it's instance type and then restart it. EBS-root let's you change hardware as fast as you can reboot with just three commands.
You are, of course, correct. Notice in my post that you replied to I was careful to use the word terminate, not stop, when referring to what I assume he did to his instance.
edit: After re-reading my post it probably wasn't clear to people who don't use EC2; so your reply is probably for the best. :)
I have to repeatedly tell people that EBS-backed is not persistent. Amazon did everyone a big disservice by calling something -backed when it was actually transient.
persistent until you terminate the instance.
Unlike a EBS volume that you explicitly attach, which persists beyond the termination of the instance you attach it to.
That was my reaction as well. Not just the fact that it's ambiguous whether they're trying to sell for 17k EUR or 17k BTC, but the fact that they're even trying to sell it at all, and then the buyer is presumably on the hook for the BTC that were lost as well!
I currently had ~13 BTC in my Bitomat account- when I login they correctly show my balance, so I'm slightly confused as apparently their database wasn't lost, but their wallet data was? How does that happen? And why was the wallet data not backed up on a non-EC2 instance?
All of this is making me lose some faith in the exchanges- between the MtGox BS and this, is TradeHill the only exchange that can be trusted? Can they even be trusted? It seems they are by far the most competent from a technical perspective, but at this point who wants to trust any of these exchanges with any significant amount of money?
I don't understand how this is news to people. Bitcoin was designed to be a secure, distributed, anonymous country. What do you expect to happen when you stop relying on Bitcoin to store your money and start relying on some random Polish webapp?
Didn't mean to disparage Polish software development; just was assuming the average user of the site wasn't Polish, leaving them even less legal security— which I see, glancing at the site now, was wrong. So apologies to any offended.
With that said, I think you could pick a better site to be the shining beacon of Poland's world class talent than the site that just lost a bunch of people's money because the only place they were keeping it was on a single EC2 instance store. If this is the top, I'm not sure I wanna see the rest of the class.
These particular developers, though, were probably not world-class. Data this important simply should not be lost due to trouble with an EC2 instance; any data on an EC2 instance, even on EBS (though somewhat less so for EBS) should be considered liable to vanish at any moment.
Google Translate is heuristics-based, and can occasionally produce strange results. For instance, when translating Irish-language text, it will sometimes translate Baile Ath Cliath (Irish for Dublin, the capital of Ireland) to London, the capital of the UK.
This is likely because its corpus for Irish is made up largely of Irish government documents (all Irish govt documents are translated into Irish, but Irish isn't used by many people and there's not all that much else written in it), and these documents are rather similar to British civil service documents.
Exactly, they screwed up big time by not knowing one of the fundamental things about EC2, now they're trying to sell the thing, and they can't decide on a currency on top of that?
However, it looks like the original page says BTC, so it looks like just a translation error.
A user's bitcoin "wallet" is really their private key. It seems they didn't lose their database, so they still have a record of how many bitcoins each user should have, but they don't have access to those coins.
Translation of post:
DECLARATION I hereby inform all users of the service www.bitomat.pl of system failure that occurred on 26 July 2011 and its consequences. At the outset I would like to apologize for such a long delay in publication of this statement. I explained that it was dictated only by the good service investigation conducted to determine the causes of failure and the people responsible for it. Unfortunately, to date, despite intensive efforts, could not determine these issues. However, I believe that the longer pause to disclose this communication at this stage would be unreasonable. I am also aware that service users are the appropriate explanation. On 26 July 2011 at about 23:00, I noticed that absorbs all Bitcoin server machine resources, and probably not used for making. So there was need to increase the amount of RAM in the server. As a result of that procedure - suddenly the whole virtual machine has been erased, all data stored on the server has been lost!, Including records concerning bitcoinowego portfolio and its backups (backups). I have taken action have established that the disappearance of the data was the result of the introduction of virtual server settings, which he never would have introduced. Amazon Web Services Company, which is located servers, website says that the machine that has been cleared has been set up in such a way as to be irretrievably destroyed automatically with the data on disks attached to it at the time of her arrest by the shutdown. We are constantly trying to determine who made changes to these settings and whether it will be able to recover lost data. Unfortunately partnership with Amazon Web Services, which was placed servers service is difficult. Once I realized that I deleted the machine have redeemed the biggest package of technical support, I talked to the manager, asked about the security of disk space, I explained, so far unsuccessfully. Still exerts pressure on the Amazon Web Services to accelerate their activities but without concrete results. At the moment I am unable to clearly determine the causes of crashes, I suppose that it is the result of actions of third parties, which are causing the server tried to cancel to hide their illegal activities, or intentionally wanted to website disappeared. If my suppositions are confirmed, the fact will let police and prosecutors. At the same time if possible take action through which it would be possible to recreate lost data. But what I need to interact with the server's owner, and that as I mentioned above is difficult. At this point I wish to inform and assure you that your cash deposited into your bank account service and not converted into BTC and unpaid cash from the sale of BTC remain safe and intact. Any further findings will you keep. At the same time I am counting on your help in solving the problem. I realize that the situation is very difficult, and you fear for the fate of their BTC. We are constantly working on a solution to the crisis, and I'm open to your suggestions. Currently going to: cancel all active orders so far, to restore service to operate to allow the performance of any operation (in particular the payment of PLN). Please your suggestions and ideas. I wish to inform you that I had several conversations with potential investors from home and abroad. Www.bitomat.pl service is on sale for EUR 17,000 BTC. If interested, please contact us at bartek@szabat.com. Best regards Bartek Shabbat Service Administrator www.bitomat.pl