"Beginning in February of 2018, communication over DigitalOcean Private Network IPs will be isolated within the account or team where they were created"
Great news, definitely a step in the right direction. The fact that DO's "private network" isn't really private is something that's not very well known I think (try googling for the topic).
Until then: Use DO firewalls to whitelist only the services connections you want. Their firewalls treat private and public networks equally -> no more publicly accessible NoSQL databases :)
Yeah, it's great news for me too. Earlier this year I tried setting up VPNs so my droplets could speak to each other in isolation. Knowing that I don't have to worry about that will be a load off my mind.
That's what happens internally (so yes, you could for example use `ProxyCommand ondevice pipe %h ssh` for other tools in the ssh ecosystem - e.g. in your `.ssh/config`).
- ondevice makes use of `ssh`'s `-oProxyCommand`, which makes ssh send its protocol data to a command's stdin and expect the responses on its stdout (you can use that for example to get around certain proxy servers using `nc`)
- `ondevice ssh` basically executes `ssh -oProxyCommand=ondevice pipe %h $@` (there's a bit more to it, but that's the gist)
- internally `ondevice pipe` creates a websocket connection to the ondevice API servers, who in turn tell the device (where `ondevice daemon` is running) that there's a connection incoming.
pretty much the same goes for `ondevice rsync` (and the - not yet released... - `scp` and `sftp` subcommands)
I recently thought about using websockets for tunneling, too, but found out that "security appliances" which do MitM and virus scanning seem to block websockets by default and even need the newest version to support them at all.
So for some scenarios, using websockets seems to be not enough
Wow, I don't think I would've even considered such an attack...
DNSSEC, HSTS and Certificate Pinning would've made it more difficult to abuse this, but I guess it would've been pretty easy to get valid SSL certificates for all your favourite .io domains.
Let's try to play malicious party here:
Phase A: First set up a simple DNS forwarder playing by the rules and answering requests as we should (as to not get any unwanted attention).
Gather usage statistics.
Phase B: Crawl the list of most-used domains to see if there are any valuable targets without HTTPS (port 443 is closed). Alternatively/additionally see if there are API subdomains used by software other than browsers (of which a few won't have annoying features like Cert Pinning - golang's DNS resolver for example afaik doesn't do DNSSEC).
Pick some medium to high level targets where the attack might go undetected for at least some time.
Phase C: MitM time! Get certificates for the target domain(s) of your choice and get to work. Start with only a few percent of the requests to not draw too much attention (and to avoid the majority of their traffic coming from a single IP (range) all of a sudden)
Obfuscate the attack by acting like a third party app or something simply doing requests for their users.
Congratulations on finding the vulnerability (and thanks for looking for that kinda stuff in the first place).
If you control the root DNS servers for .io, you can simply not answer the DNSSEC queries. Many resolvers will fail open.
HSTS requires the site is HTTPS with a valid cert. If you own all .io, you can use LetsEncrypt to get that for free. They now even support Wildcard Certs! :-) That said, you would have to choose your targets carefully and/or load balance your requests to LetsEncrypt. There is a rate limit. There are browser plugins that can tell you if a cert just changed, assuming you have been to that site prior.
Then there is Public Key Pinning. This would be great, but I suspect the number of big companies implementing this are low. I don't have numbers, but you can test your favorite sites in Qualys[1] or using testssl.sh[2] that only depends on openssl and bash.
You could proxy all requests to the real root servers for .io and only become authoritative for the ones you wish to target.
Given the small number of zones, I think a modest server could keep up, or you could balance the load on a bunch of VM's. It may take a while for anyone to notice. I am curious actually, how many fellow geeks have nagios/sensu alerts that would tell them if the root server IP's changed.
All of this said, there are BGP attacks you can do that accomplish the same thing for any TLD and the IP's wouldn't even have to change. Only more advanced monitoring tools that keep an eye on route path might notice, but probably would not alert anyone.
Let's assume an attacker who selectively hijacks .io traffic in such a magical* way that the owners of the relevant domain names do not notice the attack is happening. Assuming that, what exactly would the CT monitors notice? I assume there would be new LetsEncrypt certificates entered into the append-only log, but then what?
That could certainly be useful in the follow-up investigation and forensics. Hopefully our attacker did not spin up those VM's using a burner card or stolen CC and proxies.
There isn't anything magical about selectively targeting domains. One simply creates multiple recursors and sets the upstream forwarder to the proper IP's of the original root servers. Then one adds zones for the domains or individual records they wish to modify. Unbound DNS is great for taking over individual records. I use it for this very purpose to block advertisements and trackers. In this case however, we are just acting as a root server, so there isn't much to take over. We just point the victims to ourselves for the domains we wish to hijack. We could then have a second level of recursors to perform the above selective attacks.
I don't know if LetsEncrypt would issue that, even if you control it. That would be a good exercise to validate if there is an "easy mode" for state sponsored fun.
I've read that browsers are said to block such wildcards, but I don't know to what they are referring. I create wildcard TLD self signed certs all the time. I've never had one signed by a proper CA, so I can't tell you if the browsers have any logic to ignore them.
Your missing a ton of potential here by assuming that all DNS is good for is Web traffic. For one thing, taking over or intercepting email (remember, you now control the DNS, so you also control SPF and DKIM records) becomes trivial. you could even leverage that control of email to get SSL certificates for domains you really want to do https for (letsencrypt will even generate a wildcard cert using only dns based verification).
You could also be much more surgical, and target specific people/organizations using that .tld, ignoring dns requests for everyone that you don’t want to alert to your control. Hijack their email, and you control access to things like account recovery for domain users, and have a great method for phishing account credentials for the domains customers.
Honestly, the list of what you could do here is almost only limited by your imagination
Or, in Phase C, an attacker could proxy traffic through a botnet to avoid the single-ip-range issue; they could even find botnet participants geographically close to each user, so they might not even trigger any red flags from a close look at IP logs. Setting aside DNS hijacking altogether, the idea that phishing sites could do something like this, perfectly proxy i.e. a bank's content and remain largely undetectable, makes me very concerned that we're only just seeing the beginning of sophisticated phishing attacks.
Thanks for this; as very-much-not-a-security guy, the alarm sounded by the original post was surprisingly clear, but fleshing out a plausible attack scenario like this really helps drive the point home while also teaching me a bit.
Hadn't heard of ipset before, but yeah, that seems to be a cleaner way to do this. Maybe open a feature request?
From what I can tell, there are still a few minor details to be etched out in droplan (e.g. running as background service to avoid rewriting the rules every time, ...), but it seems to do its job reasonably well (been using it for roughly two months now
I'm not affiliated with the project (I only did a PR once involving the Dockerfile).
I don't know if it's well known, but the so called 'private' interfaces DO offers aren't really private. Any other machine in the same datacenter can access your services.
So if you have a redis/mongo/... server listening in your 'private' network, I'd suggest you secure your internal services using a firewall (and droplan is a pretty simple tool to do so automatically)
GoRef is a small Go library I've built to better monitor a project of mine.
I'm using it to measure the execution time of all my HTTP routes (and other important code paths) to track down potential bottlenecks in the application (and it helped me detect a goroutine that didn't exit properly in a certain cases)
It's designed to have as little as possible overhead (as it's meant to be used in production code)
One thing I don't quite get is their claim that the rocket would need "just some solar panels" instead of a fuel tank when in the paragraph before that they talk about exhausting argon-based plasma.
If there is a propellant, you'd need to store that somewhere first, right? Does each particle of the propellant exit the rocket with much higher energy? Or can Argon be stored in a much denser form than other fuels?
The difference is 'momentum gained per gram of propellant'.
Traditional engines burn fuel, and the expanding, hot fuel pushes itself out the back of the engine at high speed, pushing the rest of the spacecraft forward. The fuel containing the energy is also the propellant.
Ion engines are the opposite in almost every way. You spend a lot of electrical energy (solar, nuclear, whatever) speeding up a tiny amount of propellant (which is usually something non-reactive like argon). You shoot a little bit of fuel at insanely high speeds out of the engine, and it pushes the spacecraft forward a little bit- so you do it for a very long time.
Yes, ion engines still need propellant, but they need a whole lot less of it.
Thanks to you and the others for clarifying. For some reason it didn't occur to me they were talking about an ion engine. I've heard of them and their slow-but-steady acceleration characteristics before but never actually knew how they work (and for some reason didn't quite make the connection).
Yes, each particle of the propellant exits the rocket with much higher energy. Since you can only get so much energy from chemical reactions, to get to these higher exit velocities the energy from the solar panels is used. You use a higher ratio of energy to propellant mass than chemical fuels use.
I was at a demo for a plasma engine similar to VASIMR back in the 90's at a national lab. It was a long machine running the length of a large room. There was a window in the side where you could see the plasma. When we showed up there was a faint pinkish plasma barely visible in the window. The guy who was showing us around told us they were just getting ready for the demo and we should wait a few minutes. After a few minutes of the plasma being off, one of the engineers gave a thumbs up. Our guide pulled us close to the window and said "Watch this!". Nothing happened. "One more second!", he said. Then we smelled burning electronics. "Shit!", he said and ran towards the power supplies.
My co-founder and I looked at each other and smiled, because it was so nice to be on the other side of that situation. I knew exactly how the guys felt. Usually we were the ones trying to demo an experiment that sort-of, kind-of worked.
Side note: hats off to anyone working with high energy plasma.
If I make a mistake in my code, it generally doesn't escape magnetic confinement and instantly vaporize other portions of my system with the fury of the sun.
As discussed in the article, this is a very different kind of ion thruster. That category is quite large. In particular, I believe essentially all ion thrusters flown have been electrostatic ion thrusters, including both Deep Space 1 and the Hall thrusters discussed in the article.
They all need a propellant and power feed. But, you can use those same feeds for an array of multiple engines. Thus, 'scaling up' has a wide range of options and trade-offs. Vasimr let's you scale power output a lot from the same engine, but so would an array of smaller engines.
With that in mind the specific techniques are important, but the gaps are often overstated.
(IAN a physics major) But to elaborate per my understanding, there are 2 important variables you're playing with in rocket engines:
- Specific impulse / Isp (aka how much thrust you get for a given amount of propellant)
- Maximum Thrust
Due to the rocket equation [1], adding more propellant increases the weight of your vehicle, making it harder to move, requiring more propellant, etc etc. So being efficient with your propellant is very good.
That said, given a requirement against a gravity well (e.g. a planet), there's usually a minimum total thrust required for a given maneuver to be successful (low total thrust = maneuver takes longer = more time for gravity to pull you = more propellant required).
Thus far, we generally have two types of engines. (1) High maximum thrust, lower Isp (chemical rockets) & (2) high Isp, low maximum thrust (ion/electric engines). The two are currently very far apart [2, sort by Specific Impulse decreasing, then look at the Thrust column].
As examples (Isp Vacuum / Thrust Vacuum): NEXT ion thruster 4,100s/0.236N @ 6.9 kW, VASIMR 5,000s/5.7N @ 200 kW, Space Shuttle SRBs 268s/14MN.
The hope with VASIMR is that it provides a middle ground where high Isp is available with enough total thrust to actually be useful for something other than slow orbital adjustments. An example of "something useful" would generally be anything beyond Earth orbit that covers large distances, e.g. flying to Mars.
This is correct but incomplete: Vasimr may have a higher power density (be it power per thruster mass or per thruster volume) than ion thrusters but I am pretty sure it has lower power density than a Hall thruster, while being also much more complex.
The main (and not yet verified) claim of Vasimr compared to existing and proven tech like Hall thrusters is the ability to greatly vary the specific impulse.
That's what the article claims. That you can run in a less efficient mode that produces more total thrust when you're climbing out of a gravity well, then switch to the lower thrust higher efficiency mode for the long haul.
I thought one of the issues with Hall thrusters was that, in engineering-practical terms, getting enough total thrust out of them was impossible. Or would they scale effectively?
Yes, the claim is entirely wrong. The rocket still needs propellant. It just uses electricity to eject the propellant at much higher speeds than usual chemical reactions.
My guess is that especially if you're shopping in the 5$ price range, there's gonna be a much more important factor than some benchmark results: Availability and cost of extra features.
For java apps you're probably go for as much RAM as possible (giving Linode the edge here)
OVH.com for example has an S3-like object store or alternatively, physical NAS hardware (amongst a bunch of other extras).
I personally am currently looking to keep the traffic cost at a minimum (DO still doesn't charge for extra traffic but plan to do so in the future [1], Scaleway and OVH have unlimited traffic).
Some of the ones I've looked into recently also provide DDoS protection and/or load balancing while others apparently null-route your IP(s) until you can get them on the phone[2] (DDoS protection seems to be a pretty costly feature to offer).
Since all of the described (and I guess most of the discussed) services provide fast provisioning, it should be relatively easy to jump ship if you later find out you picked the wrong service.
A little anecdote:
About half a day after I've started using vultr, I got an abuse message from their SPAM detection system claiming that one of my IPs was sending loads of spam emails prompting me to respond within 48 hours (everything looked like my server was flagged automatically).
The problem was, the time of the incident was about 12hours before I even created the instance in question.
The issue was resolved quickly (there wasn't a lot to argue about after all) and I haven't had problems since, but nevertheless this leaves me wondering whether they (or another provider like them) might end up blocking my account one day simply because I missed an email.
But you can't really expect premium service for dirt-cheap products, can you ;)
I've had a quick look at your site because I'm currently using DO and Vultr with a PeerVPN[0] setup to connect the individual hosts - which brings me to my question:
Is it just a static website or do these instances communicate with each other?
If you do, I'd like to ask if you use a VPN to connect them (and which one)?
I'm relatively happy with PeerVPN (with one or two rather minor annoyances that most likely can be solved if I actually spent time on them) but am always curious about alternatives.
The site is mostly static, expect the server-side Python code to handle the submission of blog comments. I use no VPN but have automated rsync runs, over SSH, to sync data files periodically. A description of my setup is at: http://blog.zorinaq.com/release-of-hablog-and-new-design/
Great news, definitely a step in the right direction. The fact that DO's "private network" isn't really private is something that's not very well known I think (try googling for the topic).
Until then: Use DO firewalls to whitelist only the services connections you want. Their firewalls treat private and public networks equally -> no more publicly accessible NoSQL databases :)