>Blindly copying highly expensive software development processes from Google or other super-unicorn companies like Apple or Facebook is likely a prescription for failure.
Yes, this can't be stressed enough. Out here with the little people like non-Google-employed me, there's almost a desperation to copy the things that Google and Facebook do, and it's often justified with "Well, Google, Facebook, or Netflix use it!"
A huge amount of tech fads over the last decade have been the direct result of Google's publications and public discussions. Google might need BigTable, but that doesn't mean we should all bury our SQL RDBMS and use a crappy open-source knockoff "based on the BigTable paper". More than likely, Google would've been happy to keep a SQL RDBMS.
Google has the engineering staff, including two extremely technical founder-CxOs with PhDs on which Google's initial technology was based, and the hundreds of millions of dollars necessary to do things The Google Way. They can rewrite everything that's slightly inconvenient and do a lot of cutting-edge theoretical work. They have the star power and the raw cash to hire the people who invented the C language back in the 70s and ask them to make a next-gen C on their behalf.
Google has the technical and financial resources to back this type of stuff up, provide redundancy, put out fires, and ensure a robust deployment that meets their needs. They keep the authors of these systems on-staff so they can implement any necessary changes ASAP. In many cases, Google is operating at a truly unprecedented scale and existing technologies don't work well, so that theoretical cutting-edge is necessary for them.
None of those things are going to be true for any normal company, even other large public ones. Google's solutions are not necessarily going to meet your needs. Their solutions are not necessarily even good at meeting their own needs. Stop copying them!
I'm so. sick. of sitting through meetings that have barely-passable devs making silly conjectures about something they heard Google/Facebook are doing or that they read about on a random hyper-inflated unicorn's engineering blog.
You want your stack? OK, here it is: a reasonably flexible, somewhat established server-side language (usually Java, .NET, Python, Ruby, or PHP; special applications may need a language that better targets their specific niche, like Erlang), a caching layer (Redis or Memcached) that's actually used as cache, NOT a datastore, a mainstream SQL database with semi-reasonable schemas and good performance tuning, and a few instances of each running behind a load balancer. That's it, all done. No graph databases, no MongoDB, no quadruple-multi-layered-massively-scalable-Cassandra-or-Riak-clusters, no "blockchain integration", no super-distributed AMQP exchanges, ABSOLUTELY NO Kubernetes or Docker (use Ansible), none of that stuff.
Just sit down and get something done instead of wasting millions of your employers' dollars trying to make yourself feel important. If you do need an AMQP exchage, bolt it on after the fact to address the specific issue at hand, once you know why you need it, and make sure you know what you're doing before you put it in production (it's apparently difficult to grasp that AMQP servers are NOT data stores and that you can't just write directly to them and wait for some worker to pick it up if you care about data integrity).
Don't start a project with the primary intention of making a convoluted mess; let it get that way on its own. ;)
While I agree with the spirit of your post, I think you are taking the argument too far in places. For instance, we are a small team of devs and Docker solved a real problem for us. Our deployment was once on bare OS and it was a nightmare. Different distributions, different library versions (with their own sets of bugs)... Unfortunately we couldn't demand a specific environment from our customers and at the same time it drained our resources trying to accommodate all installation variants. When Docker came out we tried it out, saw that it solves all these issues beautifully and never looked back. But we use it only because it insulates our installation from Linux (apart from the kernel that is), not because it is hip.
More general advice is: take G, A, FB, NF... papers (and HN posts, while we are at it ;) ) with a grain of salt, test before use and make sure it solves more problems than it creates - for you.
Yeah, I don't mean it to be an all-encompassing prohibition. It's just important that people start with the basics and complicate their software with esoteric solutions only as a matter of necessity. This also ensures that the engineers understand the functionality that the esoteric solution needs to provide instead of making grossly incorrect assumptions and misuses that lead to major disasters (e.g., treating AMQP servers as a data store; a client lost data last week due to this silly assumption).
Yes, this can't be stressed enough. Out here with the little people like non-Google-employed me, there's almost a desperation to copy the things that Google and Facebook do, and it's often justified with "Well, Google, Facebook, or Netflix use it!"
A huge amount of tech fads over the last decade have been the direct result of Google's publications and public discussions. Google might need BigTable, but that doesn't mean we should all bury our SQL RDBMS and use a crappy open-source knockoff "based on the BigTable paper". More than likely, Google would've been happy to keep a SQL RDBMS.
Google has the engineering staff, including two extremely technical founder-CxOs with PhDs on which Google's initial technology was based, and the hundreds of millions of dollars necessary to do things The Google Way. They can rewrite everything that's slightly inconvenient and do a lot of cutting-edge theoretical work. They have the star power and the raw cash to hire the people who invented the C language back in the 70s and ask them to make a next-gen C on their behalf.
Google has the technical and financial resources to back this type of stuff up, provide redundancy, put out fires, and ensure a robust deployment that meets their needs. They keep the authors of these systems on-staff so they can implement any necessary changes ASAP. In many cases, Google is operating at a truly unprecedented scale and existing technologies don't work well, so that theoretical cutting-edge is necessary for them.
None of those things are going to be true for any normal company, even other large public ones. Google's solutions are not necessarily going to meet your needs. Their solutions are not necessarily even good at meeting their own needs. Stop copying them!
I'm so. sick. of sitting through meetings that have barely-passable devs making silly conjectures about something they heard Google/Facebook are doing or that they read about on a random hyper-inflated unicorn's engineering blog.
You want your stack? OK, here it is: a reasonably flexible, somewhat established server-side language (usually Java, .NET, Python, Ruby, or PHP; special applications may need a language that better targets their specific niche, like Erlang), a caching layer (Redis or Memcached) that's actually used as cache, NOT a datastore, a mainstream SQL database with semi-reasonable schemas and good performance tuning, and a few instances of each running behind a load balancer. That's it, all done. No graph databases, no MongoDB, no quadruple-multi-layered-massively-scalable-Cassandra-or-Riak-clusters, no "blockchain integration", no super-distributed AMQP exchanges, ABSOLUTELY NO Kubernetes or Docker (use Ansible), none of that stuff.
Just sit down and get something done instead of wasting millions of your employers' dollars trying to make yourself feel important. If you do need an AMQP exchage, bolt it on after the fact to address the specific issue at hand, once you know why you need it, and make sure you know what you're doing before you put it in production (it's apparently difficult to grasp that AMQP servers are NOT data stores and that you can't just write directly to them and wait for some worker to pick it up if you care about data integrity).
Don't start a project with the primary intention of making a convoluted mess; let it get that way on its own. ;)