Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version. But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage.
Conversely, imagine if you're using some tools developed by a far off team within the company. Every time the tooling team decides to make a change, it will immediately and irrevocably propagate into your stack, whether you like it or not.
If you were at a startup and had a production critical project, would you hardcode specific versions for all your dependencies, and carefully test everything before moving to newer versions? Or would you just set everything to LATEST and hope that none of your dependencies decide to break you the next day? Working with a monorepo is essentially like the latter.
I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" then... no, just no.
You might think this ought to be trivial by having clear API contracts, but that's a) not how things work in practice if all code is effectively owned by the same, overarching entity and, more importantly, b) now you have an enormous effort to transition between incompatible API revisions instead of just being able to do lockstep changes, for no real gain.
Even if you manage to pull that off (again, for what benefit?), it will bite you that 1.324.2564 behaves subtly different from 1.324.5234 even though the intent was just to add a new option and they otherwise ought to have no extensional changes in behavior.
It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous.
Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result.
In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it.
In a monorepo world, you don't have a choice. You've been forcefully migrated as soon as the tooling team decides to make the change on their end. They had the best of intentions, but that doesn't always translate to a good outcome.
FWIW, I'm currently working at a large famous company that uses a monorepo. Color me not-impressed. I do think that having a single repository for an entire team/project is a good idea. Hundreds of different projects and teams who've never seen one another? Not so much.
> In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it.
The correct course of action is to either reverse/fix the code change to the library you depend on, or if your code is clearly using the library wrong and can be easily fixed, to do that. Not to let the whole ecosystem slowly spiral out of control.
Either way, the point is that it will force the issue to be resolved, quickly, and the code base to move forward.
The tools/libraries you depend on are themselves dependent on other libraries and tools. They may have done changes that are necessary to continue working, which you are not picking up if you stay behind. They will do IPC and RPC and always rely on their infrastructure being current.
>In a monorepo world, you don't have a choice. You've been forcefully migrated
Yes, and that's good, because:
> and migrate only when you want to, and when you're feeling confident about it.
... does not help in moving the code forward.
If your change will break others, you need to coordinate with those others so that the transition happens gracefully, not let them live on what amounts to unsupported (and slowly more incompatible) code.
>> In a monorepo world, you don't have a choice. You've been forcefully migrated
> Yes, and that's good
In your projects, have you configured your build system to always auto-pull the latest version of every single dependency you have? If not, you're not practicing what you've claimed above.
FWIW, java-maven used to allow specifying LATEST/RELEASE versions, so that the latest version will always be auto-pulled on every build. They later removed that option entirely, because they realized how dangerous that is.
> The next morning, all your services end up broken as a result.
I mean, this is the argument for having good integration tests.
At some point someone has to figure out if the new code will break a system; if you don't have good integration tests you're basically left eyeballing the changes, and sure eyeballing changes can work fine in small teams, but at some point you need good tests.
I did some work on a ranking system last year, and by definition there's no way to roll that out incrementally, because, well, it is the central component deciding what thing to do/show, and you have to change the world at once, there is literally no other option. So you need good ways of evaluating these wide reaching changes.
"If you liked it then you shoulda put a test on it" :)
> I mean, this is the argument for having good integration tests.
Maybe companies like Google have very, very strong code hygiene, but at most places I've worked, sooner or later there's a project that had a tight deadline and someone thought it smart to cut corners on the tests. Or the test s are there but they're bad. Or incomplete. Or worse, some system was just too hard to test and not updated frequently enough, so it requires manual testing.
Having "eventual consistency" for this is quite nice. Push a breaking change, update what you can, run tests, speak with owners, get deployed what you can. Keep tab of what you couldn't. Then do what you have to do to get the stuff tested (even if it means manual) and gradually become consistent as these things get pushed to prod...and hopefully next time its easier.
> Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result.
Someone makes a commit to library code and production magically breaks? How does that happen?
I thought that Google deploys new versions gradually (first to 1% of users, and if that doesn't show errors, to more and more). Which implies that there are at least two version of an application or service running.
How does that work when you don't keep APIs stable, at least at the service boundaries?
> But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage.
Nope. In a monorepo (like at Google), you're responsible for not breaking anyone else's code, as evidenced by their tests still passing.
So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push.
And if you break their code because they didn't have tests? That's their problem, and better for them to learn their lesson sooner that later, because they're breaking engineering standards that they've been told since the day they joined. A monorepo depends, fundamentally, on all code having complete test coverage.
> So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push.
Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?
Google has a CI infrastructure which runs most of the affected tests for each commit (which they call "CL") on thousand of machines in parallel. Though even for Google, running the entire test suite every time is prohibitively expensive so they have a way to merge and run multiple CLs in a single batch run every 3 hours, which is useful for testing a CL that may affect hundreds of thousands of build/test targets. If you're interested, this paper may give you an idea how Google is doing test.
Google also regularly sees changelists break the world for hours, and each team has to sacrifice a member to serve as "build cop" and find the offending change and demand a rollback ASAFP.
> Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?
Eventually you hit a point where you need systems to run the tests for you. Making this work is part of the investment in infrastructure and tooling you need to do as a big serious company.
> A monorepo depends, fundamentally, on all code having complete test coverage.
Covering every single line of code still doesn't mean that you have complete behavioral coverage, unless your tests somehow run for all possible inputs. In practice, there will still be holes, not because someone was negligent, but because they missed a corner case specific to some state.
> Working with a monorepo is essentially like the latter.
Not really. In the dependencies analogy the author of the dependency has no way to test the dependee(s). While with monorepo this is exactly what you do, "the tooling team" will "carefully test everything" before "propagate into your stack" (and it doesn't have to be irrevocable).
In practice, at any medium/large organization, the tooling team doesn't know your system, and its nuances, nearly well enough to "carefully test everything".
Having a solid automated test suite does help. But I personally would like to be in control of when my project updates its dependencies, instead of being forced to always pull everything from LATEST.
At Google the contract is essentially infrastructure teams (and generally, your dependencies) will not break your unit tests (or will contact you well in advance to handle changes). But if you don't have a test, they might. They don't have to carefully test everything. You do. And if you don't, breakages are entirely your responsibility, because you didn't have a test for them.
they don't know it because they don't use monorepo. monorepo makes "solid automated tests" easier since basically there is only one version to test. The instinct against pulling everything from LATEST, developed in the traditional world, is perfectly understandable. However in monorepo "your" project is also tooling-team's project. "being forced" becomes "being helped". It's shared responsibility.
>In a versioned world, you can release your change as a new version, and allow your users to self-select
Repeat this process multiple times and you end up with configuration/settings hell. Been there done that. It's not black and white but "trampling over the 0.1%" could be a sensible business/architectural decision. For example how do you imagine "google maps" users selecting when/how to migrate?
I was referring only to static dependencies, like Guava for example. Static dependencies don't require ongoing "upkeep", so you really should allow your users to use an older version of your library/code, if that's what they really want to do.
When it comes to live services that you're actually running on a daily basis, like Google maps, forcing users to migrate makes a lot more sense.
Just because your older versions are available to users, doesn't mean you have to continuously maintain them. For example, I know plenty of teams still using Java 7, even though Oracle has EOLed it. The teams know the risk that they are taking, and have the flexibility to decide when they want to migrate to a newer version, and that's how it should be. I would never want to live in a world where Oracle can force me to migrate to Java 10 within a day of release.
Not saying this is how Google does it, but a monorepo doesn't prevent you from having multiple versions of the same dependency. Ideally, with a monorepo, you could update 99% of your sub-packages to the latest version while still leaving the one alone.
There are a few exceptions to the "one version" rule; the monorepo and the build system support multiple versions just fine. It's just that we don't want them.
Conversely, imagine if you're using some tools developed by a far off team within the company. Every time the tooling team decides to make a change, it will immediately and irrevocably propagate into your stack, whether you like it or not.
If you were at a startup and had a production critical project, would you hardcode specific versions for all your dependencies, and carefully test everything before moving to newer versions? Or would you just set everything to LATEST and hope that none of your dependencies decide to break you the next day? Working with a monorepo is essentially like the latter.