I only do hobby gamedev, so I don't know what I'm talking about, but when I think of a tool that's "too complex" to release, I imagine not just a conceptually difficult thing, but a dangerous thing. For example, the editor lets you shoot yourself in the foot with a bunch of a esoteric considerations that don't really make sense, but the internal people know how to work around them more or less, like making sure that there's a dummy Bar for every Foo on the map, and that they are named in some crazy way, and that the name doesn't contain the "%" symbol, and you have to do that all manually and if you don't the host computer hard locks. Internally they solve this by hard-won intution about how to use the tool, plus a huge and thorough testing scheme.
I think that's the key -- if the tool allows you to harm the user's computer in some way, then it's not a tool you can really release without polishing a lot more, to prevent the failure modes. And it's particularly bad if it's hard not to mess up.
Then you would be very suprised at a lot of what happens in game dev. Many in-house tools are mere skeletons, fleshed out just enough to get the job done. The bug squashing is focused on the primary product. Developing end-user ready tools is basically a paralell dev process. Difficult, rare, expensive but often immensely valuable and rewarding.
This is the rule in game development rather than the exception. It's why polished commercial engines like UE4 and Unity have been so successful. In house tools even at some of the biggest most successful game developers are rough and undocumented. They are also generally made by engineering teams an order of magnitude smaller than Unity or Epic.
I think that's the key -- if the tool allows you to harm the user's computer in some way, then it's not a tool you can really release without polishing a lot more, to prevent the failure modes. And it's particularly bad if it's hard not to mess up.