Or maybe in the rush to imitate the Western styles, the Japanese at the time discarded their traditional arts. Until they gained experience in that new style, they could hardly create visually integrative designs. They obviously were able to create visually integrative designs when they weren't trying to imitate someone else.
I don't think the use of a visual language impairs that part of the mind as much as you seem to think it does.
What is interesting is that it seems that there was a person or persons who developed the traditional Japanese style including modularity, scale proportional to humans, space usage flexibility, about 1000 years ago. That style was copied again and again. There are also many non-integrative contemporary buildings that do not try to imitate a style, so I'm still not unconvinced about this - but again this is clearly a pretty big generalization without any significant research, but something to think about.
I don't think the use of a visual language impairs that part of the mind as much as you seem to think it does.