You can only eliminate cross-market arbitrage by forcing everyone to trade in one place. Exchange competition lowers costs for everyone so it's not clear that this would improve things. Even on a centralized exchange, near-arbitrage would still exist between very correlated products within it or between similar but not identical products in other places (index futures vs. ETFs, different grades of oil, US treasuries vs. European bomds).

Like you said, if enough people do the arbitrage, it converges to zero very quickly. There's little profit in it and everyone reaps the benefits of near-instantaneous accurate pricing.