This works really quite well - great work, airblade! For reference, this is what I placed in my vimrc to get things how I like them:

  highlight clear signcolumn
  let g:gitgutter_enabled = 0
  nmap <leader>gu :GitGutterToggle<CR>

I prefer my buffers to start with it disabled by default and toggle as needed.