> Yeah but in reality no-one writes it down. I actually use Google-Auth for my Gmail, but I forgot were I wrote this number down(it was a couple of years ago). Keeping a scratch code for each new server is a pain.

This is your problem. I keep mine (for multiple services) in my wallet, and it's worked out excellently.