Many financial planners assume 4% or even 5% annual returns above inflation. Heck, I forget which state it was, but I remember reading about some boneheaded assumption of 8% annually written into the laws for its pension fund.
Can you clarify on why you think 4% is optimistic, but then say 4-5% appears to be a reasonable 30 year expectation?
I guess with these things I always think of past performance not being a guarantee of future performance, but on the flip side I'd like to be able to live off my investments when I retire, so making some sort of informed assumptions is probably better than not.
The expected return of a year is not independent of the year before. Rudimentary models that many financial advisors use don't account for that. Even the more sophisticated folks don't account for it enough as I think is appropriate. They think they've observed dozens of years and the evidence says it's nearly impossible for the market to go down (as a whole) over a ten-year period. I think they've observed only a handful of decades and therefore should have very low certainty.
I'm still heavily invested in the stock market. I just think you should be very cautious in your assumption of how much money you need to live comfortably on interest/dividends in perpetuity.
Historically, 4 or 5% appears to be a reasonable expectation over a 30-year timeframe after 1940 if you consider each year as an independent random variable and expect that the future will look mostly like the past.
I agree that 4% is optimistic.