The density isn't the problem in the way you're thinking. There is definitely enough people per square mile in the Valley to at least support good commuter rail (I don't think it's any less dense than the suburban areas around Chicago or NYC). The problem is density around the train lines. In Westchester County, NY, every Metro North is the hub of a small urbanized area. I live 20+ miles from Manhattan, but in a 40-story high-rise from which I can walk to the train station in under 5 minutes. Have you seen what's around the Cal Train stations? Nothing!
Which came first; small urbanized areas that the metro north serviced, or small urbanized areas that sprang up because of the metro north stations?
If you introduce light rail without changing the zoning surrounding the stations, you end up with poorly utilized transportation that does not promote the growth of urbanized areas.
Keep in mind too that there's no huge, dense attractor anywhere in Northern California that could act as a hub to a bunch of disparate smaller communities.
I personally don't see any problem with Caltrain. It's relatively reliable and goes often enough. The only thing that's missing is free wi-fi, but I think they plan to put it in there eventually.
The problem is nobody uses it. Metro North serves a region with maybe 2-3 million people (depending on how many people in Connecticut you consider part of the service area), and has a ridership of 300,000 per day. Cal Train serves a region that's at least as large, but has 40,000 riders a day.
Here is the area around a Palo Alto Cal Train station: http://maps.google.com/maps?q=palo+alto+cal+train+station...
Here is the area around a Westchester Metro North station: http://maps.google.com/maps?q=Station+Plaza+N,+New+Rochelle,...