I remember during the last winter olympics hearing stories about speed-skaters who worked at and were also sponsored by Home Depot during the off-season.
I should probably go look up their program- I've seen commercials again this Olympics where they boast about being the world's largest employer of Olympians.
My best guess is that most Olympic athletes train full time mostly year round for their sports and that in many sports they don't make a living wage doing it considering travel and training needs. I think the Pentathlon compensation outline for the US shows that "minor" sports are ones where the rewards of competing outweigh any monetary gain.
In sports like swimming and gymnastics I would guess top athletes (let's say Olympic finalists/perennial national team members) make a decent living wage in sports not considered traditional "pro" sports in most European countries in addition to possibly Asia. And in Australia and the US where they probably make a very good living.
There are fringe US swimming Olympians- national team members who may not be favorites to even make the Olympics- that have to supplement their stipends and meet winnings with irregular 9-5s also- but the stars basically are swimmers only and can afford to train and travel to meets. (unless still in college where it gets screwy because the NCAA is a bunch of facists)
That part is probably really only true in the last 12-15 years though- and Phelps type compensation as well as to a lesser extent Piersol/ Coughlin etc is even more recent than that.
edit: here's an older NYTimes link about winter olympians and Home Depot