I come back sometimes and see that I lost some of my time bank, doesn't really make me too mad. But most of the time, when the break says 5 minutes, if your back before the 5 minutes is up, you won't. Lose. DA TIMEbAnK yo.
"if you're on time you're late, if you're early you're on time."
Also, you never want to come back to your grind station while sitting out, see you are dealt aces, and see your aces auto folded.