I'm pretty sure blacks play on the 1-2 tables (if not, you'd have to have greens in your pocket instead). Not sure about purples at 2-5, but again if not you'd need to have blacks in your pocket. As long as you're using chips that are in play at the table, that's all you need to do to not miss hands.
It's not a matter of 3x100 chips slowing the game down more than 3x100 bills. If another player won't, a chip runner can break your black chips down further if you want, but in the meantime your chips are in play and you don't miss hands. This is not true of cash. And the dealers don't get involved in breaking it down at all (or at least, they don't have to), so they don't need frequent fills they way they do in rooms where dealers do sell out of the rack.
I don't think the dealer not expecting it is really an issue. If he skips you, speak up and say you're pulling chips out of your pocket, and he'll back it up and deal you in.
I don't think it's petty for you to want to play your button, but you won't always bust out on your button, and a single missed hand is not the end of the world in any case. And if you really don't want to miss a hand, you can take steps to make sure you don't (with chips in your pocket).
Again, I understand the desire to make the rebuy process as frictionless as possible, but I don't think people realize exactly how much time is lost to tray fills in rooms that sell out of the rack. If all the regs bought extra chips and sold them to newbies and casuals who don't know the rule about cash, then no fish would get tilted and leave.