In my experience I don’t travel the Friday, Sunday or Monday of Labor Day. Come in on Thursday, leave on Tuesday.
If your schedule/life just simply won’t allow for that type of scheduling then be prepared for a 7-8 hour drive that’s usually 3.5-4.5 hours.
Don’t anticipate any road closures but check Sigalert.com before you leave. Depending on when you leave and what part of OC you’re traveling from (I live in HB) the toll roads 261/241 and 91 east will save you an hour for about $15 in tolls
a real stress saver when traveling during commuting hours