Quote:
Originally Posted by RustyBrooks
We have a suite of what we call "end to end tests" which are basically... put up a full stack and run the UI through it's paces. It takes a Very Long Time to run, doesn't really cover much, and fails a lot. I am not a fan.
Yeah at our place there's a Selenium team. Keeping the tests working is a full time job for them. They also take a long time and have to be run twice due to such high failure rate. Only if the test fails both times is it marked failure. I think it takes 12 hours to run 2 passes of the entire suite.
Also they do stuff like key on CSS classes or IDs that we inadvertently break all the time. Further compounding the problem - our back end data is constantly changing stuff like tv schedules and our test users only last a few days before they break, and it's a gigantic PITA to create new test users in the myriad of back end systems (which we don't control) needed to run properly.
It's very hard to write specific tests in such an ephemeral environment. You either wind up with tests that constantly need to be curated, or tests that are so vague they pretty much just confirm a page loads.
It's always fun when some new group comes in all pumped up about some new e2e testing technology - and I get to burst their bubble that no matter what the technology, keeping the tests running will be a full time job for several people. You can see their hearts sink. Heh.
I spent many months of my life beating my head against the wall trying to get protractor (angular-aware alternative to selenium) tests stable. Gave up.
We also have this weird bug in our karma unit tests where only in the CI/CD environment they show call stack size exceeded. It's some kind of weird race condition, but exacerbated by the fact that apparently the max call stack size is lower on RHEL 6.5 than it is on our local PCs or macs.
I spent a week trying to debug that on the RHEL server, never could identify which test was causing it. It's some kind of cumulation of tests, because removing one at a time doesn't solve the problem. But removing half does. Nor could I figure out how to raise the call stack size on RHEL or lower it to recreate the problem on my local machine. Painful.
And basically all of this was my personal quest to avoid the devs just commenting out unit tests until the problem stops. No one else at the company cares - as long as they can still say they have testing as part of CI/CD. They basically shush me when I try to bring up that our testing barely works. I suspect it's this way at a lot of big companies. Unless you have a testing evangelist at the director-level holding everyone accountable, the tests go by the wayside in favor of meeting deadlines for features the business actually cares about.
tl; dr: testing, sounds great in theory but hard to work at big companies, especially if execs only care about saying they have testing
Last edited by suzzer99; 11-17-2017 at 03:46 PM.