Open Side Menu Go to the Top
Register
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

09-17-2016 , 02:29 AM
All the stuff you mentioned is bad (except perhaps the speed > quality stuff, which can be a fact of life sometimes, time is money) but none of it is very unusual. Saying it's industry standard is a bit on the hyperbolic side. Passive aggressiveness is probably the worst, makes for an unpleasant work environment.

The last job I had, when I joined there was a code base written offshore that included things like using CURSOR FOR with procedural loops in SQL for things that could be easily done with a simple set query. The C# was just as bad. I promise you it was worse than whatever you're dealing with.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 02:38 AM
Quote:
Originally Posted by ChrisV
All the stuff you mentioned is bad (except perhaps the speed > quality stuff, which can be a fact of life sometimes, time is money) but none of it is very unusual. Saying it's industry standard is a bit on the hyperbolic side. Passive aggressiveness is probably the worst, makes for an unpleasant work environment.

The last job I had, when I joined there was a code base written offshore that included things like using CURSOR FOR with procedural loops in SQL for things that could be easily done with a simple set query. The C# was just as bad. I promise you it was worse than whatever you're dealing with.
Is it weird that it seems like everyone is always trying to oneup whoever posted their worst software engineering experience. It's like...

"Oh you thought 3000 line functions were bad let me tell you about some of the code that I had to work with at my previous employer..."

That could be a good thread idea... "Post a description of the most ridiculous code you've found while working for your current employer"
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 10:01 AM
Here's mine
  • Passive aggressiveness
    We have 2 teams and they actively hate each other
  • File organization is for idiots
    Everyone makes up whatever name and place they feel like
  • Testing is bad because it takes time
    Testing?
  • Have different code conventions based on language and file type
    Having conventions at all is above us
  • Do not plan ahead
    I have no idea what I'm going to be working on sprint to sprint
  • Speed > accuracy
    No one cares about either
  • <20% of your time working should be doing "work", the rest should be faffing around doing paperwork and wasting time wading through a sea of poorly structured code (ex. 500-line function, well over 80-chars per line)
    The only thing anyone cares about is billing time to tickets, not if they actually took that long at all. I literally work 2-3 hours a day.
  • Have no style guides
    style guides?
  • Let any random person make graphic design decisions
    No one makes any decisions
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 10:32 AM
Quote:
Originally Posted by candybar
I think we're largely in agreement but someone who's in charge of hundreds of engineers is almost always going to have enough power to make that call. I don't think there are many companies in the world where you're not autonomous at that level. Dysfunction usually goes the other way (lack of common standards, information silos, communication failures, us against them mentality, etc, lack of shared cultural norms across the whole organization).







Agree with this but I think you're describing a problem that rarely exists - platform teams at big software companies simply don't have the power to impose much except in very specific cases that are driven by executive decisions. It's just helpful if certain standards become understood by the leaders throughout the organization such that they are followed regardless of whether they are directly enforceable from the top. And platform teams can incentivize this behavior by creating and providing useful services that require using shared services/tools.







A devops team at this scale would is something more like a value-added version of AWS, CI and related services - individual teams are responsible for day-to-day operations and plugging into this infrastructure. This is important because at this scale, out of the box tools don't really work very well and/or require significant customization or at least common boilerplate to allow the organization to work together. Also, even an absurdly large service like Facebook is, for the most part, a single entity to the users - this requires a lot of central coordination. Obviously if your teams are producing separate, unrelated products that don't need to be coordinated, that changes things.


I don't think we're really talking about the same thing or the same types of companies because I don't really disagree with what you're saying except maybe in certain cases.

The one thing I will add is that I think you'd be surprised how often internal teams/groups have tremendous flexibility in the devops tools they use. We sell to a number of big companies and often it starts with small teams / individual groups and only goes company wide when it's being used by a number of teams and the company wants to negotiate a better deal. Even then it's rarely pushed to other teams (as in, you must use this), just offered.

Edit: My sample size is biased to rather 'progressive' companies. Which is probably true of any company doing actual devils type things.

Edit: devils is suppose to be devops. But I'm not changing it.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 11:04 AM
Quote:
Originally Posted by jjshabado
The one thing I will add is that I think you'd be surprised how often internal teams/groups have tremendous flexibility in the devops tools they use. We sell to a number of big companies and often it starts with small teams / individual groups and only goes company wide when it's being used by a number of teams and the company wants to negotiate a better deal. Even then it's rarely pushed to other teams (as in, you must use this), just offered.
Yes! But I think that's what I said though. It's rare that teams don't have autonomy at this level. And it's not a progressive company thing - by and large you have more autonomy (or they use their autonomy more) at big non-tech companies because their internal support is not as good, the software is usually not a product as perceived by their customers and the IT department, because they are removed from how the customers perceive the company, has less power. Big software companies that essentially produce one giant product (which describes most top tech companies or at least their high-level divisions) require a much higher level of central coordination. If you're at Google, you're much more likely feel that you belong to the larger engineering organization and your decisions have to make sense at that level than if you're at an investment bank, where you're more likely to feel that you belong to the business unit you're responsible for and have virtually no connection to the other engineering teams.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 01:53 PM
As bad or worse than "testing is bad because it takes time", we currently have tests that run with every build that 5-10% of the time just straight up fail for random reasons, thus causing a build failure and ~an hour delay for anyone who cares about getting a recent build while the next one runs, or for engineers waiting to make sure our pull requests successfully build before merging. Their failure is largely ignored and "start another build" has become a way of life.

The "waiting for pull requests to build" part is especially frustrating for UI engineers, whose changes ~never touch any of the logic driving those tests but whose builds still get ****ed up by those test failures.

Why the **** have tests? I kinda feel for the team that wrote + maintains them because they're slammed with other **** right now too and the tests are pretty complex to unwind, but like, come on.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 02:07 PM
Pull requests? Man, I wish we had some form of version control on this live code we're working on.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 02:53 PM
You don't have any VCS at all? I want to eventually get in pull requests that automatically fail if the linter fails but I'm too lazy to do that.. but at least we're using git.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 03:09 PM
Quote:
Originally Posted by goofyballer
As bad or worse than "testing is bad because it takes time", we currently have tests that run with every build that 5-10% of the time just straight up fail for random reasons, thus causing a build failure and ~an hour delay for anyone who cares about getting a recent build while the next one runs, or for engineers waiting to make sure our pull requests successfully build before merging. Their failure is largely ignored and "start another build" has become a way of life.

The "waiting for pull requests to build" part is especially frustrating for UI engineers, whose changes ~never touch any of the logic driving those tests but whose builds still get ****ed up by those test failures.

Why the **** have tests? I kinda feel for the team that wrote + maintains them because they're slammed with other **** right now too and the tests are pretty complex to unwind, but like, come on.
Are they e2e tests, integration tests or unit tests? What you're describing is a classic problem when testing against live data.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 03:11 PM
Quote:
Originally Posted by Grue
You don't have any VCS at all? I want to eventually get in pull requests that automatically fail if the linter fails but I'm too lazy to do that.. but at least we're using git.
We do in the way that, say, windows 10 has a "previous versions" option for files, but not in the traditional "I messed something up so let me go back to a previous version of this file" sense, if that makes sense.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 03:19 PM
Quote:
Originally Posted by suzzer99
Are they e2e tests, integration tests or unit tests? What you're describing is a classic problem when testing against live data.
We had a fun one we solved earlier. We had these end-to-end tests that would randomly faily - fairly often. But you'd run again and it would be fine. Annoying but not critical. We finally figured out that the UI was doing a modification command immediately followed by reading the modified data. The test, of course, checked to see that the modified data was as expected.

These pass 100% of the time in the dev environment, or in a docker container, but would fail sometimes against our CI environment, which mirrors our prod env.

The reason? We have our mongo clustered, with read preference set to slave, and sometimes the UI was faster than the data replication and would read the not-yet-updated version of the data.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 03:25 PM
Quote:
Originally Posted by Noodle Wazlib
Pull requests? Man, I wish we had some form of version control on this live code we're working on.
This is an actual WTF. But I don't think this is insurmountable - the fact that the company doesn't have a centrally mandated form of version control doesn't mean you can't use it.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 03:51 PM
Yeah no version control is 100x worse than any of those other things he originally listed.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 03:56 PM
Quote:
Originally Posted by RustyBrooks
We had a fun one we solved earlier. We had these end-to-end tests that would randomly faily - fairly often. But you'd run again and it would be fine. Annoying but not critical. We finally figured out that the UI was doing a modification command immediately followed by reading the modified data. The test, of course, checked to see that the modified data was as expected.

These pass 100% of the time in the dev environment, or in a docker container, but would fail sometimes against our CI environment, which mirrors our prod env.

The reason? We have our mongo clustered, with read preference set to slave, and sometimes the UI was faster than the data replication and would read the not-yet-updated version of the data.
Our live data is constantly changing entertainment data and user data. There is almost nothing you can write a specific test against and expect it will be there in a month. So you either have to write your tests so vauge that they're almost worthless, or constantly update them.

It's always fun every time some manager at work decides we need real CI/CD and more e2e testing, then we sit them down and show them the challenges and that you basically need a team of people whose primary job is to maintain the tests. They always walk away so dejected.

Or even better - the manager just throws it over the wall to some offshore team with a hapless onshore lead here to guide them. The first time I explain to the onshore lead what he's being asked to do, you can see the horror just slowly wash over his face. Yeah buddy, you're being set up to fail - welcome to the corporate world. The good news is if you can something sorta working enough that the manager can make a slide out of it, your accountability will drop to zero after that. Deliverable = slide proudly touting CI/CD.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 04:01 PM
Our CI system is not against "live" data - it's against a fixed data set that was taken from live data at some point. Also, most of the tests are written in a way that they'll pass against any data as long as it doesn't mutate while the test is running. Testing against real live data is usually a nightmare, yeah.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 04:16 PM
Quote:
Originally Posted by candybar
This is an actual WTF. But I don't think this is insurmountable - the fact that the company doesn't have a centrally mandated form of version control doesn't mean you can't use it.
The way we're set up, it'd be pretty difficult. Would have to manually make a copy of each file I'm working on, save it, keep track of any changes that work manually, etc.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 04:37 PM
Yeah blown away by no vcs in 2016. What's their Joel test score?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 04:39 PM
I've started a new side project based off of IP that is under creative commons, does my thing need to be under it too? I don't really care just don't want to get dinged by some license nazi.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 05:08 PM
Creative commons isn't a single license, there are a ton of facets to it. Which kind of CC license does it have?
https://en.wikipedia.org/wiki/Creative_Commons_license
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 05:28 PM
Quote:
Originally Posted by RustyBrooks
Our CI system is not against "live" data - it's against a fixed data set that was taken from live data at some point. Also, most of the tests are written in a way that they'll pass against any data as long as it doesn't mutate while the test is running. Testing against real live data is usually a nightmare, yeah.
We tried that one time. They spent like a few million and a couple years developing this dummy database (ITKO was the company who sold it to us) that was supposed to constantly "record" all it's stuff from live data. It never worked right and would create annoying bugs like telling you there were 18 of something in your search result count - but show you over 100 results.

Our lead back-end developer nearly went homicidal after spending all day chasing down a weird back end bug that turned out to be because of the dummy database.

Last edited by suzzer99; 09-17-2016 at 05:33 PM.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 06:04 PM
Quote:
Originally Posted by RustyBrooks
Creative commons isn't a single license, there are a ton of facets to it. Which kind of CC license does it have?
https://en.wikipedia.org/wiki/Creative_Commons_license
https://creativecommons.org/licenses/by-nc-sa/4.0/ this guy
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 06:05 PM
I don't get it... back up production. Restore in CI. Job done. You don't need up to the minute data, you just need something representative.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 06:06 PM
Quote:
Originally Posted by Grue
If you read your link, it lays out very clearly what you need to do.

Quote:
ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 06:12 PM
Quote:
Originally Posted by RustyBrooks
I don't get it... back up production. Restore in CI. Job done. You don't need up to the minute data, you just need something representative.
Most of our data is from back-end services we don't control (the entertainment data feed, user data, sales offer feeds, etc.). It's not as simple as just backing up an Oracle DB. Also if you back up a channel guide - 12 hour old data isn't going to work right in your display client. You have to create another program to update all your times to something current that the client will display properly. Same goes for ordering a movie at a certain time.

Just to create a valid test user, we have to integrate with like 3 different systems we don't control, and it takes a lot of manual effort. Then once we create them, they tend to do a poor job of paying their bill. So they go bad after a month or two.

I do feel like we could do a better job at this, but there's never been the will to spend a lot of money on a big project to automate test data. Well except the testDB/simulator I mentioned that someone sold them as a magic bullet, which they spent $millions on and was a complete disaster.

We like to spend money on magic-bullet products for non-functional things like streamlining our operations. But a labor-heavy project where no one gets wined and dined - better have a big consumer-facing component or it won't get funded. It's a problem. Higher ups want to see something tangible, even if that tangible thing is an expensive 3rd party piece of software, hardware or service.

This is the kind of thing that makes microservices appealing to a giant company.

Last edited by suzzer99; 09-17-2016 at 06:28 PM.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
09-17-2016 , 06:15 PM
Quote:
Originally Posted by RustyBrooks
If you read your link, it lays out very clearly what you need to do.
thanks but I didn't think CC was a software license or something IDK "package json creative commons" isn't too helpful but whatever I'll throw it in there and I'm sure no one will care.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote

      
m