So i was (not so gently) told by my mentor that what I'm trying to do can be done via linux tools but I don't have proficient knowledge of them. I can observe the problem pretty well at the process level, I know how to do that already. There's a ****load of ways to do that. I want to see it broken down PER FUNCTION though, couldn't really explain it to him right.

so I'm going to use valgrind-massif visualizer tool, which is a GUI. To install though I needed to spin up an Ubuntu image and run it on there, installing on my centOS machine proved too painful for my limited patience.

It creates a pretty graph of every function and memory usage. Pretty sure I can also see it in real time. But while I've been waiting all day for this ubuntu image to download i've narrowed the problem down to a few functions.

This is WAY beyond the scope of my position and experience but if I can solve this i'm gonna get a ton of credit I feel I desperately need.

We learned all that too, including tunneling breakdown voltage and all that nonsense.

It's funny, I was originally a math/EE major, and the first few years of EE are very basic math, basically some linear algebra and a little bit of differential equations, but just the easy ones and straight to how to solve them, not much theory.

So I was taking electromagnetics and this solid state devices class in the same semester and early in the year there were some homework problems that I just got ****ing *stuck* on, like one of them was "given that electrons are applied with the following doping distribution function, how many electrons are present in the sample" which is super simple - integrate the function over the length of the sample.

But I could not make it work, I could not integrate this function and neither could anyone else I knew. So I slaved over it and eventually I went to ask the professor. He draws the function on the board and is like "well this part is kind of like a trapezoid and that's sort of like a triangle, so just approximate the area and that's it.

And that's when I discovered one of the fundamental differences between math and engineering.

So we got an analyst to look at this data and the with what we have, the match rate is awfully low with our fuzzy matching. But it may be because some data from Source A does not intersect with some Source B, but it is really hard to tell if that is the case or not. I can't believe we didn't do this kind of analyst from the very beginning before starting this project.

This whole quarter has been absolutely demoralizing. Not only have we failed on that front, our other projects are looking to fail too. Both the staff and senior engineers don't contribute much and are practically dead-weight. The only good staff engineer we have just gave notice and is leaving. So it has been me and this junior engineer running this whole show. I feel like my team is so dysfunctional. Was planning on taking a mental day tomorrow but that poor analyst set up a last minute meeting in this morning to go over some more data. I may just go to that meeting and then head home afterwards.

Most projects fail. Mine’s definitely going to

Y'all tell me what "staff engineer" means to you, because at my current gig it seems backwards. We treat it as right between junior and senior engineer, but I'm more used to it meaning between senior and like... manager or principal engineer?

Yeah, that's unusual. The one you're used to is what I expect, something like:

Senior, Staff, Principal, Distinguished, Fellow

It's caused us some hiring problems, because sometimes we're trying to hire someone who's a bit above junior, like, say, 4 years experience, and the guy thinks we're hiring him to like, lead a project.

Anyway, my company is basically AT&T now so all "our" definitions are by the wayside now. I got my "new" title just today in fact.

uh I usually equate staff with principal?

Does anyone have experience profiling Node apps especially in production? I don't mean finding memory leaks, just put out a major version of my game and people are reporting latency and I'm like uh? Now what?

Here it’s Engineer 1, Engineer 2, Senior Engineer, Staff Engineer, Senior Staff Engineer, Principal Engineer, Fellow (only 6 in the whole company that has thousands of Engineers.)

Old job was Engineer 1, Senior, Staff, Principal, Technologist, Senior Technologist, Fellow, Distinguished Engineer.

As you can see levels like Senior, Staff, and Principal aren’t comparable at all across the two companies and take much different amounts of skill and time to reach due to having many less levels at new company than old one.

grue, have you tried pm2?
http://pm2.keymetrics.io/

Got promoted to manager today. I started a year ago, got hired as part of a 4 person team, and now the other 3 work for me. One member is out having a baby, one very excited and happy for me, and the other is this crazy russian dude who seemed pretty pissed. Hopefully he quits and I can hire someone I know in.

Barrin6 I read your story the other day and then went into work and found 90 of the top 100 most common things in the data set I'm working on were bad. Luckily the data has a pretty big long tail so we only eliminated about 1/3 of the data. The first thing I thought of was your story and how I may have wasted 3 months of work because of **** data.

I use PM2 on production yeah but `pm2 monit` is pretty useless. I'll dig into more low level stuff with it. Congrats.

Finally figured out a good solution to this. Luckily, the SDK we're using represents everything under the sun with an interface, which means that for each place where the object that wasn't being destroyed had a reference passed around, I could make a wrapper object implementing that particular interface (it implemented several interfaces), forward all calls in the wrapper to the underlying pointer, and pass a reference to the wrapper instead of the underlying, and then log in that wrapper's destructor if it was actually destroyed when we told the main object to shut down. After a few cycles of doing this, I finally found a place that didn't log the wrapper being destroyed on shutdown, so that was the culprit still holding a reference.

It was actually already a place I had inspected the code and added some logging to ensure it was functioning as expected, but what I didn't account for was some special state checking before freeing its reference that returned early in this case.

Feelsgoodman to spend days tracking something tough down and finally nail it. Also feels good that I didn't have to go to the next step, which would have been to modify the C++ STL to add debug information and probably be a nightmare to dig through (let alone get our project building with a modified STL).

Is it dumb to do websites with a React frontend and python/flask (django or w/e) backend? Like if you're doing all that javascript should you just use node?

This is generally not true in the corporate world. I've only been on one failed project in 10+ years. Corporations don't shoot for the moon.

I put timers on all our middleware so we could see where the slowdowns were. I always knew what "healthy" looked like for each function. If the middleware had no backend I/O, I knew it should be in the milliseconds. But I knew all the I/O (REST, DB) calls ranged from 1 to 5 seconds when healthy. So I could immediately spot if something was out of whack.

I could reset the timers whenever I wanted, and displayed them in a special page that I always had access to, even for production. To see a different node instance I just hit refresh on the browser - super low tech but since the instances were identical it never caused a problem. I never got around to averaging over instances.

Use whatever back end and front end you feel comfortable with. My only religion is I hate any kind of stickiness between the back and front end. It should be a clean, well defined API and no shared models or anything.

But if you're a one man show even that is probably overkill.

Ours is
Software engineer 1
Software engineer 2
Senior software engineer
Staff software engineer
Senior staff software engineer

Don’t know if we have any higher than that. I worked with many staff software engineer and they are like gods. And somehow I end up with the worst one out of all of them.

This guy wanted to ETL 2 tables and then do a product between those two tables and ETL that in code. We could simply do that in SQL on big query.

Every solution to a problem he starts off with “Have you read the book X?” Where X is some obscure book and then sidetracks for 5 mins giving an explanation with bunch of buzzwords. Im starting to realize that he likes to be convulted to sound smart instead of being concise.

I think it is time for me to switch teams.

End rant.

Oh yea that is the worse. I have someone at work like that.

The part I hate most is that if you try to contradict what the say or offer a more simplistic but valid solution people are hesitant to believe you because they think "zmog this person is so smart".

It's even worse when that person probably is pretty smart just does and says things that are unnecessarily convoluted and complex.

Interesting problem and definitely not an easy one to track down. I guess a lot of places would have debug builds that could track who is using a shared_ptr and then rely on testing to find issues. Like what you did could be part of a debug build and then in production it wouldn't be compiled. MSFT had check builds for Windows (at least they used to) that basically do this. Even though shared_ptr is a "smart" pointer there are pitfalls in using it.

Nah, there's no reason the backend needs to be javascript. Pretty much all frontends are JS now, but there's a wide variety of backend technology.

Enlisted senior engineers help today and he ran a memory profiler on it and declared there was no problem. I usually cave to him but i KNEW there was something going on so i stuck to my guns. My earlier theory was right.

Eventually i discovered some structure in the server is holding on to client connections even after they are no longer being used. It eventually closes them when the server shuts down, which is why there was no leak reported.

But the thing is, each client connection is only using about 1kib. I don’t think it can really explain a crash by itself, but maybe - my VM is tiny. I have a bad suspicion this is another issue entirely. But i felt good i narrowed it down more.

Okay, got updated DBAM offer today after I said "yeah that doesn't work for me" and gave them a hint about the kind of offer ILPS was giving me, and they upped it by ~45k.

Base: current salary minus 45k
Bonus: 20k/yr
RSU: 270k

So:
- ILPS: total comp is current salary + 50k cash + 50k private stock
- DBAM: total comp is current salary plus 43k cash (treating public stock as == cash)

With refreshers, if I can generally expect (though I don't know for sure how this works) refreshers to be roughly initial grant / 4:

Year 1: current + 43k
Year 2: current + 59k
Year 3: current + 76k
Year 4: current + 93k
Year 5: hopefully promoted already or get another huge grant or whatever

I have been reluctant to share exact details of the ILPS offer with DBAM, mostly because I didn't want them to simply match it and be done with it, but I guess given that ILPS is private, I'm not sure if DBAM would try to match the cash compensation (which they're now roughly near) or the total compensation (which would still be a decent jump, and would be awesome for them to match).

ILPS, of course, also offers refreshers, just with the caveat that it's more of the private stock which obv has a riskier value attached. That said, they are in an exciting area of tech and have a viable business (imo) ahead of them, so it's hard for me to weigh whether their RSUs will be worth $0 down the road or several times their current valuation.

Still, it feels hard for me to jump aboard the DBAM train with these two offers. After two years, cash compensation would be roughly equivalent, but I'd have an additional $100k in ILPS' private stock accrued, which (if it came to a liquidity event) would likely be worth a multiplier of that. And I'd say I'm moderately bullish on the work they're doing as a company. I came into this kinda wanting to pick DBAM, but they're kinda making it hard for me to want to pick them now.

I was browsing in a bit of downtime at work today while I was waiting for some junk to compile. My new company uses svn (unfortunately) so I've been googling a little bit on alternate vcs to git. I came upon this gem today. Quote from:
from a comment from I never looked at bug fixing from that perspective. Head asplode.

1 kb seems pretty low. How many connections until it crashes?

Is it possible something else in the software is using the structure that holds on to the session and there's some sort of cascade effect to contact the client with the stale session information?

Goofy,

Personally, I am not a large company type of person at all, and think what ILPS is offering you sounds way better. You'll have the opportunity to make a much larger impact and likely deal with far less BS and a lot of flexibility once you are a respected contributor.

However, there is something to be said for post-DBAM opportunities and generally getting the experience at one, which is valuable.

Both paths are likely going to have different high points and hopefully not many low points, but ultimately the decision may mostly make sense to frame around what you want to be doing in 5-7 years from now.