We don't use 3rd party libraries. We write everything from scratch. (Well the crazy guy I have to deal with.)

I'm glad though that he was at least on the right track with multiplexing, whereas I was the confused one. It's his execution more than overall strategy that's a problem. Also he's wildly optimistic that every new layer of complexity he adds to the app will snap on in a day or two and never cause further effort to maintain.

please say you're not writing ws stuff from scratch over socket.io.. right?

I agree the pattern makes sense for a framework. I guess it's just hard to reconcile that there may be so many handlers that we don't want to explicitly attach them while at the same time, they all need to run for every socket.

Pretty sure that's what's happening. There's a lot of socket.emit and socket.subscribe going on.

I have not been involved at all in this admin app they've been cranking out. Which I'm sure is a complete disaster so I'm trying not to get sucked in.

Oh yeah - that's a very real concern of mine. I mentioned it in the last sentence of my OP on this. But at this point I'm just looking for gigantic sinkholes that will destroy the car and kill us all. Later we might look for potholes that will just tear up the suspension a little.

Aaaaand our latest fun. This little bit of code (in the aforementioned socket handlers):

Throws this error: TypeError: lib.logger is not a constructor

Turns out lib.logger is not ready yet when the module is required. Solution? Make sure all dependencies are ready at init time for modules that are being required downstream? Nope. Just require any problem modules (which is pretty much all of them now that his custom logger is going crazy everywhere) at run time instead of init time. PROBLEM SOLVED

And as a solution he suggests we add in a setTimeout of 1 second. Then lib.logger is ready apparently. Just ****ing kill me now.

I finally broke my non-working streak. It's another contract, working with Clojure and MySQL.

A "legacy" project.

Awesome stuff

How do you guys working with data scientists validate their work? With traditional software you can compare inputs to outputs with test cases and requirement documents and all that but with forecasting models that doesn't seem meaningful or possible, how do you share the confidence your data scientists have in the model with other people? Is there a way to do it that doesn't have "get a maths degree" as step 1?

I guess I have the following answers:

1. I don't. It's the data scientists job to validate their work.
2. Presumably there's a point to their work and you can tell if what they're doing is working.

in general, when people do complex analyses, there are often "sanity checks" you can do on their numbers, based on simple high-level numbers you already have. eg, they might be showing monthly growth numbers are simply not compatible with the years starting with X users and ending with Y. this can never prove they haven't made mistakes, but can catch obvious one.

another option is create some sample data that's so simple you know what the answers should be manually, and then feed it into whatever they tool they created for you, or ask them to run it, and make sure it comes back with the answers you expect.

We use sample data that has been hand classified. It's... not perfect. And we spend a lot of time wondering why our learning models don't produce the same data.

Query question :

You are trying to optimize a query that *only* powers something in the UI. You are testing your query in whatever app you use to run queries against your production db. What is your breaking point for saying "this is fast enough" vs "this is still too slow"? [in terms of ms]

lmao, the solution to all race conditions

CI or any other reasonable metric of uncertainty. That is pretty advanced tho, someone is like "what % chance is it gonna rain tomorrow" rather than a heatmap of varying levels of rainfall and probability which could be useful (in the use case of do I need an umbrella). Ultimately this is why the BI guys make like 3x the data munging folks.

Post-hoc crosschecking learning/test dataset for sampling error, making sure your model doesn't have some weird feedback loop where min/max is totally unreasonable, etc. This is a good post. I support this.

Problem is when your model shows up something that you don't understand (as all good ones should) and you have to explain why that is the case. (Computer designed circuits compared to human designed.)

That sounds terrible.

The original question is sort of impossible to answer generally because "Data Scientist" is so incredibly overloaded. But assuming we're not talking about sort-of-trivial or generally solved uses cases, the above is the problem with most of the suggestions here (aside from the sanity check ones I guess, where sanity checks are always good).

If they're doing anything interesting, they presumably have enough knowledge to figure out meaningful ways to test and evaluate their work. And ideally you have a process that can actually judge how their models are doing. An example being something like ad tech where if your model doesn't generate more revenue, it's very likely not better than the previous one.

There are political issues with this as well, it sounds like IT wants to "test" the software but doesn't know how to with forecasts. I think at the very least we'll be comparing forecasts to performance as much as possible for tests. Fortunately we'll have the historic data to do this. We'll probably be emphasizing that while these numbers are 100% real that they are also an estimate... Should be fun. Thanks for the input.

user facing? 50-100ms
internal only? maybe a second or two.

And this kinda depends on what else is going on and what you're doing. Is the user searching for something? A second or two might be fine. Is the user clicking a button to change the status of an object, or dragging something from column A to column B? Then it better seem instant, so 100ms or less.

I mean, for example, we have an endpoint that you feed a URL or a PDF or whatever into, it processes it, extracts a bunch of info out of it, validates that info, finds matches, finds recommendations based on the matches, etc, etc. It usually takes 1-5s on top of however long it takes to actually get the data at the end of the url you gave us. And this is fine - it's a rare thing that some users do as part of original research.

It kinda is. Fortunately, not my problem. One of the main things the data scientists are trying to do is take buttloads of samples and "cluster" them by things that are similar, and then find clusters that are "significant" in some way. This is something that is not too hard for a person to do, except ain't nobody got time for that, there are tens of thousands of clusters per day. The cost of missing some is maybe not too bad but the cost of false positives is high. Also I think the data is pretty dirty.

so is Angular 2 super worthwhile to learn?

at my job I have the opportunity to get put on a team that uses Angular 2. I am going to take it bc I like to learn and it should also be my dev work. but I was just wondering where Angular 2 falls in the grand scheme of the things currently and going forward.

Seems to have quite a bit of traction at this point. The job descriptions I read list it way more often than React or anything else.

I vaguely recall someone on here being anti typescript though. If I'm not making that up I'd be interested to hear why.

Dynamic typing is a scourge on production code, so I doubt anyone sane could be against typescript.

Interesting. Maybe it's regional. In the bay area react seems to have much more traction.

I've used both and prefer React but my experience with Angular 2 was from over a year ago when a lot of things were still changing and poorly documented.

Dynamic typing is hardly the devil you're making it out to be imo. I can't even remember the last time I encountered an error that would have been caught by a type system