Yeah that was the final death blow. This is the lawsuit I was talking about. http://www.latimes.com/la-me-kdday4dec07-story.html

I get now that you might want to turn down the alarm when you're in the middle of saving the patient's life. But I can't think of any case where you'd want to turn down the alarm when the patient isn't in the middle of triggering the alarm. So yeah some kind of timer - IE - the patient goes 5 minutes w/o triggering the alarm - volume goes back up.

If nurses are turning down the alarms for false positives that's a big problem.

Dave, re healthcare: I honestly don't see how what you're saying is relevant to what I'm saying. I think we're just talking past each other.


Dave, re dev environment: how do people do development at your places of business? Like let's say you have a simple web app and mysql backend. Don't they need to run their own version of the web app on their computer in order to see how the changes they make effect things?

Databases are a bit different. I have seen people (including myself) avoid a local db and only use a shared db. But its annoying and definitely not preferable to a developer getting their own local copy running to test their changes.

Hah, there have definitely been times in my life where the first real evaluation of a new employee was how long their dev environment setup took. It's kind of reasonable for some companies (based on growth rate / maturity / etc.) to have non-perfect setup instructions.

But, man, it's so much nicer when you have something like a nice vm that takes less than an hour to get started with.

My experience matchs Suzzers, the machine is "set up" by it; basically a user account and office apps then it's up to the dev and the wiki to do the rest. As more devs get onboarded the documentation gets better until it's eventually virtualized and dead simple.

There's a condition that premature babies have where their heart rate will dramatically drop, sometimes to 0, and then gradually come back up (in seconds or tens-of-seconds). This can happen multiple times a day. The vast majority of times there's nothing that needs to be done but wait. And if a nurse is responsible for 2 or 3 babies, it can happen a half dozen or more times/shift. [Edit: I've thought of at least 2 other similar examples, but I'll spare the details, just say that humans are complicated and determining what is/isn't a problem requires quite a bit of complexity past simple types of alerts.]

So what you really want is an alarm that says "If the heart rate goes below X number and stays there for Y seconds" - alarm. Or possibly a combined alarm that says "If the heart rate goes below X and the blood oxygen level goes below Y" - alarm.

We can look at an equivalent situation like if Grue wanted to monitor his database. He might say that he wants to make sure the CPU usage of the database is below 80%. But if he just set a simple alert on that threshold, every night when he ran his giant query he'd get a false positive alert. Which is why we have tools that let us do alerting conditions like "The average CPU usage over the past 1/5/15/whatever minutes is greater than Y threshold".



If I remember correctly, some monitors/machines did have a muting function. Just not all of them.

Definitely. But its often not the nurses fault. If you have no power over the tools/processes you're stuck with figuring out ways to get around the problems inherent in those tools/processes. Frequent false positives are really really bad when monitoring anything.

None of this is commentary on the particular case at MLK because I don't know the relevant details.

I'm saying that I do not have the appropriate knowledge or training to know what kind of monitoring is needed. I also understand that, emotionally, it's very difficult to be clear-headed on exactly what is appropriate and what is not appropriate because I'm not doing this for 12 hours per day, 6 days a week. I have no opinion on this. I figure those people earned their knowledge with many years of training and know better than I.

I know, nurses, etc, but nursing school is 3 years fast-tracked on learning every element of nursing, not add-ons, really. I hear nursing school is extremely difficult.

What I seen was a machine was setup with all the tools, the environment, no root access, etc. The sysadmin just set it up with all the correct OS's, versions, deps, VPNs, etc.

Right, a shared DB seems more standard than cloning it alone. Version control hasn't been solved very well yet.

As a contractor, I always have to download and clone to a local machine. If the database versions are behind my machine, I simply code down and not worry about it. If the database was changed, then I get the updated version, pop it into a different database, and diff.

I'm glad I don't program medical devices - where a bug could mean someone dies. That would be stressful.

Ok, I'm saying I do.


I think we (and lots of other people) have this thing where we view our experiences as representative. But they're not. There's a vast spectrum of programming jobs out there. The higher paid ones are definitely not doing this, but I see your point that there probably are lots of cases where this is the reality.

And I think this is a big reason for the difference.

I've talked to a bunch of people about this and their number one reaction is that they don't want to work on something like monitoring humans because they'd find it too stressful.

Production bugs/Downtimes are bad enough when you're dealing with money and angry customers. Add in the idea that people could die...

Another reason is that its much harder to make money / be innovative. Because you're talking about life and death the 'release' process is extremely slow and expensive. People generally don't like A/B testing with humans - and when we do it (drug trials / medical studies / etc.) its a very rigorous and slow process. And so on.

jj;

The numbers I said I experienced means I should have have been deep six. At the least, I shouldn't have been aware enough to hear the EMTs curse about how how those numbers weren't possible for a living person. The only reason I probably survived such a low temp is because I naturally hang around 96 to 97 degrees, which is highly unusual and possibly deadly for most people.

The human body isn't discrete parts like a computer, where you can define health into chunks like that. I'm surprised one can read a college-level A&P book and compare human monitoring to a database.

jj, one article I read recently was a huge problem in medical monitoring tech is that of data protection / security. I do expect we will see great advances in this area in the near future, though. Especially blockchain related.

Most computers aren't discrete parts. Lots of things interact and have unintended side effects. Memory / network / disk / cpu / application level metrics / etc. are all related.

But, uh, ... all of this just goes to my point. The tools we have for monitoring computers are significantly more advanced than the tools we have for monitoring humans. They allow much greater complexity than a bunch of independent machines that monitor only a small specific number of things (heart rate, pulse, blood oxygen, etc.).

I honestly have no idea what your point is, nor am I sure you understand mine.

Edit: As for the database crack... it was an example. By no means do I think a database and a person are equivalent. I can't believe I need to write this.

Oh yeah, this is definitely a big one. Most people have no problem using a SaaS provider for machines. Obviously, its more problematic for detailed human health data.

And this then leads to things like more expensive on-site solutions - which further reduces profitability / innovation / etc.

Edit: If you have the article handly, I'd love to read it.

Uncle Sam is actually very involved in the process and the process is very thorough. Rest assured that a comprehensive set of failure scenarios are addressed, FWIW.

I'm sure.

But I've seen the validation process for clinical trials - cfr pt. 11. It's very heavy on documentation and solving for past problems - but doesn't guarantee bugs won't appear.

We have a collection of highly trained humans that can stitch all of these variables together. Humans can also monitor things computers have absolutely no ability to measure, like lucidity, pain threshold, weakness, allergic reactions, irritations, etc, etc, etc.

I think you are saying that, if you have more than one variable, then you can have a clearer picture of when to sound an alarm, and you believe that turning off alarms wholesale is an unneeded danger.

My counter is simply that they see 200 people a day, and they are far less likely to make mistakes than many outsiders give them credit for because of the perceived magnitude of the situation between the patient and nurse. To the patient, it's the end of the world; to the nurse, it's the 10th time she's seen it that day.

No idea why you guys are arguing. Yes, humans are good at it, but they aren't going to get significantly better at it. Yes, it's a hard computational problem but the upside is the orders of magnitude better machines can get better at the problem given the will to tackle the problem which probably doesn't exist outside of niches like the military.

I used to develop software for "drive-by-wire" engine management systems for diesel vehicles. Lots of nasty things that can wrong with a diesel engine including runaway conditions that continue after you switch off the ignition and remove the keys. Failures are, of course, potentially life-threatening.

The safety measures we built in were many and varied both in the hardware and the software. Watchdog circuitry that monitored the hardware functionality, and software fault detection that would gracefully power down the engine.

We performed multiple FMEA reviews on each hardware component, on each pin of the microcontroller, on each pin of the unit's wiring harness and even on the input and outputs of each software subroutine. Other analysis included things like what would happen if memory locations and stack pointers were corrupted.

We also spent many hours in an anechoic chamber blasting it with electromagnetic waves of various kinds.

The software was blown into PROMs, so if we made a mistake, we couldn't apply a patch. Instead there would be a recall of 10s of thousands of units.

It was intimidating at first, but once you got into the swing of things and understood how to approach it, it was a lot of fun. Probably the most enjoyable development job I had.

I imagine developing medical software has many more safeguards in place and would be very surprised if even the most serious software bug caused significant safety problems.

Dave, the point of monitoring isn't to replace humans. At all. It's to augment the abilities they have. Make them more efficient and let them focus on quickly reacting to urgent situations, deal with developing situations in a timely manner, and things like that.

Yes, humans can monitor things machines can't. That's completely irrelevant to my point. They'll do that regardless. At literally no point have I said machines should replace humans for monitoring things like lucidity.

My whole point is about how the machines for monitoring humans and the process around how those machines are used is WAY behind how the equivalent machines/processes for computers work.

Having more than one variable available is a small part of that. Realizing that false positives are really really bad is another part of that. Realizing that the 'severity' of an alert should correspond to the urgency of the action a person has to take. And like a bajillion other things like that.

Turning off alarms wholesale is like a tiny part of what I wrote.

None of this has to do with my perception of how urgently nurses react to situations.

This isn't what we're arguing about.

I don't know what Dave thinks I'm saying (like I have literally no idea) because I'm taking about replacing current monitoring with better monitoring and not about replacing humans at all. (As an aside, humans aren't good at monitoring a lot of things. That's why we already have things like heart rate alarms, blood oxygen, etc.).

It's also not that hard of a computational problem. There are at least a dozen companies (and I suspect hundreds) that regularly deal with the amount of data to monitor all the humans on Earth. There are more individual metrics monitored in machines than humans and there are more machines then humans. It's actually fairly easily done with mostly off the shelf open source tools and a 'reasonable' amount of compute power.

And I'm not even talking about how you can do things like anomaly detection, outlier analysis, correlations, with better and combined monitoring data.

The Therac-25 example was used all the time in my engineering program to make us feel like real engineers that had people's lives in their hands.

um ok so more stupid back end problems/questions:

I need a (daily) database hit on all completed games mapped to an object so that my /stats page can show graphs etc. Prior to the game getting pretty popular it would be OK just doing a mongo collection .find({}) and then processing. Now with 21000 games and a $10/month DO box it just can't handle it - it used 100% memory then crashed. I was hoping it was just Mongoose but the native driver did the same thing.

I would think my best option is to do this in like 1000 game increments and append to a file? Any suggestions how to approach this? Obviously can set up a 4am cronjob but (FFS) if it crashes due to out of memory, PM2 never restarts automatically.

Is the easiest solution just a swap space? I have 22gb of SSD free and only 500mb of memory open.

Can you do hourly updates instead of daily?

Don't have a solution but as a crotchety old man who can't be having with this NoSQL stuff I'm gonna point out that a SQL query to calculate stats over 21,000 games would probably run in like 3 seconds and that Mongo probably wasn't a great choice considering your data is relational. You kids get off my lawn.