I'll find out next week. But given one app is a middleware between web clients and back-end CRM, and the other is a single-sign-on type thing, potential 10 second spin up times could be a real problem.

LOOL - The software engineer’s guide to asserting dominance in the workplace

Yes, Im talking about "long living" in the concept of an always available api, versus something like a cron job that runs a couple times.

Debugging/logging sucks too, as does a proper dev environment, and gl managing DB connections coherently. We had to roll our own deployment/rollback script, which sucked as well

Hm, I think you must be able to keep a hot set going, otherwise they're not even an option. We use lambdas to process kinesis queues but those always have data. Maybe lambdas work better if you always have some traffic?

We use lambda for that as well. That is fundamentally different than serving an api. Does response time even matter in that case? We use it for ETL on streams to dump to data sources. There is no user sitting there waiting for a response in most good use cases.

Suzzer's use case is every time you hit an api route a new lambda spins up to handle the request. Even if you have consistent traffic, you cannot control what server the next request will be on. Just bc it used 1 EC2 instance for the first 200 requests, doesn't mean it will for the next.

For the record, I have never seen anywhere near 10 second spin up time, I feel like thats hyperbole at this point (but maybe I am wrong). The main pain I have had is connections opening and closing on random server instances, leaving dozens to hundreds of connections hanging in the wind. You end up having to set the max instances in the pools super high and set a lower timeout (or use dynamo db for everything, which I am sure is their intention)

We don't need server stickiness for any of our apis, we just have an auto scaling ecs cluster thing, so not knowing which server a request would go to is fine. Does it really spin up a new instance every time it serves a request?

https://theburningmonk.com/2018/01/i...rts-all-wrong/

No not every request, but you don't know the context of when it jumps to a different server. So you can open a db connection on every request, and close it on finish, or you can leave 1+ open on that request, which stay open for X amount of time. When requests server hop, those connections continue on open until they eventually hit your time to live.

Hm. I figure every endpoint is going to need a database connection, so you open one the first time the lambda needs one, and keep it open. Whatever request the lambda serves uses that connection. Connections are essentially anonymous, the "next" request doesn't need to use the same connection.

I guess it could be a problem if you have a *lot* of open lambdas? But most modern databases are OK with you holding open thousands of connections.

Yes, essentially that is the issue. You have Lambda 1-100 come in on Server 1. For simplicity say you open 5 connections to the DB. Any other requests that come into Server 1 will use those already opened connections, ie not spend any time on opening a new db connection. The problem is, eventually Server 1 is no longer partitioned for your lambdas. Again for simplicity, say Lambda 101-200 come in on Server 2, which now opens 5 more connections. Server 1 may never be used again by you (It still may, it is entirely unpredictable and there is no "known" way to select a certain server, you get them assigned seemingly random). This now means you have 5 connections open on Server 1 twisting in the wind, because they get no explicit call to shut down and go on open until a time out is reached.*

So yes, imagine that on a scale of a lot of requests. And again, you don't really know when or why the next lambda may decide to cold start on a new server. They try to keep you on a hot server, but again it is random because you are essentially borrowing empty server time.

Your point about a lot of requests is true yes, that is when it becomes a huge issue. But there are a few problems there.

1. If you have lower request volume, you have to pay for more connections than you are using still. RDS lowest plan uses 100 connections. I have hit this limit on smaller scale, so I would have to pay for more, even though the scale of my actual site using the database doesn't justify it.

2. It is harder to manage a system with random open connections in the wild, so assuming you have unlimited connections, you will probably at some point want to debug just what the heck each connection is doing.**

3. The entire concept of serverless is how easy it is to handle scaling, especially bursts of traffic. This concept is actually the opposite. If i pay for 1000 connections, a burst of traffic could cause a lot of them to go unused and I may still get connection errors. And I would wager that is a tougher issue to debug in relation to scaling problems.

I would also point out in a microservice architecture, we have a ton of services living on 5-20 connections, so problem 1 would be exacerbated pretty quick. In a large monolithic service with 1000s of connections, it doesn't really feel like serverless is going to be a congruent use case anyways. So to me, it doesn't really feel like there is a correct API use case for serverless systems that use a DB connection pool.***

* You can set the timeout to a very short limit, we used 5 seconds in many cases, but that can cause errors or false warnings so I am not really sure what is best there

** This is a weaker argument but I think it is a fair one. Visibility is a big deal

*** The way connection pooling works with server hopping in general leads to a necessity for "connectionless" database systems, which obviously amazon supplies with dynamodb and elasticsearch cloud. These obviously manage connections but you talk to them from anywhere through http



Sorry for my lack of clarity, I'm a nerdy introverted programmer

You can debug what each connection is doing via the database itself, usually. Maybe that won't be good enough but it's pretty easy in mysql, for example, to see what every connection is "doing" right now, what resources it's using, etc.

I am not sure why you'd have 5 database connections open in a single lambda. Do you really open 5 database connections to serve your endpoints? Do you have 5 separate databases? Or can "one lambda" serve multiple requests at once? I really honestly have no idea.

I guess I thought if a lambda fell out of use it would just get reaped - but they might stay awake the full timeout length? It doesn't seem like it would do this unless your traffic was very low. Why would you have 1 idle but alive lambda not serving anything, while another lambda served everything?

I honestly don't see why you'd want to do serverless API stuff but I'm a dinosaur so who knows. Elastic scaling on ECS is good enough for me. Frankly, we rarely ever scale, but our services get a relatively sustained throughput all day. It goes up and down with business hours but in a slowish natural way. Most of our services are just 2 running nodes and we could get by with 1. Having 2 means you can do seamless deploys though, plus, who knows, we could get a burst or increase in traffic.

Fair, as I said it's a weak argument, but I have had situations where a connection times out and I have to figure out if that was planned due to a wild connection, or an actual timeout occurred on an active pool. I am far from a DBA

A lambda is just a function that runs on a server. The 5 connections are opened on a server. The way lambdas work is they run on idle servers. 1000 lambdas could run on the same server, using 5 connections that were opened on that server. When the next function decides to run on a different server, the former server is still alive with 5 open connections. You are conflating lambda to server.

A request comes in, a lambda (function) runs, in order to run, it picks a server with idle cpu. A 2nd request comes in, a function runs, it picks a server to run on with idle cpu. They try to make it pick the same server as often as possible, but there is a large element of randomness to it.

Again, a lambda is just a function that finds a random server somewhere in the aws ecosystem to run on. That EC2 server it decides to run on belongs to someone else. They don't spin up a new server just for lambdas. This is why they are so cheap. You are essentially paying pennies to use already partitioned servers that have spare CPU. But this means, once your function is done running, the server is still alive in the ecosystem. So the connection you opened it still alive.

I totally agree, and that was kind of my point all along. I think the whole serverless fad is kind of stupid and does not work in practice as advertised. But every time I talk to a small company (or like Suzzer's new company in this case) everyone goes "Oh I heard lambda is great, let's try that." Well we tried it in a few different ways, and it fell on its ass.

Sorry, I should also mention this is all based on a technique to essentially cache DB connections, as described here: https://www.jeremydaly.com/reuse-dat...ns-aws-lambda/

You can of course open a new connection and close it inside of every single lambda, but this adds execution time to every single request


That guy actually has an article on the exact issue: https://www.jeremydaly.com/manage-rd...ns-aws-lambda/

Also this apparently just came out to target this issue:
https://aws.amazon.com/blogs/aws/aurora-serverless-ga/

A bit late to the party on this one but I don't think anyone pointed out you can modify the html markup to an existing user's email/pw combo to be authenticated as that user. (lines 562, 563, and 565). Funny thing... I was working on a dfs app a short while back for nhl. I was initially iterating through all the players based on the html markup. It was only afterwards I realized they had exposed the full list of nhl players on a nice global variable. Ty jesus! Made my life so much easier

Don't even need to go to such effort, can just make a cookie set to "loggedin=yes"!

Good catch except that setting cookies is a *********************.

That's a long word to be censored lol

OK, so I did misunderstand. I know that a lambda is just a single "function" but I had assumed that they would allocate a unit of computing for that function, run the function once, and then keep running it for the lifetime of the lambda (i.e. 5 minutes or so). i.e. get to the end of the function, go back and do it again.

I guess a "solution" although not a great one would be to enforce usage timeouts on the server side. No queries in X seconds? I close you. But then your code has to always be ready to retry anything that fails due to a closed connection (which is actually not *too* bad, a lot of db libraries automatically support that now)

Anyway, yeah, it sounds like more of a pain in the ass than I would have thought.

Suzzer, I mean, try it if they're really set on it, but find out if they're *really* set on it. I have talked employers out of shiny stuff many many times.

How troubling is it that on day 2 of my new react contractor being here she still hasn't downloaded chrome -_-

Working off Safari?

When our first big batch of infosys contractors showed up, they all were using IE with stuff like bittorrent toolbar, yahoo toolbar, winzip toolbar. Some of them had 1/3 of the usable space covered with crapware toolbars. Really inspired confidence.

I obviously have not been doing this very long but one of my biggest immediate pet peeves is when you ask someone about an issue that's happening with something they wrote, and they immediately go on the defense, and sometimes blame other people.

Like, you're not in trouble, I just want to know what's going on here. There's only 2 people here that do that. Everyone else is just like "I'll look into it" or "I'll fix it."

Is there any other appropriate response? Lol.

I made a mistake once, in like 2001. You can't expect me to own up to 2 within one lifetime.

So here's a question... an old friend of mine is the CTO of a company here. When someone from his company interviews with mine do I...

a) say nothing to anyone
b) say "hey I know your boss's boss XXXXX (is this, like, an implicit threat, see C?)
c) call XXXX and say, uh, your minions are jumping ship, what's up

a), or b) with an assurance that the interview is confidential. c) is way out of line.

I did A. Prob going to recommend we hire this guy. How big of a bouquet to offer to my friend.