FG doesn't win. First down does. Bet he Hue kicks.

[Note: suzzer, I meant to lay this out to you the other day but forgot]

So basically the way we create our infrastructure is with a fairly simple in house tool (it's actually on github but I don't know how easy it would be for other people to use it)

There's one master git repo that describes the whole enterprise. So you start there - you make a config file for your new service, that lays out "what kind" of service it is, from a predefined list (like, api server, batch processor, sqs queue listener, maybe a few other things). You commit to this repo, which kicks off a cluster build.

The cluster build creates a git repo for your project and applies a template to it, so api servers get a skeleton created for them - some config files and a flask entrypoint file, etc. It also creates build job in bamboo. This job will look for any modifications to the git repo and
* check them out
* run the build
* run tests
* deploy to CI
* smoke test CI
* deploy to prod
* smoke test prod

You check out the new repo (actually, you use the tool and say "update any repos in this cluster that need updating)

The repo for the service has a service description file (a json file) which is basically a list of "service modifications" along with associated parameters. A service modification is basically any, uh, modification to the service. So you might add the following service modifications
* create RDS database
* create SQS queue, with such and such deadletter policy, giving RW permissions to user X
* add autoscaling policy with such and such settings

Each service modification essentially is:
* a fragment of a cloudformation template
* a cloudformation template paramter file
* a json file that describes the defaults for the service mod

So our tooling takes the base cloudformation template for your chosen service type, adds the cloudformation template fragments for each service modification, with their chosen values overlaid over the default values, and then submits that to cloudformation. A few minutes later, the service is up.

This is essentially idempotent - you keep resubmitting your cloudformation template and it'll do whatever needs to be done to change what you have into what you want.

We've given up on the "you can do anything you want" and instead just have a bunch of pre-defined things to do. Like, "we make SQS queues this way, and these are the parameters you can tweak"

Sometimes this means that to do what I want I need to **** around with making a new service modification, and I'll waste half a day on it. But generally things will fit OK into one of the existing things we have.

Let's be honest - 99% of the time you're making a service which fits into one a few categories. It's like building with LEGO instead of carving from a wooden block.

Our company is split into a few logical sections - one of the other main sections uses terraform and it seems like they spend days or weeks setting up infra. And their deploys take a long long time.

Like I said, generally about 30 minutes to first prod deploy, any given deploy, from git commit to prod is under 10 minutes.

Wow - I understand about half of that. Will remember this post for future reference. How many years of iterations did it take you to get to this point?

It's finicky, in the sense that there were a lot of iterations to get each cloudformation template where it needed to be, but none of it is *hard*

It's been iterated on starting about 2 years ago, but honestly the first version was 100% usable - some of our infrastructure still uses it. The guy who wrote it has been a VP for at least those 2 years and sort of hacks on it in his spare time. He's always adding new stuff to it.

The one ***** I have about it is that it's laid out like a java dev would lay it out (which is what the guy who wrote it is, so...). i.e. all the important stuff is in like
./src/main/service-name/service-name.py
./src/test/service-name
and so forth, wheras if I did it, it would just be
./service-name.py
because each of these directories literally has nothing in it except the next directory. That's generally how java wants stuff.

I should probably also mention that we heavily leverage docker, and each service is a docker image that inherits from a base image. The base image has an entrypoint script that has different handling depending on the service type you chose up front. So api-service instances run gunicorn on your python file, and batch services just run python on your python file, and there are other available targets. You can set run-time parameters in the Dockerfile for your service, like if you need to choose a different port or pass args to gunicorn or something like that.

I am 100% sold on docker, I use it every day at work and at home, it has simplified my life a lot. I've been doing a lot of microprocessor programming in my spare time and I used docker to get a working build environment for arduino stuff that doesn't require me to run the arduino IDE at all (because I hates it). Very simple - run python script, and it builds your project, installs on arduino (teensyduino in my case) and fires up a serial console to monitor output from the teensy. Making a non-arduino-IDE build environment means getting specific versions of various packages installed in a specific way, with files downloaded and copies to the right place, blah blah. Doing it in docker means I don't need to pollute my computer(s) with the details, and I can get a dev env up and running in a few minutes on anyone's computer - even my mac. Nothing to download / install / maintain (except docker itself). Just checkout my git repo and run script.

My previous job had sort of similar tooling except that there was really only one kind of service - it sort of assumed everything was an API server and if you wanted something different you were on your own. But, due to this simplification, everything could be done with a point and click web interface, and getting a new api server up and running was even faster. I actually have no idea what happened under the hood, I just know that it was very easy to get going (when it worked, which was not always)

We're about 8 months into building the systems to set up new services. Honestly we can have a deploy done in 20 minutes, it just wouldn't do much.

Our main problem is we seem to chase the new shiny alot, so what you need changes quicker than the templates can keep up. We also use 4+ languages so it's a bit more difficult than just building one system for one thing.

Ours is mostly just a template generator that makes a docker based project in your chosen language.

It's all fine honestly, my beef is mostly just I want to write some damn features, have some people use it and learn how different designs failed and succeeded. But making a schema for an sqs message and doing rudimentary etl to put it somewhere else is rather boring and straight forward.

End of the day, I guess the truth is most tasks are just crud and etl in the back end

leaning this way after this recent experience with "thin slices".

like, how about just doing what makes sense. you got a huge feature, ya split it up. but dont try to split up a relatively simple story card so that 5 devs can simultaneously work on it.

Office: uncomfortable chairs, have to wear pants, have to wear shoes, fluorescent lighting, can't see your family, random people wandering around talking about TPS reports, have to commute, infuriating sound of people crunching on their snacks, office gossip and small talk.

It gets boring very fast. Last year I felt there was a lot more good salary info posted. But either the market has slowed down or ppl have stopped posting salary numbers as much.

does Blind or any of these other websites verify that you actually work somewhere?

It seems to require an employer email address. Which obviously stopped me from signing up right there.

I think its a subset of conversations that are hard to find elsewhere on the internet (at least for me). If people know of other good places like that I'd be curious to check them out.

Employer email required to sign up. But they only verify email at sign up so you can create multiple accounts over time and hang out in the private groups for all former employers. Pretty useful feature to keep a pulse on the big companirs if yiu shift around frequently.

cool. Glassdoor doesn't verify you work somewhere when you make a review. lol.

I'm on blind for the luls and casual racism imo. I did find out the "range" for the next salary grade/promotion that I want though.

Here's the email I sent to our Amazon tech/rep guy. I haven't heard back from him, but I figured I'd post it here as well to see if anyone thinks anything I'm trying is crazy. I'm particularly interested in #2 - the idea of invoking the C# lambda's directly, instead of having them listen over REST - which seems excessive. Rusty kinda answered 6 and 7 already - for how they're doing it.


Hey X, thanks so much for taking the time to meet with us today. I'll try to keep this as brief as possible. I have a few big picture-type questions I've been wrestling with.

The core app we're going to build first is mostly just middleware talking to a CRM system called Blackbaud. There is an app already created (but never deployed to prod) in C# and Drupal (barf). We're replacing Drupal with react and re-using as much of the C# code as makes sense. Blackbaud exposes APIs over SOAP, so a C# or Java API wrapper is the logical choice.

A very simple extremely rough draft of the app architecture looks like this:



The node layer is optional but could be used for stuff like chaining sequential calls to the C# lambdas (which wouldn't make sense to do over the internet). I want the C# microservices layer to be as granular as makes sense - and not be dictated by the client. At the same time I don't want the client jumping through hoops to accommodate the C# API layer. So when it makes sense node could serve to chain, aggregate or choreograph calls between the client and the C# layer. Basically be the Backend for Frontend pattern. I used to call this an orchestration layer, but apparently orchestration is bad and choreography is good ��

The user SoR could be our Cognito pool, social media, or a couple of different existing systems, which we may eventually migrate to the Cognito user pool. The goal is not to create yet another logon for users. So at first only brand new users who don't opt for social media login will go into the Cognito user pool.

Blackbaud will be the SoR for most of the other stuff, but whatever data we need will go in a dedicated dynamo or RDS DB.

Questions/sanity checks:

1. We will favor SNS and step functions wherever we can use them. But I'm not really comfortable using async when the client is making a synchronous call and the user is waiting for a result. I don’t think we're quite ready for the advance microservices, full-async, CQRS, eventual consistency, etc. model yet. We want to make sure we can crawl before we get to that kind of flying. Is fully async a model you see a lot of people use?
   
2. For synchronous API Gateway to lambda calls, or lambda to lambda calls - I've been wrestling with REST vs. straight invoke. I tend to favor invoke over REST from API Gateway to the C# lambdas - since I don't need to load any http server or route listening libraries. I think I can also invoke a C# lambda from a node lambda if/when we add the node layer, right? Whether I use invoke or REST - I'd like to add the node layer without modifying the C# lambdas to accommodate it. Does that kind of architecture make sense, and have you seen it used/attempted?.
   
3. In general I've been wrestling with how much to leverage API gateway route handling vs. proxy to node. One reason I like the idea of API Gateway handling all the http and route handling - is that at first I can have it return mock data. So once we agree on the API - the front end devs can just hit API Gateway directly for their mock JSON data - which they can also edit as necessary for edge cases. Have you seen clients start out development that way - basically using API Gateway like you'd use an interactive swagger server?
   
4. I wonder if there's an easy way to turn a swagger API definition into a API Gateway CloudFormation template? Is that something you've ever heard of?
   
5. So that's for early development - for ongoing development - ultimately an important goal is for front end devs to very easily add new routes to access the API layer. So I may migrate to a node proxy eventually. Front end devs will probably be much more comfortable adding and handling routes in node, than they would be mucking with a cloud formation template for API Gateway. What I don't want is for them to have to muck with a bunch of different places just to add a route and some handler business logic on the server side.
   
6. On the question I asked you at the elevator - can you point me to a blog or something about how to keep lambdas in the same repo but still deploy them independently? How does the first line build step know what changed and what didn't? Tags maybe?
   
7. Do most people generate one giant CloudFormation template for the whole project, or somehow just the stuff they want to redeploy? Again I struggle with figuring out how the system would know which things have changed and which haven't. I just started the CloudFormation master class on Udemy. Hopefully it will answer a lot of that.

So the staff engineer is calling me out for trolling him on PRs. There has been two instances including today where I disagreed with him on a certain aspect of a PR. I expressed my opinion and he DMs me saying that I’m trolling him. He also said that it seems like I always am in contention with his views which he has been keeping track for months now. Which is odd considering I probably only disagreed with him twice.

Anyways he wants to talk about it offline one on one. Now I’m sitting here (working remotely) wondering if I have been unknowingly been bias against him. Am I supposed to wholeheartedly always be on his side and not express my opinion simply because he’s higher rank?

My other teammate was always in contention with him and another staff engineer on the team to the point where he moved teams.

Anyways, going to have to clear it up with him person when I’m in the office next week.

Sounds like he's being a little *****.

Peer vs. peer programmer battles are aways terrible. It works better when there's a clear lead to settle disputes quickly imo. Someone should be clear owner of a layer at least.

In person meeting often helps defuse things.

Yea agreed. For the first time, I think I should have brought it up to him directly.

The first time was when another developer questioned the service contract, which I also expressed the same thoughts. Looking back, I should have help him save face and talked to him directly.

This second time, I should have just let it go since I have given up on this discussion before and let him have at it many months ago.

For context for the second one: the IDE allows us to optimize imports. It’s a natural thing to hit the hot key and let the IdE sort the imports. This Staff is adamant that we don’t use it, despite the fact that we have been using it for all our other code bases. my argument is that it doesn’t scale well and we should careless what the import statements are. Instead focus on coding and shipping. The way he does imports is way different than how the IDE does it.

idk, I don't think having someone disagree with something you wrote/designed is a thing from which you have to "save face". Writing better code is a collaborative process and you have to put your ego aside for it.

Regarding programmer battles - I worked with a guy at two different jobs (I got him the job at the second place) who was a little fussy about how to do things but overall super smart and a very good dev.

At the second job, he got into a horn-locking battle with another senior dev that was about as ugly as I've seen. Didn't help that their cubicles were right next to each other. One day I went over there at like 1pm and I could smell my buddy giving off BO - he was so pissed off he was oozing testosterone or something.

It all comes down to human ****. All their battles were over stuff like re-ordering import statements or whether to throw an exception or not. I was in the meeting when it started too. My buddy was basically calling the other guy's exception strategy ****e. You have to tread delicately with that stuff,

Yep

Another coding interview, this time they asked the "island problem".

If you were stranded on a desert island and could only use one coding tool?

lol

Given a grid of 0s and 1s, determine how many "islands" there are in the grid. If anyone has ever played the Civilization games... if you imagine every in-game tile can be represented as 0 (sea) or land (1), calculate how many islands there are.