Two Plus Two Publishing LLC
Two Plus Two Publishing LLC
 

Go Back   Two Plus Two Poker Forums > >

Notices

Programming Discussions about computer programming

Reply
 
Thread Tools Display Modes
Old Yesterday, 11:28 PM   #35326
suzzer99
Carpal \'Tunnel
 
suzzer99's Avatar
 
Join Date: Nov 2005
Location: on top of the bell curve
Posts: 87,076
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Yeah he said serverless was great. I didn't know it handled stuff like that too. That's great. We'll probably use it.
suzzer99 is offline   Reply With Quote
Old Yesterday, 11:29 PM   #35327
suzzer99
Carpal \'Tunnel
 
suzzer99's Avatar
 
Join Date: Nov 2005
Location: on top of the bell curve
Posts: 87,076
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

FG doesn't win. First down does. Bet he Hue kicks.

Last edited by suzzer99; Yesterday at 11:29 PM. Reason: LOLOLOLOLOLOLLOLOL>
suzzer99 is offline   Reply With Quote
Old Yesterday, 11:49 PM   #35328
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 23,836
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by PJo336 View Post
Our setup for a new project is "fairly" automated but how do you handle permissions in AWS?

We use terraform to create roles and ecr repos and what not, which takes time to get through, we use some KMS setups to encrypt values, we use protobuf schemas. All these things are "easy" but tedious and take time. And those are a lot of the grunt work I am kind of talking about.
[Note: suzzer, I meant to lay this out to you the other day but forgot]

So basically the way we create our infrastructure is with a fairly simple in house tool (it's actually on github but I don't know how easy it would be for other people to use it)

There's one master git repo that describes the whole enterprise. So you start there - you make a config file for your new service, that lays out "what kind" of service it is, from a predefined list (like, api server, batch processor, sqs queue listener, maybe a few other things). You commit to this repo, which kicks off a cluster build.

The cluster build creates a git repo for your project and applies a template to it, so api servers get a skeleton created for them - some config files and a flask entrypoint file, etc. It also creates build job in bamboo. This job will look for any modifications to the git repo and
* check them out
* run the build
* run tests
* deploy to CI
* smoke test CI
* deploy to prod
* smoke test prod

You check out the new repo (actually, you use the tool and say "update any repos in this cluster that need updating)

The repo for the service has a service description file (a json file) which is basically a list of "service modifications" along with associated parameters. A service modification is basically any, uh, modification to the service. So you might add the following service modifications
* create RDS database
* create SQS queue, with such and such deadletter policy, giving RW permissions to user X
* add autoscaling policy with such and such settings

Each service modification essentially is:
* a fragment of a cloudformation template
* a cloudformation template paramter file
* a json file that describes the defaults for the service mod

So our tooling takes the base cloudformation template for your chosen service type, adds the cloudformation template fragments for each service modification, with their chosen values overlaid over the default values, and then submits that to cloudformation. A few minutes later, the service is up.

This is essentially idempotent - you keep resubmitting your cloudformation template and it'll do whatever needs to be done to change what you have into what you want.

We've given up on the "you can do anything you want" and instead just have a bunch of pre-defined things to do. Like, "we make SQS queues this way, and these are the parameters you can tweak"

Sometimes this means that to do what I want I need to **** around with making a new service modification, and I'll waste half a day on it. But generally things will fit OK into one of the existing things we have.

Let's be honest - 99% of the time you're making a service which fits into one a few categories. It's like building with LEGO instead of carving from a wooden block.

Our company is split into a few logical sections - one of the other main sections uses terraform and it seems like they spend days or weeks setting up infra. And their deploys take a long long time.

Like I said, generally about 30 minutes to first prod deploy, any given deploy, from git commit to prod is under 10 minutes.
RustyBrooks is offline   Reply With Quote
Old Yesterday, 11:51 PM   #35329
suzzer99
Carpal \'Tunnel
 
suzzer99's Avatar
 
Join Date: Nov 2005
Location: on top of the bell curve
Posts: 87,076
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Wow - I understand about half of that. Will remember this post for future reference. How many years of iterations did it take you to get to this point?
suzzer99 is offline   Reply With Quote
Old Today, 12:05 AM   #35330
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 23,836
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by suzzer99 View Post
Wow - I understand about half of that. Will remember this post for future reference. How many years of iterations did it take you to get to this point?
It's finicky, in the sense that there were a lot of iterations to get each cloudformation template where it needed to be, but none of it is *hard*

It's been iterated on starting about 2 years ago, but honestly the first version was 100% usable - some of our infrastructure still uses it. The guy who wrote it has been a VP for at least those 2 years and sort of hacks on it in his spare time. He's always adding new stuff to it.

The one ***** I have about it is that it's laid out like a java dev would lay it out (which is what the guy who wrote it is, so...). i.e. all the important stuff is in like
./src/main/service-name/service-name.py
./src/test/service-name
and so forth, wheras if I did it, it would just be
./service-name.py
because each of these directories literally has nothing in it except the next directory. That's generally how java wants stuff.

I should probably also mention that we heavily leverage docker, and each service is a docker image that inherits from a base image. The base image has an entrypoint script that has different handling depending on the service type you chose up front. So api-service instances run gunicorn on your python file, and batch services just run python on your python file, and there are other available targets. You can set run-time parameters in the Dockerfile for your service, like if you need to choose a different port or pass args to gunicorn or something like that.

I am 100% sold on docker, I use it every day at work and at home, it has simplified my life a lot. I've been doing a lot of microprocessor programming in my spare time and I used docker to get a working build environment for arduino stuff that doesn't require me to run the arduino IDE at all (because I hates it). Very simple - run python script, and it builds your project, installs on arduino (teensyduino in my case) and fires up a serial console to monitor output from the teensy. Making a non-arduino-IDE build environment means getting specific versions of various packages installed in a specific way, with files downloaded and copies to the right place, blah blah. Doing it in docker means I don't need to pollute my computer(s) with the details, and I can get a dev env up and running in a few minutes on anyone's computer - even my mac. Nothing to download / install / maintain (except docker itself). Just checkout my git repo and run script.
RustyBrooks is offline   Reply With Quote
Old Today, 12:07 AM   #35331
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 23,836
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

My previous job had sort of similar tooling except that there was really only one kind of service - it sort of assumed everything was an API server and if you wanted something different you were on your own. But, due to this simplification, everything could be done with a point and click web interface, and getting a new api server up and running was even faster. I actually have no idea what happened under the hood, I just know that it was very easy to get going (when it worked, which was not always)
RustyBrooks is offline   Reply With Quote
Old Today, 12:32 AM   #35332
PJo336
THRILLHOUSE!
 
PJo336's Avatar
 
Join Date: Mar 2007
Posts: 21,601
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

We're about 8 months into building the systems to set up new services. Honestly we can have a deploy done in 20 minutes, it just wouldn't do much.

Our main problem is we seem to chase the new shiny alot, so what you need changes quicker than the templates can keep up. We also use 4+ languages so it's a bit more difficult than just building one system for one thing.

Ours is mostly just a template generator that makes a docker based project in your chosen language.

It's all fine honestly, my beef is mostly just I want to write some damn features, have some people use it and learn how different designs failed and succeeded. But making a schema for an sqs message and doing rudimentary etl to put it somewhere else is rather boring and straight forward.

End of the day, I guess the truth is most tasks are just crud and etl in the back end
PJo336 is offline   Reply With Quote
Old Today, 06:03 AM   #35333
Victor
Carpal \'Tunnel
 
Victor's Avatar
 
Join Date: Jul 2003
Posts: 59,976
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by PJo336 View Post
The combination of "small deliverables" for AGILE! purposes and microservices has basically made it impossible to dig into any fun feature or task and it just feels like tiny maintenance actions all day every day.

Gimme adding a big feature to the Ole monolith any day
leaning this way after this recent experience with "thin slices".

like, how about just doing what makes sense. you got a huge feature, ya split it up. but dont try to split up a relatively simple story card so that 5 devs can simultaneously work on it.
Victor is online now   Reply With Quote

Reply
      

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Forum Jump


All times are GMT -4. The time now is 06:53 AM.


Powered by vBulletin®
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © 2008-2017, Two Plus Two Interactive
 
 
Poker Players - Streaming Live Online