** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** - Page 1565 - Computer Technical Help

Well, the parse_args and main structure pretty much yea. But that is the basic skeleton of all my large (200+ line) scripts and I just build off of it that way. I always have at least most of those elements.

I just wrote a doozie of a system using 90% bash (the rest of it was cmd.exe batch file format UGGGGHHHHHH kill me) that builds our entire software stack for 3 different OS's and 4 different languages, packages it in a way that it can be hosted on our website, and shoots it off into the void where it can automatically update the site. Ended up clocking in at 800 lines across 2 repos, jenkins, and the base runner script. I'd never be able to write anything like that without some kind of basic structure.

Quote

10-11-2019 , 09:49 PM

#39102

jjshabado

Carpal Tunnel

Join Date: Jul 2006 Posts: 22,732

Aren’t there tools for doing that?

Quote

10-11-2019 , 10:19 PM

#39103

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

Which ones?

Quote

10-11-2019 , 11:36 PM

#39104

jjshabado

Carpal Tunnel

Join Date: Jul 2006 Posts: 22,732

I don’t know. That’s why I was asking.

I’ve never been close to the release / build / deploy world. Where it’s mattered to me I just use what people give me. But it seems like an area where there’s been a lot of work - especially on build pipelines / automatic deploys / all of this stuff as code (no - bash doesn’t count).

Quote

10-12-2019 , 05:21 AM

#39105

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

I use jenkins - as well as a shitload of other tools to build. Still requires a lot of glue logic. It could be done in python but our infra favors this way. We have physical servers that I sysadmin.

I guess some of the VM’s I deploy and orchestrate could probably be offloaded to containers + kubernetes but I wrote my own kubernetes tool last year for us as an exercise. Lol. We tend to do everything the hard way. Less dependencies on outside tools is one of our design principles.

We recently started using lz4 compression and I feel like that was a big battle because the principal engineers wanted to write their own.

Here is the pipeline i built, put simply:

Step 3 probably could be kubernetes. Step 4 is quite complicated and involves compiling code for 4 seperate langs for windows,mac, and linux targets. Working with windows and mac servers from jenkins gave me a LOT of trouble.

Last edited by jmakin; 10-12-2019 at 05:32 AM.

Quote

10-12-2019 , 06:01 AM

#39106

jjshabado

Carpal Tunnel

Join Date: Jul 2006 Posts: 22,732

Wow, not using outside tools/libraries is just a terrible philosophy to have these days - especially at the level of compression algorithms.

It doesn’t need to be containers but there are other tools to help automate provisioning of machines. Things like chef/puppet/ansible/etc. Again I’m really far from an expert but feels weird there isn’t something out there that would save a lot of pain.

Quote

10-12-2019 , 07:38 AM

#39107

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

Probably, but I inherited this system and it’s a little too late to consider all that. It’d be a full time job and i can only devote like a third of my time to anything that touches dev work. We dont have a devops engineer. I’m kind of filling in until we get one.

One of the tools I co-wrote last year was basically ansible written in bash. I think the guy who spearheaded that wrote it because he was really bored.

Quote

10-14-2019 , 05:44 PM

#39108

suzzer99

Save the Cheerleader, Save the World

Join Date: Nov 2005 Posts: 99,984

Ok jmakin - the first request to create the pull request returns a massive JSON object that looks like this:

Code:

{
  "url": "https://api.github.com/repos/xxx/api-bl-donor_portal_bio_lambda/pulls/27",
  "id": 326110,
  "node_id": "MDExOlB1bGzI2OTc3MTEw",
  "html_url": "https://github.com/xxx/api-bl-bio_lambda/pull/27",
  "number": 27,
  "state": "open",
  "locked": false,
  "title": "eee",

...

So I want to capture that and parse out the number 27, which is needed in the 2nd api call to actually merge the pull request.

Here's the best I got so far. I'm reading the response of the first api call into a variable, then egrep-ping it twice. It works but I'm guessing there's a cleaner way?

Code:

pull_request_num=`echo "$api_response" | egrep '"number": [1-9]+,' | egrep -o '[1-9]+'`

Is there some way to get a value out of the regex w/o having to call egrep twice? In normal regex world you could put parens around the part you want to capture, like this: ([1-9]+). And then somehow that's output along with the regex result. I tried that but no luck.

Well I spoke to soon. I just assumed I could replace echo with a variable. Not working yet.

Ok fixed it - I had to leave the echo in and add backticks.

Last edited by suzzer99; 10-14-2019 at 06:00 PM.

Quote

10-14-2019 , 05:54 PM

#39109

RustyBrooks

Carpal \'Tunnel

Join Date: Feb 2006 Posts: 24,647

Can you use a tool like "jq"? It lets you pull out individual pieces out of json very cleanly.

ETA:

Code:

(otx) RBROOKS-MBPTE-8:labs-pulse rustybrooks$ cat foo2
{
  "url": "https://api.github.com/repos/xxx/api-bl-donor_portal_bio_lambda/pulls/27",
  "id": 326110,
  "node_id": "MDExOlB1bGzI2OTc3MTEw",
  "html_url": "https://github.com/xxx/api-bl-bio_lambda/pull/27",
  "number": 27,
  "state": "open",
  "locked": false,
  "title": "eee"
}
(otx) RBROOKS-MBPTE-8:labs-pulse rustybrooks$ cat foo2 | jq .number
27

Quote

10-14-2019 , 06:28 PM

#39110

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

I use jq for parsing json. Way better to use a tool than to try to do it in bash - it’s a nightmare

Quote

10-14-2019 , 06:28 PM

#39111

suzzer99

Save the Cheerleader, Save the World

Join Date: Nov 2005 Posts: 99,984

Apparently this is what it takes to set a variable equal to a previously defined array:

Code:

a=('a' 'b' 'c')
b=( "${a[@]}" )

Unless your array has spaces in the values, then you need to basically just loop over everything:

Code:

declare -a b=()
for i in ${!a[@]}; do
    b[$i]="${a[$i]}"
done

#BASHTHINGS

Quote

10-14-2019 , 06:29 PM

#39112

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

Question, why not do this in python?

Quote

10-14-2019 , 06:57 PM

#39113

suzzer99

Save the Cheerleader, Save the World

Join Date: Nov 2005 Posts: 99,984

I don't know python. But I should probably learn.

I could do it in node/grunt fairly easily. But ideally this will live in a CI/CD build job some day and it's annoying to have all those dependencies.

Quote

10-14-2019 , 07:16 PM

#39114

microbet

Solar Powered

Join Date: Jan 2005 Posts: 56,673

Python is a LOT more like java/javascript/almost everything else than bash is.

Quote

10-14-2019 , 10:35 PM

#39115

Larry Legend

Celtic Pride

Join Date: Jul 2009 Posts: 42,920

Quote:

Originally Posted by jmakin

Question, why not do this in python?

This is correct

Quote

10-15-2019 , 07:11 AM

#39116

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

Any time I find myself trying to do heavy string parsing or array manipulation, I think python wins. Python is just better for actual computation. Bash is better as glue logic for combining a lot of pieces together, and work in conjunction with the OS.

I think whatever you’re trying to do with arrays could probably be done in an easier way. I like to use the IFS variable which acts as a delimiter when you do these types of ops.

It’s funny I use arrays all the time and have never had to “copy” over one like in your example. I’m really curious now what your use case is - if you’re trying to do something that’s gonna be more than 200 lines I guarantee you’ll be happier tearing it down and rewriting in python.

Quote

10-15-2019 , 09:47 AM

#39117

suzzer99

Save the Cheerleader, Save the World

Join Date: Nov 2005 Posts: 99,984

The use case is I want the script to create and merge the lib repos, unless a special 4th parameter is added, then it should create and merge the app repos. So I would declare an array of lib repos, and an array of app repos, then decide which one runs through my loop based on the run time parameter.

I just ended up declaring the array inside the check for the parameter. A little messier but it works.

What I wanted:

Code:

declare -a lib_repos=(
  "api-bl-donor_portal_lambda_layers"
  "api-bl-csharp_lambdas_parent"
)

declare -a app_repos=(
  # REACTE WEBSITE 
  "app-donor_portal_web"

  # API GATEWAY
  "inf-cpl-dp_api_gateway"

  # NODE LAMBDAS
  "api-bl-cognito_access_lambda"
  "api-bl-donor_portal_admin_lambda"
  "api-bl-donor_portal_bio_lambda"
  "api-bl-donor_portal_db_lambda" 
  "api-bl-donor_portal_user_lambda"
  "api-bl-donor_portal_util_lambda" 
  "api-bl-idm_cognito_triggers"

  # CSHARP LAMBDAS
  "api-bl-crm_bio_lambda"
  "api-bl-giving_history_lambda"
  "api-bl-lookup_crm_user_lambda"
)

repos_to_merge=$lib_repos

if [[ "$4" == "MERGE_APP_REPOS_YES_I_ALREADY_MERGED_LIB_REPOS_AND_WAITED_FOR_THEM_TO_FINISH_BUILDING" ]]; then
  repos_to_merge=$app_repos
done


for i in "${repos_to_merge[@]}"
do

   ... create and merge pull requests ...

done

What I had to do instead:

Code:

declare -a repos_to_merge=(
  "api-bl-donor_portal_lambda_layers"
  "api-bl-csharp_lambdas_parent"
)

if [[ "$4" == "MERGE_APP_REPOS_YES_I_ALREADY_MERGED_LIB_REPOS_AND_WAITED_FOR_THEM_TO_FINISH_BUILDING" ]]; then
  repos_to_merge=(
    # REACTE WEBSITE 
    "app-donor_portal_web"

    # API GATEWAY
    "inf-cpl-dp_api_gateway"

    # NODE LAMBDAS
    "api-bl-cognito_access_lambda"
    "api-bl-donor_portal_admin_lambda"
    "api-bl-donor_portal_bio_lambda"
    "api-bl-donor_portal_db_lambda" 
    "api-bl-donor_portal_user_lambda"
    "api-bl-donor_portal_util_lambda" 
    "api-bl-idm_cognito_triggers"

    # CSHARP LAMBDAS
    "api-bl-crm_bio_lambda"
    "api-bl-giving_history_lambda"
    "api-bl-lookup_crm_user_lambda"
  )
fi

for i in "${repos_to_merge[@]}"
do

   ... create and merge pull requests ...

done

Quote

10-15-2019 , 01:57 PM

#39118

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

Ahhhhh. I see. So, this is gonna be a LOL bash thing but I would just use strings. You can iterate through strings like arrays in bash, using the "IFS" variable (Internal file separator?). By default it's whitespace.

So, I would just declare your array simply as arr="val1 val2 val3 ... valN" (no declare keyword, I don't really use that).

and use expansion to modify strings inside of it using this syntax:

${parameter/pattern/string}

Look up "parameter expansion" in shell.

Quote

10-15-2019 , 02:53 PM

#39119

jmakin

banned

Join Date: Jan 2008 Posts: 30,120

I have to brag because my first major project ($1mm+) is on track to have a successful release by the end of this month. I started out the project by losing my entire team and the product to be integrated for the stakeholder wasn't even nearly alpha ready. But I just knocked one fire out at a time until i got here. I didn't manage it particularly well and was pretty disorganized, lessons to learn for next time - but it got done, with only slight bitching by my stakeholders, who I am slowly learning to seem like colossal pains in the ass.

This one was tricky because I had to manage 2 teams from 2 different companies. I don't particularly want to do this again though. I don't know how to put it as a big win on my resume - I want to tailor it more for an individual contributor type of role in my next gig. I don't want to seem too managery.

Quote

10-19-2019 , 04:11 PM

#39120

PJo336

THRILLHOUSE!

Join Date: Mar 2007 Posts: 22,158

Well I finally got off my ass and checked out my market value. Just signed on for a new gig for ~40% more money. Hard to leave friends, but pretty happy overall

Quote

10-19-2019 , 04:22 PM

#39121

suzzer99

Save the Cheerleader, Save the World

Join Date: Nov 2005 Posts: 99,984

Nice!

Quote

10-19-2019 , 08:48 PM

#39122

jjshabado

Carpal Tunnel

Join Date: Jul 2006 Posts: 22,732

Congrats!

Quote

10-20-2019 , 01:00 AM

#39123

RustyBrooks

Carpal \'Tunnel

Join Date: Feb 2006 Posts: 24,647

Quote:

Originally Posted by PJo336

Well I finally got off my ass and checked out my market value. Just signed on for a new gig for ~40% more money. Hard to leave friends, but pretty happy overall

I guess this is something folks should check in on periodically. Whatever you make, there might be a job paying 40% more.

Congrats

Quote

10-20-2019 , 08:08 AM

#39124

PJo336

THRILLHOUSE!

Join Date: Mar 2007 Posts: 22,158

Thanks guys!

Quote:

Originally Posted by RustyBrooks

I guess this is something folks should check in on periodically. Whatever you make, there might be a job paying 40% more.

Congrats

Yeah. We're having a bit of a tech boom in Denver between Boulder and San Francisco companies moving in as much as possible. Prices have gone up a lot for devs all over town the last 2 years.

40 percent is probably pretty rare, but the writing was on the wall when most devs leaving my company were getting a fair bit more