Just wait till they beat down all their competition and raise the prices

You can add a tag and then have that tag added to the cost report. So adding things like team/role is a good idea so that you can actually watch costs for specific things over time and know who should be bothered.

The opaqueness isn't even AWS being evil. It's just a natural consequence of offering all these services (many of which involve usage of other services) and usage based pricing. Throw in things like reserved instances, spot prices, usage based discounts, etc. and its just a crazy amount of raw data.

There's also zero chance of AWS beating down their competition and raising prices. Google and Microsoft aren't going away.

Like most places we have a guy who I think wrangles our AWS stuff as a full time job. He's always looking for stuff to cut, figuring out where our money goes, figuring out how to save money on what we're usng, handling complaints (we do cyber security so we are often doing stuff that gets interpreted as shady behavior)

A friend of mine told me about how he spent some time reducing spend greatly from one service, only to have someone in the same week make an oopsie that cost 10x that much, wiping out his savings for a year.

ETA: someone owned one of our accounts once and spun up as many super large instances as possible to mine for bitcoin or something. That cost us $60k in about 12 hours.

Damn.

There is no better industry to be in than one with opaque pricing.

I'm about to call the drug store to try to find out how much two prescriptions I get will cost w/o insurance, which I have none right now. Think I'll get an easy straight answer?

IME, yes. I lost a prescription bottle and needed a replacement. Initially they sort of shrugged and said "your insurance won't cover a 2nd one in a month"

After a beat I paused and asked what it would cost me to pay myself. He tapped for a second and said "8 dollars." Uh, ok, let's do that.

Granted, he didn't know before checking, but it was easy for him to do.

I mean they already have like half the market share.

what jj said. do you guys really not tag all your stuff and then complain later that you don't have a way of seeing exactly what's costing what?

I'd be interested in hearing how those who are having issues with AWS billing have structured their accounts/organizational units/tagging strategies. With departments broken into OUs, IAM roles delegated to those who need access to the billing info for each OU, and use of Cost Explorer, Budgets, and tags, one should have a solid grasp of where funds are going and what the anticipated spend should be in the coming months. And if there is outsized consumption you should be getting alerts about it.

Their competition is going so far as buying data centers from prospective customers to get them on their platform so it'll be a long wait.

It's more like 1/3 and all the other big players have increasing market share.

I don't know the exact numbers (mostly because 'the cloud' is so hard to define) but AWS isn't gaining significant market share - and I suspect is probably treading water or even dropping slightly (as a share - not in absolute value obviously).

But aside from that, Azure is a solid chunk of Microsoft's revenue now - and growing crazy fast. GCP is smaller (both absolutely and as a percentage of Google's revenue), but still growing fast.

Both companies have crazy deep pockets and major incentive to keep competing. There's literally no chance they go away and let AWS have the business.

AWS is also pretty open about this. They know this is a low-margin business where you make your money on scale.

Funnily enough, the company with aws problems that I am working on is a health insurance one.

CloudHealth was a client of mine at one point (they also got bought yesterday for what I felt like was short money) and they basically had a sophisticated tagging system.

Data is something we control. Every api call has the potential to throw an error because something is passed incorrectly, is omitted when its required, etc. I'm a fan of trying to catch errors as early as possible. If an api call is going to fail (or should fail if they've validated everything) then why not escape right away?

One issue we've had a lot is api calls fail then give no reason for why they failed. That usually means trying to hunt down who wrote it then ask why it failed. That can add up to quite a bit of lost productivity over time. Catching errors early allows us to always know why it would have failed.

Yea i pissed off problem person yesterday for a similar issue i brought up, she was redirecting stdout and stderr to dev/null so i had to run a trace just to figure out why things were silently failing

I've always been kind of conflicted on the CloudHealth type companies (or say Spotinst, or ...). They're valuable, but I wouldn't want to be in the position of being so dependent on the big cloud companies. The cloud companies are constantly building features that compete with what they offer and can introduce features that totally screw these guys (for example, AWS made major changes to how spot instances work last Fall that ruined a couple of features that a bunch of companies were promoting).

Our other problem with CloudHealth is that they wanted too much money for the value they created after a certain scale. It quickly became worth it for us to invest in our own tooling around cloud costs and reporting.

Just got roped into a meeting to discuss our billing. But they won't give me a detailed view of our bill. I guess I can just randomly turn **** off till our bill goes down.

Our problem isn't which solution we're using, it's the finance guys not giving us anything at all and not even listening to us about getting the data out. But I will recommend CloudHealth to them.

If data can only be null or undefined due to an internal coding error, then you shouldn't check for it at all. Just let the system break and force it to be fixed at the source.

If data can be null or undefined as part of normal operation, then it should be part of your normal flow. You shouldn't use try/throw/catch to wrangle it, as that is basically for unexpected, or expected but unfixable, error conditions.

I'm going to kill someone.
We have a data product that is a few gig in size and is downloaded by about a dozen customers. Once a month we run a job to update the data that takes a few hours to run. Then the file is pushed out to IT who puts in on an FTP server.

We have multiple large RDS instances and EC2 boxes that run 24/7 to support this costing us over $2500 a month. A $130 a month m4 large could run the file in less than a day and be shut down for the other 29 days of the month.

blacklab, I guess the other thing you can do is just try to get a proxy of the usage that you actually have. If you have monitoring tools you should be able to get a decent sense of what you're running. The cost is going to flow from usage anyway.

I have no idea if that's something you have or can get - but at least it doesn't involve working with finance (unless you need to pay for it!).

LOL, they don't let us have that.

My group does proofs of concepts and one off tests/projects for other groups. We are constantly trying out different things to see if we can do it faster/better/stronger. Some days we install a dozen different open source packages/projects in a day to see if they would work for us.

We had a pretty funny meeting with IT after I started. AWS was apparently the wild wild west before I got here and they were reeling it in. Our permissions were revoked and we got told we had to open a ticket and they would build the instance for us. I ask how long does it take to get a ticket answered and they said 2-4 weeks. I tell them that's unacceptable and I need to spin up and spin down things all the time, I can't wait 4 weeks to run a 15 minute test.

So we end up having a meeting with them and they go through this whole hoopla about how their getting tickets answered faster and what not. I ask if I'll have root access to the boxes and they say no, whatever software you need installed you'll have to open a ticket. I ask if I have to open a separate ticket for each individual thing I want installed and they say yes.

Restart apache? ticket
Open a port? that's a ticket

So I say ok, so the current web server box we have has apache, mysql, php, about 30 php modules, perl, about 30 perl modules, java, tomcat, node, npm, dozens of npm modules and about 20 other things.

I am going to have to open 100 tickets just to get a web server up? They answer yep.
Install something with npm? open a ticket.

I'm like cool, stand by for some tickets to be opened. I'm going to open a couple of hundred right after this meeting and a few dozen every day.

Finally the head guy asks my background. I tell him I bought my first web server in 1999 and have been running web servers ever since. He hems and haws and says well maybe since it's just POC's and your background we can make a special exception for you.

So TL;DR, no they don't let us do **** like that.

Best argument for containers.

And lol so much for the cloud democratizing OPS.

What's my background? **** you, that's my background

GET /****ed HTTP/1.1

How do you guys manage your versions and deployments? I like to keep master the most up-to-date branch, that way any other dev working on a distant feature can fetch master and branch/rebase off of it without asking (and avoid merge conflicts). But I know others who create separate branches for versions they intend to deploy.


Pros/Cons seem to be:

Master most up-to-date - It's obvious where the most up-to-date code is, but if you need to deploy an immediate fix/feature, you have to branch off of the last tag, which may be dozens of commits behind (awkward, feel like I'm in no one's land), then merge into master later.

Version branch most up-to-date: More transparency and organization around deploying immediate intermediate versions. Master is not the most bleeding edge, rather it's the latest deployed code.


Maybe it's a function of release frequency. If you're super agile, you keep master up-to-date, but if you spend weeks/months on a release, perhaps the 2nd option is better.


Btw, this article has nice pictures, but I'm not 100% converted:
https://nvie.com/posts/a-successful-...anching-model/

.