You definitely want to decouple the processing of the music from your backend web server.

Without a ton of thinking about it, I think what you can do is something like this:

* On the front end get the file and some basic metadata (size of file - most importantly)
* Send that metadata to your back end.
* Have your back end verify that its ok to upload.
* Then have your back end construct a pre-signed url for uploading the file (This is what S3 calls it, not sure what's available for other providers). I don't remember all of the details, but you can basically restrict it to only work for a specific file and have it expire after a period of time. Should take care of most of Gull's concerns.
* Once the upload is complete, kick off some processing job that's independent from your web server (ideally), or just throttle-able in some way
* Once processing is done, kick off a notification of some sort or just wait for the user to come back.

There's some work here in maintaining state. You always want to know what stage of that pipeline you're in. Especially needs to handle random failures at any point of the pipeline.

100% use a domain, but you can use a subdomain. EG, your main domain is www.example.com (always use www) which is behind CF. You can create uploads.example.com which points to another environment (or the same environment) and accepts the upload, but is not shielded behind CF.

Difficulty will come if you're using and caching etc later down the line. This was a real pain point for our new website but we've gone with the idea that 200mb will be enough for forseeable future so CF will be fine.

Something you might want to look at is encoding wav to FLAC on the users browser before uploading. Storage costs are tiny so I wouldn't be worried about storing WAV's, but you want to improve upload times for end users.

Thanks, you two. It's good food for thought. Super helpful.

You'd be proud to know I never used a naked domain.

I feel like I shouldn't bother with a second server then. I'll have to do all the things that I was trying not to do by using Heroku, so it probably makes sense to keep everything localized. Would remove a lot of problems.

Regarding the caching, I would probably use nginx or varnish.

I'm not sure what the context of the project is. But there's actually a lot of 'secret sauce' type services from AWS (and I assume probably others) that would make this a lot easier.

For example, you could have the upload to S3 kick off a lambda function that does your processing, that kicks off a notification to you or your web app with the results. All fairly easy to set up and scale easily.

Sort exactly what I said, which roughly translates to I don't really know yet.

Basically, I don't want to maintain a server, though it looks like I may have to.

I don't think speed and super hot caching is a concern for this specific item. The initial upload and retrieval wouldn't​ be that fast, and my intuition thinks later retrievals will happen either a few times for curiosity or not at all. Adding a way to share file links feels a bit pointless. The person is really practicing scales and other small stuff.

Basically, what you describe is what I'm after. Send a file, process, send back XML to the database, serve up the music file.

jj,

which ones have you used personally?

S3 (including pre-signed urls) and the notification services.

Never used lambda, but seemed fitting. The old school version (which I have done) is to send a message to SQS with a separate worker app that pulls work from there. Think of it as your own lambda.

I wasn't really advocating for those specific tools - as much as trying to make the point that there are lots of times where you shouldn't bother re-inventing the wheel and should just use a cloud provider for all of the plumbing. I included those specific tools above to try to make it a bit more concrete.

An application that uploads files, processes them, and returns some results to the user isn't particularly novel. There are lots of tools/services out there that can get you up and running quickly and with a scalable solution. I'm not even particularly convinced the AWS (or other) approach would be more expensive in the short-term.

yeah, agreed. AWS is just one of those things I've only used a tiny bit, but I know it has infinite services so was just curious.

I've used quite a few AWS services over the years. Most of the data/big-data related ones (Redshift, Dynamo, RDS, EMR) and a fair number of the 'plumbing' type services. I haven't touched most of the newer services or those focused more on consumer apps.

In general, they're usually pretty good at what they say they do. They just often don't do something that you would assume it would do. AWS is very good at the MVP. And they have horrendous UI/Usability.

Come on, the one giant dash board to rule them all is simple the 5th time you find the widget you're looking for... I don't understand how S3 can require unique names, I must have read that wrong.

It's just a matter of limiting the # of buckets a customer can have. Lots of unique names possible.

This might actually work.

Those numbers are so large -vs- micro-pennies and my project is definitely not "big." I'm planning to open up beta for 10 people max to start, so at least can get some baselines.

If I did the DO route, it would be minimum $20 / month. I doubt I could hit that on AWS unless I create a miracle of user zombification. Give it a month-long test before I can return with a data point of 1 person.

Anyone used Railgun before? It's insanely, insanely good (if you can get it going, having teething issues!) and is a really clever bit of infrastructure.

Our azure environment is in east US, (with google analytics disabled) can cold load our homepage in ~250ms from London, ~190ms from Dallas. (21 requests, 240kb)

hey friends,

so I had an interview with a company about 2 weeks ago, where at the end they clearly told me that I've past this stage of the process and the next stage will be an online assessment, however its been 2 weeks and I haven't heard anything from them? whats the best way to play this? do I contact them or wait a little longer? <3

I'm waiting for the same thing, I don't think there is any way you can talk yourself into a job in this situation and is a small chance you could talk yourself out of one. Keep looking for a job until you have an offer. Worst case you can contact the first company with the offer from a second company.

just prod them. HR is probably being lazy; guy took a vacation last week and stuff got dropped on the floor.

+1 to prod them. Just be polite and professional.

The tiny chance of talking yourself out of a job is offset by the tiny chance you fell through the crack and you end up getting a job when you wouldn't have without prodding.

yeah, i'll probs email them tomorrow

DrSo after 3 days of lectures and moving what I felt like was slow, they just dropped a homework assignment that torched the class.

As of right now only 5 (out of 21) people have any code on github and only me and 1 other student have working solutions (mine is much better but who's paying attention to that )

It was a word problem to build a program that solved for number of stops between trains, having 3 different lines that shared a common station. Had to figure out how to store the data, and how to accept arguments in the form of numberOfStops(departing line, departing station, arriving line, arriving station).

I'm at the point where once i have the "solution" in my head it takes an hour or two to get it down into working code. Need to improve with things like grunt, writing tests, and debugging immediately.

Just don't be the douchebag who thinks his code is always the best plz

We have enough of those

I will definitely avoid that. I wanna just be a chill programmer, not a douche brogrammer.

This homework is setting up for a contentious day tomorrow.

We have a few people in the class who are really defensive of their lack of knowledge while also being somewhat arrogant. I'm not sure if that is the right way to put it but here are some comments from a pull request that just came in. This dude actually seemed chill from speaking to him at lunch:



I've been improving my code all night and am passing all tests now and _dave_ even helped me optimize a couple lines. If things slow down again tomorrow, I'm gonna try and keep my individual pace going.

Eh, when you first start, why not pat yourself on the back a little bit?

Larry, put this comment at the top of every file you write:

You may not know why today, but you will know in 6 months.

While your at it, don't forget to put reasonable limits on the inputs or some way to time out, preferably between now and the heat death of the universe.

Hopefully I didnt come off like the dbag there...lol

To me that was actually the most fun part of learning/school, was getting crazy **** thrown at you because figuring it out feels so damn good. Plus thats kind of how engineering works anyways. That said its a pretty stressful process so I can totally understand people freaking out/complaining a lot.

Good to see you back dude

+1

The attitude that student has is a very bad sign for that person. I have been 'tutoring' students of an online code school since October and I have noticed a few students who find it absolutely incredulous that a lesson didn't outline everything to the T required to complete a lab. Often times these lessons/labs actually do include links to supplemental reading or videos which actually explain the methods or concepts more in-depth. It's one thing to learn a bit slower and need clarification about concepts, it's another to be douchey about OMG they had to use google and actually learn something.

The best part of classroom learning is showing up everyone else, you can't do it at work, you can't do it amongst your friends, but in a class go ahead and break the curve. It's also funny that someone is taking a book camp and complaining that it's hard.