I don't remember actual numbers, but I remember at the AWS conference a guy showing how CloudFront improved download speeds vs downloading directly from s3 by a significant margin (want to say something like 30-40%). I'd imagine you would get even bigger benefits comparing CloudFront to your own server.

I was going to qualify my statement that I haven't done any research on this so I'm sort of talking out of my ass. Which then gave me the impression of what this thread would look like if we all qualified our statements the way lawyers always seem to do.

I wonder if lawyers have some lawyer-only forum where they're allowed to say whatever they want free of any obligation to qualify their statements or proclaim how their opinions/advice aren't actually legal advice.

Thanks, think we'll just pony up and go with Amazon/Rackspace. Quite surprised how much bandwidth still costs nowadays, (they price on BW usage). Was under the assumption it was a lot cheaper nowadays.

Here's my untested theory:

It totally makes sense for css/js because downloading a 100-300kb file is quite fast, but a 200ms delay to travel half way around the world is going to be a big chunk of the total time spent to transfer the file.

On the other hand with a 100mb+ file, I'm not sure how much of a real difference a CDN will make for speed. If I download a file from California or Germany, it's still going 2mbs/s.

Maybe it'll download like 20% faster but at 2mb/s it's already done in 50 seconds or so. If it finishes in 40 seconds or 50 seconds makes little difference.

Basically the more time you spend waiting for the download, the less effect a CDN will have if all you care about is speed. I'm sure cloudfront will be more reliable than your server but it's not like your server is going to be garbage, I'm sure a static file server would have good uptime if it's configured properly?

Fair points.

Is this something you can test? Set up something on Amazon/Rackspace/whatever and do an A/B test on where people download from. I'm not sure what the right metrics are or how easily you can track which people did which downloads, but might be worth thinking about before investing a chunk of change.

I'm fairly certain if a download is served more geographically locally, the download should be faster as there are less weak points/overloaded points between the client and the file. I'm not sure what actual real life benefit this would have for end users in terms of seconds, but measuring it might be a good idea, give me some data to work out if it's worth an extra $1k a month.

Download intiation time is pretty much meaningless, good point.

40s vs 50s download time is pretty important imo. 40s is 20% less than 50s and I'm sure that would result in more installs for volumes of 80k downloads per month.

Edit: Going with dedicated for now, when I get time will test the speeds. Dedicated is 10TB, 1gbps, 64gb ssd for $60 p/m, +$20 p/m for 1TB extra bandwidth

True, but I was thinking more of the point that in a lot(?) of cases it'll be user's individual bandwidth on the last hop into their home that is limiting their download speed.

All our users appear to be pretty evenly dispersed across the entire world, we know a lot of them have poor/slow connections.

20% was just a rough guess, it's probably less in most cases. Unless you have an awful route you can transfer files pretty fast across the US or near by continents.

Try downloading something cross continent. It's quite good. You'll most likely cap your connection as long as the host is reasonable or you have some mutant fast broadband like 50mb/s+.

Hey guys,

I want a program that scrapes information off a website and enters it into excel at regular intervals. Who would I be looking to hire in this situation, and how much would a job like that cost?

Thanks!

So I had to do this mortgage calculator for my Java class for homework. I got it working, however I was really surprised at how much you had to pay monthly to keep up with the interest rate.

Then I thought about student loans. Then I got

How do people pay for schoool????

Slowly.

They get jobs as programmers.

This depends on how complicated the website you're scraping is, but you should be able to get something done for less than $1000. I've paid $300 for something similar.

One issue that you're going to run into is that many websites change a lot. And any one of those changes can break your scraper, so you might have to regularly rehire the guy to make updates.

Python + Beautiful Soup

^^^ yup, I love Beautiful Soup

Social contract aka pwning the taxpayer for it.

Anyone written an http server from scratch? My goal is start with the RFC 2616 spec and implement as much as possible without help/looking at other code. Deciding between Java/Python/C. C++ I use everyday and I know C# well. C would be fun but might be too tedious with text parsing and processing.

At work we are getting upgraded from 24 GB to 96 GB RAM. Management finally succumbed to the age of big data.

If you ever do it and do it in Python, look at the Twisted source code after you're done

Thanks Max. Website in question is Amazon....

Also if anyone is interested in setting this up for me, I'm willing to offer a fair price.

Googling stackoverflow html regex will help a lot

loool parsing a website with regex

You could use this: http://www.kimonolabs.com/

I built something like that in python to crawl trulia for home value data recently. If you can pm some more details about the project I'll be happy to look at it for you