I still think the OS X user interface is bad. Finder sorting still tilts me and most things aren't intuitive. I've counted I think 3 different ways of what you have to do to actually install a program once it's opened in the installer thingy (drag to Programs folder, double click and something I don't remember...some sort o click through).
The dialogues are also horrible quite often. I've seen the equivalent of "Do you want to cancel? Ok, Cancel" a couple of times.

UX may be slightly better than UI because ZOMG pretty, good battery life etc. But the UI is bad.

The differences between mac and windows ui/ux really aren't that different. Yes, things are in different places and it takes a bit to get used to if you switch from one to the other, but the whole "mac ui > Windows ui" trope started so long ago (back at a time when they were really different) and has become so often repeated that it's just accepted as fact.

Pretty sure the Linux partition tool lets you make a virtual partition that you can separate into as many partitions as you need.

@clown from 20372
There some GB/TB mix ups in your post?

I'm not up to date on the whole history of the i5 but I'd be surprised if the air doesn't beat your laptop in standard benchmarks.

Measuring just the basic specs like clock speed, amount of memory, and amount of disk isn't really enough.

Clown, use S3.

Edit: LK I'm assuming his external HD is 4TB.

Eh yeah 4 TB HD of course heh

Edit: Can't use any external provider for now...like I said bureaucracy. Will take 2-4 weeks to get approval for anything that costs money (or is in our IT department) and I don't feel like paying for it.

I researched something in the cloud for this a while back and it was pretty expensive but maybe I completely miscalculated. Like I said I'm creating 20GB/day and I think Amazon stuff came out fairly expensive (+the script runs 24/7)

S3 is almost certainly cheaper than a home grown solution, assuming you're paid even remotely fairly.

I guess your solution is reasonable if the only approved place to store the data is basically an external usb drive. But I'd be figuring something else out pretty quickly. Because this solution seems heavy on manpower and likely to lose/drop some amount of data.

Edit: It's not clear to me what you're doing with the data. You're writing to a local mongo database and then copying to a mongo database in the external drive?

Seems easier to just store the data in a file locally - maybe partitioned by hour and then have a second job that moves/loads those files to the external disk on a regular basis.

What do you do with your external hard drives? If you want to query a years worth of data do you have to plug each one in separately?

Don't use cloud for this imo.

Getting a basic root server w/ 2x3TB HDD and more than enough included traffic (10TB/m iirc) is <40€/month without setup fees:
https://robot.your-server.de/order/market

So this means there isn't an upgrade path I guess.

I don't know - but I doubt it.

This is the kind of stuff that ends up being penny wise and pound foolish.

You end up with a machine that needs to be set up and maintained (monitoring, software upgrades, fixing failures, etc). Documentation needs to be written or else the first time clown goes on vacation (or changes jobs) there's going to be a failure and nobody knows anything about it. The average programmer's cost per hour is such that this approach ends up costing way more than just using a cloud storage solution.

Someone was saying earlier about how there's an X as a service for everything now a days. And sure there are some silly examples but in lots of cases it really is useful to pay a subscription fee for a specific service than it is to pay someone to build and maintain a home grown solution.

Maybe I got the wrong impression, i thought this was basically a relatively simple job with negligible maintenance where you setup a machine once and move the data every other month. Plus costs appeared to be a big factor.

He mentioned he already has a script for it anyway? In that case I see little upside of cloud vs root server. You'll basically never need to scale to >1 machine for this task and setting up the root server takes like 15 minutes if he is familiar with linux (which he is iirc).

[For some reason i assumed clown is working at a university when writing the reply, maybe I just remembered that wrong though. If this is in some corporate environment it's a different story i guess.]

15 minutes to set up and no maintenance is the ideal and really unlikely to actually happen.

And that still leaves you with a random machine running that nobody knows about but you and that you need to manually check on occasionally to make sure things are ok.

I completely agree with you if there is an existing service and costs are not a huge issue.

But cloud vs root server doesn't make much sense to me, what's the upside of running this on e.g. amazon? (Setting up monitoring w/ email/text notifications takes a few minutes in the hetzner web interface, if that's a factor.)

My favorite version of windows was 2000 Professional.

I'm saying cost favours Amazon. And I don't think it's even particularly close.

The number of times I've been told a server can be set up in minutes is probably in the hundreds. The number of times said server arrived in minutes can be counted on one hand.

Edit: At least for an initial setup of a machine. I can get chef-ified servers in minutes.

Just to avoid confusion: By <15min setup i meant setting up the OS etc after the machine has been delivered, but getting the machine usually takes 12-24h.

I haven't tried EC2 in a few years, but iirc it was ridiculously expensive for high CPU/memory tasks compared to using a normal server. Main upside is scalability, but this doesn't really matter in this scenario.

I don't think he needs an ec2 server (although I'm probably in favour of that too instead of using the machine he has now).

I thought we were just comparing data storage solutions.

Edit: And yes, I'm talking set up time, not waiting for hardware to arrive or be made available to you.

Oh. I assumed he wanted to completely replace the current setup, not just storage.

This didn't give the impression that he was talking about anything close to ~40€/m:
Not much point talking about this further before clown chimes in, lol.

Edit: I've setup like 50 servers with hetzner, the standard ones come pre-configured with a selected OS and can be re-installed via web interface in a few clicks. For the marketplace servers i linked you may need to select an installation image via recovery console. Either way, they have things automated very well and 15min is actually a high estimate if you don't need anything else installed. Maybe a bit longer if you aren't used to their web interface though.

open source self hosted Slack-compatible alternative:

http://www.mattermost.org/
https://github.com/mattermost/platform

no longer have to worry about your company secrets being in the hands of a 3rd party

No you don't remember wrong. I work at a university in a research project and thus all the needless hassle. I know there's better options just checking to make sure my hacked together option which allows me to start collecting data now doesn't have a massive flaw (since I've never collected data en masse)

Right now I'm just interested in a quick sanity check (just in case there's something obvious I'm missing like ARE YOU CRAZY YOU CAN'T EASILY MERGE MONGO COLLECTIONS).

Long term I'm def. looking for a better/robust solution though. Using the Amazon calculator I arrived at roughly 65$/month (Frankfurt, 1xt1.micro, 100% usage, 550GB/month general purpose SSD). About 40$ with magnetic storage.
If I also want to transfer the data out it's 75$...all those options are too much for me right now. But I guess it's good to know that you can log and backup the public twitter stream for ~900$/year. 8TB local backup storage is another ~300$ (2x4TB disks or something).
http://calculator.s3.amazonaws.com/index.html

If anyone happens to know the rough prices of (ex)Gnip or similar services (access to Twitter data as a service) that would also be helpful.

I'm still not really clear how this data is being used - because that has a pretty big impact on what you should do. If you're just archiving it then I don't think mongo is particularly useful. Just save and compress the data. If you need to actually query it regularly than the usb drive solution seems really onerous.

I'm kind of surprised someone isn't archiving the Twitter stream already. The other thing to consider is how bad it will be if you either drop some data or lose a chunk of it. Because your current solution isn't very robust.

Losing some data isn't a big issue (even entire days wouldn't be that bad). It's basically a first exploration. We want to calculate some pretty trivial stuff like "what percentage of tweets was in German (roughly)", how many tweets/week contain keyword X. We have a list of keywords/keyword combinations for which we'll analyze all tweets that match manually. I did a quick check with the data already collected and we're talking about keywords that only match ~25 tweets/day.
Mongo queries are already pretty slow (couple of minutes for a count() with one keyword filtered over a 40GB dataset)

We're not trying to predict anything or do any fancy statistical analysis (I have already checked and can play with the data in R if I want to). Queries can take a while, too. It's not critical if say getting all tweets that match X from the DB would take a couple of days.

An example would be having a human wade through all German tweets for a time period that mention Merkel and manually rate them on a scale (max negative to max positive). Then we'll compare that with some sentiment algorithm.
Small stuff like that.

The main thing that we eventually want to do with the data is get a rough estimate of how worried people are about certain things. We have a standard questionnaire that is currently being developed for this and want to see if we can somehow mimic a mass poll with some sort of twitter data analysis (basically a passive, constant questioning of the twitter hivemind of sorts). We'll know in advance what we want to query about (basically experts construct the query and manually wade through the tweets as a first step)

It's being done, Twitter bought one of the companies that did it. They can also get more than the public 1%. You can buy access to the data, prices unknown/"ask us" but pretty high. If we can build anything useful with the data we collect ourselves we might eventually get a quote from one of these companies. But it's very possible that our ideas are dumb in the first place :P

This is a pretty stupid question, but how far away is speech recognition technology away from being able to tell if someone is slurring (and is drunk or having a stroke or something).

Upgrading CPU yourself will void the warranty.

You can get the Apple store to upgrade RAM after purchase as long as you bought the RAM from an Apple store (including online store). That's about the only non-voiding post-purchase upgrade option I am aware of.

There may be other options, but pretty certain CPU is not one of them.