Hehe, yeah that sounds very similar (I think these might have had a full keyboard to let you program assembler rather than machine code but it's so long ago I can't be sure).

I do remember one of the guys who taught us was an ex telephone engineer and he decided to make his own serial port for them: was a huge bird's nest of wires around a serial chip (RS232?) connected via these big "physics style" wires to the parallel port of the marc board and amazingly it actually worked!

Juk

http://rubyonrails.org/applications

I'm not sure how you can say RoR hasn't led to anything (especially to startups). Maybe I'm missing the point.

I think teaching lower level abstractions is often overrated - especially for people focusing on high level cs theory. If you come across the rare situations where you need to know about it in your job - you can always learn it then. Just like hundreds of other things.

This. It allows tons of people to realize ideas. If your idea of innovation means a breakthrough in CS theory or something, then ok. But if it means innovation in the larger sense of human endeavor, obviously these abstractions are incredibly valuable.

I see both points. I'm not arguing against the idea that building on abstractions doesn't allow progress in business (websites), marketing and statistics (R, Python, etc), AI (Lisp, Octave, Python), communication (blogs). I'm just saying that problem that can't be solved with the abstractions cannot be solved without going under the abstractions. I simply disagree with the idea that more "stuff" is equivalent to progress.

I may by under the incorrect assumption that the problems solved by facebook, twitter, google, the video game industry, cellular phones, et.al., were not able to solved by the available abstractions, and thus progress was caused by a need that could not be solved by abstractions. The wider adoption of these ideas don't really count as progress in my opinion, but rather reuse. Of course, this reuse led to modifications and improvements which is a form or progress, and this is the differentiating factor for me: has prior knowledge been built upon to create a truly distinct idea or product? I don't think that plugging in code in lieu of anything new is progress, even if the business idea itself is unique.

I also disagree with the notion that learning lower-level stuff is over-rated. Personally, I wouldn't be able to understand many things I work on now without the simplistic view of lower-level coding to refer to and expand upon. As I pointed out before, I think it's easy to forget what its like to be a beginner at this stuff and not realize how much of that knowledge sort of seeps into the higher-level stuff and how strange even the most basic questions are. Lower-level coding also greatly simplifies the view of the code you are learning to create and build upon, and without lower-level code and how to create abstractions on it, how can one learn what an abstraction is, and by extension, the reasoning for using one in the first place? I mean, you pick on Java for having a bloated "hello world!" but isn't that program really a culmination of abstractions?

Funny to note: I know which company made a few of these listed websites. That dev house moved over to Django, which I'm sure they had full confidence that whatever could be done in RoR could be done in Django.

How do you measure progress then? I view the main purpose of software development to be building "stuff". And some of that "stuff" has changed the world in a very profound way.

This is back to the previous argument. Your argument might as well be that we don't need anything but machine code since every abstraction above that isn't really necessary. But the abstractions make things easier/cheaper to build - which make the amount of "stuff" that can be built much higher. Compare the investment cost of a start-up that builds a product with a team of developers writing machine code on their own machines with a start-up that builds a RoR site hosted on Heroku.


Did you go down to the assembly level? digital circuits? Transistors? Are all of these things worth learning? I also don't really agree that most abstractions are a more complex view of their lower level.

No. Compare Java and Python. Two different abstractions - one of which still makes it really easy for people to learn.

Anyway, I don't think its worthless to learn about lower level stuff (I don't think its worthless to learn almost anything). I just don't buy that its really important or necessary for someone to be a strong programmer.

The more abstractions the larger the set of people who can be productive with the language. IE it's easier to learn. The larger that set is the more likely it is that something cool is going to be built with it. I disagree that low level code provides a simplified view of your project. By definition low level code provides more detail than is needed and because of the added, unnecessary complexity the number of people who can be productive with it is diminished.

If that is where you want to spend your time no one is saying you shouldn't, but thinking that is the only way to be able to code a business application is a bit elitist. For every NASA app that requires hand tuned assembly there are a thousand apps that don't.

One more point about abstractions - CS is a very Darwinian world. If an abstraction (be it a language, framework, library, whatever...) isn't making it easier for some group of people - it'll die (or live a small little life as a cool/intriguing/academic thing that people pull out in forums). There's a reason nobody is using Brain**** to write applications.

On a related note - what my company is trying to do is to basically build a new abstraction layer for engineers to use when working with big data. It's very cool (imho) but if we don't actually make things easier for people - we'll go out of business. In fact one of the most important pieces of feedback that we try to get is when do people stop using our abstraction layer and go down a level. If they're skipping our product and going down - it means we're missing something important in our abstraction.

wikipedia is asking for donations, help out

Let's not and say we did.

Is your company similar to karmasphere?

Very much needed. I can't seem to get Apache Whirr to configure my Hadoop clusters. Abstraction please.

Most programming is about building up the basic blocks of your language in to higher level abstractions that are easier to work with. You use statements to build functions to build classes, and put together classes in packages (in some cases) or higher level groups (eg, "Models", "Controllers") in order to actually build the thing that you're trying to build.

The only difference between a "low level" and a "high level" language is about where you start building your own abstractions, and which ones come prepackaged. It's turtles all the way down.

This was pretty much my point. Your team saw a serious issue with the status quo, so you decided to roll back the abstractions to improve the process. I assume you are creating a new item from the (sort of) ground up because adding an extra layer to the tools out there wouldn't do anything useful. I'm saying that what your team is doing is progress.

I think of it the same as an engineer. Someone can use black-box abstractions to create the diagram for a bridge, but most certainly would prefer it if said engineer knew about statics. Even if said engineer never wrote a differential again, it still is nice to know that my life is in the hands of people who understand the *why* of the math and when to use said math. When geometry isn't enough, I would hope they use the correct formulas when they plan and design a bridge and don't decide to take away a few poles because it will look more beautiful. I understand that programming isn't (usually) a life and death situation, but still. People's lives depend on programs functioning correctly more than ever before, yet people are fudging things up left and right via bandwagon on the latest _____ craze.

Something more mundane: If people didn't break abstractions, we'd all still be styling in '57 Chevy's with no internal computers, large gas-guzzling engines, etc, or, if they didn't break abstractions, our computer chips would be the size of the Empire State Building.

It's interesting that you bring up circuits. No, I have not exactly dived into circuits, but I have had the (dis)pleasure of writing circuit programs and I'll argue that there are tons of lessons that are learned from this exercise that are low-level representations that seep well-into high-level languages. Here, from 3 basic circuits, you can represent so many things! I'll try to do the following from memory, so forgive mistakes:

You start with 4 items: And Gate, Or Gate, Inverter, Wire.

Then with a few basic concepts:

And Gate takes two wires, and one output and set up the following truth:

If input A == input B, return the value of A, otherwise, return 0

Or Gate takes two wires and generates one output.

If input A == input B, return the value of A, otherwise, return 1

Inverter takes one wire and generates one output.

Take the input A and return the opposite of A.

Very basic so far.

Now add an Inverter to the output of Or Gate to create Nor Gate. Add and inverter to And Gate and create a Nand Gate.

Now combine the output of an Or Gate with an inverter on the carry and create a Half Adder. Combine two Half Adders to create a Full Adder. Then with just 4 primitives, you can represent many things and even combine these to create adders of (theoretically) infinite size. This combination of primitives -> abstractions is a very good exercise, wouldn't you agree?

Then how to move all that chuff to higer-level stuff?

Think about the decisions inherent to the program: do you need to track the state of the wires? If so, how? If not, do you ever have to specify the wires themselves? If you only need to see the output, then why bother specifying a wire at all?

Moving along, we see there are multiple ways to represent such items:

Full Adder -> Full Adder -> Inverter
Half Adder -> Full Adder -> Inverter
Inverter -> Half Adder -> Full Adder

And alas, we see a pattern here. Can the system, if represented some other way be abstracted? What about:

Multiply -> Append -> Sum All
Divide -> Map -> Multiply All
Divide by PI -> Append -> Print All

And there is yet another common pattern that can be abstracted into a generalized function. This goes beyond the idea of if you find yourself copy-pasting, it's time to refactor. This says that even if you are using the same patterns over and over again, even if your program follows DRY to a T, it's time to refactor.

How would anyone be able to see these patterns unless there was a very simplistic, non-abstract, and low-level model to begin with? I know the above takes more preponderance than (--install) or slopping an answer into map(), but this knowledge is stuff I think you are taking for granted that is wholly unfamiliar to people who "abstraction" is a foreign term.

With all that said, I'm not saying abstractions are inherently bad, as I most certainly don't agree with this, but I think there is something to be said about at least being able to visualize the abstraction layers you are building on.

I think that your assumption is probably partially wrong to begin with - in particular, I'd say that facebook/google/video games were probably examples of incremental improvements on the current accepted abstraction level at the time. To pick a subject that I have personal knowledge about, I really don't think Google's search algorithm was necessarily a great step forward at the time. It was probably better than Alta Vista and Pagerank was likely an improvement over the algorithms in use at that time, but the big innovation Google had was building cheap clusters with unreliable commodity off the shelf components and using those to do significantly more processing than other search engines at the time could do. (Google probably still has a significant infrastructure advantage over basically everyone outside of NSA-level organizations, with respect to being able to throw a ton of machines at a problem).

Beyond that, I just think there's more value to incremental improvements than you think. Not every project can be cast as a huge paradigm shift - I'm generally impressed with small improvements that somehow end up having massive effects (e.g. Facebook crushing Myspace and Livejournal despite not having really significant technical improvements over either).

dave i honestly have no idea what point you are trying to make anymore.

i do think it's funny how 70% of all discussions on this forum devolve into an argument about the importance of knowing extremely low-level concepts.

Yes, incremental changes until someone decides they had enough of it. You have to admit that the span under 10 years is pretty quick though. I mean, when you consider the discovery and slow adoption of things like the number zero.

I do agree that tipping points are interesting to ponder.

Must have went off the deep end.

I think the discussions aren't really about low-level vs high-level, though that has been the representation of it. I read it as frustrations at how things currently function. Why should certain frameworks force you into corners, or why aren't CMS's as easy as they are advertised, or why should someone have to download a 5 gig file when all they want is a drop-down list, and where does bad code come from, and how would I, or anyone, ensure that bad logic, code, etc, doesn't describe me or anyone, or why do people still use Dreamweaver to abstract the difficulty of table-layouts?

I'm glad I wasn't the only one.

Not really. We're more focused on people that have your problem. As in make it so that you don't really need to worry about your hadoop cluster at all to get started with processing your data. PM if you want more info. That goes for others too, I just don't want to post my company name publicly on 2+2 since it means google could be used pretty easily to find me and my 2+2 account. I like the freedom of being able to post whatever I want without worrying about how it looks to people that don't know me.

If you're just playing around with Hadoop/Whirr for fun feel free to post your problem since I've used Whirr a bit and might be able to help.

We use Hadoop and I'm not really sure what it is, can you give a short description?

The short version is that we offer Hadoop as a service. Sort of like Elastic MapReduce but more user friendly for developers and more focused on the whole project life cycle (from development to production). Right now we support projects based on apache pig and the pig udf languages (mostly python).

Dave, I gave your post another shot and saw this (missed it before):

That's where you're 100% wrong (both in what we're doing and the general case).

Maybe dave has a point. The top topic on HN right now is "Why is processing a sorted array faster than an unsorted array?" and unless I'm somehow underthinking it the answer is pretty obvious for anyone that has done any low level work

I'm somewhat scared to click on it.

Kind of makes me wonder...does Intel still ship the printed manuals. I think they stopped doing it and only proved PDFs now. Always tried to get AMD ones but they didn't ship to Europe or something. Those 5 blue books are probably my most beloved books in my collection simply because it was so cool when they arrived

I wish I was able to discuss the intricacies of this, but I understand it is a proprietary solution.

Maybe I could try a correlation. There was two tools that I was considering recently: a dependency migration tool and a SQL DSL.

I passed on the dependency migration tool because I couldn't see how it was any improvement over using raw SQL to create the table schema and triggers.

I decided to use the DSL for two reasons: It looks more like "native code" than raw SQL and it fills a knowledge gap. For example, I do understand how to create joins, but I'm not totally clear on when to choose LEFT JOIN over INNER JOIN, to be stupidly simple. In this case, I don't have to worry about optimizing or testing query speeds. This is an immensely helpful tool, but it doesn't replace actual SQL knowledge.

The question is, then, is your tool a wrapper over Hadoop + stack or is it a redesign of the stack itself? Are you looking more at something similar to creating the DSL described or are you creating something that completely redesigns the workflow? I was under the impression, initially, that you were creating a Hadoop replacement, but I guess I'm wrong there.

I would consider a new stack that makes life easier + tunes the speed of Hadoop progress, but where the crux is that it doesn't sound like your tool can replace general knowledge, which may be the basis of my own confusion (?) on the matter. I wouldn't say you are simply creating more "stuff" because you are attempting to solve a dissatisfaction with the current systems that are available, so in the name of progress, I assumed you are breaking some common workflows and abstractions to do this better.

I saw that thread a long time ago. It's a really cool read. I would have never known the answer to that question, and your guess is correct.

Pretty interesting read off HN about branch predictions, especially how different compilers produce different results

http://stackoverflow.com/q/11227809/399268

MrWooster rides a slow pony.

daveT - I think we'll have to agree to disagree. By your reasoning (I think) you don't think of things like Rails, Heroku, Github, ... progress because they're just wrapping an existing product and making it easier to use. I disagree.

Edit: And you probably wouldn't like what we're doing.