Heh, I meant to reply to this.

I agree, in part, but I disagree in many other parts, but this may be because I use a lot of SQL, so I keenly aware of the trade-offs between speed and readability.

For me, this

is much more readable than

and in fact, any intro to databases class will teach you the first one. But, the second version is way faster than the first, and I'm not talking about a small distance. Where the first one may take 5 minutes, the second one will take under 5 seconds on the same data set.

How about this one?

It certainly isn't the way I think, and omg(!) look at that "order by" clause. Can't I do better? As far as I know, not really. I could do many other queries that reflect how I think, but the vast majority of them would perform worse.

This is, in my opinion, the danger of using a bunch of functional programming ideas. Since functional is so opaque, the danger lies in not really "seeing" what is happening underneath, and that could cause code that doesn't really reflect your actual thought.

Take the Clojure code I posted. Since you don't know the language, you wouldn't know that each (vec n) is a function, and served as a somewhat abstraction to (into [] n). You can't mutate in FP, so those (vec)'s are not coercions. But here is something you wouldn't know at all: while (vec) and (into []) resolve into the same thing, (vec) isn't as fast as (into []). (vec) just looks and mind-maps to thinking better, but in reality, it is the worse option and I probably shouldn't be using it in an n*n matrix equation.

The #() is another function. So, to build the matrix, I used a ton of functions, 3 that you may not have seen, and one for loop. When I look at that code, I expand all of that out in my brain, so I "see" all of the functions and 2 loops.

Honestly, I hate it. What's up with the (for x (range n) ...) and then that is the last we see of x?

I'm on a programming break, so I have free license to write bad code.

dave, i dont think that long SQL example was needed to make the point that sometimes speed matters, and when it does you might have to sacrifice clarity and maintainability. i'd never argue against that either. but your default position should be prioritize readability over speed until your code proves too slow. pre-optimization root of all evil, etc.

I haven't taken the MIT class, but just took a peek at some of the lectures and it looks pretty intense, with a lot of mathematical proofs and analysis.

I would say the Roughgarden MOOC is a lite version of MIT's course. He said that in his real Stanford class there are a lot of proofs and little coding. I don't think Sedgewick's course is directly comparable, since it's much less analytical, but has a bunch of useful practical advice and covers more ground. If I was starting out I would take Roughgarden first, then Sedgewick.

I agree with you somewhat, but I'm on the fence about it in many instances. I think that the abstractions offered by many FP ideas, especially when baked into a normally non-FP language, often dissolves into "look ma! no loops" swagger and ultimately doesn't fit context, maintainability, or readability, to be honest. Just because you can read the nested maps better than nested loops doesn't mean you aren't doing n*n operations, and I often suspect that people don't realize this when they show off functional in language X.

As for SQL: it's a funny place where the readable answer is never the correct answer. Certainly merits the point, I think.

I've been doing a bit of light reading this week. The book is "Gamification by Design." About 50% of the way through, and it is virtually a play-by-play of every strategy used by Stack Exchange.

Anyways, the book discusses some app called Foursquare. I never heard of it, but apparently you win random badges when you visit various local dives. I guess that, when you visit one place, you get this badge of (dis)honor:



I'll admit it: I look much better than that guy.

the point you should take from that is its evidence of poor language implementation. you could write your own SQL interpreter which took the SQL that you find more readable and converted it to the SQL that executes fast. Or the vendors could just fix it, assuming there aren't other reasons for keeping the slow version.

Your conclusion, when confronted with a conflict between speed and readability, seems to be: "Readability isn't everything. I care about speed."

My conclusion is: "The language is broken. Fix it so you have both."

I don't know enough about the innards of SQL or why the standard is that way. I find most of it pretty logical in its own way.

Obviously I'm not talented enough to build my own SQL. Sure, I could write PL/pgSQL and views, if that is what you meant. Otherwise, I prefer using DSLs.

I didn't mean it literally. It's just a different attitude that might influence your priorities differently is all.

Fair enough. I just didn't understand the comment about the nested loops. I should trust you come from the appropriate place.

Option B sounds pretty dangerous. You're obviously not me but when I played poker full time I was basically "obsessed" with the game i.e. my brain was thinking about hands when I was not playing etc.
Was really hard to do anything else on the side.

Also: not sure how good you are at poker obviously but 5-8k/month sounds like a lot in the current environment of games if you dive in from playing part time/not at all.

If it's financially feasible I'd rather go with
C: Quit poker, grind out the job and invest the time you would have played poker in "career building". Identify areas where you think you should get better (often algorithms for self learners for example), pick up an interesting new language and finish a couple of projects and put them on github.
Since you do C# and JS I'm assuming JS is mostly filler. You could invest in getting great at JS i.e. use it as the sole language for a project. Getting better at databases is also something you could consider.
You could pick up one of the trending languages + web framework (Ruby+Rails, Python+Django) but I think the JS route is more fruitful. Alternatively you could pick up a language that will generate some interest in a job interview just because you know it (my guess these days it's Go or Erlang, possibly a Lisp or something functional depending on who you interview at)

----
dave: pretty shocked you haven't heard of foursquare it is/was one of the startup darlings. Quite a bit of buzz. I mostly make fun of my friends who use it for providing free data about their whereabouts.

Dave, in your SQL examples isn't it up to the database platform to turn your readable but less efficient code into the more efficient path?

I remember some article on HN the other week about this where a guy compared a query on postgres, mysql, mssql and oracle. Oracle generated the best query plan given the current query which resulted in it completely destroying postgres and others in performance.

I think the moral of the story was you could get postgres to execute it as just as fast but it required a more complicated query to get the same plan.

I'm not really sure if functional vs non-functional language styles even play a performance role in web development. Maybe in the 0.00000000001% case?

For example if you decide to use map vs a for loop in a language that supports both then maybe the map version will only do 20 million iterations per second while the for loop can do 50 million iterations per second but does that matter when as soon as you introduce IO it drops to 300 iterations per second for both?

Or if you're only drawing 50 elements on the screen, does it really matter that one of them completes 2 nanoseconds faster than the other?

I agree with your conclusion completely.

This and some of the earlier posts about processor architectures has me doing a fair amount of thinking about processor architectures of the future that will facilitate higher level, more readable code and run it even faster (with less power drain while we're at it).

Honestly, I don't find the stuff interesting enough to care. I've heard of the super hits like Facebook, Twitter (I have accounts on neither), and Angry Birds, obviously. I've heard something about that Tinder app, SnapChat, and that other one that FB just bought for 16b (names slips my mind?) and Flappy Bird, but I've never bothered to use them. I don't find the concepts appealing or "disruptive", tbh.

Wanna guess how many social and games apps I have on my phone?

Shoe, I hope someone with more knowledge answers your questions here, but this is my answer:

Yes, I think it should, but it is hard to fault the implementation in all cases. Some (most) database schemas are far from normalized, so it wouldn't be possible to optimize for all queries on all data sets.

I think that PostgreSQL does a pretty decent job of optimizing queries overall, though. I have quite a few not-so-good queries that run fine. A classical example is using cartesian products where using inner joins would clearly be superior.

As far as I've been able to measure, these two queries run in about the same time:

The first one uses (visually at least) a Cartesian product, which is basically like running two for loops, but the query optimizes it as an inner join, which is similar to running one loop.

Oddly, many beginners are told to use this one:

It does resolve an output just fine and gets incredible speed gains compared to not using the "distinct" clause, but oh my....

This wouldn't surprise me one iota for many reasons. The most obvious is that Oracle is built by a large corporation and has been around longer than PostgreSQL. PostgreSQL is, of course, well-built, but it is open source and built on passion. Just can't compete with the compilers on a project like this.

The next reason is that PostgreSQL works in the opposite fashion as Oracle. In PostgreSQL, you have to consider the proper hardware and software foundation, and thus it takes some dirty work to get the full potential if you are aiming to use micro benchmarks. Oracle compiles down to a Virtual Machine, which of course, is optimized to work with Oracle. Without information on how the machines are set up, these benchmarks don't mean much. This of course goes for MySQL and any other database that doesn't compile to a VM. This doesn't exactly disagree with the query optimization problem, but it is something to consider when looking at benchmarks.

The final caveat is that Oracle also demands optimizations, and they are often at odds with what you would do with other databases, so it can be cherry-picking queries to make a point (not making an accusation, but it sounds a bit ingenuous to me to extend one case to all cases). I can just as well pick a query that blows up an Oracle database to prove PostgreSQL is better at optimizing queries. A good book that explores various optimizations on different databases is Refactoring SQL Applications. The first thing the author write is "don't take the benchmarks as proof that one database is better than another." The takeaway is that there are too many variables to consider.

I guess I sort of answered this one above.

I fear that too many developers take this attitude and take it some odd point of no return. I have 30+ mbp internet, and many sites take forever to load and often cause my fan to start spinning. I run various programs to prevent resource downloads and even turn off site-defined fonts to speed up my browsing. I can't imagine the hell people on 5mbp are living with these days.

So, imagine how many times they said "meh" and now they have a crap site that is barely usable. Imagine the time it would take them to reverse-engineer a site to be in the top 80% speed once they are sitting at sub-50%?

As for writing an algorithm for some NxN matrix, I don't think you should eschew using faster algorithms. A major point, in my interpretation, is to assume that N can be extremely large.

I hate when new people move into my apt building and take the same channel as my router and I start getting negative pings. GTFO my channel br0s

Dave, I wouldn't know what others do. I only take it to the point I described. I still pay huge attention to latency and the perceived load time of sites I create but now instead prematurely optimizing things that don't make sense to do I just look at the final results and take care of the low hanging fruit.

I mean, I'm using rails which is supposedly the slowest thing in the world but the sites I create usually score low 90s in those google/yahoo speed tests and routinely get 40-60ms response times in chrome's network tab while serving dynamic content from an ec2 micro instace that I ping 20ms to.

So another db modeling question related to a for fun project.


Theres a TEAM, which has PROJECTs, which has things like CONTACTs.

The guy Im working with likes to cram team_id into pretty much every sub node, so like CONTACT will have a project_id AND a team_id. Im wondering if this is a positive, a negative, or a big who cares? Obviously you can query the project it belongs to, then query which team that project belongs to, so it feels like were giving it a weird 2 parent setup, but I really dont know the pros or cons.

so basically, he wants to **** up your database structure because he's not good at writing join queries?

also, it seems like teams, projects, and contacts can vary independently. a contact, in particular, could belong to multiple projects. so with projects and contacts, at least, there should be a many to many relationship and a bindery table to implement it. that *may* not actual be the case for you guys, but that seems like the most natural model to me.

I think the issue he/Im having since we're NoSql newbs is that if you are a user, you should only be able to see contacts that belong to a project, which belongs to your team. IE you cant see other teams projects. So you find what team the current user belongs to, then you need to find what team the contact belongs to, so I assume he figures its easier to just have that team_id sitting there in the contact object. I admit, when we get like 5 children deep, its getting a bit confusing/annoying finding the team.

you didn't mention it was nosql before... my response was assuming a traditional relational db. i've not used nosql much so i can't speak to that.

separately from that, your explanation above is confusing to me. users belong to teams but contacts also belong to teams? the "team" concept implies to me a group within a business organization. "contact" implies people outside the businesses, ie, clients or potential sales leads, etc. you should probably explain what the business is generally and what the teams do and who these contacts are. it's hard to give advice otherwise.

Yeah I don't know either. I'm seeing this in rails lingo so far:

Project has_many contacts
Contact belongs_to project

Project has_many teams
Team belongs_to project

But if he wants to answer questions like "show me all contacts who belong to xyz team" then I think a `through` relationship should exist so he can reach through the contact's project to pull out the team.

I really do need to work on being more clear. (And naming conventions!)

A team is the main object, Id prefer it be called Company but thats neither here nor there. Actually **** it, Im going to change it to Company right now.

Its contact management for a very specific media company. The company is the main object, which has users who work for the company. Users create projects. When they do, the project belongs to the Company that user belongs to. Project is essentially work to be done with another media company. The project name would generally speaking be the name of the company you are working with. The contacts are people inside of that "project".

Contacts belong to a project, 1 project to many contacts.
1 Company/Team owns many projects.
1 Company/Team has many users.

But yeah sorry for not clarifying about NoSql. I dont sleep much and thought I mentioned this recently but it was actually January lol. All of my experience is with MySql and Oracle so this would be much easier there, but Im trying to expand my horizons so for some reason I picked a language Id never used and a DB id never used.

This through thing sounds interesting, thats pretty much exactly what im looking for. Ive never heard of that term though, can you elaborate? Right now Im using parent references and traversing the parents till I get to the Company, but this is gross.

Materialized Paths look pretty cool. Think Im gonna mess with that next. This stuff is crazy!

Email I just sent to all leads and management in our department. Should be interesting to see if a big company can really change it's stripes. I tend to doubt it.

Once again, I don't take much stock in benchmarks. I guess I'll pull framework shootout that pops up on HN every few months: http://www.techempower.com/benchmarks/

Now, I can't talk much about all of these frameworks, but I can talk about the Clojure ones. There are 4 different Clojure "frameworks" that are discussed here: Jetty, http-kit, Compojure, and Luminus (if there are others, forgive me). First issue at hand is that none of these are frameworks. Jetty and http-kit are nothing more than servlets. Basically, all they do is compile down into a .jar file and serve your resources to :3000 or whatever you tell it to. Jetty and http-kit can be used with any java-based system, including Java, JRuby, Jython, Scala, etc.

Next, Compojure is sort of a framework, but only in the nominal sense that you are given the ability to create routes and few extra tools to work with. The real framework here is Ring. Luminus is nothing close to a framework. It is simply a way to layout out your files. Finally, no sane person deploys Compojure (and by extension: Luminus) raw: it is explicitly stated on Compojure not to serve up raw .clj files and it explains why, thus, you wind back to getting utterly worthless and meaningless benchmarks since you have to compile to some other servlet (Jetty, http-kit, which once again are not frameworks) to get something dependable.

I'm not sure if RoR has some intermediate step or if you deploy it directly. The big issue is that there are a million variables, and I wouldn't know how using RoR -vs- SomethingElse makes much of a difference. You have to consider the server, resource size, image sizes, javascript and css frameworks, and of course, database optimization and code -> db -> code latency. I suppose one small improvement would be gained if you built the same site with mostly Ruby code and compared the same site to nothing but gem installs, but that is something you would know more about.

The point is that I don't believe that RoR is inherently "slow." I imagine you still enjoy minimal and clean code, thus your work would be much easier to optimize and is likely fundamentally clean and optimized from the start. Granted, there are going to be some issues, but unlikely a spaghetti blob of junk and complex dependencies. I suspect though, that if you were attempting to "speed up" RoR, you'd be well using some JRuby and compiling down to a java servlet, but the very last place I'd look is RoR itself.