Open Side Menu Go to the Top
Register
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

07-26-2012 , 06:18 PM
That structure to me just seems really difficult to work with but maybe that's just because I'm conditioned to relational.

I find this bit interesting:

Code:
  votes : 5,
  voters : [ "jane", "joe", "spencer", "phyllis", "li" ],
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 06:19 PM
Quote:
Originally Posted by jjshabado
db.posts.distinct("tags")

Edit: So you'd query the documents and not create a separate collection. If you're expecting a large result set I think you need to use a map-reduce job.
Wouldn't this sort of thing scale horribly?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 06:23 PM
Quote:
Originally Posted by Neko
So if you wanted to query for all Tags, would you have to look through every single blog post to find out what tags are defined? Or do you store a collection of your tags elsewhere?

edit: or are those tag names, keys to retrieve information about a tag (like a foreign key)?
This is where it gets fun. It depends on your use case.

If you think you will need to query tags or comments on their own without the blog content then you'll likely want to separate them into their own collection (sort of like a table).

Now you're trading performance for flexibility because you're running multiple queries instead of 1.

You could also do both at once and depending on what area of your page is being read, you could choose to pick the most efficient one. Now you're trading disk space for performance (reasonable IMO on a very high concurrency page).

You could also start caching stuff at the application level so your multiple queries are just stored in redis and now reading them is insta. This trades performance for code complexity/maintenance/worrying about cache bs.

I think a lot of people really under estimate what a good host is capable of serving. If you step away from $5/month shared hosting, you can actually serve a lot of requests per second.

TBH a blog post is kind of a bad example because really it's highly connected data. I would use a graph DB for a bloggy type of site.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 06:24 PM
Quote:
Originally Posted by Gullanian
Wouldn't this sort of thing scale horribly?
It's web scale.

It depends a lot on your use cases. If you're doing tons of different analytics then mongo might not be the best solution. If you're doing a standard blog server - then the benefits easily outweigh the costs.

I haven't done much map-reduce querying, but I suspect they've got the performance covered.

Also, if you're doing a web blog server the structure of the document in the data is probably going to match the structure of the document on the client - which is pretty freaking awesome.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 06:45 PM
Mongo configuration and monitoring is the biggest pain in the ass, though.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 06:49 PM
Quote:
Originally Posted by kyleb
Mongo configuration and monitoring is the biggest pain in the ass, though.
I can't remember any major problems, but we moved to https://mongolab.com/home awhile ago so we don't have to worry about much.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 06:53 PM
I never had any problems either. I didn't have to configure anything other than telling it which directory the db should live in.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 06:56 PM
CouchDB also has free cloud hosting.

I set one up for 2+2:
http://2p2.iriscouch.com/_utils

Feel free to do whatever you want, it's free. I figured this would save people the 5 seconds it took to sign up (no need to verify your e-mail).

Ignore the _users and _replicator databases. Those are internally used by couch.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 07:04 PM
for those who've used both mongo and couch, what would you recommend learning? no specific project in mind, so i guess ease/fun/interestingness of use would be a big factor
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 07:12 PM
There's Riak too. IMO Mongo is better for general purpose, that is why it's so popular. You give up small edges for general usage but the things you give up kind of don't matter because chances are your web app is never going to be the biggest thing ever.

Throwing a few hundred million documents in it and having a couple hundred thousand users is fine as long as you budgeted your hardware correctly and your full web stack is setup properly. If you grow past this point then congrats, you can afford to hire the best guys in the industry and let them worry about everything.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 07:39 PM
Quote:
Originally Posted by kerowo
Still waiting for my app credit for picking up the new MBP.... damit!
Apple support says I probably won't get it until Saturday. I don't know if I want to wait that long to save 20 bucks...
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 07:42 PM
@Shoe Lace: and anyone else that's interested. I put sample companies in the database from the Crunchbase API so you can see what the data looks like. I'll put some sample MapReduce funcitions in javascript in their too.

Here's the script in python to upload(although you need a list of what are called permalinks). These function as sort of the 'strong keys' of the database

Code:
# save all the json objects to couchdb
server = couchdb.Server('http://2p2.iriscouch.com')
db = server['companies']

# point program at links file and pick out company permalinks

permalinks = [line.strip() for line in open('C:\Documents and Settings\Luke\Desktop\Programming\Python\py web programming\crunchbase_test.txt', 'r')]

# loop over the list fetching all json objects. Then save to couchdb
for name in permalinks:
    h = urllib.urlopen('http://api.crunchbase.com/v/1/company/%s.js' % (name))
    try:
        data = json.loads(h.read())
    except:
        continue
    if not 'name' in data:
        continue
    db.save(data)
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 07:42 PM
Quote:
Originally Posted by Shoe Lace
There's Riak too. IMO Mongo is better for general purpose, that is why it's so popular. You give up small edges for general usage but the things you give up kind of don't matter because chances are your web app is never going to be the biggest thing ever.

Throwing a few hundred million documents in it and having a couple hundred thousand users is fine as long as you budgeted your hardware correctly and your full web stack is setup properly. If you grow past this point then congrats, you can afford to hire the best guys in the industry and let them worry about everything.
+1.

The main reason we used Mongo is because it makes development fast. Our team has a lot of experience with traditional databases but if you're building something where you're adding/deleting tables/fields regularly then Mongo is awesome. It's quite possible that we'll get to a point where we'll need to change databases - but if we make it to that point its probably a good thing overall.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 08:02 PM
To add a very subjective fact to the debate...the main downside of using a noSQL DB would be that Postgres is really ****ing awesome

I really don't get why companies even bother with Oracle or MSSQL or the like (well I kinda get it stuff like lol-SAP only run on those and there's this and that but really comeonson).
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 08:16 PM
MS SQL makes me want to vomit

Last edited by anononon; 07-26-2012 at 08:16 PM. Reason: oracle only slightly less so
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 08:34 PM
Quote:
Originally Posted by LA_Price
@Shoe Lace: and anyone else that's interested. I put sample companies in the database from the Crunchbase API so you can see what the data looks like. I'll put some sample MapReduce funcitions in javascript in their too.
Cool man, thanks for filling it with sample data. Also for people viewing this, in couch's web based admin panel you can view the design document here:

http://2p2.iriscouch.com/_utils/docu...design/company

Then expand each view. That will show you the map and reduce (reduce isn't present atm) code.

Then go back to the main document list:
http://2p2.iriscouch.com/_utils/data...nies/_all_docs
Select one of the views from the drop down on the top right'ish. The 2 bottom most options.

Alternatively the following links will show you the results of running the map on each view. I just directly linked them:

View: company_tags_descripton
http://2p2.iriscouch.com/_utils/data...ags_descripton

View: location_if_ipo
http://2p2.iriscouch.com/_utils/data...ocation_if_ipo
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 08:53 PM
Here's a map reduce view of categories on the attribute founded year using the built in _stats function

http://2p2.iriscouch.com/_utils/data...dedyear__stats

you just have to hit the reduce check box in the top right
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 09:17 AM
Feel free to ramble/provide ideas regarding the design of hand histories for a poker server, including DB stuff:
http://forumserver.twoplustwo.com/19...s-etc-1227234/

it's mostly a thread to keep track of my own thoughts but any input will be gladly appreciated
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 10:56 AM
Maybe a mod could break out this database discussion to a thread? Seems like good stuff.

Are there mods in this forum?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 11:00 AM
10gen offer a free MongoDB monitoring service at https://mms.10gen.com
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 11:49 AM
Couchdb has admin party(which is basically just let anyone do anything at the start). What does Mongo have?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 03:48 PM
Quote:
Originally Posted by LA_Price
Couchdb has admin party(which is basically just let anyone do anything at the start). What does Mongo have?
Mongo has native drivers for whatever language you're working with (well, the popular ones at least). You could however build your own http api on top of one if you wanted. There's likely already open source projects in most languages where this was done.

The main reference guide for drivers:
http://www.mongodb.org/display/DOCS/Drivers
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 04:20 PM
If anyone wants to take a swipe at this, that'd be awesome:

http://stackoverflow.com/questions/1...5-3-threadsafe

I can't connect to MSSQL (not my choice, trust me, I hate MSSQL) from PHP's libraries from Mac/Linux/Windows/whatever. My first choice is obv to use sqlsrv_connect() under Linux, but that is basically a ****ing pipedream. I will settle for using it in Windows, which should be easy, but hasn't been.

Goddamn I hate Microsoft products, particularly MSSQL. Also PHP but that is another debate entirely.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 05:03 PM
Quote:
Originally Posted by MrWooster
Just initiated my first Facebook ad campaign. Anyone had any experience with FB ads? I have set it up to be very specifically targeted and the advert points to the Facebook page of my site. Basically doing a minimum spend to see what happens and try and asses the worth based on the results.
If you're doing a banner-type ad, you're doing really good if you get 1 click per 1,000 impressions. That's actually common advertising data for web interfaces: usually 1/10% click-through is your "expectation" with almost every web portal that has that metric available. So, extrapolate your b/e and go from there.

That data-point made little sense to me until I read something along the lines of this: "Think about it a TV commercial you seen recently. Easy, isn't it? Now think about the last web banner you seen, or any web banner you've seen in the past week. Can't do it, can you?"

Edit to ad: I need to be specific that a banner ad is usually, "My Company Logo and Why We Rule," which shouldn't be confused with a product ad, which is "Click Here to Buy this Brand New POS."

Another Edit: There's a reason that banner ads are so low cost-per-impression. It would behoove you to remember you get what you pay for, and if one is 8c per impression compared to 4c per impression, buy the 8c per impression.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-27-2012 , 07:25 PM
Quote:
Originally Posted by kyleb
If anyone wants to take a swipe at this, that'd be awesome:

http://stackoverflow.com/questions/1...5-3-threadsafe

I can't connect to MSSQL (not my choice, trust me, I hate MSSQL) from PHP's libraries from Mac/Linux/Windows/whatever. My first choice is obv to use sqlsrv_connect() under Linux, but that is basically a ****ing pipedream. I will settle for using it in Windows, which should be easy, but hasn't been.

Goddamn I hate Microsoft products, particularly MSSQL. Also PHP but that is another debate entirely.
I'm more familiar with Postgres but I've been using SQL Server 2008 with python/pyodbc for a medium sized Django app at work and it's been working out great so far. Seems to be pretty decent software IMO.

Anyways, I know next to nothing about PHP but have you tried PHP's odbc_connect? It should work fine from Linux too I think.

edit: Also, I could have sworn you were defending PHP recently?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote

      
m