Open Side Menu Go to the Top

07-25-2012 , 10:27 PM
The initial debates about how to store a HH in a db were a bit misguided IMO. Reason being that people were talking about the poker server vs a tracker like fpdb.

A Server isn't going to have any performance concerns about storing/querying HHs b/c it does not affect actual gameplay.

Trackers an HUDs have totally different priorities as they are constantly querying.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **
$25m Guaranteed WPM on CoinPoker
Join the action now
Daily Rewards • Splash Pots • CoinRaces
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **
07-25-2012 , 10:57 PM
Quote:
Originally Posted by Shoe Lace
So a dict directly maps to json 1:1 without any encoding or decoding? That does makes things extremely easy if you can end up working with the db results directly and writing dicts straight to the db.
it directly maps conceptually in that json files are just nested key-value pairs (like python dictionaries (which btw are just hashtables)). you still need encoding/decoding, but it's super lightweight/cheap compared to, say, xml.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-25-2012 , 11:05 PM
Quote:
Originally Posted by tyler_cracker
it directly maps conceptually in that json files are just nested key-value pairs (like python dictionaries (which btw are just hashtables)). you still need encoding/decoding, but it's super lightweight/cheap compared to, say, xml.
Just for the record I was talking specifically about python/mongo where the driver takes care of any encoding issues for you.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-25-2012 , 11:40 PM
oic. and that obv makes sense.

on the plus side, what i said is generally true of the cozy relationship between json and modern scripting languages. so... go team?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 12:11 AM
The performance discussion ITT was super interesting. Now that I work almost entirely with big data, everyone expects it to be very slow so I'm mostly off the hook

Though I just got my RasPi and writing optimized code for ARM is super tricky
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 12:16 AM
Quote:
Originally Posted by sorrow
No good reason, except we didn't know. Starting the project again today I probably would.

When we had a user attempt to convert FPDB to using an ORM we were getting import speeds of 1-2 hands per second, so work in that direction stopped pretty quickly.
Yeah I could see reasons to avoid it in the application itself, but didn't see any not to use one to create the SQL code
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 01:46 AM
HTTP question for you guys.

Setup: An API to a consumer website that is used by mobile apps. Has calls like "login", "create_user", etc. The API is public in the sense that it is open to public traffic, but it is not published publicly (ie, it's intended purpose is only for use by the mobile apps). Not sure if that even matters.

Consider the "login" call: The mobile app makes a POST to the api, and the body of that post contains JSON like {"username":"john","password":"secret"}. That is, this is a custom login system and not http basic authentication or digest authentication.

Happy Path: The happy path is 200 http response, returning JSON like: {"success":true, "user_id":203482, "session_token":asfl2oajofjwasljfopwo3}

Question: Is the by the books correct sad path for a correctly formatted request that has invalid credentials a 401 http response, or a 200 http response with JSON like: {"success":false, error:"The username and password did not match"}? Or is either option consistent with the http specification?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 01:53 AM
Quote:
Originally Posted by kyleb

Though I just got my RasPi and writing optimized code for ARM is super tricky
Just got mine too, such a neat piece of kit
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 01:59 AM
Quote:
Originally Posted by gaming_mouse
HTTP question for you guys.

Setup: An API to a consumer website that is used by mobile apps. Has calls like "login", "create_user", etc. The API is public in the sense that it is open to public traffic, but it is not published publicly (ie, it's intended purpose is only for use by the mobile apps). Not sure if that even matters.

Consider the "login" call: The mobile app makes a POST to the api, and the body of that post contains JSON like {"username":"john","password":"secret"}. That is, this is a custom login system and not http basic authentication or digest authentication.

Happy Path: The happy path is 200 http response, returning JSON like: {"success":true, "user_id":203482, "session_token":asfl2oajofjwasljfopwo3}

Question: Is the by the books correct sad path for a correctly formatted request that has invalid credentials a 401 http response, or a 200 http response with JSON like: {"success":false, error:"The username and password did not match"}? Or is either option consistent with the http specification?
A 401 would be the correct response.

Basically, you should not be returning "success":"true". This should be determined by Looking at the response header. 200 = success, non 200 = fail.

In practice however, a lot of APIs return 200 with a "success" entry in the JSON, even tho this is technically incorrect.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 02:02 AM
Regarding the noSQL vs SQL argument going on, I have a couple on things to add but currently typing on an iPad so it will have to wait until tomorrow.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 02:03 AM
Yeah, I return 401 for my APIs in that case.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 02:11 AM
Quote:
Originally Posted by MrWooster
A 401 would be the correct response.

Basically, you should not be returning "success":"true". This should be determined by Looking at the response header. 200 = success, non 200 = fail.

In practice however, a lot of APIs return 200 with a "success" entry in the JSON, even tho this is technically incorrect.
sorry, the success:true should have been success:false in the sad path case.

in any case, can you "prove" why it should be 401 from the specification. i'm not arguing, just trying to understand. because my thinking would be in the case of say a page request using basic authentication, you make a request for a resource but don't provide the credentials needed to view that resource.

this case seems different. i am perfectly allowed to view this resource, because the resource precisely IS a service which tells me if u/p match, and gives me an error message. i mean, if i go to a website and try to login with invalid u/p, i don't get a 401, i get a 200 with the login form back displaying an error message. why is this different?
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 02:58 AM
Quote:
I "lead" FPDB and think SQL is a ****ing awful language propped up by 40 years of theory, 25 years of experience and a lack of solid alternatives. That said, my experience is that it performs very well with structured hh data. DB normalisation also seems to pretty closely match OO design, hence the prevalence of ORMs appearing in the last few years.
Since I have worked with ZODB (the object database used by Zope) a bit one thing I'm wondering is if you couldn't simply serialize some sort of "hand object". Depending on how you'd design a poker server you just call hand.new or something for each hand and after that one is done hand.new again for the next and so forth then you'd just write the serialized hand objects to ZODB. Would probably be neat for hand replayers and so forth.

I do know that if you want to use ZODB in hugh poerformance environments some hackery is needed (we used to index via mysql to optimize etc.)

sorrow: Good to know there's a FPPDB dude ITT. I meant to look at this for quite some time. I'll probably install it and test it over the weekend or something.

Last edited by clowntable; 07-26-2012 at 03:05 AM.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 04:01 AM
Quote:
Originally Posted by MrWooster
A 401 would be the correct response.

Basically, you should not be returning "success":"true". This should be determined by Looking at the response header. 200 = success, non 200 = fail.

In practice however, a lot of APIs return 200 with a "success" entry in the JSON, even tho this is technically incorrect.
I think you may be mistaken on this.

Quote:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html:

The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource.
A 401 response is only if you're using basic authentication. If not, it's incorrect b/c a 401 requires a WWW-Authenticate header to be included. If you aren't using basic authentication, it makes no sense to send a WWW-Authenticate header.

Quote:
i mean, if i go to a website and try to login with invalid u/p, i don't get a 401, i get a 200 with the login form back displaying an error message. why is this different?
Exactly. If you're not using basic authentication, it's nothing more than submitting a form and then getting results from the server.

Also, responding with json on success and 401 on failure pretty much negates the usefulness of json.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 09:11 AM
Quote:
Originally Posted by Shoe Lace
The thing is though, I don't even see that as something the db would be doing. That would be the job of the hand parser. The hand parser would take the hand from any supported network and transform it into a json structure. Then really it's as simple as directly doing a "save" into the db with that json object. That would probably get you 95% of the way.

Once you have the hands in there, you would use whatever tools the document db gives you to query it. These might be ad-hoc queries that you can run, or map/reduce jobs that you can write, etc.. Running standard filters like "show me hands where I have AA and I'm on the button" would be super ultra trivial in both complexity and performance.
Hi. FPDB developer here and operator of Pokeit.co. Been meaning to respond to this thread for a couple of days. Over the past month I've been playing around with ideas for how to store records of hand histories themselves with the primary goal of reducing the storage footprint of each user's database/hand histories. Instead of storing hands as records in a large table, we'd store just the relevant info in a file that could compressed, indexed, and accessed later when a user wants to review the hand. After implementing a good portion of this I got to wondering if this was opening the door for a NoSQL solution. That's about as far as I've gotten, but i'm planning on putting in some more time to figuring this out. I've already got it spitting out json object from this stripped down hand history. I'll post it here when I get another free moment.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 09:20 AM
Totally happy with Mountain Lion. Notification Center is welcome. Expose is BACK.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 09:33 AM
Quote:
After implementing a good portion of this I got to wondering if this was opening the door for a NoSQL solution.
It might be. json supports arrays of objects so you could store a few hundred hands for each json file. Most document dbs support bulk inserts too and in the ones that do it's much more performant to bulk insert when you can.

Last edited by Shoe Lace; 07-26-2012 at 09:39 AM.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 09:43 AM
Pretty good explanation from a Stackoverflow question on why 403 probably isn't the right answer for failed logins:

Quote:
Receiving a 403 response is the server telling you, “I’m sorry. I know who you are–I believe who you say you are–but you just don’t have permission to access this resource. Maybe if you ask the system administrator nicely, you’ll get permission. But please don’t bother me again until your predicament changes.”

In summary, a 401 Unauthorized response should be used for missing or bad authentication, and a 403 Forbidden response should be used afterwards, when the user is authenticated but isn’t authorized to perform the requested operation on the given resource.
Link: http://stackoverflow.com/questions/3...http-responses

Last edited by sdturner02; 07-26-2012 at 09:44 AM. Reason: Heh. Nevermind.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 10:58 AM
Quote:
Originally Posted by gaming_mouse
sorry, the success:true should have been success:false in the sad path case.

in any case, can you "prove" why it should be 401 from the specification. i'm not arguing, just trying to understand. because my thinking would be in the case of say a page request using basic authentication, you make a request for a resource but don't provide the credentials needed to view that resource.

this case seems different. i am perfectly allowed to view this resource, because the resource precisely IS a service which tells me if u/p match, and gives me an error message. i mean, if i go to a website and try to login with invalid u/p, i don't get a 401, i get a 200 with the login form back displaying an error message. why is this different?
Sorry, I misread a bit. If the API request is valid, but the result of the API request is "login failed", then you should return status 200, because the system requesting did have permission to make the actual API request.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 11:10 AM
Quote:
Originally Posted by Neil S
Totally happy with Mountain Lion. Notification Center is welcome. Expose is BACK.
Still waiting for my app credit for picking up the new MBP.... damit!
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 11:15 AM
Quote:
Originally Posted by ChazDazzle
Hi. FPDB developer here and operator of Pokeit.co. Been meaning to respond to this thread for a couple of days. Over the past month I've been playing around with ideas for how to store records of hand histories themselves with the primary goal of reducing the storage footprint of each user's database/hand histories. Instead of storing hands as records in a large table, we'd store just the relevant info in a file that could compressed, indexed, and accessed later when a user wants to review the hand. After implementing a good portion of this I got to wondering if this was opening the door for a NoSQL solution. That's about as far as I've gotten, but i'm planning on putting in some more time to figuring this out. I've already got it spitting out json object from this stripped down hand history. I'll post it here when I get another free moment.
I have been using MongoDB to store hand data for PkrSess, and it works great. I would NOT recommend it however for any application based around statistical analysis. PkrSess does very little actual analysis on the hands, so there is no need for a relational database. Something like Pokeit and FPDB need to be able to link hands to statistics to players. Doing this using a NoSQL database would be very messy and you would probably see a performance hit because you are attempting to join collections. If you need to join collections, either your DB design is wrong, or you should be using a relational DB.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 11:32 AM
Quote:
Originally Posted by MrWooster
I have been using MongoDB to store hand data for PkrSess, and it works great. I would NOT recommend it however for any application based around statistical analysis. PkrSess does very little actual analysis on the hands, so there is no need for a relational database. Something like Pokeit and FPDB need to be able to link hands to statistics to players. Doing this using a NoSQL database would be very messy and you would probably see a performance hit because you are attempting to join collections. If you need to join collections, either your DB design is wrong, or you should be using a relational DB.
I agree with your general point, but I don't think its quite as strong as you say. There are times where you'll want to do small joins and that's totally reasonable and acceptable.

For example, it could be totally reasonable to have a HH collection and a player collection. If you have a query where you want to calculate some statistics about a small group of players (like those at your table) you would be fine. You could look up the player data, query the HH for the players you want doing whatever super complex stuff you want, then 'join' the data in memory where you match the players to the appropriate results.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 12:11 PM
Correct me if I'm wrong, but I see no reason to use a non-relational database for storing HH's if the purpose is to generate reports/stats on those HH's.

To generate a simple report as to your profit in the dealer position would take a lot of time to generate on a non-relational database but a well designed relational database with indexes set up correctly would be easier to write queries for AND should be faster to actually calculate the data.

Why would you use a non-relational database for this task? It just looks like it would make everything harder and slower. The only benefit I can see is faster insertions? But I mean that's not really going to be much of an issue, in real time you might get up to 100 hands per hour @ 30 tables = 0.8 hands inserted per second. And if you're inserting bulk data from past sessions people are usually fine waiting for an hour or so for that.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 12:18 PM
Quote:
Originally Posted by Gullanian
To generate a simple report as to your profit in the dealer position would take a lot of time to generate on a non-relational database but a well designed relational database with indexes set up correctly would be easier to write queries for AND should be faster to actually calculate the data.
3 questions:

1. What makes you think it's going to be slow for the document db?
2. What makes you think you cannot index fields in a document db?
3. Is it fair to make claims that it's easier without having worked with a document db before? A few posts back I wrote an example query on how to get hands on the button, it cannot be easier. Calculating the sum of a field would also be ******ed easy.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
07-26-2012 , 12:19 PM
Quote:
Originally Posted by Gullanian
To generate a simple report as to your profit in the dealer position would take a lot of time to generate on a non-relational database but a well designed relational database with indexes set up correctly would be easier to write queries for AND should be faster to actually calculate the data.
I don't see why the non-relational database query is so hard to write or so slow to execute in this example. Profit and position would be easily calculated from a single document - then you just have to add up the rows.

And I think you skip over the fact that "a well designed relational database with indexes set up correctly" is a lot harder to set up than a document store of hand histories.
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD ** Quote
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **
$25m Guaranteed WPM on CoinPoker
Join the action now
Daily Rewards • Splash Pots • CoinRaces
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

      
m