Two Plus Two Publishing LLC Two Plus Two Publishing LLC
 

Go Back   Two Plus Two Poker Forums > >

Notices

Programming Discussions about computer programming

Reply
 
Thread Tools Display Modes
Old Yesterday, 10:15 PM   #28676
Grue
Pooh-Bah
 
Grue's Avatar
 
Join Date: Mar 2004
Location: It is pitch black.
Posts: 4,806
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

idk anything about back end/databases really so yeah. These games are pretty big though. They contain all chats and actions and such.

Suzzer every call (right now) gets all games i.e. the whole collection. Hourly would make it crash hourly
Grue is offline   Reply With Quote
Old Yesterday, 10:31 PM   #28677
suzzer99
Carpal \'Tunnel
 
suzzer99's Avatar
 
Join Date: Nov 2005
Location: on top of the bell curve
Posts: 80,788
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Yeah you're gonna need some way to just store the stats you need in another DB, then add to your stats collection once an hour or something. Grabbing all games is not a sustainable solution obviously. You'll probably have to shard at some point as well. Thus concludes everything I know about Mongo.
suzzer99 is offline   Reply With Quote
Old Yesterday, 10:35 PM   #28678
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,309
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by jjshabado View Post
I don't know what Dave thinks I'm saying (like I have literally no idea) because I'm taking about replacing current monitoring with better monitoring and not about replacing humans at all.
I guess i was / am confused. If all you are saying is unify the blood, sugar, etc, monitor into one system, I'm not sure why they didn't do that long ago, as it seems obvious. I'm guessing there is a reason they haven't yet. I'd trust the people who create this stuff is far far more intelligent than I am on this subject.

Quote:
Originally Posted by ChrisV View Post
Don't have a solution but as a crotchety old man who can't be having with this NoSQL stuff I'm gonna point out that a SQL query to calculate stats over 21,000 games would probably run in like 3 seconds and that Mongo probably wasn't a great choice considering your data is relational. You kids get off my lawn.
According to MongoDB, no one needs any of that old theory trash until the data gets too large to deal with (16mb to be exact).
daveT is offline   Reply With Quote
Old Yesterday, 10:46 PM   #28679
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 33,120
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by Grue View Post
idk anything about back end/databases really so yeah. These games are pretty big though. They contain all chats and actions and such.

Suzzer every call (right now) gets all games i.e. the whole collection. Hourly would make it crash hourly
Can you not just retrieve only the data you need, or is that not a thing in mongo? That the games contain chats, actions etc is a complete irrelevancy in SQL.
ChrisV is offline   Reply With Quote
Old Yesterday, 11:08 PM   #28680
Grue
Pooh-Bah
 
Grue's Avatar
 
Join Date: Mar 2004
Location: It is pitch black.
Posts: 4,806
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by ChrisV View Post
Can you not just retrieve only the data you need, or is that not a thing in mongo? That the games contain chats, actions etc is a complete irrelevancy in SQL.
I honestly didn't think of that but um mongo has to have some sort of map built into it I would guess but still not sure that that would prevent the out of memory problems. Something to do tomorrow. I was obviously just doing the map inside of the node cb itself.
Grue is offline   Reply With Quote
Old Yesterday, 11:19 PM   #28681
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 33,120
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I mean, from what I understand you're essentially loading your entire mongo database into memory? That sounds like a good candidate for what is causing the out of memory problem, lol.
ChrisV is offline   Reply With Quote
Old Yesterday, 11:22 PM   #28682
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,807
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

So there are 3 things you need to do. I don't know what language you're writing in, but probably all 3 things are available.

I use mongo with python most often. So you can iterate over a mongo find query, without loading the whole thing into RAM. You can do like

Code:
for x in mongo.db.collection.find(whatever):
   ... do something with x
This means you only have to be able to hold each entry in memory at a time, plus whatever structure aggregates results.

Secondly, you can limit mongo queries to just what you want. In pure mongo, this is something like

Code:
find({..query..}, {'_id': 0, 'field1': 1, 'field2': 1})
which means "only fetch field1 and field2, and leave out the _id field". This dramatically limits the amount of **** you fetch.

And finally, you need to look into aggregation. Mongo has ways to do processing on rows to develop summary data. It's not the fastest thing ever and it's a little mind bending but basically you can probably do 90% of what you want to do via aggregation queries and not have to load all the **** into RAM.

Long story short, but, it's time for you to actually learn how mongo works.
RustyBrooks is offline   Reply With Quote
Old Yesterday, 11:23 PM   #28683
Grue
Pooh-Bah
 
Grue's Avatar
 
Join Date: Mar 2004
Location: It is pitch black.
Posts: 4,806
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

It was just the games db! I have a whole accounts and logs db!
Grue is offline   Reply With Quote
Old Yesterday, 11:36 PM   #28684
blackize5
adept
 
Join Date: Sep 2012
Posts: 1,046
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

That for x in loop sure looks like it's loading everything into memory. I guess it could only be loading a batch of the data at once but that seems unexpected.

Like if the resulting collection from that find call doesn't contain everything, then the loop is hitting the DB n times, no? That seems unexpected too.
blackize5 is offline   Reply With Quote
Old Yesterday, 11:48 PM   #28685
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,807
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by blackize5 View Post
That for x in loop sure looks like it's loading everything into memory. I guess it could only be loading a batch of the data at once but that seems unexpected.

Like if the resulting collection from that find call doesn't contain everything, then the loop is hitting the DB n times, no? That seems unexpected too.
The whole point of a cursor is to maintain your place in a database and just load the "next" part.

Some database drivers don't accomodate this and just load everything. But the good ones only load the next record.
RustyBrooks is offline   Reply With Quote
Old Today, 12:22 AM   #28686
goofyballer
 
goofyballer's Avatar
 
Join Date: Jun 2005
Posts: 58,422
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Work/negotiation question, curious to get some thoughts. Sorry for verbosity.

Last year I got a ~10% raise (which I asked for) when our founders asked me to switch projects to help our other project get across the finish line for its release. This year I got a ~inflation-sized raise. This happened early in the year during our standard salary review period, and shortly after that my manager (our project lead) swapped projects to become the technical director of our other project. He kinda had two roles on our project, as both overall project lead and lead of a small team of core engineers, including me; one of our founders asked if I'd like to take over the latter role and manage that small team. I said yes.

Perhaps naively (this seemed like an obvious shoe-in to me but maybe my expectations were off), I assumed this promotion would come with a raise, but after asking about that following a month or so in my new role I was eventually told that it's not. The reasons I was given make very little sense to me:
- our company has two different progression ladders, technical and management
- because my advancement so far has been on the technical side, I make far more than what a first time lead would be paid and he said I'm probably about 20k over average lead compensation

...and that's basically it. It seems that this distinction about technical vs management stuff is important to them.

I discussed this first with my new manager (new project lead) before asking our founder about it, and he basically sided with the company. My impression of that conversation was that he had some Stockholm Syndrome about instances in his past when he didn't got promoted and also didn't get accompanying raises, so he didn't see why I should necessarily get one, and this ladder system or whatever seemed to make sense to him (it does not to me).

So, I have some rather obvious counterarguments:
1. as an extremely basic principle, if I'm doing more work/taking on more responsibility then I should get paid more for it
2. there's no reason for me to agree to manage more stuff if I don't get anything in return (a fancier title, which I've never cared about, isn't going to pay my bills)
3. this casts a lot of doubt about my ability to get raises in the future (at least by advancing in management, which I'm now having second thoughts about) if I'm going to continue to be compared salary-wise to people less senior than me

I'm working on a diplomatic email outlining these, #2 is a little tricky because it makes me sound like a mercenary/not a team player/whatever but the other two are easy enough. Kinda just want to sanity check this situation with you guys and make sure my expectations/view of this situation aren't grossly out of line; it makes me question my judgement/perception when something like this happens that leaves me so confused.
goofyballer is offline   Reply With Quote
Old Today, 01:00 AM   #28687
leavesofliberty
Pooh-Bah
 
leavesofliberty's Avatar
 
Join Date: Jul 2010
Location: Without the powers of Deep Blue
Posts: 5,061
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Good luck.
leavesofliberty is offline   Reply With Quote
Old Today, 01:02 AM   #28688
muttiah
Carpal \'Tunnel
 
muttiah's Avatar
 
Join Date: Aug 2004
Posts: 23,019
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I would phrase 1) and 2) from perspective of the value you add to the company, not the extra work you're doing. Responsibility and work load are kinda irrelevant to salary discussion.
muttiah is offline   Reply With Quote
Old Today, 01:43 AM   #28689
jjshabado
Carpal Tunnel
 
jjshabado's Avatar
 
Join Date: Jul 2006
Posts: 21,468
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

There's a lot of context that's probably important.

But it's not crazy to me to have two different progression ladders and to have good experienced engineers paid more than a fairly new team lead engineer.

I think more companies are recognizing that there's a class of good engineer that wants little to do with management but they still have more value in what they do then an inexperienced lead. Especially if you're someplace where there are a lot of duties assigned to senior engineers regardless of if they're a manager or not (being on call, doing code reviews, participating in architecture reviews, etc).

So one way to look at it is that by doing team management you're actually providing less value (at this point) then when you were just an experienced engineer who was good at their job. Again, context here is probably important.

That being said, if this is actually their philosophy it should be transparent to everybody. And you should have had conversations about which track you were on and where your career could progress on each one.

For your questions I think the responses might be:

1. You're not really doing more work or have more responsibility. You're trading work/responsibilities you had as an experienced engineer for work/responsibilities you have as an inexperienced manager. Again, context is important here because this could be true or it could be complete bull****.

2. Yes. But maybe phrase it as you want to have the conversation about where each ladder can take you. If you don't enjoy/want the leadership responsibilities you should know what your alternative is on the technical ladder. Like is the leadership role a role that tops out way above the engineer role? Is your compensation 'stuck' for some period of time? Etc.

3. I guess I cover this in 2.

I don't think 2. Makes you sound like a mercenary. I think you phrase it as though you didn't know about these two tracks and now you want to know what your future can look like on either one. If the technical one looks better to you overall (short and long term) then I think you just phrase it as you want to follow the non-management career arc.
jjshabado is offline   Reply With Quote
Old Today, 01:52 AM   #28690
jjshabado
Carpal Tunnel
 
jjshabado's Avatar
 
Join Date: Jul 2006
Posts: 21,468
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Also, if you're close enough to the founders have this discussion with them unless you think it would really cause problems.

There's a thing that can happen where 'leadership' can have this great compensation/promotion philosophy and strategy that actually makes a lot of sense and could work really well but a lack of transparency/documentation means each step removed from them loses a lot of the concepts.

So the founders might have a totally reasonable argument for why you shouldn't get a raise. But your direct manager doesn't really understand it and can't talk knowledgeably about it.
jjshabado is offline   Reply With Quote
Old Today, 01:56 AM   #28691
lostmypw
newbie
 
Join Date: Mar 2017
Posts: 32
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

any node.js experts here? take a look at this code,

https://github.com/LiskHQ/lisk/blob/...rs/sequence.js

lisk is blockchain and these Sequence objects basically execute a callback on every tick of the event loop. If they have a task in the queue, they run a single task.

This strikes me as bizarre. Is there any reason to write code like this?
lostmypw is offline   Reply With Quote
Old Today, 02:27 AM   #28692
suzzer99
Carpal \'Tunnel
 
suzzer99's Avatar
 
Join Date: Nov 2005
Location: on top of the bell curve
Posts: 80,788
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Looks like they wrote their own event loop. Not sure why you would do that - but I'm going to assume these guys know what they're doing as they have their own specialized version of node you have to run.

suzzer99 is offline   Reply With Quote
Old Today, 02:37 AM   #28693
goofyballer
 
goofyballer's Avatar
 
Join Date: Jun 2005
Posts: 58,422
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by muttiah View Post
Responsibility and work load are kinda irrelevant to salary discussion.
I'm curious, why? Assuming you're getting it all done, I would find it natural that X+1 work/responsibilities should get paid more than X, and for those things to correlate directly with the value to the company I'm providing.

Quote:
Originally Posted by jjshabado View Post
So one way to look at it is that by doing team management you're actually providing less value (at this point) then when you were just an experienced engineer who was good at their job. Again, context here is probably important.

That being said, if this is actually their philosophy it should be transparent to everybody. And you should have had conversations about which track you were on and where your career could progress on each one.
This seems potentially reasonable, though I think they've got a bit of work to do on the part where they communicate how that works to their employees. This year is the first I've heard of this stuff. I definitely enjoy sitting at my desk and writing code more (like, infinitely more) than sitting in meetings and worrying about what tasks each individual on my team should work on.

Quote:
Originally Posted by jjshabado View Post
Also, if you're close enough to the founders have this discussion with them unless you think it would really cause problems.
goofyballer is offline   Reply With Quote

Reply
      

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Forum Jump


All times are GMT -4. The time now is 03:13 AM.


Powered by vBulletin®
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Search Engine Optimisation provided by DragonByte SEO v2.0.33 (Pro) - vBulletin Mods & Addons Copyright © 2017 DragonByte Technologies Ltd.
Copyright ę 2008-2010, Two Plus Two Interactive
 
 
Poker Players - Streaming Live Online