Two Plus Two Publishing LLC Two Plus Two Publishing LLC
 

Go Back   Two Plus Two Poker Forums > >

Notices

Programming Discussions about computer programming

Reply
 
Thread Tools Display Modes
Old 08-17-2017, 02:49 PM   #29351
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by RustyBrooks View Post
In mysql, you have to acquire a "metadata lock" in order to alter a table - this includes adding an index. OK. You also have to acquire such a lock to perform a select on a table. This prevents a table from being altered while a query is running.
Well, I dived deeper into this and finally figured it out. It's weird though.

There's a mysql command called "show full processlist" that shows all the threads and what they're doing. This never showed anything untoward.

There's other queries you can do though, that show varying levels of detail. I finally landed on
SELECT * FROM information_schema.innodb_trx order by trx_started;
and noticed that I had a few rows with trx_started that were like... 15 or 20 minutes old. They had no associated query, they claimed that nothing in them was locked (trx_tables_locked=0, trx_rows_locked=0 etc)

Except for 1 or 2 old ones, every other transaction would be less than 1s old.

But, if I delete the old ones, and then create my index, poof, it works immediately.

I wonder if there is something that is starting a transaction and then having an error occur without every doing any queries, and subsequently not getting closed? My connections have a 30 minute TTL so even if that's the case, they don't "pile up"

Anyway, annoying but at least not insurmountable.
RustyBrooks is offline   Reply With Quote
Old 08-17-2017, 07:12 PM   #29352
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,476
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

PostgreSQL can also freeze up if there is a transaction lock running. This is important for MVCC and ACID, though I don't like the idea of messing with information_schema to alter behavior. I suppose RDS doesn't give a mysql prompt?
daveT is offline   Reply With Quote
Old 08-17-2017, 07:34 PM   #29353
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I'm not altering behavior using information_schema, just selecting for what transactions are open. I don't really know what "open" means in this sense, since there is no running query, and according to the results I get, nothing is actually locked. Somehow there is an open transaction not doing anything, blocking access to getting a metadata lock.

I "killed" the threads that were holding these bogus locks.

You can get a prompt via RDS, sure. I was using a local client which is more or less the same thing.
RustyBrooks is offline   Reply With Quote
Old 08-17-2017, 07:53 PM   #29354
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,476
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Sounds possible that there is a piece of code in the application that does:

-- open transaction

-- run insert / update / delete query

-- (forgets to close transaction).

This would block everything else.
daveT is offline   Reply With Quote
Old 08-17-2017, 08:10 PM   #29355
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by daveT View Post
Sounds possible that there is a piece of code in the application that does:

-- open transaction

-- run insert / update / delete query

-- (forgets to close transaction).

This would block everything else.
Right, but this would also show up in several places. The query that shows open transactions also shows what they're doing (what query is being run in the transaction).

What I suspect is actually happening is
-- open transaction
-- don't run any sql queries
-- forget to close transaction
like maybe an error is happening before it even gets a chance to use the connection.

Anyway, hopefully I can get that figured out.
RustyBrooks is offline   Reply With Quote
Old 08-17-2017, 08:21 PM   #29356
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,476
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I also suspect that wrapping a SELECT in a transaction, even if it's closed, could possibly be causing an issue. I'm not entirely sure of this, but I think that caused some issue with me once before. It may have just returned an empty tuple and I could be wrong on the transaction state.

Is it possible that a return tuple is sent back before closing the transaction? I feel like this could show a completed / empty query even though the system is locked up.
daveT is offline   Reply With Quote
Old 08-17-2017, 08:33 PM   #29357
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I guess I couldn't really say. It actually "shouldn't" be possible for a transaction to not get closed at all. There's a piece of middleware that is supposed to run at the end of every web request that cleans up open connections. But clearly we're beyond should/shouldn't here because things aren't acting like they "should"

It would be nice if I could find out what used a given mysql thread id last, so maybe I can find a way to start logging that or something.
RustyBrooks is offline   Reply With Quote
Old 08-17-2017, 09:45 PM   #29358
Craggoo
culled
 
Craggoo's Avatar
 
Join Date: Sep 2006
Location: R.I.P. ItzPenzoo 12-09-11
Posts: 12,226
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Is there some way to log all the PIDs? That would be a good start. If you could output spawn time/kill time to terminal that would give you a good idea of where to look.

Welcome to the world of

Quote:
Originally Posted by TheOne View Post
relying on 3rd party code to work as advertised
Craggoo is offline   Reply With Quote
Old 08-17-2017, 10:34 PM   #29359
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I can probably log them all, although I'm a little concerned about performance. I guess I'll try it and see how it goes. Actually I guess first I'll have to figure out how to get the pid of the connection I've been given.
RustyBrooks is offline   Reply With Quote
Old Yesterday, 12:28 AM   #29360
blackize5
adept
 
Join Date: Sep 2012
Posts: 1,077
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I thought MySQL allowed reads without lock by default. A bit of research suggests that this is for innodb tables only. Anyone who knows more about dbs know any more on the subject?
blackize5 is offline   Reply With Quote
Old Yesterday, 12:34 AM   #29361
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,476
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I'm not sure if RDS has it, but is it possible to access server logs? That tends to give a lot of information...
daveT is offline   Reply With Quote
Old Yesterday, 10:18 AM   #29362
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by blackize5 View Post
I thought MySQL allowed reads without lock by default. A bit of research suggests that this is for innodb tables only. Anyone who knows more about dbs know any more on the subject?
This is true but we're not talking about row-level or table-level locks here. The problem I'm having is that in order to change the structure of a table (add or remove a column, etc, which includes adding indexes) you have to pass a "metadata" lock. This metadata lock prevents you from altering the structure of a table while a query is running on it. This was actually only added in like 5.5 (current version is 5.7.something)

So anyone, something is holding onto metadata locks way longer than they should, preventing me from adding indexes.
RustyBrooks is offline   Reply With Quote
Old Yesterday, 07:02 PM   #29363
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,476
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by Grue View Post
So day 2 at new job at GIANTCO.

Given a skylake HP with 16gb ram and... windows 7. Git bash is ~5 second delay between commands so doing command line stuff with cmd.exe. Windows 7 or bloatware, who knows?

Cannot download anything or run .exes but no surprise there.

Usual netnanny sites blocked, but not reddit or imgur. OK?

Cannot set default browser?

Cannot globally install npm packages i.e. update the PATH env variable? But can run them if I type out their real path. Fun.

No problem installing git from their "appstore" though. Because I couldn't just clone in a virus and run it? I couldn't just use curl from git bash to curl in an exe virus?

No problem with people bringing in their personal cell phones because they couldn't just take a picture of their screen to steal sensitive information?

And heres the worst: chrome extensions blocked. Firefox? No problem!

There used to be a great little feature in Chrome that allows all computers on a network to share and entire browser config, same session cookies, and so on. This means you can look at other people's emails, grab other people's session data, see their browsing history, and download porn on everyone's computer.

****

I guess I can share my recent Windows story. I was working a contract and some system needed to talk to the database via ODBC. I seriously thought about declining this because a) it's Windows and b) other people already tried to configure this.

I connect to the machine and you guessed it; whoever worked on this last downloaded a propriety driver, that driver was not paid for, and good luck getting anything to work. I immediately uninstall said driver and continue on.

Windows and ODBC are violent enemies, as far as I can tell. ODBC makes an effort by creating MSI files, but... it doesn't always work as expected. Some drivers are x64, some x86, some are both. The trick is figuring out which driver is the correct one, and if you guessed that a 64 bit Windows machine takes a 64 bit driver, nope, very wrong on that one. And if you think ODBC is easy to configure, let it be known that exactly one source on the internet shows the correct technique. I spent a good 40 hours of my life figuring it all out.

The x86 driver didn't work, so I tried to install the x64. I wasn't completely surprised that the download process was corrupted and had to think of some reasons why:

1- I used the wrong ODBC driver, but I know it can be one of either versions 1 to 6, either x86 or x64 or dual, and I don't know if I want the original or updated release. To be sure, i'd have to install, uninstall, and test each of 25 drivers.

2- The propriety driver successfully uninstalled, but there is still a bunch of cruft in the registry which has to be manually deleted, and until this very dangerous process is done, I won't be able to install ODBC drivers.

3- Who the **** knows why and it won't ever work on this machine.

Even I, who has done this for about 50 computers across Windows, Mac, and Linux, ended up taking over 4 hours to get this issue figured out. If I was in control of 50 dote heads running on a network of Windows computers, I'd lock their stuff down tighter than a library computer.

Last edited by daveT; Yesterday at 07:27 PM.
daveT is offline   Reply With Quote
Old Yesterday, 07:56 PM   #29364
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Oh god, ODBC can bite me. Some of the worst experiences of my life.
RustyBrooks is offline   Reply With Quote
Old Yesterday, 08:43 PM   #29365
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,476
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Oh, I think JDBC is even worse. I'll never associate pom poms with zeal again in my life.
daveT is offline   Reply With Quote
Old Today, 09:53 AM   #29366
JustASpectator
veteran
 
Join Date: Mar 2005
Posts: 2,070
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by jjshabado View Post
I have a co-worker who has a healthy holy crusade against 'elses' and I'm coming around to his way of thinking (sort-of-similar to what Rusty wrote above). So he would do:

Code:
If x
    # Optional Something
    # Return

#Do untrue stuff
This post and the others related to it have totally changed how I code, and I've called this style GOOD Programming, where GOOD stands for "Get Out Of Dodge".
JustASpectator is offline   Reply With Quote
Old Today, 11:39 AM   #29367
Larry Legend
Isaiah GOAT
 
Larry Legend's Avatar
 
Join Date: Jul 2009
Location: Let's go celtics
Posts: 40,813
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

I don't get everyone freaking out about the react.js license.

Don't sue anyone for about patents and things are good.
Larry Legend is offline   Reply With Quote
Old Today, 01:12 PM   #29368
_dave_
_Pooh_Bah_
 
Join Date: Feb 2005
Location: UK (or what remains of it)
Posts: 12,206
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

indeed, it's a complete nothing / people (perhaps intentionally) not understanding / parroting of wrongness

Last edited by _dave_; Today at 01:14 PM. Reason: or maybe I'm the wrong one, lol
_dave_ is online now   Reply With Quote
Old Today, 03:28 PM   #29369
daveT
S.A.G.E. Master
 
daveT's Avatar
 
Join Date: Jun 2005
Location: Why didn't I use Clojure instead?
Posts: 21,476
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Rusty, I had a thought about your issue. If you have a middleware that closes all transactions, then perhaps my suggestion is wrong.

However, if you have manually open transactions, using open() or whatever, then is it possible that you are running a query outside of a transaction because the programmer forgot to open a transaction? It seems logical to me that a query running in a function is a session-level transaction**, but this case may end up creating a system-level transaction, if that makes any sense. In other words, if the query is running, it is automagically wrapped in a transaction within the database but not within the app itself. To discover something like this, find functions that return "none" regardless of the params.

** I would think forgetting to open a transaction would completely fail at every level, but I'm not really sure, to be honest.
daveT is offline   Reply With Quote
Old Today, 05:03 PM   #29370
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 22,989
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

The way I have sqlalchemy configured, which is more or less the recommended/default way, every time you start using a connection, it's a transaction. So even selects are in a transaction. It's supposed to automatically roll back if you close the connection without ending the transaction. Again, "supposed to", so it's possible that isn't happening, or the connection isn't getting properly closed.

I suspect the latter, because any time I look, I can find a few of these phanton transactions. They are never more than 30 minutes old, which is the timeout I have to recycle connections in the connection pool. I am guessing an error or bug in the middleware itself, or something that somehow is skipping the middleware.

I've put the issue aside for now. The urgency is much lower than before because at least now I can get around it when I need to.

Also, a complicating factor is that "other things" connect to the database too - my own local SQL sessions through Sequel Pro. There's a read-only account that our product manager uses - probably with sequel pro or whatever tool he likes. There's a process that does backups, and so forth. it *could* be one of those things. It's sometimes easy to forget that your website is not the only thing connecting to your database.
RustyBrooks is offline   Reply With Quote

Reply
      

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Forum Jump


All times are GMT -4. The time now is 06:35 PM.


Powered by vBulletin®
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright ę 2008-2010, Two Plus Two Interactive
 
 
Poker Players - Streaming Live Online