Two Plus Two Publishing LLC
Two Plus Two Publishing LLC
 

Go Back   Two Plus Two Poker Forums > >

Notices

Programming Discussions about computer programming

Reply
 
Thread Tools Display Modes
Old 01-19-2019, 02:29 PM   #37301
_dave_
_Pooh_Bah_
 
Join Date: Feb 2005
Location: UK (or what remains of it)
Posts: 12,812
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

You don't really need anything super complex Java Python soup to archive two threads really.

The URLs are formed in a predictable manner, so grab the last page, e.g.

https://forumserver.twoplustwo.com/2.../index176.html

change 176 all the way down to 1, give urls to wget - there we go, job done.

Note, thread opened in an incognito window to get default pagination and no need for cookie handling.
_dave_ is offline   Reply With Quote
Old 01-19-2019, 02:39 PM   #37302
jmakin
 
jmakin's Avatar
 
Join Date: Jan 2008
Location: Streaming
Posts: 29,744
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

haha you genius that's really good. That'll probably be involved in my load process. But I want to save the files in a format where you can click a page link and it takes you to the next page in the series of HTML pages, for easy reading. A little ambitious probably. There's probably a messy shell script solution for that too.
jmakin is online now   Reply With Quote
Old 01-19-2019, 03:19 PM   #37303
jmakin
 
jmakin's Avatar
 
Join Date: Jan 2008
Location: Streaming
Posts: 29,744
** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

damn, when you give a URL with index*.html, if it's larger than the number of pages, the site just gives you the last page. No idea how to end that loop.
jmakin is online now   Reply With Quote
Old 01-19-2019, 03:25 PM   #37304
jmakin
 
jmakin's Avatar
 
Join Date: Jan 2008
Location: Streaming
Posts: 29,744
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Theres a really filthy way to do it. Nvm
jmakin is online now   Reply With Quote
Old 01-19-2019, 04:04 PM   #37305
kerowo
lolcat
 
kerowo's Avatar
 
Join Date: Nov 2005
Posts: 36,307
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by jmakin View Post
damn, when you give a URL with index*.html, if it's larger than the number of pages, the site just gives you the last page. No idea how to end that loop.
Parse out the page number and wait for it to repeat.
kerowo is offline   Reply With Quote
Old 01-19-2019, 06:47 PM   #37306
adios
Carpal \'Tunnel
 
Join Date: Sep 2002
Location: Russian Troll
Posts: 21,300
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by jmakin View Post
I ruled out the other tests causing a problem- that was one of the first things I tried yesterday. I still saw this issue even if it was the only test in the suite that was running. Right before i wrote this post i tried running that test on a different device than the other tests and got the same results.

I emailed my boss and told him basically that I give up because this is so low reward and I have a lot of other things I can work on. But heís really paranoid that something is wrong with our software so he wants me to chase it down - i already wasted like an hour of his time today with this. The dev in me wants to spend another 16 hours chasing this but the PM in me is saying itís just really not worth the time.
Totally understandable that your boss wants to know. Put yourself in his position. Pretty sure he won't be satisfied with your explanation.

Quote:
Originally Posted by jmakin View Post
It is not the DB. Compiling the exact same code on its own main method runs fine. It is something goofy with the maven build - it has to be. Thereís no other explanation.
I know little about Java and less about maven. A Google search reveals

Increasing Heap Size When Building with Maven
Quote:
1. Open your mvn.sh file and add the following line:
export MAVEN_OPTS=-Xmx512m
this will allow to specify java heap space based on your project needs.


How to increase Java Heap space for Maven on Windows
1. go to C:\apache-maven-2.2.1\bin
2. open your mvn.bat file and add the following line :
set MAVEN_OPTS=-Xmx512m
next time you run maven commands like mvn clean or mvn install it will use this heap size.
This article is in continuation of my earlier post on java heap 10 points on java heap space ....
Have you compared the amount of memory allocated to the heap with the builds that are working with what is allocated with MAVEN_OPTS?



Quote:
I am on the smallest piece. The slowdown happens after N insertions. Doing the jvm pstack command (i forget what its called) les me to believe most of the time is being spent in GC. Which is why i figured it was the heap approaching its limits set by the JVM. If the JVM senses its running out of heap space it aggressively tries to GC. My program is supposed to use a lot of memory, this is expected.
I'm guessing you are probably correct but rooting the problem would go a long way in satisfying your boss and probably enhancing his opinion of your capabilities.
adios is offline   Reply With Quote
Old 01-19-2019, 09:42 PM   #37307
Victor
Carpal \'Tunnel
 
Victor's Avatar
 
Join Date: Jul 2003
Posts: 60,677
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by jmakin View Post
i don't want to say if it's for politics or not but that is a use case. I don't know if that violates some 2.p2 TOS and I don't want to get banned. I was just going to open source it somewhere after I was satisfied of not embarrassing myself if it doesn't get me in hot water.

The main reason it's been on my todo list was because of the OOT drama, ni.ma/bayareabeast BS looking very likely to possibly end up with me in a courthouse someday and I wanted an archive in case it got nuked. Google cache is not very reliable for these things, pages go missing very easily.

I also want to archive my blog, it's kind of an autobiography at this point and it'd really ****ing be a huge blow to me if the site disappeared or I got OTBC'd or something.
Wait wheres the nima drama? Main thread has been dead
Victor is offline   Reply With Quote
Old 01-20-2019, 02:32 AM   #37308
jmakin
 
jmakin's Avatar
 
Join Date: Jan 2008
Location: Streaming
Posts: 29,744
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

No itís very old drama. I almost had to sue him. Plus thereís a good chance of future drama. Just covering my bases.

Adios,

Thatís one thing i havent tried. I tried setting the heap size with surefire-plugin but it didnt seem to be actually changing anything.
jmakin is online now   Reply With Quote
Old Today, 03:03 PM   #37309
microbet
Solar Powered
 
microbet's Avatar
 
Join Date: Jan 2005
Location: SoCal
Posts: 51,550
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Say I have a webpage that works with this:

Code:
<script src="https://api.whatever.com/js/?params=values&key=932i8ghja-09uetc></script>

function onLoadWhatever() {
   console.log(whatever.api.somefunction(x, y, z));
}
How do I

a) just test that in a new console? The stuff in the script tags loads (?) all the functions into the browser (I can do "whatever.api" in the console on that page to get a list of api functions) So, like go to any old page and get that api from console. (something like "var whatever = require("https://api.whatever.com")")

b) how do I use that api in node? - it's not available to npm install

Hopefully these questions make sense.
microbet is offline   Reply With Quote
Old Today, 03:42 PM   #37310
Grue
Pooh-Bah
 
Grue's Avatar
 
Join Date: Mar 2004
Location: It is pitch black.
Posts: 5,520
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by microbet View Post
Say I have a webpage that works with this:

Code:
<script src="https://api.whatever.com/js/?params=values&key=932i8ghja-09uetc></script>

function onLoadWhatever() {
   console.log(whatever.api.somefunction(x, y, z));
}
How do I

a) just test that in a new console? The stuff in the script tags loads (?) all the functions into the browser (I can do "whatever.api" in the console on that page to get a list of api functions) So, like go to any old page and get that api from console. (something like "var whatever = require("https://api.whatever.com")")

b) how do I use that api in node? - it's not available to npm install

Hopefully these questions make sense.
a) whatever (object) should be globally defined in the console on any page you load that script tag

b) uh copy and paste the js file into something, add module.exports = whatever at the bottom, require it in node?
Grue is offline   Reply With Quote
Old Today, 04:15 PM   #37311
suzzer99
Carpal \'Tunnel
 
suzzer99's Avatar
 
Join Date: Nov 2005
Location: on top of the bell curve
Posts: 90,038
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

You can use a tarball or a git url in npm dependencies (but not a straight JS file): https://docs.npmjs.com/files/package.json

If you really don't want to copy the file (IE - you want the code to always pick up updates) you can fetch it from within your node app.js and go from there. But keep in mind your node app (not just wherever you run npm install) will always need access to punch out to the internet - which probably isn't a problem for a small DIY app.
suzzer99 is offline   Reply With Quote
Old Today, 04:43 PM   #37312
microbet
Solar Powered
 
microbet's Avatar
 
Join Date: Jan 2005
Location: SoCal
Posts: 51,550
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

a) I do get that and can use the functions in the console on a page that was loaded that had that script tag. I think that's what you're saying. I was wondering if I could just go to console on say - Google - or 2p2 - or w/e - and I guess load the api from there.

b) Something like this has to work and I'll get it, but it's really effing hard (for me anyway). The js file has a bunch of window.*** and document.*** that it's generating, but won't work and I see it does a bunch of stuff to grab a bunch of other js files. I can see the requests and responses in the network traffic of the console.
microbet is offline   Reply With Quote
Old Today, 05:27 PM   #37313
suzzer99
Carpal \'Tunnel
 
suzzer99's Avatar
 
Join Date: Nov 2005
Location: on top of the bell curve
Posts: 90,038
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Window isn't going to work in node - it's a browser construct. Why do you need this to run on the server-side and not in the browser?

Seems like that API thing is a dynamically generated script based on the parameters you send. So you're not gonna be able to run it on the server side. Is it open source?
suzzer99 is offline   Reply With Quote
Old Today, 05:47 PM   #37314
microbet
Solar Powered
 
microbet's Avatar
 
Join Date: Jan 2005
Location: SoCal
Posts: 51,550
Re: ** UnhandledExceptionEventHandler :: OFFICIAL LC / CHATTER THREAD **

Quote:
Originally Posted by suzzer99 View Post
Window isn't going to work in node - it's a browser construct. Why do you need this to run on the server-side and not in the browser?

Seems like that API thing is a dynamically generated script based on the parameters you send. So you're not gonna be able to run it on the server side. Is it open source?
I'm basically reverse-engineering a web page that takes user data, uses these functions and renders the page, only I don't know how I'm going to receive the data and it won't only be from web pages. I just need to receive the same data and functions, but my output is a pdf. My input could come from anywhere and my output could go to anywhere (it's the stream, not the file).

I don't know where my app will run, if it will be on a server of some kind or if someone will just run it as a script.

I just finished a similar project, but they had an npm package for that and I just had to install npm, require it and then I could call all the functions.
microbet is offline   Reply With Quote

Reply
      

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Forum Jump


All times are GMT -4. The time now is 06:09 PM.


Powered by vBulletin®
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Copyright © 2008-2017, Two Plus Two Interactive
 
 
Poker Players - Streaming Live Online