Two Plus Two Publishing LLC Two Plus Two Publishing LLC
 

Go Back   Two Plus Two Poker Forums > >

Notices

Programming Discussions about computer programming

Reply
 
Thread Tools Display Modes
Old 11-14-2017, 10:21 PM   #1801
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 23,264
Re: Programming homework and newbie help thread

Code:
select t.thread_id, max(c.last_updated)
from threads t
join comments c using (thread_id)
order by max(c.last_updated) desc
limit 5
That should do it. For this to be scalable, you'll want the comments table to have an index on thread_id,last_updated (a joint index)
RustyBrooks is offline   Reply With Quote
Old 11-14-2017, 10:47 PM   #1802
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

That is super vendor specific SQL. I have never seen that formulation before, where you're using MAX in the select list without a GROUP BY, is that a thing you're allowed to do in whatever version of SQL that is? Joining to threads table also seems pointless, but maybe it facilitates doing that. USING and LIMIT are vendor specific.

This would be my version in SQL Server, which I think is valid ANSI SQL:

Code:
SELECT TOP 5 thread_id, MAX(last_updated) AS most_recent
FROM comments
GROUP BY thread_id
ORDER BY MAX(last_updated)
ChrisV is offline   Reply With Quote
Old 11-15-2017, 12:12 AM   #1803
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 23,264
Re: Programming homework and newbie help thread

using is veeeeeeery common

The join is not necessary in that specific query, but he is almost surely going to want to get thread title etc at the same time.

I left off the group by on accident.

"top 5" is no more or less vendor specific than "limit 5"
https://en.wikipedia.org/wiki/Select...#Result_limits
RustyBrooks is offline   Reply With Quote
Old 11-15-2017, 12:29 AM   #1804
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

Yeah OK. The top/limit clause in ISO SQL 2008 seems unnecessarily verbose (in SQL?! surely not).
ChrisV is offline   Reply With Quote
Old 11-15-2017, 01:07 AM   #1805
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 23,264
Re: Programming homework and newbie help thread

The point is well made, though.... every database has it's own way of doing something that was not part of standard sql in 1999. That is one thing ORMs are good for... they will typically hide some finer points of syntax for you.

The bad part is, they hide some finer points of syntax from you.
RustyBrooks is offline   Reply With Quote
Old 11-15-2017, 10:31 PM   #1806
RustyBrooks
Carpal \'Tunnel
 
RustyBrooks's Avatar
 
Join Date: Feb 2006
Location: Austin, TX
Posts: 23,264
Re: Programming homework and newbie help thread

Anyone good with Java and Spring? I am driving myself ****ing nuts.

I have a spring boot application that has both an HTTP and HTTPS port configured. 8080 and 8443 respectively. I have the certs all set up and they both work fine - I can access any endpoints from either HTTP or HTTPS.

But what I actually want is to serve SOME from HTTP and the rest from HTTPS. The docs for this look simple and the examples look simple, and it builds and runs, but it doesn't ****ing work, I can still access all the endpoints from either HTTP or HTTPS.

So now I'm trying something even simpler - expose both ports but only allow connections on HTTPS. Doesn't work, still allows both.

This is probably the relevant section but I can probably throw up a minimal POC if need be:

Code:
@Configuration
@EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        System.out.println("CONFIGURE CALLED");
        http.csrf().disable();
        http.requiresChannel().anyRequest().requiresSecure();
    }
}
RustyBrooks is offline   Reply With Quote
Old 12-05-2017, 05:36 AM   #1807
locknopair
journeyman
 
Join Date: Dec 2010
Posts: 380
Re: Programming homework and newbie help thread

taking a shot here hoping that someone can help. i am writing a program in java using sports lines and i currently need to extract url data from the cells in this table (fwiw i am using netbeans as my ide and jsoup to parse text).

clicking a cell in the table directs to a "line history" page which contains the data that i am looking for; however from what i can tell the only way to access a line history page is by clicking on a cell in the table.

basically i'd like to to turn each row of the table into a game object containing line histories for each individual sportsbook. using jsoup i can retrieve column/row attributes such as id, name and class although i am unable to find a way to get the url to each cell's line history page, which is what i really need to continue writing my program.

i have read the table's html and each cell has an event listener using javascript and jquery (both of which i have not fully learned yet) to direct to each individual line history page. i have wondered maybe about ways to simulate a click, but i don't know. essentially i'm just hoping someone can direct me, even if it means having to learn a new programming language.
locknopair is offline   Reply With Quote
Old 12-05-2017, 06:51 AM   #1808
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

Good news! Writing sports analysis stuff in Java, including a ton of scraping, is what I currently do full time.

Code:
https://free.sportsinsights.com/linehistory.aspx?v_date=12/5/2017&v_nss=711&sport_id=12&sportsbookId=2
That v_nss thing is the away team rotnum for the game, which you can get in the Team cells. This one is Ball State. sport_id and sportsbookId I imagine won't vary, so if the table you're pulling up has the same sport every time, that's all you need to construct your urls.

If you want to dynamically read what's there and you don't know what the sport_id or sportsbookIds you want are in advance, things are a little more complicated. The page does not store those ID numbers in the html, they're stored in a Javascript object.

Open the table up in Chrome. Right click one of the cells and choose Inspect. In the top right, click the Event Listeners tab. Expand the "click" section. One of the handlers is something like td#blahblah.sportsbook. Click the link to Main.js next to it. That shows you the Javascript code which retrieves the id numbers when you click the cell.

So it's using this eventModel thing. After a brief sojourn through the Javascript source, I discovered where it populates that from. Do a View Source on the main table page and search for "var rawData". It's a JSON document in some horrifyingly gross format which has all the data you need in it. So your choices are:

1) Parse that JSON to get the information you need. It probably won't be too bad, you just need to get those IDs, you don't need to understand all of whatever is going on with all those fricking underscores.

or

2) Use a headless browser - I use HtmlUnit - to load the page and inject Javascript into it. You can then use their Javascript machinery to access whatever you want (eventModel is a global variable). HtmlUnit is capable of returning that data into Java for you to use. It's a technique worth bearing in mind even if you don't use it here.

Godspeed!

Last edited by ChrisV; 12-05-2017 at 07:01 AM.
ChrisV is offline   Reply With Quote
Old 12-05-2017, 08:15 PM   #1809
locknopair
journeyman
 
Join Date: Dec 2010
Posts: 380
Re: Programming homework and newbie help thread

wow. thanks man, what an awesome post. so now that i can get the rotation number, the sport_id and the sportsbookId, i can just pass them into a method as parameters and return a concatenated string url to use in jsoup.

earlier though i discovered one other small issue and i figure i'll just ask instead of getting lost in a google rabbit hole. on a line history page, there is a drop-down menu in the top right-hand corner which contains "spread", "moneyline", "over/under", and "additional spread". since the table is on "spread" by default, i've been able to use this path to access and loop through the table with jsoup:

Code:
html body form#f div.content table.layout tbody tr td table.tableStatic tbody
and here is (i think) the drop-down menu path with the second selection, "moneyline":

Code:
html body form#f div.content div.title div#options div select#LineTypeDropDownList.ddl option#LineTypeDropDownList > option:nth-child(2)
i've been trying for a while now to create a path back to the table using the new menu selection without any success. sorry if this is a basic question but keep in mind i'm not a pro at this.
locknopair is offline   Reply With Quote
Old 12-05-2017, 08:59 PM   #1810
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

I'm not familiar with jsoup but why do you need a full path like that? Can't you just use doc.select("table.tableStatic > tbody")?
ChrisV is offline   Reply With Quote
Old 12-05-2017, 10:36 PM   #1811
well named
Carpal \'Tunnel
 
well named's Avatar
 
Join Date: Jun 2007
Location: esse est coesse
Posts: 73,494
Re: Programming homework and newbie help thread

I've used jsoup in scala (with scala-scraper) and I'd assume it works the same way, in which case yeah, you don't need to use the full path through the document. The fragment should be fine.
well named is offline   Reply With Quote
Old 12-05-2017, 11:32 PM   #1812
locknopair
journeyman
 
Join Date: Dec 2010
Posts: 380
Re: Programming homework and newbie help thread

Quote:
Originally Posted by ChrisV View Post
I'm not familiar with jsoup but why do you need a full path like that? Can't you just use doc.select("table.tableStatic > tbody")?
i'm new to jsoup so i've just been using full paths inside .select() to access the tables because i can just copy the paths via 'inspect element' in firefox, however your query worked as well and is much simpler. how would i alter your statement to get table data based on what option is selected in the line history drop-down menu?
locknopair is offline   Reply With Quote
Old 12-06-2017, 02:04 AM   #1813
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

That's going to be a problem. Changing the drop down list reloads the page. The technology being used is ASP.NET Webforms. The selection in the drop down list is sent back to the server (bundled up with a bunch of other information about everything on the page) in an encoded field called __VIEWSTATE that you can see if you view-source on one of those tables. ViewState has tamper protection built into it. You can file all this under "things that Microsoft thought were a good idea in like 2001".

The only way I can really think of to do it is going to be to use a headless browser like HtmlUnit and get it to change the drop down list just like a user would, then wait for the page to reload each time. You can then grab the HTML and give it to jsoup.
ChrisV is offline   Reply With Quote
Old 12-06-2017, 02:12 AM   #1814
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

From memory ViewState is base 64 encoded and it sends a hash of ViewState to the server as well as the ViewState itself, so the tamper protection is really against man in the middle attacks. I think you can spoof the viewstate if you try hard enough, but I think that's going to be a nightmare compared to just using a headless browser.
ChrisV is offline   Reply With Quote
Old 12-06-2017, 02:15 AM   #1815
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

I have no idea what jsoup's capabilities are btw, so maybe it can do this stuff and you don't need HtmlUnit.
ChrisV is offline   Reply With Quote
Old 12-06-2017, 05:10 AM   #1816
locknopair
journeyman
 
Join Date: Dec 2010
Posts: 380
Re: Programming homework and newbie help thread

well what i like about programming is that there's a way to do almost anything, so i'm going to get htmlunit and learn what it can do. as things stand i can get all the spread histories so i'm in a pretty good spot. the important thing is that thanks to your help i now know how to construct the urls and am no longer at a dead end in my project.

Last edited by locknopair; 12-06-2017 at 05:15 AM.
locknopair is offline   Reply With Quote
Old 12-13-2017, 08:58 AM   #1817
locknopair
journeyman
 
Join Date: Dec 2010
Posts: 380
Re: Programming homework and newbie help thread

hey guys really hit a roadblock here and hoping someone would be kind enough to spend a few minutes and have a look at this. i am working with numerous tables, yet only one is giving me a problem i just cannot solve despite my endless hours of trying. i know this is long, so please bear with me.

i have written a loop (included at the end of this post) that i am using to work with this table (the same one i posted before) whose url contains an attribute called "SportGroup", which determines which sport table to display; for example: "&SportGroup=sg6" is ncaab, "sg1" is nfl, "sg2" is nba, at the end of a table url:

Code:
https://free.sportsinsights.com/free-odds/free-odds-frame.aspx?MaxColumns=100&LineOption=Combined&EventOption=undefined&SortBy=undefined&Previous=&Yesterday=&ShowPercents=&SportGroup=sg6
however no matter which sport table i connect to (in this case ncaab, or "sg6"), i can only extract table data from all sports tables despite trying any and every combination of pseudo-selector. clicking on 'inspect' in the ncaab table, you can see each row uses one of two classes: ".row-odd" for rows relevant to the current ncaab table, and ".row-odd hidden" which hides rows related to all other sports. you'd think using a pseudo-selector for ".row-odd" while connecting my doc to the ncaab sport url above would extract only ncaab results, but instead i get results for all sports (from all tables).

also under 'inspect' you can see each table row has an "id" attribute and a "name" attribute; however the "id" attributes for all sport types are just a bunch of randomly generated numbers which appear to have no distinct pattern toward a given sport (and thus appear essentially useless). and every "name" attribute = 0, so it seems there is no way to sort with either of these two remaining row attributes.

i have studied the html and some of the .js files and cannot seem to find anything which would allow me to sort these tables by sport; however i did notice that in the ncaab table url above there are two "undefined" attributes: one called "EventOption=undefined" and the other called "SortBy=undefined". i tried plugging in values like 1, 2, and 6 as well as ncaab, nfl, and nba with no success and also tried searching the page source for these attributes but found nothing useful... although maybe defining these attributes will solve my problem?

i'm out of ideas here and have really tried everything i can think of so maybe someone here can help. usually i would just avoid a table causing this much trouble however i need the available aggregated percentages listed for each team. hopefully there is a common solution to this problem that i'm just overlooking and not something related to __viewstate or something more difficult. huge thanks to anyone that helps or provides some insight. here's my table loop in case someone sees something i'm missing:

Code:
	try
	{
	    Document doc = Jsoup.connect("https://free.sportsinsights.com/free-odds/free-odds-frame.aspx?MaxColumns=100&LineOption=Spread&EventOption=undefined&SortBy=undefined&Previous=&Yesterday=&ShowPercents=&SportGroup=sg6").get();

	    for (Element table : doc.select(".tableOdds"))
	    {	
		for (Element trs : table.select("tr:not(.row-odd hidden)"))
		{
		    Elements tds = trs.select("td");
		    if (tds.size() > 13)
		    {
			System.out.println(tds.get(0).text() + "   " + tds.get(1).text() + "   " 
				+ tds.get(2).text() + "   " + tds.get(3).text() + "   " 
				+ tds.get(4).text() + "   " + tds.get(5).text() + "   " 
				+ tds.get(6).text() + "   " + tds.get(7).text() + "   "
				+ tds.get(8).text() + "   " + tds.get(9).text() + "   " 
				+ tds.get(10).text() + "   " + tds.get(11).text() + "   " 
				+ tds.get(12).text() + "   " + tds.get(13).text() + "   "
				+ tds.get(14).text());
		    }
		}
	    }
	}
locknopair is offline   Reply With Quote
Old Yesterday, 07:20 AM   #1818
md46135
journeyman
 
Join Date: Mar 2015
Location: where they don't respect my raises
Posts: 222
Re: Programming homework and newbie help thread

I have a really weird problem with Python 2.7, it looks like I can't run any kind of script! copy-pasting into shell works normal, but when I try to run a script (even the simplest script you can imagine), I get this error.
Code:
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python27\lib\lib-tk\Tkinter.py", line 1541, in __call__
    return self.func(*args)
  File "C:\Python27\lib\idlelib\MultiCall.py", line 166, in handler
    r = l[i](event)
  File "C:\Python27\lib\idlelib\ScriptBinding.py", line 149, in run_module_event

    if PyShell.use_subprocess:
AttributeError: 'module' object has no attribute 'use_subprocess'
I tried uninstalling and reinstalling different versions, but I always get this. Is there something that can be done about this? seems like noone on the internet had this problem
md46135 is offline   Reply With Quote
Old Yesterday, 07:30 AM   #1819
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

locknopair, i will attempt to understand your post at some juncture at which I am not stoned.
ChrisV is offline   Reply With Quote
Old Yesterday, 07:38 AM   #1820
ChrisV
Carpal \'Tunnel
 
ChrisV's Avatar
 
Join Date: Jul 2004
Location: Adelaide, Australia
Posts: 34,953
Re: Programming homework and newbie help thread

Right off the bat though, without having looked through everything, here's a problem:

Code:
table.select("tr:not(.row-odd hidden)")
Spaces in selectors select descendant elements, what ".row-odd hidden" would select is the bolded here:

Code:
<tr class="row-odd">
    ...
    <hidden></hidden>
    ...
</tr>
But you're also specifying that it's a tr element, so your whole selector means "find me tr elements which are NOT also <hidden> elements which are descendants of elements with a row-odd class". Since no tr elements match that second part, jsoup obligingly selects all tr elements.

What you want is simply:

Code:
table.select("tr:not(.hidden)")
Class attributes allow multiple space-separated classes, so class="row-odd hidden" just gives the element both classes.
ChrisV is offline   Reply With Quote
Old Yesterday, 07:48 AM   #1821
Mr.mmmKay
Drugs are bad...
 
Mr.mmmKay's Avatar
 
Join Date: May 2011
Posts: 13,730
Re: Programming homework and newbie help thread

Quote:
Originally Posted by md46135 View Post
I have a really weird problem with Python 2.7, it looks like I can't run any kind of script! copy-pasting into shell works normal, but when I try to run a script (even the simplest script you can imagine), I get this error.
Code:
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python27\lib\lib-tk\Tkinter.py", line 1541, in __call__
    return self.func(*args)
  File "C:\Python27\lib\idlelib\MultiCall.py", line 166, in handler
    r = l[i](event)
  File "C:\Python27\lib\idlelib\ScriptBinding.py", line 149, in run_module_event

    if PyShell.use_subprocess:
AttributeError: 'module' object has no attribute 'use_subprocess'
I tried uninstalling and reinstalling different versions, but I always get this. Is there something that can be done about this? seems like noone on the internet had this problem
The exception at the bottom of this page http://level1wiki.wikidot.com/errormessages looks kind of similar, could it be same problem?

Are you running the scripts from the command line?
Mr.mmmKay is online now   Reply With Quote
Old Yesterday, 08:04 AM   #1822
md46135
journeyman
 
Join Date: Mar 2015
Location: where they don't respect my raises
Posts: 222
Re: Programming homework and newbie help thread

Quote:
Originally Posted by Mr.mmmKay View Post
The exception at the bottom of this page http://level1wiki.wikidot.com/errormessages looks kind of similar, could it be same problem?

Are you running the scripts from the command line?
no, I open PyShell, then file->open and run the script from there.
md46135 is offline   Reply With Quote
Old Yesterday, 08:07 AM   #1823
Mr.mmmKay
Drugs are bad...
 
Mr.mmmKay's Avatar
 
Join Date: May 2011
Posts: 13,730
Re: Programming homework and newbie help thread

Could be a PyShell problem, maybe try running the script from the operating system command line.

Last edited by Mr.mmmKay; Yesterday at 08:24 AM.
Mr.mmmKay is online now   Reply With Quote
Old Yesterday, 08:17 AM   #1824
md46135
journeyman
 
Join Date: Mar 2015
Location: where they don't respect my raises
Posts: 222
Re: Programming homework and newbie help thread

Quote:
Originally Posted by Mr.mmmKay View Post
Could be a PyShell problem maybe try running the script from the operating system command line.
yeah, looks like PyShell is broken. in idlelib there is also a file called "idle" which when ran looks exactly like PyShell, I tried to run the script with that and it works. wish I knew about that file before.
md46135 is offline   Reply With Quote
Old Yesterday, 07:58 PM   #1825
Loki
Formerly Noodle Wazlib
 
Loki's Avatar
 
Join Date: Nov 2015
Location: Participating in buffoonery
Posts: 11,593
Re: Programming homework and newbie help thread

Anyone familiar with google apps scripts?

I’m curious if this is even possible, but I have a sheet I want tracked. There's a field that changes based on the date and I want an email to be automatically sent when the change happens. Problem is, I don’t want the sheet to actually have to be open for the script to be run.

Is that possible?

https://developers.google.com/apps-s...uides/triggers indicates doGet and doPost can be used to run your trigger/script (hopefully those really are basically the same thing, I’m new to this) when a web app receives a post/get request. Could I set up a recurring task on my computer to send a get request that would make the script process?

Thanks for any advice or direction

Edit

I guess there are time-driven triggers, so that may work too, if the actual sheet doesn’t need to be up and running.
Loki is offline   Reply With Quote

Reply
      

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Forum Jump


All times are GMT -4. The time now is 10:12 AM.


Powered by vBulletin®
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © 2008-2017, Two Plus Two Interactive
 
 
Poker Players - Streaming Live Online