That should do it. For this to be scalable, you'll want the comments table to have an index on thread_id,last_updated (a joint index)

That is super vendor specific SQL. I have never seen that formulation before, where you're using MAX in the select list without a GROUP BY, is that a thing you're allowed to do in whatever version of SQL that is? Joining to threads table also seems pointless, but maybe it facilitates doing that. USING and LIMIT are vendor specific.

This would be my version in SQL Server, which I think is valid ANSI SQL:

using is veeeeeeery common

The join is not necessary in that specific query, but he is almost surely going to want to get thread title etc at the same time.

I left off the group by on accident.

"top 5" is no more or less vendor specific than "limit 5"
https://en.wikipedia.org/wiki/Select...#Result_limits

Yeah OK. The top/limit clause in ISO SQL 2008 seems unnecessarily verbose (in SQL?! surely not).

The point is well made, though.... every database has it's own way of doing something that was not part of standard sql in 1999. That is one thing ORMs are good for... they will typically hide some finer points of syntax for you.

The bad part is, they hide some finer points of syntax from you.

Anyone good with Java and Spring? I am driving myself ****ing nuts.

I have a spring boot application that has both an HTTP and HTTPS port configured. 8080 and 8443 respectively. I have the certs all set up and they both work fine - I can access any endpoints from either HTTP or HTTPS.

But what I actually want is to serve SOME from HTTP and the rest from HTTPS. The docs for this look simple and the examples look simple, and it builds and runs, but it doesn't ****ing work, I can still access all the endpoints from either HTTP or HTTPS.

So now I'm trying something even simpler - expose both ports but only allow connections on HTTPS. Doesn't work, still allows both.

This is probably the relevant section but I can probably throw up a minimal POC if need be:

taking a shot here hoping that someone can help. i am writing a program in java using sports lines and i currently need to extract url data from the cells in this table (fwiw i am using netbeans as my ide and jsoup to parse text).

clicking a cell in the table directs to a "line history" page which contains the data that i am looking for; however from what i can tell the only way to access a line history page is by clicking on a cell in the table.

basically i'd like to to turn each row of the table into a game object containing line histories for each individual sportsbook. using jsoup i can retrieve column/row attributes such as id, name and class although i am unable to find a way to get the url to each cell's line history page, which is what i really need to continue writing my program.

i have read the table's html and each cell has an event listener using javascript and jquery (both of which i have not fully learned yet) to direct to each individual line history page. i have wondered maybe about ways to simulate a click, but i don't know. essentially i'm just hoping someone can direct me, even if it means having to learn a new programming language.

Good news! Writing sports analysis stuff in Java, including a ton of scraping, is what I currently do full time.

That v_nss thing is the away team rotnum for the game, which you can get in the Team cells. This one is Ball State. sport_id and sportsbookId I imagine won't vary, so if the table you're pulling up has the same sport every time, that's all you need to construct your urls.

If you want to dynamically read what's there and you don't know what the sport_id or sportsbookIds you want are in advance, things are a little more complicated. The page does not store those ID numbers in the html, they're stored in a Javascript object.

Open the table up in Chrome. Right click one of the cells and choose Inspect. In the top right, click the Event Listeners tab. Expand the "click" section. One of the handlers is something like td#blahblah.sportsbook. Click the link to Main.js next to it. That shows you the Javascript code which retrieves the id numbers when you click the cell.

So it's using this eventModel thing. After a brief sojourn through the Javascript source, I discovered where it populates that from. Do a View Source on the main table page and search for "var rawData". It's a JSON document in some horrifyingly gross format which has all the data you need in it. So your choices are:

1) Parse that JSON to get the information you need. It probably won't be too bad, you just need to get those IDs, you don't need to understand all of whatever is going on with all those fricking underscores.

or

2) Use a headless browser - I use HtmlUnit - to load the page and inject Javascript into it. You can then use their Javascript machinery to access whatever you want (eventModel is a global variable). HtmlUnit is capable of returning that data into Java for you to use. It's a technique worth bearing in mind even if you don't use it here.

Godspeed!

wow. thanks man, what an awesome post. so now that i can get the rotation number, the sport_id and the sportsbookId, i can just pass them into a method as parameters and return a concatenated string url to use in jsoup.

earlier though i discovered one other small issue and i figure i'll just ask instead of getting lost in a google rabbit hole. on a line history page, there is a drop-down menu in the top right-hand corner which contains "spread", "moneyline", "over/under", and "additional spread". since the table is on "spread" by default, i've been able to use this path to access and loop through the table with jsoup:

and here is (i think) the drop-down menu path with the second selection, "moneyline":

i've been trying for a while now to create a path back to the table using the new menu selection without any success. sorry if this is a basic question but keep in mind i'm not a pro at this.

I'm not familiar with jsoup but why do you need a full path like that? Can't you just use doc.select("table.tableStatic > tbody")?

I've used jsoup in scala (with scala-scraper) and I'd assume it works the same way, in which case yeah, you don't need to use the full path through the document. The fragment should be fine.

i'm new to jsoup so i've just been using full paths inside .select() to access the tables because i can just copy the paths via 'inspect element' in firefox, however your query worked as well and is much simpler. how would i alter your statement to get table data based on what option is selected in the line history drop-down menu?

That's going to be a problem. Changing the drop down list reloads the page. The technology being used is ASP.NET Webforms. The selection in the drop down list is sent back to the server (bundled up with a bunch of other information about everything on the page) in an encoded field called __VIEWSTATE that you can see if you view-source on one of those tables. ViewState has tamper protection built into it. You can file all this under "things that Microsoft thought were a good idea in like 2001".

The only way I can really think of to do it is going to be to use a headless browser like HtmlUnit and get it to change the drop down list just like a user would, then wait for the page to reload each time. You can then grab the HTML and give it to jsoup.

From memory ViewState is base 64 encoded and it sends a hash of ViewState to the server as well as the ViewState itself, so the tamper protection is really against man in the middle attacks. I think you can spoof the viewstate if you try hard enough, but I think that's going to be a nightmare compared to just using a headless browser.

I have no idea what jsoup's capabilities are btw, so maybe it can do this stuff and you don't need HtmlUnit.

well what i like about programming is that there's a way to do almost anything, so i'm going to get htmlunit and learn what it can do. as things stand i can get all the spread histories so i'm in a pretty good spot. the important thing is that thanks to your help i now know how to construct the urls and am no longer at a dead end in my project.

hey guys really hit a roadblock here and hoping someone would be kind enough to spend a few minutes and have a look at this. i am working with numerous tables, yet only one is giving me a problem i just cannot solve despite my endless hours of trying. i know this is long, so please bear with me.

i have written a loop (included at the end of this post) that i am using to work with this table (the same one i posted before) whose url contains an attribute called "SportGroup", which determines which sport table to display; for example: "&SportGroup=sg6" is ncaab, "sg1" is nfl, "sg2" is nba, at the end of a table url:

however no matter which sport table i connect to (in this case ncaab, or "sg6"), i can only extract table data from all sports tables despite trying any and every combination of pseudo-selector. clicking on 'inspect' in the ncaab table, you can see each row uses one of two classes: ".row-odd" for rows relevant to the current ncaab table, and ".row-odd hidden" which hides rows related to all other sports. you'd think using a pseudo-selector for ".row-odd" while connecting my doc to the ncaab sport url above would extract only ncaab results, but instead i get results for all sports (from all tables).

also under 'inspect' you can see each table row has an "id" attribute and a "name" attribute; however the "id" attributes for all sport types are just a bunch of randomly generated numbers which appear to have no distinct pattern toward a given sport (and thus appear essentially useless). and every "name" attribute = 0, so it seems there is no way to sort with either of these two remaining row attributes.

i have studied the html and some of the .js files and cannot seem to find anything which would allow me to sort these tables by sport; however i did notice that in the ncaab table url above there are two "undefined" attributes: one called "EventOption=undefined" and the other called "SortBy=undefined". i tried plugging in values like 1, 2, and 6 as well as ncaab, nfl, and nba with no success and also tried searching the page source for these attributes but found nothing useful... although maybe defining these attributes will solve my problem?

i'm out of ideas here and have really tried everything i can think of so maybe someone here can help. usually i would just avoid a table causing this much trouble however i need the available aggregated percentages listed for each team. hopefully there is a common solution to this problem that i'm just overlooking and not something related to __viewstate or something more difficult. huge thanks to anyone that helps or provides some insight. here's my table loop in case someone sees something i'm missing:

I have a really weird problem with Python 2.7, it looks like I can't run any kind of script! copy-pasting into shell works normal, but when I try to run a script (even the simplest script you can imagine), I get this error.
I tried uninstalling and reinstalling different versions, but I always get this. Is there something that can be done about this? seems like noone on the internet had this problem

locknopair, i will attempt to understand your post at some juncture at which I am not stoned.

Right off the bat though, without having looked through everything, here's a problem:

Spaces in selectors select descendant elements, what ".row-odd hidden" would select is the bolded here:

But you're also specifying that it's a tr element, so your whole selector means "find me tr elements which are NOT also <hidden> elements which are descendants of elements with a row-odd class". Since no tr elements match that second part, jsoup obligingly selects all tr elements.

What you want is simply:

Class attributes allow multiple space-separated classes, so class="row-odd hidden" just gives the element both classes.

The exception at the bottom of this page http://level1wiki.wikidot.com/errormessages looks kind of similar, could it be same problem?

Are you running the scripts from the command line?

no, I open PyShell, then file->open and run the script from there.

Could be a PyShell problem, maybe try running the script from the operating system command line.

yeah, looks like PyShell is broken. in idlelib there is also a file called "idle" which when ran looks exactly like PyShell, I tried to run the script with that and it works. wish I knew about that file before.

Anyone familiar with google apps scripts?

I’m curious if this is even possible, but I have a sheet I want tracked. There's a field that changes based on the date and I want an email to be automatically sent when the change happens. Problem is, I don’t want the sheet to actually have to be open for the script to be run.

Is that possible?

https://developers.google.com/apps-s...uides/triggers indicates doGet and doPost can be used to run your trigger/script (hopefully those really are basically the same thing, I’m new to this) when a web app receives a post/get request. Could I set up a recurring task on my computer to send a get request that would make the script process?

Thanks for any advice or direction

Edit

I guess there are time-driven triggers, so that may work too, if the actual sheet doesn’t need to be up and running.