Which Politics regs write at the highest grade level?
04-25-2014
, 04:57 PM
04-25-2014
, 07:27 PM
"
04-25-2014
, 09:24 PM
04-25-2014
, 09:26 PM
04-25-2014
, 09:29 PM
04-25-2014
, 09:32 PM
04-26-2014
, 03:37 AM
04-26-2014
, 03:38 AM
04-29-2014
, 01:35 AM
Quote:
So apparently you got yourself banned, but I'll answer briefly anyway.
The scraper, written in java, is really just a series of tedious brute force extractions from the html pages based on based on deterministic triggers for each piece of data. Although on first glance the html source code for this web page looks like a monstrous pile of random ****, the forum software actually produces content that's highly structured and predictable.
Ultimately, it's just a matter of automatically walking through menu pages, extracting thread URLs, and then walking through the pages of each thread extracting content. The post data is inserted into a very simple database table via a very simple DAO.
Although I doubt my operations would have any impact on a web-based ecosystem as gigantic as twoplustwo, I still throttled my requests to one every 2 seconds.
As for the presentation, I know the charts are not stellar, but for personal project work, I'm basically stuck with an old version of Apple Numbers (part of iWork '08). It gets the job done, but apparently more recent versions have a richer set of chart features (including bubble charts, which I would have liked to apply here also).
I played around with the scatter plot, but I couldn't figure out how to label the individual points, so what we're left with is more of a holistic view of the distribution. Here's one where I added a few labels for some notable outliers. These are the same top 50 posters from earlier charts.
The Politics plot looks compressed compared to Unchained, but that seems to be due to ikestoys putting a leftward squeeze on everybody else (visually at least).
The scraper, written in java, is really just a series of tedious brute force extractions from the html pages based on based on deterministic triggers for each piece of data. Although on first glance the html source code for this web page looks like a monstrous pile of random ****, the forum software actually produces content that's highly structured and predictable.
Ultimately, it's just a matter of automatically walking through menu pages, extracting thread URLs, and then walking through the pages of each thread extracting content. The post data is inserted into a very simple database table via a very simple DAO.
Although I doubt my operations would have any impact on a web-based ecosystem as gigantic as twoplustwo, I still throttled my requests to one every 2 seconds.
As for the presentation, I know the charts are not stellar, but for personal project work, I'm basically stuck with an old version of Apple Numbers (part of iWork '08). It gets the job done, but apparently more recent versions have a richer set of chart features (including bubble charts, which I would have liked to apply here also).
I played around with the scatter plot, but I couldn't figure out how to label the individual points, so what we're left with is more of a holistic view of the distribution. Here's one where I added a few labels for some notable outliers. These are the same top 50 posters from earlier charts.
The Politics plot looks compressed compared to Unchained, but that seems to be due to ikestoys putting a leftward squeeze on everybody else (visually at least).
If you send me your data I can overlay the poster names instead of the crosses. Alternately I can just show you how to do it in Excel.
05-03-2014
, 10:59 PM
05-04-2014
, 02:56 PM
