Open Side Menu Go to the Top
Register
Programming homework and newbie help thread Programming homework and newbie help thread

12-14-2017 , 08:35 PM
It seems like triggers are going to work, and I think I got it mostly figured out. Ignore me, unless someone has some super great tips and tricks for this
Programming homework and newbie help thread Quote
12-15-2017 , 12:11 AM
Quote:
Originally Posted by ChrisV
locknopair, i will attempt to understand your post at some juncture at which I am not stoned.
thanks man. really hoping you can help.

btw i tried ".hidden" in the selector like you suggested, however the output was oddly all the games from the nfl table, despite connecting my doc with the ncaab table. something is weird about this site, since other tables i am working with (at other sites) have given me zero issue.
Programming homework and newbie help thread Quote
12-15-2017 , 12:31 AM
I tried out tr:not(.hidden) in Chrome and it works fine. The problem is that the sports are dynamically shown or hidden by script after the page loads. Pages loaded in a browser can end up looking very different than if you just download the HTML document. This is why using HtmlUnit and injecting script is more flexible. If you don't want to go that route, you'll need to iterate over all the rows and find tr.row-group td.team elements which have NCAAB in the cell text. Then continue to iterate, parsing rows, until you hit another tr.row-group element. No clue how you go about doing that in jsoup.
Programming homework and newbie help thread Quote
12-15-2017 , 12:33 AM
You could also just use HtmlUnit to load the page, wait for all the script stuff to happen, then dump the resulting HTML to jsoup. That would probably be the quickest thing to do as you wouldn't need to change the rest of your code. I'm assuming there that HtmlUnit is capable of dumping DOM documents into HTML. I don't know for certain if it can, I'd be surprised if it can't though.
Programming homework and newbie help thread Quote
12-15-2017 , 12:41 AM
To see the actual document you're giving jsoup, open Chrome devtools (F12) with the sportsinsight page open. Go to Network, up the top, then click Doc in the gray bar below that. Reload the page. Click the "free-odds-frame.aspxblahblahetc" thing. Under Response, after the page loads, you'll see the raw document you get when you request that page.
Programming homework and newbie help thread Quote
12-15-2017 , 12:52 PM
Quote:
Originally Posted by Loki
It seems like triggers are going to work, and I think I got it mostly figured out. Ignore me, unless someone has some super great tips and tricks for this
I may ping you about this in the next 2 weeks if I run into trouble. I run a playoff fantasy football league on google app engine, and for the past few years I've been manually locking lineups every week sometime on gameday. I'm going to look into having it auto-lock 5 mins before kickoff to reduce my overhead. While it's not quite the same platform as yours it might be done similarly.
Programming homework and newbie help thread Quote
12-15-2017 , 01:17 PM
Yeah, the sheets API looks like it does have something for your purpose. Would appear to be a pretty easy foreach loop, but some of the setup is a little weird.

I saved links in case you have questions.
Programming homework and newbie help thread Quote
12-16-2017 , 05:39 AM
Quote:
Originally Posted by ChrisV
You could also just use HtmlUnit to load the page, wait for all the script stuff to happen, then dump the resulting HTML to jsoup. That would probably be the quickest thing to do as you wouldn't need to change the rest of your code. I'm assuming there that HtmlUnit is capable of dumping DOM documents into HTML. I don't know for certain if it can, I'd be surprised if it can't though.
hey man it works. although after using htmlunit for a day i can see i've got a lot to teach myself. however i did find some cool stuff that i might use in my program. so thanks again for all the help.

Code:
	    WebClient client = new WebClient(BrowserVersion.CHROME);
	    client.getOptions().setJavaScriptEnabled(true);
	    
	    HtmlPage webPage = client.getPage("https://free.sportsinsights.com/free-odds/free-odds-frame.aspx?MaxColumns=100&LineOption=Spread&SportGroup=sg6");
	    client.close();
	    
	    String webPageToParse = webPage.asXml();
	    Document doc = Jsoup.parse(webPageToParse);
Programming homework and newbie help thread Quote
12-16-2017 , 09:58 AM
Programming homework and newbie help thread Quote
12-23-2017 , 12:08 PM
Anyone know why awk will start printing a different column than asked for in large csv files?

I'm saying something as simple as awk 'print $71' will have big sections of what is column 72.

Never mind, it's extra commas in an address field in the stupid data.

Last edited by kerowo; 12-23-2017 at 12:18 PM.
Programming homework and newbie help thread Quote
12-28-2017 , 02:22 PM
Related to my google apps script question previously, does anyone have any experience using Microsoft Office 365/Sharepoint REST API?

I would love to incorporate some updated data directly from Sharepoint into my sheets doc, and since apps script uses JavaScript I feel like maybe this is possible.

So, say I’m keeping track of a list of documents. I want to update my sheets spreadsheet automatically if the "last edited by" field changes in Sharepoint. Doable?

Just looking into this now, so maybe it’s easy, but at my skill level I highly doubt it.
Programming homework and newbie help thread Quote
12-31-2017 , 05:34 PM
I need a mentor to help/give guidance with using Python Dash/plotly. Willing to pay a reasonable rate. Hesitant to put code on here as it is work related and I figure having a mentor will enable to ask a lot more questions/pick up things a lot quicker. Please PM if interested.
Programming homework and newbie help thread Quote
04-03-2018 , 04:07 PM
I need to write a selection algorithm to find the k'th largest elements in an array.

Caveat - the values of the array can only be accessed through a compare(a, b) method which returns a value depending if a < b or a > b. All elements are unique.

The hardest thing is that the expected worst case # of compare(a, b) calls needs to be < 12,000 for n = 10,000 and k = 40.

Quick select algorithm can be done using buckets and "median of medians" method for choosing pivot for an expected run time of O(n) but with the current constraints I think this will result in way too many compares.

my idea is to keep a BST array of size 40 and keep track of the largest and smallest elements but this can result in far too many compares whenever you need to update the array - log(40) which is ~5. Doing some quick and dirty EV calcs and I think the EV of compares for an update to the array is ~2.5.

I think this will still result in far too many compares. Professor says there's an O(n) solution that exists.

any ideas without telling me a solution?
Programming homework and newbie help thread Quote
04-04-2018 , 02:53 AM
BST array, what you described is a max heap and with my napkin math you will hit 50k comparisons.

I would advise you do brute force and use your benchmarks and go from there. Priority is getting a working solution and then you can venture and try different methods.
Programming homework and newbie help thread Quote
04-04-2018 , 06:47 AM
I guess it just has to be an observed worst case of <12k over 1000 runs, so something with fairly high probability could work
Programming homework and newbie help thread Quote
04-04-2018 , 07:04 AM
i think a modified tournament selection tracking the winners of each comparison probably results in the fewest compares but will use a huge amount of space. but that isn't really a problem for this project
Programming homework and newbie help thread Quote
04-04-2018 , 03:23 PM
I’m urgently looking for someone that knows Delphi(pascal) language . Please PM me or email me gaysaak@gmail.com TY
Programming homework and newbie help thread Quote
04-04-2018 , 07:39 PM
I’m going to try to take advantage of the fact that there’s no real limitations on anything but calls to compare, so if i track the results of each tournament, say an array for each index and all the other indeces that it beat, and check those arrays each time there’s a new comparison, (if a < b and b < c then a < c) i think i can run k = 40 tournaments and by the 10th one or so there will be basically no new comparisons left to do.

First tournament is 10k comparisons, then the second will be a few hundred, by the end practically 0. That’s the plan anyway
Programming homework and newbie help thread Quote
04-16-2018 , 02:06 PM
i finished it with an avg case of ~10,550

Gotta go below 10,500 to get an extra 5 points credit, I know how i can do it but it may make the program much slower.

this might be the most beautiful piece of code i've ever written. It was easy -

basically you just run a large tournament on the entire array, storing the results in an array of integer arrays. for example, if index 3 beats index 2, you go to results[3] and store index 2 in that array (first slot available).

This requires maintaining current size of the sub-arrays, which I used a separate array for. I could have used the 0 index for size, but this made things really messy on my tournament algorithms.

once you have the largest index from n elements, you run a smaller tournament on that index's subarray. The 2nd largest is guaranteed to be in that array.

Then you run a tourney on the 3rd largest, etc.... all the way to k'th largest.

Things I learned - allocating memory on the heap is REALLY ****ing slow. with my heap implementation, the n = 10,000 and k = 40 would run in about 0.4 seconds, which seems fine, but this algorithm is supposed to run 2000 times.

So i used global static variables for the arrays, but ran into issues with them retaining previous run data. So my initial solution to that was to just go through each index of each array and resetting the values, but this ended up in a really nasty n^2 loop that ran over 10,000^2 array indeces, lol.

so i did some C trickery and fixed that and now it runs all 2000 trials in about half a second.

my partner was worthless, I should show you the code he sent me. I just told him to implement a function that takes 2 indeces of an array and return the largest element. It was an unreadable disaster and didn't work so I just rewrote it myself.

not sure if i should tell him b/c i don't want hurt feelings.
Programming homework and newbie help thread Quote
04-23-2018 , 01:36 PM
Noob question regarding a simple Java simulation I wrote, with some background first. I'm on a 2-core laptop with hyper-threading turned off. Originally I wrote the simulation as a single thread, then decided I wanted to double the speed by using 100% cpu instead of 50%. I did some googling on multi-threading (which I'm new to) and was able to imitate the examples I found. To keep it as simple as possible, I avoided using thread synchronization and I think I successfully did so without problems (but correct me if I'm wrong).

I wrote two multi-threaded versions. They both use close to 100% CPU as intended, and their numerical outputs are the same.

Version A uses one thread per each element of FRAC[ ] (splitting into chunks if FRAC.length > #cores).

In Version B, FRAC is not an array. Version B splits the sample into subsamples of size N/(#cores). It uses one thread for each subsample.

Here's where I think it gets weird. I used a stopwatch to compare the speeds using the same parameters (a 1-million trial sample for each FRAC). When I test a single FRAC, Version A finishes in about the same time as Version B despite using a single thread and half the CPU. Likewise, when version A has a 2-element FRAC array, it finishes in about the same time, making it twice as fast as B when both are multi-threading.

So my question is, why isn't B roughly twice as fast as a single-threaded A, and equally fast (per #trials) as a multi-threaded A? How in general do I achieve speed gains with multi-threading instead of just doubling my CPU usage for no benefit?

Feel free to criticize any/all aspects of my code while you're at it.

Version A:
Code:
import java.util.Random;
import java.util.Arrays;
import java.util.concurrent.Callable;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;

public class LoanKelly{
   static final int SAMPLE = 1000000;
   static final int ROUNDS = 1000;
   static final int INTEREST = 770;
   static final int BUYINS = 4;
   static final int WIN_POSSIBILITIES = 51;
   static final int POSSIBILITIES = 100;
   static final int INSTALLMENT = 5000;
   static final double MINBET = 1.0;
   static double INITBR = (double)BUYINS * INSTALLMENT;

   public static void main(String[] args){
      final double[] FRAC = {.01,.02};
      final int CORES = Runtime.getRuntime().availableProcessors();
      int threads = Math.min(CORES, FRAC.length);
      int threadsRemaining = Math.max(CORES, FRAC.length);
      int index = 0;
      ExecutorService[] pool = new ExecutorService[(int)Math.ceil((double)FRAC.length/CORES)];
      
      for(int p=0; p<pool.length; p++){
         pool[p] = Executors.newFixedThreadPool(threads);
         if(p>0)
            while(true)
               if(pool[p-1].isTerminated())
                  break;
         threads = Math.min(threads, threadsRemaining);
         for(int c=index; c<index+threads; c++)
            pool[p].submit(new compareMedian(FRAC[c]));
         pool[p].shutdown();
         index += threads;
         threadsRemaining -= threads;
      }
   }
   private static class compareMedian implements Callable<Void> {
      private short[] br = new short[SAMPLE];
      private int busts, outcome;
      private int ruins = 0;
      private int total_busts = 0;
      private double bet, stack, bankroll, fixedFrac, medianGrowth;
      Random rand = new Random();
      
      public compareMedian(double f){
         fixedFrac = f;
      }
   
      @Override
      public Void call() throws Exception{
         for(int e=0; e<SAMPLE; e++){
            busts = 0;
            stack = (double)INSTALLMENT;
            bankroll = INITBR;
         
            for(int r=0; r<ROUNDS; r++){
               bet = fixedFrac * bankroll;
               if(bet>stack) bet=stack;
               else if(bet<MINBET){
                  if(stack>=MINBET) bet=MINBET;
                  else bet=stack; //Allows one extra bet.
               }
               outcome = rand.nextInt(POSSIBILITIES);
               if(outcome < WIN_POSSIBILITIES){
                  bankroll += bet;
                  stack += bet;
               }else{
                  bankroll -= bet;
                  stack -= bet;
                  if(stack==0.0){
                     busts++;
                     if(bankroll>0.0) stack=(double)INSTALLMENT;
                     else break;
                  }
               }
            }
            if(busts>0){
               total_busts += busts;
               if(busts==BUYINS) busts--;
               bankroll -= (double)busts*INTEREST;
            }
            br[e] = (short)Math.round(bankroll*.01);
            if(bankroll<=0.0) ruins++;
         }
         Arrays.sort(br);
         medianGrowth = (br[SAMPLE/2] + br[SAMPLE/2-1])*100.0/(2*INITBR);
         System.out.println("N="+SAMPLE+": After "+ROUNDS+" rounds of "+fixedFrac+" ,");
         if(medianGrowth>0) System.out.println("Median growth per round = " + Math.pow(medianGrowth, 1.0/ROUNDS));
         else System.out.println("Median growth per round = 0");
         System.out.println((double)total_busts/SAMPLE+" busts on avg, with ruin occurring "+100.0*ruins/SAMPLE+" %");
         System.out.println("");
         return null;
      }
   }
}
Version B
Code:
import java.util.Random;
import java.util.Arrays;
import java.util.concurrent.Callable;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;

public class KellyLoan{
   static final int CORES = Runtime.getRuntime().availableProcessors();
   static int SUBSAMPLE = (int)Math.round(1000000.0/CORES);
   static int SAMPLE = SUBSAMPLE*CORES;
   static short[] br = new short[SAMPLE];
   static final double MINBET = 1.0;
   static final int POSSIBILITIES = 100;
   static final int WIN_POSSIBILITIES = 51;
   static final int BUYINS = 4;
   static final int INSTALLMENT = 5000;
   static final int INTEREST = 770;
   static final int ROUNDS = 1000;
   static final double FRAC =.02;
   static double INITBR = (double)BUYINS * INSTALLMENT;
   static int[] subset_busts = new int[CORES];
   static int[] subset_ruins = new int[CORES];

   public static void main(String[] args){
      int total_busts = 0;
      int total_ruins = 0;
      double medianGrowth;
      ExecutorService pool = Executors.newFixedThreadPool(CORES);
      
      for(int c=0; c<CORES; c++)
         pool.submit(new compareMedian(c));
      pool.shutdown();
      while(true)
         if(pool.isTerminated())
            break;
      for(int c=0; c<CORES; c++){
         total_busts += subset_busts[c];
         total_ruins += subset_ruins[c];
      }
      Arrays.sort(br);
      medianGrowth = (br[SAMPLE/2] + br[SAMPLE/2-1])*100.0/(2*INITBR);
      System.out.println("N="+SAMPLE+": After "+ROUNDS+" rounds of "+FRAC+" ,");
      if(medianGrowth>0.0) System.out.println("Median growth per round = " + Math.pow(medianGrowth, 1.0/ROUNDS));
      else System.out.println("Median growth per round = 0");
      System.out.println((double)total_busts/SAMPLE+" busts on avg, with ruin occurring "+100.0*total_ruins/SAMPLE+" %");
   }
   
   private static class compareMedian implements Callable<Void> {
      private double bet, stack, bankroll;
      private int busts, outcome, mindex, maxdex, batchNum;
      Random rand = new Random();
      
      public compareMedian(int c){
         batchNum = c;
         subset_busts[c] = 0;
         subset_ruins[c] = 0;
         mindex = c*SUBSAMPLE;
         maxdex = mindex+SUBSAMPLE;
      }
   
      @Override
      public Void call() throws Exception{
         for(int e=mindex; e<maxdex; e++){
            busts = 0;
            stack = (double)INSTALLMENT;
            bankroll = INITBR;
            
            for(int r=0; r<ROUNDS; r++){
               bet = FRAC*bankroll;
               if(bet>stack) bet=stack;
               else if(bet<MINBET)
                  if(stack>=MINBET) bet=MINBET;
                  else bet=stack; //Allows one extra bet.
               outcome = rand.nextInt(POSSIBILITIES);
               if(outcome < WIN_POSSIBILITIES){
                  bankroll += bet;
                  stack += bet;
               }else{
                  bankroll -= bet;
                  stack -= bet;
                  if(stack==0.0){
                     busts++;
                     if(bankroll>0.0) stack=(double)INSTALLMENT;
                     else break;
                  }
               }
            }
            if(busts>0){
               subset_busts[batchNum] += busts;
               if(busts==BUYINS) busts--;
               bankroll -= (double)busts*INTEREST;
            }
            br[e] = (short)Math.round(bankroll*.01);
            if(bankroll<=0.0) subset_ruins[batchNum]++;
         }
         return null;
      }
   }
}
Programming homework and newbie help thread Quote
04-23-2018 , 01:41 PM
Could the compiler be optimizing them both to the same code?
Programming homework and newbie help thread Quote
04-23-2018 , 01:48 PM
Multi-threaded version A is twice as fast as B, so I doubt it. Also, functionally their behavior is slightly different, since A lets you compare different FRAC values in one execution.
Programming homework and newbie help thread Quote
04-23-2018 , 02:15 PM
I see a potential bug in one version (I was just comparing the compareMedian classes to see if they were identical, more on that below):

Code:
               else if(bet<MINBET)
                  if(stack>=MINBET) bet=MINBET;
                  else bet=stack; //Allows one extra bet.
In one version you added braces so it looks like maybe you fixed that, but I honestly don't know without running something if the "else bet=stack" line would do an "else" for the "else if" two lines above it (that's my guess), or the "if" one line above. Always use braces for that, don't leave ambiguity.

Regarding this:

Quote:
Originally Posted by heehaww
Here's where I think it gets weird. I used a stopwatch to compare the speeds using the same parameters (a 1-million trial sample for each FRAC). When I test a single FRAC, Version A finishes in about the same time as Version B despite using a single thread and half the CPU. Likewise, when version A has a 2-element FRAC array, it finishes in about the same time, making it twice as fast as B when both are multi-threading.
A couple things:
- build timing into your program so you don't need a stopwatch - Java probably has more detailed objects in their standard library you can use, but the most basic way to time something is - get a timestamp (down to milliseconds or nanoseconds) when your program starts, get another timestamp when it finishes, take the difference, print it out.
- so, you're saying that when the FRAC[] array only contains one element, that A runs at the same speed as when it contains two? Is it only doing half as much work in that scenario, though? (which would indicate that threading IS speeding up your work, because double the work takes the same time? but I might not be understanding what the program is doing or your description above)

One thing I'd recommend to try nailing down the problem - as well as being a good idea in general - is to isolate different parts of your program. In this case, you have an algorithm that's doing a set of work on some data (the compareMedian class), and it's taking in some input and doing some work and generating some output. That algorithm shouldn't have to be aware of how the threading in your program works, it should be the same in both cases, and be told about the problem space it's working in simply by its inputs (for something like this with a varying work size, maybe an array with bounds telling it where it's supposed to work). That would make it easier to reason about what your program is doing based on the different ways version A and B are using it.
Programming homework and newbie help thread Quote
04-23-2018 , 02:52 PM
Thanks for the quick reply.
Quote:
Originally Posted by goofyballer
In one version you added braces so it looks like maybe you fixed that, but I honestly don't know without running something if the "else bet=stack" line would do an "else" for the "else if" two lines above it (that's my guess), or the "if" one line above. Always use braces for that, don't leave ambiguity.
It's the same behavior in both. Without the braces, "else" gets applied to the very last "if". And if the behavior weren't the same, Version B would be much faster because it would be breaking out of the ROUNDS loop much earlier.

Omitting unnecessary braces has always been my style but yeah, I would have guessed that it's unconventional and probably not something professional developers do. In the future, for the reader's sake I'll include braces when sharing code.

Quote:
- build timing into your program so you don't need a stopwatch
True. I've done that in the past but forgot how and was too lazy to google it. The results were so much different than I expected that the difference can't be chalked up to reaction time. Nevertheless, I'll add built-in timing now if for no other reason than practice.

Quote:
- so, you're saying that when the FRAC[] array only contains one element, that A runs at the same speed as when it contains two? Is it only doing half as much work in that scenario, though? (which would indicate that threading IS speeding up your work, because double the work takes the same time?
Version A is properly speeding up the work by about 2x. However, Version B is not, and that's the part I don't understand.

Quote:
That algorithm shouldn't have to be aware of how the threading in your program works, it should be the same in both cases, and be told about the problem space it's working in simply by its inputs (for something like this with a varying work size, maybe an array with bounds telling it where it's supposed to work). That would make it easier to reason about what your program is doing based on the different ways version A and B are using it.
I don't understand what you're proposing. What does the bolded mean?

As coded, Version A gives each thread a 1M sample to work on and Version B gives each thread a 500k sample to work on.
Programming homework and newbie help thread Quote
04-23-2018 , 04:13 PM
I figured it out. Adding a built-in timer (which was very simple: System.currentTimeMillis() ) required me to use a while-loop in Version A that was previously unused for FRAC.length < 3 on my machine. This while-loop added about 8 or 9 seconds to the multi-threaded time.

Code:
while(true)
   if(pool.isTerminated())
      break;
Version B always uses that loop, which is why it's slower than A.

Just now i tested increasing the #trials to 10M with and without the loop. Now the loop takes 90 seconds which makes sense.

So when multi-threading, if I want a speed boost I either have avoid making the code do things after the threads, or I have to find a more efficient way to make it wait for them to terminate. There is probably a better way than checking pool.isTerminated() every nanosecond (or w/e) until the thread is done. This had crossed my mind when writing the while-loop, but then when comparing the versions it didn't occur to me that one of them wasn't using the loop as often.
Programming homework and newbie help thread Quote

      
m