Slow Model Performance Over Large Forecast Datasets

0

Hey Everyone,
I’m working on redoing an old Excel system for forecasting production at an industrial mineral plant and I was hoping to get some advice on optimizing model performance.
The mine engineers output a data file for what they anticipate the site can produce in terms of tons and quality, its then brought into the model where that output file is translated (all simple equations) into human readable values. Where I’m running into problems is that these forecast sheets have daily data per machine, and there ends up being some very large DataLink tables. I’ve increased the RAM allotment to 8GB (it doesn’t appear to go higher than 3GB) and it looks like Quantrix is fully utilizing all 8 threads available to it on the i7 I’m running.
Does anyone else work on large data sets and have any advice on maximizing performance?
The two main equations its running are:

  1. a weighted average of the daily machine production to a total site value (sum for tons, w-avg for quality data)
  2. a weighted standard deviation for quality data

It runs those two equations to roll up production to both a standard week and monthly Quantrix Timelines.
I’d attach the model file, but forward looking mineral production statements tend to be very sensitive and I don’t have legal clearance to post it.
It appears to run quickly and normally with the smaller scale data sets I’ve used in the past, its primarily an optimizing for large data sets problem.

Edit:

I’ve added a picture of the equations, they’re weighted averages and standard deviations (there’s also a sum calculation for total tons but that’s simple and performant). The data comes into two tables in Quantrix which list tons per machine per day per site (for TonSource), and chemical quality per machine, per day, per site (StatSource). The calculation matrix is linked to the source tables through categories which auto generate based on the input (the using as statements). The Selectbetween statements are to relate the source tables’ date columns to the Timeline category stored in the Calendar matrix (Start and End are timelinedate() and start+7 respectively to for weekly “Other” ranges).

I’m also attaching this on weighted standard deviations since these aren’t the most common thing out there. We’re using weighted values because a given amount of production tonnage has a sampled chemical quality and when the tonnage outputs of several machines are aggregated together, they need to be based on the relative tonnage of each machine since they don’t produce in equal rates.

immeralcode Answered question May 6, 2019
0

I have worked with some large models.  I recommend you get more RAM on your machine.  I currently have a machine with 32GB of RAM and I adjust Quantrix to have full access to all of it, if necessary.  I am curious to see the formulas for your 2 main equations, can you post a image of them?

braddo Posted new comment July 20, 2020

Hey QA,

I just updated the post with an image from the formula bar. unfortunately is looks like the container these forum feeds are in displays it pretty small. if your having issues reading it let me know and I’ll try to work something else out with smaller snips.

Also I’ve had the RAM allocation at 8gb’s for a while and its still bogging down I’ve got more RAM to give it, I’m just not sure if that’s the bottle neck or not. I’ll try upping the allotment to 15 and see how things improve.

You are viewing 1 out of 3 answers, click here to view all answers.