Parallel calculation on single formula

114 viewsIdea Portalcalculation engine
0

Hi,

I’m having big models, running for several hours to calculate. Obviously I run these models on big computers with multiple CPUs. Unfortunately, depending on the model, all these nice CPUs are hardly used, often it runs only on one processor instead of 10. So, potentially, I could speed up my models up to a factor 5-10, which would be enormous.

That said, speed optimisation in Quantrix is a tricky subject. But what I found out is a big factor is the following: If I have a heavy calculation (big matrix doing a lot of calculations a lot of times) and this matrix has a nice multidimensional structure, so I can nicely write the formula for A LOT of INDEPENDENT calculations in one formula, well … this formula can’t be parallelised, and thus will be a bottleneck in my calculation, and since the depending calculations what for this formula to finish, a lot of time is spent just on one CPU instead of multiple.

What I do if things get really bad: I try to find out in which of all my formula a lot of “Single-CPU-time” is spent. And then I go and split the formula up manually: Yes, you understand right: I take this nice single formula, and split up along one or several categories, and repeat the formula once per item. I sometimes also use scripting to create the bunch of formulas by script. This then allows Quantrix to process all these “different” formulas in parallel on different CPUs.

Whatif Quantrix could do this on it’s own? Find out if a formula acts independently on items, split it up, and then parallelise it? Without the modeler having to worry about manually parallising the formulas (first find out which formula is the bottleneck, split it up, maintain all the formulas, etc.).

Question to you folks out there: Who has models out there which run long enough that you think doing deeper parallelism would benefit you? Am I the only one trembeling in the evening because I had to drink so much coffee while the calculation is running on a single CPU?

Thanks for considering this idea.

Gilbert

Quantrix Marketing Changed status to publish May 9, 2024