Using --cpus-per-task=128 is the correct way to specify the thread count. Since we don not have control over how Setonix operates, we must work with its scheduling behavior as best we can.

I still cannot see the C code, so I am not sure how you measured the execution time. However, it is possible that Setonix's scheduling scheme could make wall time an inaccurate measure of execution time.

I suggest trying this for measuring CPU time:

#include <unistd.h>

struct timespec t;
clock_gettime(CLOCK_THREAD_CPUTIME_ID, &t);
return t.tv_sec + t.tv_nsec / 1e9;