Given that the project involves working with large values of N, there's a risk of integer overflow if the program doesn't handle numerical limits carefully. I would suggest that fellow students take care to avoid this issue, and the marker should be mindful of it as well, as integer overflow can lead to inaccurate performance reports, making the ultimate task of the project meaningless.
Additionally, what should we do if N is so large that we cannot gather the results into a single process? When N is large, even if the original matrix has a low non-zero rate, the resulting matrix is likely to have a non-zero rate close to 1. So for MEM = 230g, we can only do N < 2500000.
Should we write the results to a file? But will Setonix allow us to create such a large file? Since the time required to combine the results is also a critical factor in performance, I would appreciate clarification on this matter.
These are both great observations.
Please do not write the results to files, as we don't want to use Setonix's storage for these large random matrices. Other than this, I'm happy for you to address the issues in whatever way you want -- as long as you explain it in your report.
Finding the limitations and how they could be solved is exactly why I made the assignment a little more open ended. You can write about these in the report, and explain some strategies you could use if you needed to scale up to larger matrices across more nodes.
