I’m constructing a graph with ~10000 nodes, each node having metadata that determine which other nodes it will be connected to, using an edge.
Since the number of edge possibilities (~50M) is far greater than the actual number of edges that will be added (~300k), it is suboptimal to just iterate through node-pairs with for loops to check if an edge should be added between them. Using some logic to filter out many pairs to not have to check, with the help of numpy‘s rapid methods I quickly reduced the possibilities to an array of ~30M pairs only.
However, when iterating through them instead, the performance did not improve much – in fact iterating through a bigger 2D boolean matrix is twice as fast (compared to my method, which previously collected the True values from the matrix and only iterates through these ~30M instances). There must be a way to get the desired performance benefit, but I hit a deadend, looking to understand why some methods are faster and how to improve my runtime.