mpi Combining 2-D arrays into a master array on all processes using MPI Allgatherv

[Smiffy_]

Hi,

 

I'm attempting to combine row/column 2-D arrays outputted from each process into a single complete 2-D array on all processes. Essentially, I have a large NxN 2-D array (4000x4000 +) that requires the same operation to be carried out on all elements. My intention is to break this down into either sections of rows or columns that each process will complete. I need each process to have the entirety of the array once all sections have been completed.

 

I have looked at multiple examples of using these MPI instructions but could not find one that combined rows/columns from N processes. Could someone please inform me as to how I can implement this?

 

Below is a boilerplate example of what I'm trying to achieve. Each process creates a master matrix and then calculates the set of rows it's responsible for. It then creates a 2-D array of that size and copies the data from the master. It then carries out its operation on each element of the 2-D array. I then need to use Allgatherv to collect each process's 2-D array and combine them to overwrite the master matrix. Please note that it is not important whether I use rows or columns from my point of view, I believed sticking to one and not trying to create multiple submatrices would allow me to easily increase the number of processes running without added complexity.

 

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

double **alloc_2d_array(int m, int n) {
    double **x;
    int i;

    x = (double **)malloc(m*sizeof(double *));
    x[0] = (double *)calloc(m*n,sizeof(double));
    for ( i = 1; i < m; i++ )
        x[i] = &x[0][i*n];
    return x;
}

void main(int argc, char *argv[]) {
    int n = 8;
    int rank, size;
    int root_rank = 0;

    MPI_Init(&argc,&argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    // Report active to console
    printf("Rank: %d, reporting!\n", rank);

    // Make master matrix
    double ** master_matrix = alloc_2d_array(n, n);

    // Set starting values in master matrix
    for (int i=0; i<n; i++) {
        for (int j=0; j<n; j++) {
            master_matrix[i][j] = i*n+j;
        }
    }
    
    // Calculate each ranks section of matrix
    int interval = n/size;
    int section_end = interval*(rank+1);
    int section_start = section_end - interval;
    if (rank == size-1) {
        section_end += n % size;
    }
    int section_length = section_end-section_start;
    printf("Start: %d, End: %d\n", section_start, section_end);

    // Make local rows
    double ** local_sect = alloc_2d_array(section_length, n);

    // Set local rows to master_matrix rows
    for (int i=0; i<section_length; i++) {
        local_sect[i] = master_matrix[i];
    }

    // Carry out operation (in this example, adding 7)
    for (int i=0; i<section_length; i++) {
        for (int j=0; j<n; j++) {
            local_sect[i][j] = local_sect[i][j]+7;
        }
    }

    // Use Allgatherv to overwrite master matrix to new complete matrix

    // MPI_Allgatherv(my_values, my_values_count, MPI_INT, buffer, counts, displacements, MPI_INT, MPI_COMM_WORLD);

    // Print new master matrix out on all processes
    printf("NEW MASTER MATRIX\n");
    for (int i=0; i<n; i++) {
        for (int j=0; j<n; j++) {
            printf("%f ", master_matrix[i][j]);
        }
        printf("\n");
    }
    

    MPI_Finalize();
}

 

[Sauron]

If I'm reading this correctly you can use the displacements array to define where each process will write its data in the collective receive buffer. A matrix is just an array of arrays, meaning the nth row is offset from the beginning of the matrix of (n-1)*row_length bytes. You can therefore populate the displacements array with

 

{0, row_length, 2 * row_length, [...], (number_of_rows - 1) * row_length}

 

and the gather function should then write each process' data in the correct spot.