Problem with parallel_copy

[avergnaud002]

Hello,

I am trying to access data at a given location with parallel_copy (using Fortran Interface) inside a loop. Possibly, those data are on another proc than the current proc and on different level of refinement. I tried something like bellow but it didn't work when data (on the intersection of the box bx with the boxarray data(lev)%ba) are not on the current proc...

! here I am inside a loop over cells inside a box
bx = amrex_box((/i,j,k/), (/i+1,j+1,k+1/))
call amrex_boxarray_build(ba,bx)
pmap = amrex_parallel_myproc()
call amrex_distromap_build(dm, pmap) ! put this data on the current proc
call amrex_multifab_build(mf, ba, dm, 1, 0)
call mf%parallel_copy(data(lev), amrex_geom(lev))

In other words, I want to have access to data at the location given by the box bx, regardless of the proc that owns them...
I am missing something on how parallel_copy work ?

Alban

[WeiqunZhang]

amrex::MultiFab is a distributed container for data. So it cannot be used in this way. It's very likely the case that you should be able to do the communication outside the loop over cells. (Even if we could do communication inside the loop over cells, the performance would be terrible.) If you can show more codes around the code snippet shown in the question and the code works without MPI, we might have more concrete suggestions.

[WeiqunZhang]

Looks like what you need is ghost cells. When you build a MultiFab, you can specify the number of ghost cells. You can take a look at this (https://github.com/AMReX-Codes/amrex/tree/development/Tests/FortranInterface/Advection_F) as an example. In particular, you can look at https://github.com/AMReX-Codes/amrex/blob/development/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90 for an example of using fillpatch.

[avergnaud002]

Yes I already use ghost cells. In fact fillpatch can help in the case of accessing coarse level data from fine level data. But the difficulty I have is accessing fine level data from coarse level data... Typically, at a coarse/fine interface, from a given coarse cell, how can I access the data of the 2 adjacent fine cells (assuming I am in 2D and with a ref_ratio=2) ? I don't see how fillpatch can do this...

[WeiqunZhang]

Maybe you can take a look at how amrex::YAFluxRegister and amerx::EBFluxRegister do things similar to what you have described.

[avergnaud002]

Ok I will look at this.
But to be sure, is there any "function" in AMReX that can copy data in a MultiFab from another MultiFab, in the regions of intersections of these two MultiFab ? Here by "regions of intersections" I mean the intersection of their respective geometrical domain (included ghost cells !) AND regardless the distribution of the data on the different procs. I thought as first sight that parallel_copy could do this, but I apparently misunderstood the way this routine works...

[WeiqunZhang]

No there is no such function. It seems that you want to copy from a fine level MultiFab to a coarse level MultiFab in the intersection region. But the coarse level MultiFab (which is conceptually just a multi-d array) does not have enough space to hold the fine data. You could refine the coarse level BoxArray and use that new BoxArray and the coarse level DistributionMapping to build a new special "coarse level" MulitFab. You could then copy from the fine level MulitFab to the special "coarse level" MultiFab. You could do that, but it's very inefficient in memory usage. Then you might say why don't we only allocate memory in the place we actually need the data (i.e., coarse/fine interface), not everywhere. Well, that's what FYAFluxRegister and EBFluxRegister do. Note that the coarse/fine interface can be a complicated shape. So I am not sure how a function (assuming it's pure without internal data) can do it.