Kokkos N-dim view with ADIOS2

[hr87]

I am trying to write an output function for my radiation transport code using ADIOS2. The code uses Kokkos::Views to store the data. The view has 3 dimensions, one for vertices (N-dimensonal structured mesh), one for directions, and one for energy.

I want to write the data as N+2 dimensional array, where N is the number of dimensions. N is known at compile time. I can transform the 3D view I use internally to a N+2 dimensional Kokkos view.

The documentation on using Kokkos views with Adios2 is a bit short: https://adios2.readthedocs.io/en/v2.9.1/advanced/gpu_aware.html#using-kokkos-buffers. Does the put method take a multi-dimensional view that has the same extent per dimension as the variable? Or does the view need to be one-dimensional? If so, what ordering does it need?

Thanks for the help.

[hr87]

Also, is there an easy way to translate a Kokkos memory space into an adios2 memory space? Or obtain the adios memory space from a Kokkos view?

[anagainaru]

We translate the Kokkos view memory space to adios memory space in ../../bindings/CXX11/adios2/cxx11/KokkosView.h

You would need something like (basically saying if the view is HostSpace than we set the adios memory space to Host, otherwise the adios memory space is Device):

template <>
struct memspace_kokkos_to_adios2<Kokkos::HostSpace>
{
    static constexpr adios2::MemorySpace value = adios2::MemorySpace::Host;
};

template <typename T>
struct memspace_kokkos_to_adios2
{
    static constexpr adios2::MemorySpace value = adios2::MemorySpace::GPU;
};

mem_space = detail::memspace_kokkos_to_adios2<typename Kokkos::View<T, P...>::memory_space>::value;

By default we detect automatically where the buffer has been allocated so you don't need to set the adios memory space unless you are concerned about performance.

[anagainaru]

Sorry for the delay. You can use multi-dimensional views and pass them directly. I tested using something like this, but I think you can play with the adios dimensions as long as it can store the total amount of data you have.

Kokkos::View<float ***, MemSpace> gpuSimData("simBuffer", N1, N2, N3);

 const adios2::Dims shape{N1, N2, N3};
 const adios2::Dims start{0, 0, 0};
 const adios2::Dims count{N1, N2, N3};

This would also work:

    const adios2::Dims shape{N1 * N2 * N3};
    const adios2::Dims start{0};
    const adios2::Dims count{N1 * N2 * N3};

Internally we copy the data from your View to our internal buffers (but we use the default layout) so we expect the same layout for the reading view as for the writing view (otherwise you might get transposed data). We do not support strides right now.

Our Kokkos backend is still very young and not tested a lot, but on the up side if there is something you need we are looking forward to seeing this used in some apps so we'll help you and potentially implement whatever functionality is missing.

[anagainaru]

I can add an example with this if you want

[hr87]

Thanks for the answer. I decided to use the span and write directly to that. I had to make a copy of the internal data because it is stored differently than the output needs, so this is more performant. I might use the Kokkos interface at other locations. I let you know if I run into any problems.

[anagainaru]

Ok, I'll mark this as answered but let us know if you start using the Kokkos backend for anything else. We plan to make it more flexible in the future.