Particle diagnostic swaps particle id's, impossible to recreate trajectories

[aveksler1]

After the transition to SoA particle attributes #4653, I am no longer able to recreate particle trajectories using ParticleDiagnostics with the openpmd backend. Admittedly, I haven't tried other backends, since openpmd-viewer is my main analysis tool.

When there are multiple particles, the positions for a given particle id do not always correspond from one iteration to the next. I see this using openpmd-viewer's ParticleTracker, and a clunkier way to reproduce trajectories that isn't shown here.

Below is a minimal reproducing example python input script and analysis script. I launch 100 ions starting at different positions (with do_not_deposit=True so I have a simple particle tracer) in a uniform magnetic field expecting to see perfect Larmor orbits. The plots below show how some trajectories "jump" around, mapping to another particle's trajectory.

analysis.txt
PICMI_inputs_2d.txt

[roelof-groenewald]

attn: @RemiLehe @ax3l

[ax3l]

Thanks for the ping! I'll take a look if there is a mismatch between id and cpuid now :-o
(Or an initialization error.)

[ax3l]

Thank you for the details!

Does the issue appear equally on CPU and GPU? How many MPI ranks did you use, if any?

[RemiLehe]

Thanks for reporting this @aveksler1
Btw, did you see this when running on CPU or GPU (or both)? Are you running on multiple MPI ranks?

[aveksler1]

I saw this when running on GPU, with 1 MPI rank, and 1 grid. I will check on CPU once it finishes compiling

[ax3l]

@RemiLehe has a good guess that I think will be it: we forgot to move the idcpu when we sort particles in various ways.

We look into it now. cc @atmyers

update: see below, looks ok actually...

[roelof-groenewald]

Thanks guys!

[ax3l]

It looks like sorting should do the right thing:
https://github.com/AMReX-Codes/amrex/blob/76d09f554c3e5b8b4ecd4b8bf0c39d0fefddffa4/Src/Particle/AMReX_ParticleContainerI.H#L1118-L1131

Continuing search...

[ax3l]

Could it be a GPU race condition in swapping of idcpu?
@aveksler1 @roelof-groenewald does setting export CUDA_LAUNCH_BLOCKING=1 still show the issue?
https://warpx.readthedocs.io/en/latest/usage/workflows/debugging.html

[aveksler1]

@ax3l Yes, it still shows the issue.

[ax3l]

Verified in parallel: openPMD write uses the correct data set:

WarpX/Source/Diagnostics/WarpXOpenPMD.cpp

Lines 904 to 909 in c5a5732

	// here we the save the SoA properties (idcpu)
	{
	// todo: add support to not write the particle index
	getComponentRecord("id").storeChunkRaw(
	soa.GetIdCPUData().data(), {offset}, {numParticleOnTile64});
	}

[roelof-groenewald]

Just guessing here but could there be a problem with openPMD's ID tracker not disentangling the id and cpu bits appropriately? In the above it still looks like the full idcpu 64 bit number is stored as a single quantity.

[ax3l]

For openPMD, ids are just 64bit numbers. It does not deconstruct them.

[RemiLehe]

I just compiled and ran on CPU (WarpX (24.04-51-gc5a5732721b4)), and I do not see the issue in this case.

@roelof-groenewald I think that this excludes the possibility that openPMD-viewer is misinterpreting the ID

[aveksler1]

Issue does not show up when running with CPU! both on 1 and 2 MPI ranks.

[RemiLehe]

@aveksler1 I just compiled for NVIDIA GPU (WarpX (24.04-51-gc5a5732721b4)) and ran with 1 MPI rank, and I again do not see the issue.

I wonder what the difference with your case is. What was the platform that you ran on? Was it an AMD or NVIDIA GPU?

[aveksler1]

@RemiLehe @ax3l Sorry to waste your time, my WarpX build was out-dated, I should be better about rebuilding often, especially when facing bugs. It seems like AMReX-Codes/amrex#3890 is likely the fix that I was missing. Thank you all for your help.

[RemiLehe]

OK, no worries. Thanks for letting us know, and thanks again for reporting potential bugs here.