CHANGES

# 22.08

  -- Let `selectActualNeighbors` return right after starting if there are no
     particles for communication. (#2886)

  -- Add Comm Sync to Redistribute (#2891)

  -- Multi-materials and derived variable output (#2888)

  -- Fix host / device sync bug in PODVector (#2890)

  -- MinLoc and MaxLoc Support (#2885)

  -- HIP: Remove the call to hipDeviceSetSharedMemConfig (#2884)

  -- Add Frontier to GNU Make (#2879)

  -- Add option to derefine to AMRErrorTag (#2875)

  -- Fix the segmentation fault in selecting actual neighbor particles. (#2877)

  -- Workaround to bypass issue observed at very large scale with Fujitsu MPI (#2874)
     `TagBoxArray::collate`: Fujitsu Clang (#2889)

  -- Allow zero components MultiFab and BaseFab (#2873)

  -- New EB optimization parameter: eb2.num_coarsen_opt (#2872)

  -- SENSEI 4.0: Fix Build for Particles (#2869)

  -- Cache the neighbor comm tags for the CPU implementation of fillNeighbors. (#2862)

  -- Remove some hard checks in check_mvmc for 3D (#2864)

  -- Carry over fix for ngbxy.smallEnd typo (#2868)

# 22.07

  -- Adding control APIs and namespacing for core algorithm paths like SpGEMM, SpMV, and SpTrans. (#2859)

  -- update the SENSEI in situ coupling for SENSEI v4.0.0 (#2785)

  -- Write runtime attribs to checkpoints on GPUs (#2856)

  -- Fix gnu make on Crusher for mpi_gtl_hsa (#2857)

  -- CMake: FindDependency CUDAToolkit (#2849)

  -- NERSC Programming Environment prototype (#2848)

  -- GNU Make: No need to query mpif90 if Fortran is not used. (#2852)

  -- Remove f90doc (#2851)

  -- Explicitly invoke python3 (#2850)

  -- Maintain the high end of the 'roundoff domain' in both float and double precision (#2839)

  -- add Ok to coordsys (#2844)

  -- ParamParse: Add Files at Runtime (#2842)

  -- Fix a pathological case for 2d EB (#2840)

  -- add fvolumesum to GNUmakefile (#2836)

  -- Clamp particles shifted from plo boundary against rhi, rather than back to plo (#2814)

  -- Fix: CMake NVTX not only Hypre (#2837)

  -- Update sensei CI container for sensei v4.0 integration (#2834)

  -- HIP Memory Advise : Set managed memory to coarse grain (#2835)

  -- CMake: Fix `export` with `AMReX_INSTALL=OFF` (#2838)

  -- make PODVector work with PolymorphicArenaAllocator (#2829)

  -- Re-implement FaceLinear::interp() for InterpFromCoarseLevel (#2831)

  -- Make regrid method of Amr class public (#2833)

  -- amrex::Any (#2827)

  -- Fix line integral computation (#2830)

  -- Fix a bug in multigrid grids (#2823)

  -- Add html, additional sections to README.md (#2775)

  -- Allow StateDataPhysBCFunct to operate on face-centered data (#2819)

  -- Fix Parser ODR (#2820)

  -- CMake: Cleanup old nvToolsExt (#2817)

  -- Handle the case where we don't have enough device memory for the snd_buffer (#2705)

  -- CMake: 3.17+ (#2813)

  -- Landon/fix bug ghost particles (#2812)

  -- Follow-on to 2809; update selectActualNeighbors as well. (#2810)

  -- Generalize the type of callables that can be passed into the neighbor list build function (#2809)

  -- Add AVX2 instructions flag. (#2803)

  -- Avoid M_PI because it's not in the C++ standard (#2807)

  -- In the array version of FillPatchTwoLevels, allow specifying an (#2800)

# 22.06

  -- Fix solvability issue in the nodal solver RAP approach (#2783, #2801)

  -- Profiler Sync Timers (#2784)

  -- Implement Serial ParallelDescriptor::Gather (#2793)

  -- Renamed new internal class variables using m_ convention to fix compiler warnings in debug mode. (#2790)

  -- Make IntVectFromLocation const (#2789)

  -- make sure m_particles is sized properly when numLocalTilesAtLevel is called (#2782)

  -- Fix for small cells (#2781)

  -- Add some timestep controls to the AMReX TimeIntegrator class for its integrate() driver function. (#2780)

  -- Fix the bug in the CMake build with AMReX_BASE_PROFILE. (#2774)

  -- configure value of AMReX_GPU_RDC flag for use in cmake find_package(AMReX ...) (#2770)

  -- Add an optional volume weighting to AMRErrorTag (#2772)

  -- Fix: AmrCore Move (#2773)

  -- Update particle << operator after changes to id/cpu (#2769)

  -- this updates to recent Hypre API changes (#2765)

  -- multilevel version of writeplotfiletoascii (#2742)

  -- Avoid the use of null stream (#2754)

  -- add scomp and ncomp arguments to IntegratorOps functions. (#2759)

  -- Add HDF5 H5Z-ZFP support in CMake (#2753)

  -- CUDA On Cray: More Robust w/o Wrapper (#2757)

  -- Reimplement amrex::min and max to Work around an nvcc bug (#2756)

  -- Revert "Turn on managed memory by default in The_Arena for HIP (#2734)" (#2752)

  -- Add methods for resetting the ParGDB of a ParticleContainer (#2732)

  -- Switch nvtxRangeStart to nvtxRangePush (#2746)

  -- Eb flow diffusive solve (#2741)

# 22.05

  -- Update Required SUNDIALS version (#2743)

  -- EB: Geometry generation from STL file (#2728)

  -- Drop CUDA 9 support (#2736)

  -- Disable AsyncOut by default for HIP and DPC++ (#2735)

  -- Turn on managed memory by default in The_Arena for HIP (#2734)

  -- Update WarpX's MLEBNodeFDLaplacian for 2D RZ (#2733)

  -- Adding more recent Hypre APIs to initialize the library. (#2729)

  -- Allow the computation of neighbor lists between particles of different types. (#2727)

  -- Add option for face linear interpolater in fortran interface (#2726)

  -- Allow some EB code to handle single precision (#2723, #2719)

  -- Allow `FillBoundary` to cast messages from double to single precision. (#2708)

  -- Fix FluxRegister::SumReg for GPU builds (#2718)

  -- Swap hip lib and include ordering (#2717)

  -- Fix integer overflow warnings in FabConv (#2716)

  -- Make AsyncOut::Finish safe to call when async_out is not enabled. (#2715)

  -- flushParForInfo: remove erroneous assertion (#2712)

  -- Link particles in ghost cells to closest box (#2685)

  -- Generalize the type of lambdas that can be passed into ParticleReduce (#2697)

  -- Return non-zero if nans were found when using fnan. (#2704)

  -- Reorganize Make.package in Src/Particle (#2703)

  -- FillPatchUtil edge interpolater fix (#2701)

  -- Adds SUNDIALS integrator options for flexibility. (#2700)

  -- Only error if HDF5 is not parallel if MPI is enabled. (#2699)

  -- Make the ParticleReduce functions not rely on ParIter. (#2695)

  -- Semicoarsening in WarpX Linear Solvers (#2690)

  -- FillBoundaryAndSync (#2683)

  -- macOS: Fix Warning (ranlib, profiler) (#2688)

# 22.04

  -- Fix parser expressions (#2682)

  -- fixes to build CUDA with Clang (#2681)

  -- CI: Apple Silicon (#2672)

  -- Use the roundoff domain in enforcePeriodic. (#2679)

  -- Catch CUDA 11.6 bug at compile time (#2677)

  -- MPI+OMP+HIP build capability (#2676)

  -- EB Flow (#2661)

  -- BackTrace workarounds (#2667)

  -- Redefine ParticleContainter multifab after setting particle boxarray and dmap (#2673)

  -- Handle the case where we lose a level in RedistributeGPU (#2670)

  -- Add int overflow assert check before Bcast wrapper (#2649)

  -- Allow ParticleToMesh and MeshToParticle to access runtime particle components (#2668)

  -- Fix bugs in the MultiBlock test (#2662)

  -- Separate version strings for particle plotfiles and checkpoints. (#2663)

  -- Add support for SZ compression in HDF5 output (#2644)

  -- Removed the use of H5Aget_storage_size (#2656)

  -- Add prefetchToHost and prefetchToDevice for GPU containers (#2655)

  -- Introduce `AMReX_Version.H` (#2653)

  -- Add move assignment operator to Fab (#2652)

  -- Added an option to set fast MRI dt directly instead of using the integer ratio of slow/fast dt. (#2651)

  -- AMReX_Config.H: AMREX_EXPORT_DYNAMIC (#2650)

  -- Fix bug affecting neighbor proc calculation with mesh refinement. (#2646)

  -- Update flags for DPC++ AOT (#2643)

  -- Doc + CMake: AMREX_EXPORT_DYNAMIC (#2635)

  -- Update SpackSmokeTest to run with Fortran and/or HIP. (#2629)

  -- Simplify CoordSys::IsRZ and IsSPHERICAL (#2642)

  -- Take advantage of async copies in PODVector. (#2641)

  -- HIP: atomicAddNoRet (#1809)

# 22.03

  -- ParticleTile: push_back_real/int w/ Vector (#2634)

  -- Update Docs to reflect newer options in MLMG (#2620)

  -- Allow the ghost-particle exchange to communicate different variables (#2627)

  -- Fix AmrParticleContainer::AssignDensity when ncomp == AMREX_SPACEDIM+1. (#2632)

  -- Use async version of cuda/hipMemcpy in PODVector::resize() (#2631)

  -- ParticleContainer::make_alike (#2630)

  -- Update checkpoint format to account for expanded particle ids. (#2624)

  -- Update FabArray::copyTo (#2603)

  -- explicitly request python3 for makebuildinfo_C.py (#2621)

  -- store the CUDA version in build info (#2617)

  -- missing check for mapped solver support when not using GPU (#2616)

  -- Remove ifdefs for older compiler version. (#2614)

  -- Allow a descriptor component's BndryFuncFabDefault to be stateful. (#2612)

  -- TableData: fix missing return in operator= (#2613)

  -- SUNDIALS MRI Interface for AMReX TimeIntegrator class (#2600)

# 22.02

  -- Fix I/O bug in Amr class (#2609)

  -- Spack install instructions for Building section. (#2602)

  -- Kernel fusing in Geometry (#2606)

  -- Kernel fusing in FluxRegister (#2605)

  -- TableData: A Multi-Dimensional Array Class (#2601)

  -- FabSet::multiFab (#2604)

  -- Disable EB solver's phi-on-centroid for hip for now (#2598)

  -- Update locations for tutorials and minor formatting. (#2583)

  -- CI: AMD `hip::device` also for Fortran (#2595)

  -- fix vol initialization for Cartesian in fvolumesum (#2596)

  -- Add headings and steps to plot vector field in ParaView doc section (#2588)

  -- Kernel fusing in FabArrayUtility (#2593)

  -- Change FPinfo's fact_fine_patch to always have ng=1. (#2591)

  -- Support arbitrary refinement ratio in FaceLinear Interpolater (#2590)

  -- Default Arena for MultiFab/iMultiFab/FabArray (#2589)

  -- `add_par` precedence in Docs. (#2575)

  -- Fix typo in 1D version of AMREX_LAUNCH_DEVICE_LAMBDA_DIM (#2587)

  -- CMake: remove nonexistent header file, update minimum sundials version (#2586)

  -- Enable CUDA in CMakeLists.txt for Spack smoke test. (#2581)

  -- Fix warning from hiprand header (#2585)

  -- HIP: amdclang++ (#2582)

  -- Update sundials sycl memory helper interface (#2580)

  -- HIP: Add -munsafe-fp-atomics to CMake (#2577)

  -- ParmPaser::queryAdd (#2573)

  -- Fix regression test configuration (#2574)

  -- Convert switch statement to if statement (#2570)

  -- remove erroneous BL_ASSERT in sundials initialization (#2568)

  -- Support implicit function class defined on host when using device memory explicitly.  (#2563)

  -- HIP: Add -munsafe-fp-atomics to GNU Make (#2567)

  -- GNU Make: Add a new machine, crusher (#2566)

  -- Mapped solver support (#2088)

  -- Update to SUNDIALS 6 (#2551)

  -- Return a zero-size vector in ParallelDescriptor::Gather. (#2560)

  -- Add option not to remove particles with negative ids when calling Redistribute. (#2561)

  -- Update FillPatchTwoLevels for single component face-centered data (#2539)

  -- Kernel fusing in FabArray Comm (#2559)

  -- Use MF ParallelFor in error tagging (#2558)

  -- iMultiFab I/O (#2495)

  -- Embed more EB data in EBFArrayBox (#2550)

  -- CpuBndryFuncFab: Face Data (#2545)

  -- Fix bug in Parser (#2555)

# 22.01

  -- Add Gatherv for USE_MPI=FALSE. (#2549)

  -- Resize vector inside GatherLayoutDataToVector (#2548)

  -- Add missing FillBoundary_nowait (#2546)

  -- add a tool to compute the volume integral of a plotfile field (#2541)

  -- Update MLEBNodeFDLaplacian (#2538)

  -- Update NVector_MultiFab functionality from ATPESC-codes and add to Src/Extern/SUNDIALS (#2505)

  -- Projections migrated to AMReX-Hydro (#2530)

  -- Fix a bug in Robin BC (#2477)

  -- Add ParallelCopyToGhost and fix a bug in FillPatch (#2523)

  -- CMake 3.22+: Policy CMP0127 (#2525)

  -- Add function for make a dmap that is similar to an input MultiFab. (#2526)

  -- Specify dimensions that can be refined with `refine_grid_layout` (#2516)

  -- DPCPP: AMREX_INTEL_ARCH (#2459)

  -- Generalize problem size specification (#2518)

  -- Use specific targets for make uninstall (#2471)

  -- Move Basic Time Integration into AMReX (#2503)

  -- allow the use of SuperParticle in amrex::ParticleToMesh (#2514)

  -- ANSI Escape Codes (#2509)

# 21.12

  -- AMREX_RELEASE_NUMBER (#2448)

  -- Add a function for detecting whether amrex::Initialize has been called or not. (#2501)

  -- Fix fcompare's --abs_tol (#2504)

  -- HDF5: Fix an issue when some MPI ranks have no box at all (#2494)

  -- Signed distance function in 2D (#2490)

  -- Reduce ghost particles in particle communication. (#2480)

  -- Assign initial value to prevent (likely spurious) -Wmaybe_unitialized compiler warning. (#2484)

  -- Fix a corner case in fixing small cells. (#2489)

  -- Added special case for Mac when using rpath (#2479)

  -- Fix multi-level EB nodal solver solvability fix (#2473)

  -- Disable warning on atomicAddNoRet in HIP (#2466)

  -- DPCPP: workaround for the scan issue (#2462)

  -- DPCPP: AOT in GNU Make (#2463)

  -- Implement HOEXTRAPCC for more than single ghost cell (#2456)

  -- Spack Smoke Test (#2460)

  -- Support fmod in Parser (#2395)

  -- HIP: ftrapv causes issues with DEBUG linking (#2455)

  -- DPCPP:  Enable early optimizations (#2454)

  -- New HDF5 schema with separate datasets for different variables (#2432)

  -- InterpFaceRegister (#2452)
    
  -- Fix WarpX EB solver's grad phi computation (#2453)

# 21.11

  -- Set to never write fill values to the dataset for HDF5 output (#2450)

  -- Lower Hypre minimum version to 2.20. (#2446)

  -- Fix use-after-free in the flush of parallel copy meta-data cache. (#2443)

  -- Update MPI config checking to include Intel MPI wrappers for DPCPP (#2435)

  -- Add EBCellFlagFab::getNumRegularCells, getNumCutCells and getNumCoveredCells (#2422)

  -- ExclusiveSum on GPU: return 0 when size of pointer is 0 (#2440)

  -- Multi-component overset solver (#2429)

  -- Use -rpath=dir instead of -rpath dir for dpcpp. (#2439)

  -- DPCPP: Use -g1 for non-debug build (#2438)

  -- DPCPP: Fix warning on tautological-constant-compare (#2436)

  -- Tests: Safe-Guard Module Dirs (#2434)

  -- Remove deprecated BaseUmap (#2425)

  -- Update WarpX EB solver (#2421)

  -- MLNodeLaplacian: Fix max coarsening level for enclosed EB (#2431)

  -- Fix two IO overflow bugs uncovered by NyX. (#2417)

  -- Cleanup of amrdata (#2430)

  -- Set options to allow other Doxygen docs to link to AMReX. (#2424)

  -- Remove DArena and EArena (#2426)

  -- Update MLNodeTensorLap for WarpX (#2416)

  -- Reduce peak memory usage when sorting particles. (#2414)

  -- Extend CellQuadratic Interpolater to 3D (#2409)

  -- Fix 2D bug in EB2::SplineIF (#2407)

  -- Cleanup Tools/Postprocessing/C_src (#2398)

  -- Port CellQuadratic to C++ with GPU support (#2369)

  -- added bamg option to set restriction type (#2402)

  -- Fix dot product mask in EB nodal solver (#2404)

  -- Move Tools/C_util/Convergence/WritePlotFile to Src/Extern/amrdata. (#2397)

  -- Arena Document and PrintUsage (#2403)

  -- BArena: Make a static BArena object (#2401)

  -- Make fcompare print usage and exit if only one plotfile is provided. (#2388)

  -- Add Arena::PrintUsageToFiles for debugging (#2392)

  -- Update c util (#2390)

  -- Adding ILU smoothing parameters for Hypre interface. (#2382)

  -- Avoid free after amrex::Finalize (#2389)

  -- Workaround for nvcc bug that (sometimes) incorrectly complains about unused TinyProfiler variables. (#2383)

  -- Add floor and ceil to parser (#2380)

  -- CI: Groups Include Branch (#2379)

  -- Fix bug in the particle communication routines when the number of chars in the message is > MAXINT. (#2376)

  -- ROCm now supports __managed__ (#2377)

  -- Fix read unitialized memory issue in MLMG solver (#2373)

# 21.10

  -- Update for oneapi 2021.4.0 (#2368)

  -- Port CellConservativeQuartic to C++ with GPU support (#2362)

  -- Port CellConservativeProtected::protect() to C++ with GPU support (#2347)

  -- CMake CUDA Debug: -G by Default (#2355)

  -- OverlapSync nowait and finish. (#2346)

  -- Add another overload for PC::Checkpoint (#2353)

  -- Call setLevelBC from getFluxes in case we have set up the projector
     class but not yet called project (#2345)

  -- Function to atomically set particle ids (#2344)

  -- Remove macro: CRSEGRNDOMP and AMREX_CRSEGRNDOMP (#2339)

  -- Don't rely on mananaged memory in PC::InitOnePerCell (#2337)

  -- TinyProfile parser that saves data in a machine-readable format (#2335)

  -- Fix return type of MLEBNodeFDLaplacian::setEBDirichlet (#2329)

  -- ParReduce (#2327)

  -- Remove some versions of ParallelFor(MF) (#2326)

  -- Update MLMG solver to allow variable number of ghost nodes for variable refinement ratios (#2319)

  -- CMake: Always CUDA Perf-Neutral Debug (#2321)

  -- Add version of SumBoundary that takes src_nghost (#2322)

  -- Remove the Old Fusing Approach (#2308)

  -- Add const to Array's sum and product functions (#2316)

  -- Fix signed/unsigned comparison in diagnostic neighbor list function. (#2312)

  -- HDF5 I/O and Compression  (#2220)

  -- Use ParallelFor(Tag) in Linear Solver Preparation (#2297)

  -- TagBoxArray: Remove omp parallel inside GPU launch region (#2309)

  -- Update DiffMultiFab tool (#2307)

  -- Kernel fusing in linear solver's applyBC (#2295)

  -- Adding new parallel for into normalize method of node laplacian. (#2299)

  -- Gpu::Atomic::If (#2301)

  -- add adaptor and bridge for amrMesh+Particles (#2285, #2364)

# 21.09

  -- Migrate Fortran kernles in the Advection_AmrLevel test to C++. (#2296,
     #2268)

  -- Move slope routines out of amrex into AMReX-Hydro (#2294)
    
  -- GNU Make: Fix GNU builds on Perlmutter, + some bug fixes. (#2286,
     #2293)

  -- Use ParallelFor(Tag) (#2292, #2267)

  -- Optimization of thread safety check in building communication metadata
     (#2288)

  -- SENSEI: Remove amr adaptor/bridge when AMRLEVEL is off (#2258, #2283)
    
  -- ParallelFor(Tag) (#2281)
    
  -- Protect against nullptr when calling PODVector dtor. (#2272)

  -- Parser: Hide local symbols (#2274)
    
  -- Add EditorConfig section for rst files (#2273)
    
  -- Fix bug in the dual grid PC restart in the case where there are no
     particles on some levels. (#2276)

  -- Fix FabArray move-ctor and move-= (#2270)

  -- Fine-tune FabArray::setBndry (#2269)

  -- MF ParallelFor: PETSc (#2262)
    
  -- MF ParallelFor: HYPRE (#2257)

  -- MF ParallelFor: MultiFabUtil (#2256)

  -- Fix Particle CI Tests (#2252)

  -- Github Action Concurrency (#2254)
    
  -- MF ParallelFor: MultiFab (#2249)
    
  -- Reimplement AddParticlesAtLevel for GPU (#2236)

  -- Enable mixed (face) GpuBndryFuncFab (#2238)

  -- Use the new ParallelFor in some FabArrayUtility functions (#2246)

  -- Use new ParallelFor in FabArray (#2245)

  -- CI: Build Embedded Boundaries (#2232)

  -- Also remove empty map entries when calling clearParticles (#2248)

  -- Bug fix of rarely met issue from Tools/CMake/AMReXUtils.cmake file (#2244)

  -- Move MF ParallelFor out of experimental:: (#2241)

  -- Option for dynamic tiling in CPU build of MF ParallelFor (#2235)

  -- Add sum(), product(), size(), {x,y,z}{lo,hi,len}() member functions to
     GpuArray and Array?D (#2217)

  -- Fix derive data in AmrLevel::writePlotFile (#2234)

  -- Add MFNodeBilinear for nodal linear interpolation (#2202)

  -- Adding experimental parfor into the FSmooth method (#2230)
    
  -- Parser: Exception handling (#2233)

  -- Parser: Noinline transcendental functions (#2229)

  -- Adding experimental parfor into the restriction operator (#2204)

  -- Fix PArena (#2221)

  -- Test Build Variant (#2223)

  -- EB: Fix small cells and multicuts (#2225)

  -- Make CreateVirtualParticles use GPU functions for aggregation_type=Cell (#2219)

  -- Change getgradphi in nodalproj (#2222)

  -- Fix EBInterpolater (#2214)

  -- Add EBMFCellConsLinInterp for EB cell data interpolation (#2203)

  -- Bilinear interpolation for cell data (#2205)

  -- CMake: Prebuild & SetupCUDA (#2209)

  -- Reduction of MulitFab to 1D data (#2182)

  -- YAFluxRegister with blocking factor of 1 (#2155)

  -- Fix lincc_interp (#2201)

  -- Add MFPCInterp for piece-wise constant interpolation (#2200)

  -- Fix Coarse/Fine Boundary BoxArray generated in FPInfo for FillPatch (#2207)

# 21.08

  -- Add AsyncVector. (#2197)

  -- Make CreateVirtualParticles use GPU functions for aggregation_type=None
     (#2195)

  -- Change mask for ctr to ctroid interp (#2196)

  -- Add OpenMP support to DenseBins::build (#2193)

  -- Integration of experimental parfor into NodeLaplacian Fapply. (#2190)

  -- Fix filter functions for callables that return bool. (#2192)

  -- MFCellConsLinInterp (#2189)

  -- Introduction of the experimental parfor into NodeLaplacian
     interpolation (#2183)

  -- Enable default construction for AmrParticleContainer and add
     isDefined() method. (#2188) A Particle container is considered
     "defined" if it has been given a set of grids, otherwise not.

  -- Make CreateGhostParticles use GPU functions (#2185)

  -- Add ROCMClang as an LLVM compiler to cmake (#2184)

  -- Scale uniform random interval to account for gridbox length in
     ParticleInit (#2179)

  -- Optimization of ParallelFor for single box FabArray (#2176)

  -- Add ns to gpu cns (#2177)

  -- CI: IntelLLVM known to CMake (#2136)

  -- Add default move constructors / assignment operators for other pc
     types. (#2174)

  -- Gradient in WarpX EB nodal solver (#2171)

  -- Update MacProjector::project to check if m_umac is nullptr (#2166)

  -- EB nodal solver (#2145) New EB nodal solver with Dirichlet EB.  This
     uses a finite-difference stencil.  It will be used by WarpX's static
     solver.

  -- Workaround for a DPC++ reduction issue (#2165)
     More reliable workaround for DPCPP reduction issue (#2167)

  -- Add ParmParse::remove (#2160)

  -- Remove MFGhostIter (#2162)

  -- Fix typos with the help of codespell (#2163)

  -- Parser Precision (#2154) Use double internally.  If the arguments are
     all floats, the final result will be converted to float.

  -- IParser: Integer Parser (#2158)

  -- Disable `jn` in Parser on Windows (#2152)

  -- Fix the size of the BC Vector in the multi-level version of
     ParticleToMesh. (#2149)

  -- Make ParticleToMesh and MeshToParticle support iMultifab. (#2137)

  -- Fix crashes with Conduit Blueprint Code (#2144)

  -- CMake: HIP Modernizing & RDC (#2031)

# 21.07

  -- Parser: This can be used at runtime to evaluate math expressions given
     by strings. It's ported from WarpX. (#2121, #2129, #2132)

  -- Bug fix in neighbor particle list construction. (#2131)

  -- Bug fix in multi-level nodal solver with non-uniform cell size. (#2126)

  -- Add SENSEI Adaptor for Particles Containers (#2016)

  -- Enhance DiffSameGrid2.cpp (#2114)

  -- Don't rely on managed memory in ParticleCopyPlan (#2116)

  -- HIP: Increase to C++17 (#2113)

  -- Port remaining Fortran and CPU kernels in FluxRegister to GPU (#2112)

  -- Reimplement partitionParticlesByDest (#2109)

  -- Fix a bug in in-place ExclusiveSum (#2105)

  -- Add AUTO_BUILD_SOURCES to the dependencies for GNU make (#2101)

  -- Add a number of forward declaration headers (#2100)

  -- CMake 3.20+ CUDA: Fast Math (#2094) Honor CUDA fast math option
    `AMReX_CUDA_FASTMATH` (default: ON) in modernized CUDA CMake logic.

  -- Remove VecReduce and FabReduce (#2091)

  -- Fix indexes bug for particles output in GPU version of packIOData (#2087)

  -- Remove deprecated Reduce functions (#2086)

  -- Disable kernel fusing if gpu rdc is off. (#2083) Because the kernel
     fusing depends on rdc.

  -- ParallelFor for MultiFab/FabArray (#2073, #2089)

  -- ParticleContainer-level reduction functions that work on tuples of data at once. (#2082)

  -- AMReX_PROBINT: Depends on AMReX_AMRLEVEL (#2081)

  -- Arena aliases (#2078)

  -- CMake 3.20+ CUDA: Policy & Include (#2074)

  -- GPU RDC Support (#2061)

  -- Generalize fextract allowing to specify coords perpendicular to slicedir (#2035)

  -- Add interface between sundials SUNMemory and the Arena (#1800)

  -- Do not rely on managed memory in ParticleContainer::WriteAsciiFile (#2070)

  -- Particle plotfile/checkpoint without managed memory. (#2063)

  -- Support preprocessing of AMREX_SPACEDIM in input files. (#2049)

# 21.06

  -- Add roctx ranges to TinyProfiler (#2057)

  -- CMake: Link with HIP_LIBRARIES (#2055)

  -- Extrapolator with nGrow > 1. (#2054)

  -- Perlmutter make (#2058)

  -- Fix to correct CUDA flag and add missing libraries in hypre cmake (#2056)

  -- Remove IndexSequence (#2052)

  -- CMake link with roc::rocprim for HIP (#2050)

  -- Fix PODVector to work without managed memory. (#2048)

  -- When building neighbor masks, perform setVal on device so it will work
     without managed memory. (#2046)

  -- Async Output Docs (#2041)

  -- Option to set amd gpu targets (#2038)

  -- Work-Around: oneAPI 2021.3.0 SYCL with oneDPL (#2025)

  -- ParamParse: Find Entries under Prefix (#2043)

  -- Add relative gradient to AMRErrorTag (#2044)

  -- Workaround argument mismatch in MPI reduce for gfortran >= 10 (#2042)

  -- Wrappers for builtin clz extensions and tests thereof. (#2036)

  -- HIP: AMREX_AMD_ARCH Env Variable (#2034)

  -- Add spock support to GNU make (#2028)

  -- nvcc --display-error-number --diag-error errNum (#2033)

  -- Build neighbor masks without managed memory (#2030)

  -- Option to use oneDPL (#2027)

  -- CMake: add support for CMake >= 3.20 (#2012)

  -- Turn on RDC support for HIP (#2029)

  -- Device profilerStart and profilerStop APIs (#2026)

  -- Launch bounds (#2013)

  -- Arena initial size (#2007)

  -- Use CPU memory in memory pools for OpenMP threads (#2006)

  -- No casting to unsigned long long needed in iMultiFab::sum (#2005)

  -- Arm Clang (#2002)

  -- Templated implementation of ostream operator for both Dim3 and XDim3 (#1997)

  -- Generalize elemwiseMin and elemwiseMax. (#1996)

  -- Optimizing away temporary BoxArrays in AmrMesh::MakeNewGrids (#1994)

  -- Non-blocking SumBoundary (#1991)

  -- Deprecate FabArray::copy() (#1985)

  -- Fix proper nesting issue for large error buffer (#1987)

  -- Update HDF5 I/O to use new HDF5 Async VOL APIs (#1948)

# 21.05

  -- Support OpenMP offload with the NVHPC compiler in GNU make (#1980)

  -- In GNU make, use -MMD -MP instead of -MM to generate dependency if
     supported (#1967)

  -- Workaround compile error on Apple M1 (ARM64) (#1969)

  -- Changes to EB slopes (#1965)

  -- In GNU make, use `LOG_BUILD_TIME=TRUE` to log build time. (#1963)

  -- Using rocPRIM in HIP Version of Scan (#1962)

  -- Reimplementation of ReduceOps (#1955) The new approach does reduction
     in two passes and therefore does not require atomics.

  -- CMake: implement logic to support compiler ID IntelLLVM (new in CMake
     3.20) (#1959)

  -- CPU version of TagBox::buffer (#1957)

  -- Fix out of bound error in AsyncWrite (#1753)

  -- Use pinned memory ad buffer in Fab data I/O (#1956)

  -- Allow user to opt out of renaming MTMPI suffix (#1954)

  -- Port FluxRegister::ClearInternalBorders to GPU (#1952)

  -- Skip temporary in ParticleToMesh when it is not needed. (#1949)

  -- Multilevel version of ParticleToMesh (#1945)

  -- Use CUB for block reduction (#1943)

  -- Reimplementation of PrefixSum with cub (#1942)

  -- Cell-Centered Overset Solver Coarsening (#1915)

  -- Hypre Nodal Setup (#1906) Allow CoarseningStrategy::Sigma in Hypre
     nodal solver and port setup to GPU.

  -- CMake: add support for BOUND_CHECK (#1934)

  -- Alias HostVector to PinnedVector and update documentation. (#1931)

  -- Make CNS work on CPU too (#1930)

  -- std::isfinite/inf for SYCL (#1929)

  -- Update RNG for latest oneAPI (#1928)

  -- Use pinned memory for data transfer between host and device (#1923)

# 21.04

  -- ParmParse parameter, nodal_proj.normalization_threshold, for setting
     the normalization threshold of the bottom linear solver. (#1917)
     The default value of the threshold has been changed from 1e-10 to
     1e-8. (#1914)

  -- Fix a bug in BaseFab::resize. (#1916)  This caused performance issues
     on systems with managed memory and crashes if not.

  -- Option to not zero out input multifab in amrex::ParticleToMesh (#1910)

  -- CMake: add test_install (#1894)

  -- Reimplement increment and incrementwithtotal for particles to take
     advantage of GPUs and OpenMP (#1899)

  -- Return const& in Particle rdata and idata and disallow returning copies
     for rvalues. (#1895)

  -- Update ALaplacian for multi-component and reuse (#1891)

  -- CMake: IPO/LTO (#1890)

  -- Make FabArray::setFab safer (#1884) Instead of taking a raw pointer, it
     takes std::unique_ptr or rvalue FAB, so that the ownership is very
     clear.  It also deletes the old FAB if not null.

  -- Flags: RelWithDebInfo (#1879) Add warning control to CMake Build mode
     `RelWithDebugInfo` (currently default for WarpX).

  -- Build: Windows with Shared Libs (.dll) (#1847)

  -- Remove tutorials documentation from main repo; it now lives at
     https://github.com/AMReX-Codes/amrex-tutorials (#1877)

  -- 2D Poisson and ALaplacian Solvers in 3D Build of AMReX (#1805) New
     capability of solving 2D cell-centered Poisson's and ALaplacian
     equation with 3D build of amrex.

  -- Add FabArray::release and Make BaseFab virtual (#1873)

  -- Robin BC in Linear Solver (#1848)

  -- LevelBld (#1852) This is a breaking change for Amr/AmrLevel based
     codes.

  -- Optimization of CArena::freeUnused (#1865)

  -- CI for macOS: Build a shared library. This is the default in
     Spack. (#1836)

  -- Remove tabs and trailing white spaces (#1860)

  -- Free unused memory (#1861) If Arena is asked to allocate more than
     what's available in the system, try to free unused memory first.

  -- Fortran Interfaces for new FillPatch (#1793)

  -- Arena Release Threshold (#1857)

  -- Feature/non local parallel copy (#1842) This PR introduces "Multiblock"
     capabilities to AMReX.

  -- AMREX_NO_PROBINIT (#1846) For Amr/AmrLevel based codes, there is now a
     compile time option AMREX_NO_PROBINIT=[TRUE|FALSE] for the support of
     the probin file and amrex_probinit function.

  -- Add EditorConfig (#1841)

  -- Fix ncomp bug in linear solver (#1832)

  -- CMake: AMReX_INSTALL (#1831) Add an `AMReX_INSTALL` option that
     controls `install()` generation.

# 21.03

  -- Support multiple components in the ABecLaplacian solver. (#1825)

  -- Add query functions to Arena. (#1823) They can be used to find out the
     memory type and if the memory is accessible on device or host.

  -- CMake: add -Wno-pass-failed to Clang-based compilers (#1815)

  -- Option to cover multiple cuts in EB generation (#1810)

  -- The_Async_Arena and Elixir::append (#1804)

  -- Add function name output option to TinyProfiler. (#1803)

  -- DPCPP: Use codeplay_host_task (#1797)

  -- Add ParticleArray classes (#1796)

  -- ParallelCopy_nowait & ParallelCopy_finish (#1765)

  -- Add FabArray::tags() to return the tags. (#1794)

  -- Bump minimum C++ standard from 11 to 14. (#1787)

  -- Update the preferred short name of NVIDIA HPC SDK to nvhpc (#1788)

  -- Fix precision issue in ParmParse::add (#1783)

  -- Overset Solver with Refinement Ratio of 4 (#1778)

  -- Refinement Ratio of 4 Support in Nodal Solver (#1774)

  -- MSVC: Proper __cplusplus macro (#1773)

# 21.02

  -- Abort when MLMG is detected as failing regardless of verbosity
     setting. (#1762)

  -- Update DPC++ version of Anyof (#1757)

  -- Use ParticleReal for particle interpolation (#1763)

  -- Remove obsolete functions from AMReX particle classes (#1754, #1755)

  -- Make ParticleContainer::Geom always refer to the particle one. (#1470)

  -- Gpu Docs Update (#1758)

  -- Add IntVect versions of numParticlesOutOfRange. (#1751)

  -- Improve the interpolation from cell centroids to face centroids:
     (#1750)

  -- In ParticleToMesh and MeshToParticle, use nGrowVect() to accommodate
     different a different number of grow cells in each dimension. (#1748)

  -- HIP: New Warp Size Macro (#1742) Use new the HIP macro for the warp
     size on various AMD devices.

  -- Add allocator template parameter to ParticleContainer (#1416)

  -- Add ability to calculate slopes on face centroids (#1707) And apply the
     explicit stencil when computing the laplacian.

  -- HIP indirect function and kernel fusing (#1739) Add a GNU Make flag,
     HIP_INDIRECT_FUNCTION=[TRUE|FALSE], and modify the Gpu kernel fusing
     test.

  -- When using CMake 3.15+ allow for MSVC runtime library changes (#1736)

  -- RoundRobin: option to not sort processes (#1721)

  -- EB: edge centroid (#1730) Add edge centroid to the database and make it
     available via EBFArrayBoxFactory.

  -- Fix RegridOnly to regrid even with single level grids (#1722)

  -- Define: _OPENMP -> AMREX_USE_OMP (#1560) Replace the define `_OPENMP`
     with `AMREX_USE_OMP` for all parallel "backend" implementations and
     control of MFIter loops.

  -- Update Make.nrel to accommodate MPT MPI with CUDA enabled (#1726)

  -- Cell-centered Hypre and Petsc setup on GPU (#1709)

  -- ParmParse::hasUnusedInputs & getUnusedInputs (#1716) Add functions to
     find if there are unused `ParmParse` parameters starting with
     `prefix.`.  If `prefix` is empty, all unused `ParmParse` parameters are
     included.

  -- CMake: offer option to exclude Src/Amr from build (#1714)

  -- Make bisect more robust (#1708)

  -- Support single precision build with configure (#1706) Add new arguments
     `--single-precision` and `--single-precision-particles` to `configure`.

  -- Support when CMake 3.18 policy 105 is set to new (#1696)

  -- Optimized function for filling BaseFab of GpuArray (#1692)

  -- BaseFab::minmax (#1691)  Add BaseFab::minmax function and use it in
     VisMF.

  -- New feature in the fextract tool to directly export in the csv format
     (#1634)

# 21.01

  -- Optimization of MLCellLinOp::applyBC (#1690)

  -- Optimization of iMultiFab sum (#1684)

  -- Sanitize the input of mkconfig (#1685)

  -- New versions of Reduce::Min, Max and MinMax taking callable (#1679)

  -- Only issue prefetch if arena uses managed memory (#1678)

  -- GNU Make: Fix the library flags (#1641)

  -- PolymorphicArray4, a thin layer around Array4 allowing us to access
     both AoS and SoA data with operator() (#1643)

  -- CMake: dowstream projects must enable Fortran if AMReX_FORTRAN=ON
     (#1669)

  -- Optimization of FaceLinear interpolater (#1663)

  -- Allow alternate form of "check_pair" function (#1667)

  -- ParallelFor for Reduction (#1658)

  -- Add features to fextract (#1657)

  -- Fix parallel add bug when both USE_GPU and USE_OMP are defined (#1654)

  -- Write a plotfile to ASCII (#1638)

  -- Fix NodalProjector::getGradPhi for multilevel case (#1642)

  -- Remove GPU pragma support (#1650) (#1649)

  -- Remove custom MFIter reduction code (#1648)

  -- Remove support for old versions of DPCPP (#1644)

  -- Fused FillBoundary (#1570)

  -- Divergence Free FillPatch (#1483)

  -- Optimization of constant coefficient MacProjector (#1636)

  -- Optimization of nodal solver with constant coefficient (#1629)

  -- Expose the index of the neighboring particle to users of the
     iterator. (#1628)

  -- Implement overset mask for MLPoisson too (#1604)

  -- Fuse nodal jacobi smoother for GPU (#1625)

  -- Support for SoA data for neighbor particles. (#1590)

  -- Fix a bug that causes errors when writing HDF5 plotfile and particle
     (#1621)

  -- Fortran Interface: Porting more MultiFab functions (#1609)

  -- Remove alternative operators from AMReX (#1599)

  -- EB from STL (#1592)

  -- AMReX_Config.H generated at built time is included in all
     headers. (#1566)

  -- MacProjector: Update setDomainBC API (#1589)

  -- Reimplement NumParticlesAtLevel (#1581)

  -- Add msvc cuda support (#1573)

# 20.12

  -- EB2::PolyIF (#1567) The existing PolynomialIF contains a dynamic size
     vector.  This makes it hard to support it on GPU.  This new PolyIF
     class uses fixed size array and has GPU support.

  -- Tools/Plotfile works for for single precision now. (#1577)

  -- Arena: Add accessor to query Arena attributes through info object
     (#1575)

  -- Allow MacProjection reuse (#1574)

  -- Remove SUNDIALS Fortran interface support (#1559)

  -- Allow MLEBABecLap to solve for multiple components (#1557)

  -- DPCPP: MPI (#1556) Allow MPI for dpcpp build in gnu make assuming
     mpiicpx is used.

  -- Non-local boundary conditions (#1544)

  -- If TheZeroVector() is passed in to SortParticlesByBin, do nothing
     instead of crashing. (#1564)

  -- Scan size type (#1521) Make the size type a template parameter to
     support the scan of more than INT_MAX elements.

  -- DPCPP: Fortran (#1554) Switch from gfortran to ifx that comes with
     oneapi in GNU make.

  -- CMake: generate_buildinfo target (#1545) This commit generates a static
     library `buildInfo::<target>` for the passed user target. This target
     is then automatically linked.  Also, the generated `.cpp` file is now
     scoped properly, avoiding collisions.

  -- nvcc --Werror cross-execution-space-call (#1540) This is turned on by
     default in GNU make, and can be turned on in CMake.

  -- fcompare: Add option for an absolute tolerance check (#1537)

  -- Remove call to cudaDeviceSetCacheConfig (#1536) No longer set CUDA
     device cache configuration to prefer L1 cache.

  -- Assert no nested MFIters (#1530) Add assertion to catch nested MFIters
     (e.g., MultiFab functions are called inside MFIter).

  -- Free MPI_Datatype defined by AMReX (#1527) This fixes a one-time memory
     leak in MPI_Datatypes defined by AMReX as reported in #1525.

  -- make clean & make cleanconfig (#1520) `make clean` now does what `make
     realclean` does.  `make realclean` is kept.  `make cleanconfig` is
     introduced to do what `make clean` used to do.  For most users, `make
     clean` is the one that should be used.

  -- hypre: Adjust singular matrices before hypre solves (#1519)

  -- CMake: overhaul options (#1490)

  -- New Reduce::Sum function taking lambda function (#1508)

  -- GNU Make: AMREX_CUDA_ARCH (#1507) Option to use AMREX_CUDA_ARCH in
     addition to CUDA_ARCH to set CUDA arch.  If AMREX_CUDA is set, it will
     be used over CUDA_ARCH.

  -- Add option for device sync at beginning and end of TinyProfiler region
     (#1505) Now when setting
     `tiny_profiler.device_synchronize_on_region_end = 1` in the inputs
     file, we will synchronize before calling nvtxRangePop() and
     nvtxRangePush(), which means that TINY_PROFILE regions will include the
     full kernel time, rather than just the kernel launch time.

  -- CMake; bump up hypre minimi required version (#1503)

# 20.11

  -- The default hypre interface in MLMG is now IJ matrix for both EB and
     non-EB.  Previously, it was semi-struct for non-EB. (#1492)

  -- USE_SAVE_TEMPS option in GNU Make system for HIP. (#1492)

  -- The device version of `Error`, `Abort` and `Warning` is no-op, unless
     `NDEBUG` is defined. (#1492)

  -- Ascent: Use Default Name for Mesh Ghost (#1488)

  -- Add fill method to GpuArray (#1472)

  -- Add Gpu::Atomic::AddNoRet (#1469)

  -- Option to make the code NOT enforce solvability in MLMG for singular
     problems. (#1471)

  -- CMake: improve HDF5 support (#1468)

  -- CMake: FindPETSc must not overwrite CMAKE_Fortran_FLAGS (#1464)

  -- CMake: prevent in-source builds (#1453)

  -- change char * to char const* so SWFFT code compiles with USE_CUDA=TRUE
     (#1454)

  -- Fix when small_volfrac is used -- we need to adjust the data in cut
     cell neighbors as well as regular neighbors. (#1451)

  -- Allow users to set a Geometry object for ParticleContainer
     independently of the AmrCore / AmrLevel object. (#1446)

  -- HDF5 plotfile write bug fix and performance improvement (#1448)

  -- CMake: add HIP support (#1316)

  -- NodalProjector: use volume-weighted average down (#1444)

  -- Fix race conditions in EB interpolation from cell centers to faces
     (#1443)

  -- Allow tagging value to vary by level (#1441)

  -- Hypre with overset (#1439)

  -- Hypre IJ interface: Enable access to additional solvers and
     preconditioners available in Hypre (#1437)

  -- Fix CPU version of uninitailiedFillNImpl in PODVector (#1435)

  -- CMake: Fix missing compile time definitions for HYPRE/PETSc (#1436)

  -- Fix bug in EB extdir slopes (#1434)

  -- Make the id and cpu members of amrex_particle private, as they should
     no longer be accessed directly. (#1433)

  -- Disable OpenMPI C++ binding in GNU make system (#1398)

  -- Fix a long standing bug in eb levelset (#1432)

# 20.10

  -- CMake: fix ENABLE_PROFPARSER and ENABLE_SENSEI options (#1428)

  -- Remove EB LSCore and levelset, and reimplement signed distance function
     capability. (#1429 & #1425)

  -- Remove flags setCGVerbose and setCGMaxIter since those are more
     accurately called setBottomVerbose and setBottomMaxIter.  The
     "setCG..."  flags are misleading since they actually apply to non-CG
     bottom solvers as well. (#1413)

  -- Remove GetPosition and SetPosition from ParIter.  These were added for
     WarpX, but they are not used any more. (#1408)

  -- Only check the requested MPI Threading level if AMReX also initialized
     MPI. (#1406)

  -- DPCPP support for multiple GPUs (#1392)

  -- Random Number Generation on Device (#1363) In order to support RNG on
     device with DPC++, we have to change the API because DPC++ does not
     support global device variables.

  -- Remove redundant particle tests and old/obsolete/non-test tests (#1389
     & #1381)

  -- Gpu kernel fusing is integrated in MFIter. (#1332)

  -- New runtime parameter, amrex.max_gpu_streams. (#1386)

  -- Switch to `use mpi` from `include 'mpif.h'` (#1385)

  -- Remove reliance on managed memory from
     AMReX_ParticleCommunication.H/.cpp (#1380)

  -- CMake: re-organize tutorials (#1333)

  -- Implement Particle in a way that does not involve UB. (#1337)

  -- amrex::EB_average_down(): Fix indexing error. (#1360)

  -- Only add one copy of the ghost particle regardless of how many isects
     we have (#1359)

  -- Add non-EB slopes and extend the generality of EB slope routines in 2D
     and 3D (#1350)

  -- Async IO: Check Runtime Threading (#1351)

  -- Changes to make AMR codes compiled with EB but not building any
     Geometry work as if it is all regular. (#1349)

  -- Fix mkconfig.py (#1344)

  -- Add -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored to
     suppress spurious warnings from nvcc. (#1342)

  -- EB extend_domain_face (#1321)

  -- Add parameter that stores OpenMP support version (#1325)

# 20.09

  -- Fix interpolation coefficients for EB stencil in cell-centered linear
     solvers when the EB surface at the domain boundary is tilted.

  -- Add a parameter to `AMReX_omp_mod` that can be used by Fortran code to
     check whether AMReX was built with OpenMP support, and, if yes, what
     the version number (_OPENMP) was at the time AMReX was compiled.

  -- CMake: re-write genex evaluation functions.

  -- Extend the number of unique particles per cpu we can have at once.

  -- Add a new method to the `amrex_distromap` type in
     F_Interfaces. `get_pmap` fills a caller-owned array of PEs.

  -- Fix a long standing bug in GNU make system on the use of HOST and
     HOSTNAME.

  -- Add Scan::InclusiveSum and ExclusiveSum for CPU to avoid ifdef.

  -- Add check for empty probin_file in Amr::restart.

  -- Update Sundials interface, documentation, and build to be version
     agnostic.

  -- Updates for ROCm 3.6 and 3.7.

  -- Add section to Make.unknown for intel mpi.

  -- CMake: re-factor third party libraries setup.

  -- Port TagBoxArray to GPU.

  -- Fix a bug in matching of COMP_VERSION to correctly treat CCE > 9.

  -- By default, EB outside the domain will be generated by extending from
     the domain faces.

  -- Fix an issues due to roundoff error in determining whether particles
     are inside the domain.

  -- Remove Perilla because it's incompatible with our GPU strategy and it's
     no longer being maintained.

  -- Switch the overset mask convention to that 1 means unknown and 0 means
     known.  The new convention is more convenient for AMR-Wind.

  -- Remove old solvers, Src/LinearSolvers/C_CellMG and C_TensorMG.  They
     have been superseded by Src/LinearSolvers/MLMG.

  -- MacProjector: allow for re-use of the object and enhance multi-level
     algorithm.

  -- Add asserts for bounds checking to Array1D and Array2D.

  -- Optimization of the construction of SFC.

  -- Refactoring of particle buffer map.

  -- Optimization of FPinfo and complementIn.

  -- Move the Amr/Extrapolater used in IAMR/PeleLM to C++/GPU compliant.

  -- Less surprising behavior for the ok() method of FabArray. This makes
     the `ok()` method of FabArray return `false` instead of crashing if the
     `define()` method has yet to be called.

  -- Optimization of regrid.

  -- Special FillBoundary for Almo.

  -- Fix the average_down_faces calls in linear solvers to see periodicity.

  -- Make sure the calculation of divu at inflow face in nodal projection
     does not use tangential velocities on an inflow face.

# 20.08

  -- New average_down_faces functions that take periodic boundary into account.

  -- Update the documentation for Nsight.

  -- Fix a bug with HDF5 boxcenter data.

  -- New runtime parameter, `eb2.extend_domain_face`, that can be used to extraploate EB information
     from domain faces.

  -- New GNU Make options for GCC and Clang, `WARN_ALL` and `WARN_ERROR`. `WARN_ALL` turns on more
     compiler warnings.  `WARN_ERROR` turns warnings into errors.

  -- Fix various compiler warnings.

  -- Fix memory fragmentation issue with communication buffer.

  -- Semi-coarsening support in linear solvers.

  -- Fix virtual particle issues.

  -- Capability to fuse GPU kernel launches.

  -- Make ghost particles support soa data.

  -- CMake: ability to export build tree.

  -- Reset amrex_geometry_module initialization state when finalizing.

  -- Remove Long version of Gatherv.

  -- Optimization of TagBoxArray::collate.

  -- CMake: fix missing PETSc setup in Config file.

  -- BndryFunc for nodal data.

  -- MLMG Hypre fix for singular problems.

  -- Add particle copy filters.

  -- Async output support for particles.

  -- BackgroundThread class.

# 20.07

  -- Nodal hypre solver now supports coarsening before hypre is called.

  -- A new ParmParse parameter, `amrex.the_arena_is_managed` is introduced to set the default memory
     type of `The_Arena`.

  -- MFIter is now OMP thread safe when it is compiled with GPU support.

  -- Overset support is added to cell-center and tensor linear solvers.

  -- Remove compile time flag, BACKTRACE, for simplicity.

  -- Updates on Blueprint and Ascent.

  -- HDF5 support in GNU Make and CMake.

  -- CMake support for DPC++.

  -- Add -DNDEBUG for non-debug build.

  -- Support for Windows.

  -- Support for ParallelContext in Particles.

  -- Non const iterator for neighbor list.

  -- Tool for comparing single-level plotfiles of all nodalities.

  -- GNU Make: Use -M instead the perl script to generate dependencies.

  -- Particle: Remove directly accessing the m_idata and m_rdata structs internally.

  -- Extend the makeSFC method to take an additional argument where the user can override the number
     of processes a BoxArray is distributed to.

  -- Turn tiling off by default for gpu.

# 20.06

  -- Set a non-default CMAKE_INSTALL_PREFIX only if AMReX is the
     top-level CMake project. Otherwise, the default of a downstream
     project using AMReX as subproject will be placed in a quite
     confusing, temporary directory.

  -- C order support for Array2D and Array3D.

  -- Disable unqualified abs function within amrex namespace.

  -- Updates for SENSEI 3.2.

  -- CMake: link to OpenMP::Fortran only when needed.

  -- DPC++ support.

  -- Add non-const data() to GpuArray.

  -- Fix HIP version of scan.

  -- Return immediately from GPU memcpy if size is zero.

  -- Documentation on EB linear solvers.

# 20.05

  -- AMReX's git history has been rewritten.  If you have problems due
     to this, follow the instruction here. https://xkcd.com/1597/  We
     also have a new CONTRIBUTING.md.

  -- EB linear solvers: You now need to specify the location of
     diffusion coefficient (face center or face centroid) and the
     location of the unknowns (cell center or cell centroid).

  -- Optimization of makeKnapSack and makeSFC.

  -- Unification of GPU and CPU versions of neighbor particle /
     neighbor list implementations.

  -- New HostDeviceParallelFor functions that can run on either host
     or device depending on the launch region flag.

  -- Add CUPTI trace for CUDA kernel timing.

  -- Implement AMReX_FORTRAN (BL_NO_FORT) option in CMake.

  -- Fix CUDA build of tools.

  -- Fix invalid git version in CMake.

# 20.04

  -- BaseFab can no longer be used on device.  Many BaseFab functions
     now requires RunOn::Device or RunOn::Host template parameter when
     GPU support is enabled.

  -- FillPatch* functions are now templated to support FabArray in
     addition to MultiFab.

  -- Fix -pthread for nvcc by default in Make.unknown

  -- EB now has the option to build coarse level from GeometryShop
     when coarsening from fine level fails.  This could be used for
     hyperbolic system with EB resolved to the finest level

  -- CMake: add C++ 17 among valid options for AMREX_CXX_STANDARD

  -- Add option to set fixed number of iterations in the Fortran
     interface of linear solver

  -- Fabs in alias FabArray now carries EB information.

  -- Add AMREX_LOOP_3D and 4D macros.

  -- Make amrex::Random() return numbers in the same interval for both
     CPU and GPU: [0:1)

  -- More flux register functionality for flash

  -- Chunked partition in particle redistribution to use less memory.

# 20.03

  -- AmrCore can now be built without using ParmParse.

  -- Improve BL_NO_FORT support.

  -- Fix OMP bug in MFIter constructors taking BoxArray and
     DistributionMapping.

  -- Cray compiler has been re-enabled in CMake.

  -- For device only code, the users can now use #if AMREX_DIVICE_COMPILE.

  -- Fix an integer overflow issue in TileArray cache.

# 20.02

  -- RealVect updated for GPU.

  -- Support for restarting from HDF5 particle checkpoint file.

  -- Improvement of MacProjector and NodalProjector

# 20.01

  -- GPU host device version of swap.

  -- Fix an edge case of gpu block reduce where the block size is less
     than a warp.

  -- Always check MPI buffer size for overflow.

  -- CMake support for PETSc.

  -- Include C sources in the CUDA proprecessing.

  -- It's now safe to construct a zero components Fab.

  -- CXXSTD: option to set c++ standard in GNU make system

  -- Improve performance of the nodal linear solver on GPU.

# 19.12

  -- Fix potential deadlocks in amrex::Random.

  -- MultiFab::Swap that does shallow swap if it can, does deep swap
     otherwise.

  -- FabArray::arena function that returns the Arena used to build.

  -- Nodal projection sync residuals are implemented for RAP.

  -- GPU support for initializing Fabs to NaNs.

# 19.11

  -- AmrCore::regrid will now simply return instead of abort if the
     base level is already the maximal level.

  -- Add NodalProjector that wraps MLMG for convenience.

  -- Enforce in AmrCore that blocking factor must be a power of 2.

  -- Add support to write plotfile in HDF5 format.

  -- FabArray's Reduce functions can now take lambda working on Array4
     in addition of FAB.

  -- Add AMREX_REAL_MIN, AMREX_REAL_MAX and AMREX_REAL_LOWEST for
     convenience.

  -- GPU launch functions and macros are now safe with empty boxes.

  -- Remove FabView from BaseFab.

  -- Avoid race conditions in GPU reduction code called multiple
     times.

  -- MLMG tensor solver is fixed for AMR.

# 19.10

  -- Add inhomogeneous Neumann BC to MLMG for convenience.

  -- Fix a corner case of tensor solver.

  -- Various fixes for single precision and particle real.

  -- Add amrex::Long that has a size of 8.

  -- Add int8_t support in typecheck.

  -- C++ user literal _rt for amrex::Real and _prt for
     amrex::ParticleReal.

  -- EB MAC projector now requires the user to specify whether the
     face velocities are defined on face centers or centroids.

# 19.09

  -- First pass of Sr/EB on GPU.

  -- Optimization of particle redistribution.

  -- EB integral: add integral of x*y*z

  -- HostDevice::Atomic::Add that does atomic add for both device and
     host omp code.

  -- makeFineMask: optional arguments specifying coarse and fine flag
     values.

  -- Optimization of FabArray communication for GPU by merging small
     kernels.

# 19.08

  -- MacProjector has the option to set maximal coarsening level.

  -- OwnerMask and OverlapMask are now built on GPU.

  -- Random number generation on GPU no longer requires an call to
     initialize seeds.  They are done in amrex::Initialize and resize
     is not needed when launching with more threads than the size of
     seed array.

  -- Optimization of NeighborParticles.

# 19.07

  -- Geometry/CoordSys: Static class members have been removed.
     Functions such as `isPeriodic`, `isAnyPeriodic`, `isAllPeriodic`,
     `ProbLo`, `ProbHi`, `IsCartesian`, `IsRZ`, and `IsSPHERICAL` are
     now non-static.  The default Geometry can been obtained via
     `DefaultGeometry()` function.

  -- Function amrex::Loop and amrex:LoopConcurrent are added to help
     looping over cells in a Box.

  -- Nodal solver: A bug in nodal owner mask that could affect
     covergence was fixed.

  -- Fortran amrex_amrcore_module: add amrex_set_boxarray,
     amrex_set_distromap, and amrex_set_finest_level.

  -- Option to allocate MultiFab data in arenas different from
     The_Arena().

  -- Option to tag MultiFabs and query their memory usages.

  -- AsyncFab has been deleted.

# 19.06

  -- Geometry/CoordSys: New constructor and define function are added.
     We plan to remove static data from Geometry and its base CoordSys
     in the next release.  This means most of the static functions in
     Geometry class will become non-static.  These include
     `isPeriodic`, `isAnyPeriodic`, `isAllPeriodic`, `ProbLo`,
     `ProbHi`, `IsCartesian`, `IsRZ`, and `IsSPHERICAL`.  You can
     prepare codes now for the upcoming changes.  In almost all cases,
     all you have to do is replace `Geometry::function` with
     `DefaultGeometry().function`.

  -- Add tensor linear solvers for viscous flows with and without EB.

  -- Algoim: A modified version of Algoim is added.  This removes the
     dependence of 3d EB nodal MLMG solver on the official Algoim code
     and Blitz.

  -- EB: Fix the potential ownership issue is IndexSpace.

  -- Add a new FillPatchTwoLevels for EB.

# 19.05

  -- Interface of virtual function Interpolater::interp has changed.

  -- Add Fortran funcntions for shared memory reductions.

  -- EB nodal linear solver: The default bottom solver is now bicgcg.
     It starts with bicgstab and switches cg if bicgstab fails.

  -- EB nodal linear solver: Finished the implementation of matrix
     coarsening with RAP.

  -- Particle: Option for adding runtime struct of array components.

  -- Add a trilinear interpolation option for 3D.

  -- Add more support for non-uniform grow vectors to MultiFab and
     FabArray.

  -- AmrMesh: n_error_buf is now IntVect.  This allows for different
     number of error buffer cells in different directions.

  -- Using PETSc as bottom solver no longer requires Hypre.
 
# 19.04

  -- SUNDIALS 4 support.

  -- Option to disable device sync in MFIter.

  -- Functions for prefetch unified memory.

  -- GNU Make system: It is now an error if a function with return
     type does not return.

  -- GNU Make system: Pass C++ flags to nvcc when compiling C codes.

  -- Allocate 3/4 of gpu memory upfront in The_Arena.

  -- BaseFab now has Allocator template parameter.

  -- Random number generation on gpu.

# 19.03

  -- Deprecated Fortran framework F_BaseLib and linear solver F_MG
     have been removed.  A number of tools written in F_BaseLib have
     been migrated to Tools/Plotfile/.  Note that the Fortran wrappers
     to C++ (F_Interfaces) are still there.

  -- In addition to launch macros, there are now functions for
     launching, amrex::launch, amrex::For and amrex::ParallelFor.
     These improves debuggability. 

  -- If compiled with BASEFAB_IS_PINNED, pinned memory will be used
     for BaseFab meta-data.  On systems with nvlink, this improves the
     performance a little bit.

  -- Performance improvement in particle redistribution.

  -- Particle NeighborList has been moved from tutorial to AMReX
     proper.

  -- Cell-centered MLMG linear solver have been ported to GPU.

# 19.02

  -- Array4 class that contains the important information in BaseFab
     class.  It can be captured by lambda and used on gpu.

  -- A number of new macros for launching for loops: AMREX_FOR_1D,
     AMREX_FOR_3D, AMREX_FOR_4D, AMREX_PARALLEL_FOR_1D,
     AMREX_PARALLEL_FOR_3D, and AMREX_PARALLEL_FOR_4D.

  -- Disable cuda aware MPI by default.  It can be enabled with
     parameter amrex.use_gpu_aware_mpi.

# 19.01

  -- Documentation and Tutorials on GPU

  -- EB implicit function using spline

  -- FabArray::fabPtr() function takes MFIter and returns a device
     pointer to BaseFab.

# 18.12

  -- More GPU supports

# 18.11

  -- GPU support in Base and Particle

  -- BaseFab is no longer polymorphic

  -- EB now supports ghost cells outside physical domain

# 18.10

  -- MacProjector class for MAC projection

  -- Support for Dirichlet EB in linear solver

  -- Option to use PETSc as bottom solver

  -- Multi-level linear solver with EB not crossing coarse/fine
     boundary. 

  -- Sundials 3 support

# 18.09

  -- Cell-centered linear solver with homogeneous Neumann embedded
     boundary. 

  -- Hypre as a bottom solver has a IJ matrix interface and supports
     homogeneous Neumann boundary condition on embedded boundary.

  -- Old embedded boundary codes have been removed.

# 18.08

  -- Faster geometry generation is available at Src/EB2.
     Tutorials/EB/CNS is now based on EB2.

  -- AMReX is now fully compliant with xSDK.

  -- Fortran interface for particles.  See Tutorials/Amr/Advection_F/
     for an example.

# 18.07

  -- Pre- and post-interpolation hooks in FillPatch.

  -- NDEBUG is no longer defined in xSDK mode.

# 18.06

  -- When amrex::Initialize is called, optional std::ostream arguments
     can be passed in and their defaults are std::cout and std::cerr.

  -- ParmParse inputs files can now contain Fortran namelist that can
     be accessed from Fortran (and Frotran only).  These Fortran inputs
     are stored in amrex_namelist character variable in
     amrex_parmparse_module, and can be read with standard Fortran
     I/O.
     
  -- There is a new non-blocking parallel copy function in
     MultiFab/FabArray called ParallelCopy_nowait.  It returns a
     CopierHandle object and can later be used to finish the
     communication with CopierHandle::finish().

# 18.05

  -- FillBoundary and ParallelCopy functions can now take IntVect
     arguments as the number of ghost cells.  That is the number of
     ghost cells could be different for different directions.

  -- MultiFab, iMultiFab and FabArray can have different number of
     ghost cells in different directions.

  -- Fortran module mempool_module has been renamed amrex_mempool_module.

  -- Many macros have been renamed.
     * USE_PARTICLES and PARTICLES have been renamed AMREX_PARTICLES
     * DIMENSION_AGNOSTIC has been renamed AMREX_DIMENSION_AGNOSTIC
     * USE_CVODE has been renamed AMREX_USE_CVODE
     * DEBUG has been renamed AMREX_DEBUG

 -- amrex::Array is now an alias to std::array in stead of amrex::Vector.

# 18.04

  -- New BoxList constructors making a given number of Boxes from a
     single Box.

  -- Optimization of particle redistribution

  -- MLMG nodal linear solver: add solvability fix

  -- Modification to amrex::Initialize and Finalize to make it
     suitable as a library in an application that does not use AMReX
     as a framework.

# 18.03

  -- STL support for EB

# 18.02

  -- Fortran interface: added new functions amrex_multifab_build_alias
     and amrex_imultifab_build_alias.

  -- Documentation in reST/Sphinx.  See Docs/Readme.sphinx for more
     details.

  -- Src/LinearSolvers/MLMG now supports nodal based data.

  -- Fortran module AMReX_fillpatch_module takes a user provided
     callback function for filling physical boundary conditions.  That
     function used to use 0-based index for component (i.e., the last
     index).  For consistence with 1-based index for component in
     other Fortran modules, this has been changed to 1-based index for
     component.

# 18.01

  -- New linear solver MLMG at Src/LinearSolvers/MLMG/.  Currently it
     supports cell-centered single level and multi-level composite
     solve.  Fortran interface is also available at
     Src/F_Interfaces/LinearSolvers/.  See Tutorials/LinearSolvers for
     tutorials.

# 17.12

  -- Ported more features (including dynamic scheduling and explicit
     tile size) of MFIter from C++ to Fortran.

  -- Added procedures to Fortran amrex_multifab_module to add,
     subtract, multiply, divide, saxpy and lincomb (linear
     combination).  Note that we call it saxpy, but it doesn't mean
     single precision.

  -- The `configure; make; make install` build approach enables linear
     solvers by default now.

  -- MultiFab and iMultiFab now support move assignment operator.
     This allows for swapping two MultiFabs without deep copying.  For
     instance, std::swap(mf1, mf2);

  -- Tutorials/Amr/Advection_AmrCore changes from using
     `Vector<unique_ptr<MultiFab>>` to `Vector<MultiFab>`.

# 17.11

  -- AMREX_ASSERT_WITH_MESSAGE and AMREX_ALWAYS_ASSERT_WITH_MESSAGE
     are new macros for assertion when it fails additional message
     will be printed.   For example:

         AMREX_ASSERT_WITH_MESSAGE(x > y, "reason why x > y must be true");

  -- amrex::Array is deprecated and replaced by amrex::Vector.
     Currently Array is an alias to Vector.  So this should not break
     any codes.  In the future, we will remove Array completely.
     Therefore it is recommended that application codes should start
     replacing Array with Vector.  There is a script,
     amrex/Tools/Migration/amrex_array_to_vector.sh, to help.  But use
     it with caution.   If it is completely gibberish to you, you
     probably shouldn't use the script. 

  -- functions for printing to a file in a parallel and thread-safe manner
     have been added. For example:

        amrex::AllPrintToFile("output") << "Hello from rank " << rank << 
                                           " and thread " << thread << "\n";

     will print out the requested information to separate files for each 
     rank and thread.

# 17.10

  -- Added a load balance approach to Amr/AmrLevel.  The runtime
     parameters for this is amr.loadbalance_with_workestimates and
     amr.loadbalance_level0_int.  The latter is for single level
     (i.e., amr.max_level=0) only.  For multi-level runs, load balance
     is done during regrid and thus the load balance interval is
     controlled by amr.regrid_int.  To use this approach, AmrLevel
     derivative must has a StateData for work estimates and its index
     is returned by virtual function int WorkEstType().

  -- MFIter now supports dynamic scheduling of OpenMP threads.  For
     example,
 
       for (MFIter mfi(mf, MFItInfo().SetDynamic(true).EnableTiling(tile_size);
            mfi.isValid(); ++mfi)
       { ... }

  -- Added EBFluxRegister to Src/EB for refluxing and
     re-redistribution in EB.

  -- amrex.signal_handling = 1 is a new runtime parameter that can be
     used to control whether AMReX should handle signals like SIGSEGV
     etc.