Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new land only initial conditions #2403

Open
1 of 5 tasks
wwieder opened this issue Mar 6, 2024 · 51 comments
Open
1 of 5 tasks

Create new land only initial conditions #2403

wwieder opened this issue Mar 6, 2024 · 51 comments
Assignees
Labels
enhancement new capability or improved behavior of existing capability priority: high High priority to fix/merge soon, e.g., because it is a problem in important configurations

Comments

@wwieder
Copy link
Contributor

wwieder commented Mar 6, 2024

As we move to CTSM5.2 datasets and introduce CLM6_0 physics options, it seems like it would also be helpful to provide new initial conditions files. @dlawrenncar noted, this may be most important for coupled model simulations, so we can provide more realistic land states in F and B cases as we move forward with CESM3 development runs.

Given very high productivity biases in CRU-JRA forced runs, I'd suggest we provide 1850 initial conditions forced with GSWP3 using CTSM5.2 surface data. It may be most straight forward to do this after #2348 comes to main?

Looking through the LMWG-Dev issues, I don't think we have a GSWP3 forced run that's similar to this case, #54, do we @olyson? If not, we can create a new issue on LWMG-Dev for this.

We should make sure that isotopes, tillage, and residue removal are all on for these spin ups.

Definition of done:

  • Decide on which LND_TUNING_MODES will have initial conditions (clm5_0_GSWP3, clm6_0_TRENDY, clm5_0_cam6, clm6_0_cam7?)
  • Which resolutions will be run? ne30pg3, f09, f19, f45? What about NEON sites?
  • Run all the configurations needed
  • Preliminary finidat file in place for clm5_1/clm6_0 all forcing options: lnd/clm2/initdata_esmf/ctsm5.2/clmi.I1850Clm60BgcCrop-ciso.1361-01-01.ne30pg3_mg17_c240317.nc
  • Put files in place and in XML and into a tag
@ekluzek
Copy link
Collaborator

ekluzek commented Mar 6, 2024

Since, this requires CTSM5.2 it actually should wait for #2372. If we need it sooner it could be added to CESM3_dev. But, I'd like to avoid that if we can. This was also on the CTSM5.2 board as a post-5.2 activity, so I'm linking this there.

Also this relates to #554, so linking it here.

@olyson
Copy link
Contributor

olyson commented Mar 6, 2024

@wwieder and I discussed this and thought that we could just clone the most recent deadveg branch simulation and switch from CRUJRA to GSWP3. In that recent simulation, tillage and residue removal was on. And I have been using CTSM5.2 surface datasets, i.e., for that run I used

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_78pfts_CMIP6_1850_c230517.nc

I know that @slevis-lmwg has recently re-generated all of the datasets, so I was planning to use that version which appears to be:

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_1850_78pfts_c240216.nc

Is there another reason to wait for #2372 ?

@ekluzek
Copy link
Collaborator

ekluzek commented Mar 6, 2024

Nope that makes perfect sense @olyson. Better to not have to wait when you don't have to.

@olyson
Copy link
Contributor

olyson commented Mar 6, 2024

See NCAR/LMWG_dev#57

@olyson
Copy link
Contributor

olyson commented Mar 6, 2024

I initially set this up as an f09 (1deg) simulation, but I think it should actually be an ne30 since we'll use it as initial conditions for the ne30 F- and B-cases coming up. I don't think the finidat interpolation will handle going from 1deg to ne30. Is that a correct assumption @ekluzek and @billsacks ?

@ekluzek
Copy link
Collaborator

ekluzek commented Mar 6, 2024

@olyson it should be fine to interpolate from f09 to ne30np4.pg3 grid. The interpolation is just nearest neighbor so it will work between any type of grid.

But, I agree that we should spinup the land with the workhorse ne30 grid rather than f09. We'll interpolate to f09 from it for the land only simulations.

@billsacks
Copy link
Member

I agree with @ekluzek .

@olyson
Copy link
Contributor

olyson commented Mar 17, 2024

Here is an ne30 spunup initial file for 1850 with configuration as noted above:

/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm51_ctsm51d166deadveg_ne30pg3ne30pg3mg17_GSWP3V1_ABsnoCDE_blk_A5BCD_1850pAD.clm2.r.1361-01-01-00000.nc

@wwieder
Copy link
Contributor Author

wwieder commented Mar 18, 2024

Thanks @olyson should this be the default finidat file we point to for simulations with 'modern' tags? Here I'm especially thinking about the CESM alpha (or beta) 17 tag that's upcoming?

@olyson
Copy link
Contributor

olyson commented Mar 18, 2024

Yes, I think it will be suitable for the upcoming CESM tags.

@wwieder wwieder added this to the ctsm6.0.0 (code freeze) milestone May 2, 2024
@wwieder
Copy link
Contributor Author

wwieder commented May 8, 2024

@ekluzek and @slevis-lmwg, can we make it so this initial conditions file is used out of the box with the next round of coupled model experiments?

/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm51_ctsm51d166deadveg_ne30pg3ne30pg3mg17_GSWP3V1_ABsnoCDE_blk_A5BCD_1850pAD.clm2.r.1361-01-01-00000.nc

@wwieder
Copy link
Contributor Author

wwieder commented May 8, 2024

looking at the project board, I wonder if this should come in with #2492, or actually #2501?

@wwieder wwieder added the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label May 8, 2024
@ekluzek
Copy link
Collaborator

ekluzek commented May 9, 2024

@wwieder it looks like that's GSWP3 forcing finidat file. Do you want that with I1850Clm60BgcCrop and also with B1850/F1850 or just the former because it's for GSWP3 forcing?

@wwieder
Copy link
Contributor Author

wwieder commented May 9, 2024

all of the above B, I, & F1850 until we get new cpl.hist files to try a spinup for the B case. We want to provide living arctic vegetation as much as possible.

@ekluzek
Copy link
Collaborator

ekluzek commented May 10, 2024

It looks like the fsurdat file used for that IC file was with a preliminary version of ctsm5.2.0 datasets, and is different by at least roundoff level. That means that it will likely require use_init_interp set to TRUE.

Also @olyson note that in CSEG Meeting Notes Mike Levy tells us that they will be going to the t232 mask for MOM. So we should start using it for our simulations as well.

@olyson
Copy link
Contributor

olyson commented May 10, 2024

Ok, sounds like it is a showstopper, so I'll start another spinup with the latest code and the latest surface dataset (which out of the box seems to be /glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_ne30np4.pg3_hist_1850_78pfts_c231026.nc) and the t232 mask.

@ekluzek
Copy link
Collaborator

ekluzek commented May 10, 2024

@olyson I don't mean to say it's a showstopper. I mean to say we need to transition over to it. I think we need this in place by end of June for the science capability freeze. But, actually we should talk as a group about this in CTSM as well as CESM.

@olyson
Copy link
Contributor

olyson commented May 10, 2024

Ah, ok, sorry, I didn't read your comments carefully enough.

@wwieder
Copy link
Contributor Author

wwieder commented May 10, 2024

FWIW it also seems like @slevis-lmwg is creating some new initial condition files for matrix-CN work, although these also are unlikely to have the correct ocean mask.

@ekluzek
Copy link
Collaborator

ekluzek commented May 10, 2024

New finidat file name will be:

lnd/clm2/initdata_esmf/ctsm5.2/clmi.I1850Clm60BgcCrop-ciso.1361-01-01.ne30pg3_mg17_c240317.nc

And used for standard 1850 control and hist compsets starting at 1850 with clm5_1 and clm6_0 physics.

@ekluzek ekluzek removed the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label May 10, 2024
@slevis-lmwg
Copy link
Contributor

Keith shared his script from 2014 where we ran the model at 0.5 (resolution of CRUNCEP data) forced by Qian data (at T42) and saved 3-hourly coupler history files and then blended that into CRUNCEP forcing data.
/glade/u/home/oleson/cruncep/blend_Qian_CRUNCEP.ncl

@slevis-lmwg
Copy link
Contributor

Discussed at CTSM SE meeting today.

Needed for coupled model runs that Cecile will start hopefully in the next few weeks. Longest-term fix is to have a version of CRUJRA we can support that includes Antarctica—just put GSWP3 data there.

Decision: Go ahead and do that; skip intermediate steps. @wwieder, @slevis-lmwg, and @adrifoster to work on this. Can be done on Izumi next week during Derecho/Casper downtime if the datasets needed get copied—Sam L. will do this. Even just 1901-1920 to have something.

Since Thursday's meeting I have tried copying files to a common space on izumi (/project/tss02); however, permissions continue to fail, so I'm resorting to copying the files that I think we will need here:

/project/tss/slevis/atm_forcing.datm7.GSWP3.0.5d.v1.c200929
/project/tss/slevis/TRENDY2024/inputs/three_stream

@wwieder
Copy link
Contributor Author

wwieder commented Sep 16, 2024

see also CRUJRA files that include Antarctica mentioned here

I realize I didn't copy the mesh file if we're able to test longer simulations on Izumi, so hopefully you have these, Sam?

@wwieder
Copy link
Contributor Author

wwieder commented Sep 16, 2024

Here's what the new fields look like, for all us visual people.
image

And their location on izumi: /project/tss/TRENDY2024/inputs/three_stream

@slevis-lmwg
Copy link
Contributor

From 2024/9/16 stand-up meeting
@slevis-lmwg will try two I1850SP simulations with hourly output with:

  1. the new datm data /project/tss/TRENDY2024/inputs/three_stream
  2. the gswp3 data /project/tss/slevis/atm_forcing.datm7.GSWP3.0.5d.v1.c200929
  3. why not a third case with the orig. TR24 data, for an extra sanity check...

@slevis-lmwg
Copy link
Contributor

I realize I didn't copy the mesh file if we're able to test longer simulations on Izumi, so hopefully you have these, Sam?

I think I have the mesh files that I need for (2) and (3) which means gswp3-only and trendy-only runs. I will need to generate a new mesh file for the blended product.

@slevis-lmwg
Copy link
Contributor

slevis-lmwg commented Sep 17, 2024

Update

  • (2) and (3) are running (will need restarting twice)
  • To generate mesh file for (1):
ncks --rgr infer --rgr scrip=/project/tss/slevis/TRENDY2024_GSWP3_blended/scrip_all1.nc clmforc.CRUJRAv2.5_0.5x0.5.TPQWL.1901.nc /project/tss/slevis/TRENDY2024_GSWP3_blended/foo_all1.nc
/project/esmf/PROGS/esmf/8.2.0/mpiuni/gfortran/9.3.0/bin/binO/Linux.gfortran.64.mpiuni.default/ESMF_Scrip2Unstruct scrip_all1.nc mesh_all1.nc 0

I'm running the mesh_mask_modifier tool now.

@wwieder
Copy link
Contributor Author

wwieder commented Sep 17, 2024

fwiw @adrifoster suggested just using the mesh_maker tool in tools/site_and_regional to generate the mesh, Sam.

@slevis-lmwg
Copy link
Contributor

Thanks for the reminder @wwieder. I tried mesh_maker just now and got the same error that I saw with all my attempts. I'm starting to wonder whether the problem is with the datm data somehow.

No idea whether this would be a problem, but here's the difference between a CRUJRA file that works:

        float lon(lon) ;
                lon:units = "degrees_east" ;
                lon:long_name = "longitude" ;
                lon:mode = "time-invariant" ;
        float lat(lat) ;
                lat:units = "degrees_north" ;
                lat:long_name = "latitude" ;
                lat:mode = "time-invariant" ;
        float LONGXY(lat, lon) ;
                LONGXY:units = "degrees_east" ;
                LONGXY:long_name = "longitude" ;
                LONGXY:mode = "time-invariant" ;
        float LATIXY(lat, lon) ;
                LATIXY:units = "degrees_north" ;
                LATIXY:long_name = "latitude" ;
                LATIXY:mode = "time-invariant" ;
        float FSDS(time, lat, lon) ;
                FSDS:_FillValue = 1.e+36f ;
                FSDS:units = "W/m**2" ;
                FSDS:long_name = "incident shortwave radiation" ;
                FSDS:mode = "time-dependent" ;
                FSDS:missing_value = 1.e+36f ;

and one of the blended files:

        float lon(lon) ;
                lon:_FillValue = NaNf ;
        float lat(lat) ;
                lat:_FillValue = NaNf ;
        float LONGXY(lat, lon) ;
                LONGXY:_FillValue = NaNf ;
                LONGXY:units = "degrees_east" ;
                LONGXY:long_name = "longitude" ;
                LONGXY:mode = "time-invariant" ;
        float LATIXY(lat, lon) ;
                LATIXY:_FillValue = NaNf ;
                LATIXY:units = "degrees_north" ;
                LATIXY:long_name = "latitude" ;
                LATIXY:mode = "time-invariant" ;
        float FSDS(lat, time, lon) ;
                FSDS:_FillValue = NaNf ;

For now I will switch my attention back to ctsm5.3 tests as we discussed.

@wwieder
Copy link
Contributor Author

wwieder commented Sep 17, 2024

ah, that seems like an issue.

@slevis-lmwg
Copy link
Contributor

By the way... in case it matters... it probably doesn't at this point:
I accidentally deleted clmforc.GSWP3.c2011.0.5x0.5.Solr.1920-12.nc
in my izumi directory /project/tss/slevis/atm_forcing.datm7.GSWP3.0.5d.v1.c200929

@slevis-lmwg
Copy link
Contributor

slevis-lmwg commented Sep 19, 2024

List of nco commands that make a blended file more like a CRUJRA file. First I will test 1901 to see if the run will work.

Summary:
Delete _FillValue from most variables.
Update time attributes.
Update lon attributes.
Update lat attributes.
Update FSDS attributes.
Order time before lat dimension.

ncatted -O -a _FillValue,time,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,lon,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,lat,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,LONGXY,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,LATIXY,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 

ncatted -a units,time,m,c,'days since 1901-01-01 00:00:00' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a long_name,time,c,sng,'observation time' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc

ncatted -O -a units,lon,c,sng,'degrees_east' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a long_name,lon,c,sng,'longitude' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a mode,lon,c,sng,'time-invariant' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 

ncatted -O -a units,lat,c,sng,'degrees_north' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a long_name,lat,c,sng,'latitude' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a mode,lat,c,sng,'time-invariant' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 

ncatted -a _FillValue,FSDS,m,float,1.e+36 clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a units,FSDS,c,sng,'W/m**2' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a long_name,FSDS,c,sng,'incident shortwave radiation' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a mode,FSDS,c,sng,'time-dependent' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a missing_value,FSDS,c,float,1.e+36 clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc

ncpdq -a time,lat clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901_new.nc

I made scripts that execute the above for Solr, Prec, and TPQWL automatically. If all this works, then I can modify the scripts to operate on all the years rather than just 1901.

@slevis-lmwg
Copy link
Contributor

slevis-lmwg commented Sep 19, 2024

Good news. The above updates to the 1901 files made a difference, and the new simulation is running now. I will wait until we look at results before I also update 1902-1920.

Meanwhile the hourly output is on izumi here (except the third simulation is in progress):

/scratch/cluster/slevis/archive/ctsm52028_f19_GSWP3
/scratch/cluster/slevis/archive/ctsm52028_f19_TR24blended
/scratch/cluster/slevis/archive/ctsm52028_f19_TR24bl_actual

The second simulation should NOT say "blended" in its name (sorry).

@wwieder
Copy link
Contributor Author

wwieder commented Sep 19, 2024

Thanks Sam, I'll correct this in the notebook when we make the full dataset

@slevis-lmwg
Copy link
Contributor

@olyson and I were troubleshooting (in the last hour) a new issue that I came across with the blended data, and we discovered that the mesh file has an additional longitude. In particular, both 360 and zero appear, instead of just 0, and the file has a larger node count than expected as result.

I have the following hypothesis at the moment:
I generated the mesh file from one of the datm files before "fixing" them. One of the "fixes" was a reorder of the (time, lat, lon) dimensions. I will regenerate the mesh file with a "fixed" datm file to see if it resolves the issue.

@slevis-lmwg
Copy link
Contributor

Previous hypothesis: wrong.

However, good news:
Seems I misspoke at this morning's meeting when I said that all 3 methods by which I generated mesh files resulted in identical files. From what I see now, the mesh_maker tool added an unnecessary longitude (360) in the mesh file that the other methods did not. It didn't crash the model but changed answers in the first column (0-degree meridian).

I just looked at a new simulation using one of my other mesh files and confirmed this.

@olyson you can now start the spin-up: The new mesh file has the same name that I told you earlier, but you can see that it doesn't have the extra longitude at 360.

@olyson
Copy link
Contributor

olyson commented Sep 20, 2024

As @slevis-lmwg and I discussed, I ran two cases with hourly output on Derecho to verify the mesh file. One with the original atm forcing and mesh file (/glade/derecho/scratch/oleson/temp_work/TRENDY2024/inputs/three_stream, copied from /project/tss/slevis/TRENDY2024/inputs/three_stream/) and the other with the blended atm forcing and new mesh file (/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream). The forcing was bfb except over Antarctica.

@slevis-lmwg , the two cases are here:
/glade/work/oleson/ctsm5.3.n04_ctsm5.2.028/cime/scripts/ctsm53n04ctsm52028_f09_blendedhourly_AD
/glade/work/oleson/ctsm5.3.n04_ctsm5.2.028/cime/scripts/ctsm53n04ctsm52028_f09_orighourly_AD
The diferenced hourly history file is here:
/glade/derecho/scratch/oleson/ANALYSIS/blendedhourly-orighourly.nc

So I've started the ne30 and f09 spinups with the blended atm forcing and mesh file.

@ekluzek
Copy link
Collaborator

ekluzek commented Sep 20, 2024

@adamrher in the table above linked here...

#2403 (comment)

There are finidat files for f19 for 2011 and 2003 as well as 2013 for the CONUS grid. I want to simplify our initial conditions so we are just supporting 1850, 1979 and 2000. Since 2013 is specific for the CONUS grid it sounds like that is important to have for the CONUS grid alone. But, I wanted to make sure you see it as important to have.

CAM does start simulations for a variety of dates outside of 1850, 1979, 2000 (and 2013) including: 1950, 1969, 1974, 1980, 1995, 1997, 1999, 2004, 2005, 2006, 2010, and 2015. We can't sensibly support all of those, but if we have IC files within about 15 years of the start date that seems pretty reasonable. The only outlier would be 1950 which is only used for the FWHIST compsets and I assume WACCM simulations aren't going to be as sensitive to the initial conditions from the land model.

How does that sound? Is there anyone else you'll want to weigh in on this?

@ekluzek ekluzek modified the milestones: ctsm5.3.0, cesm3_0_beta04 Sep 25, 2024
@olyson
Copy link
Contributor

olyson commented Oct 1, 2024

Candidate initial files from latest spinups and historicals:

ne30:
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_pSASU.clm2.r.0121-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_hist.clm2.r.1979-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_hist.clm2.r.2000-01-01-00000.nc
f09:
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_pSASU.clm2.r.0161-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_hist.clm2.r.1979-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_hist.clm2.r.2000-01-01-00000.nc

@slevis-lmwg
Copy link
Contributor

@olyson thank you for these.

Question:
Is there a similar list of actual f19 files that I could use. If not, I will point to the f09 files for f19 cases.

@olyson
Copy link
Contributor

olyson commented Oct 8, 2024

There is an 1850 f19 file from the PPE spinup (NCAR/LMWG_dev#70) here:
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

My understanding based on the last post of this issue is that the PPE group is testing this file now. I haven't run a historical yet which would provide year 1979 and year 2000 initial files.

@wwieder
Copy link
Contributor Author

wwieder commented Oct 8, 2024

I think we're just expecting to provide 1850 ICs for the f19 grid at this point, Keith. Thanks for posting its location here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement new capability or improved behavior of existing capability priority: high High priority to fix/merge soon, e.g., because it is a problem in important configurations
Projects
Status: On the grill (work in progress)
Status: In Progress
Development

No branches or pull requests

8 participants