Add implicit tendency for canopy temperature

[kmdeck]

Purpose

Add implicit tendency for canopy temperature
Address #696

Previously, all canopy variables were stepped explicitly. Now temperature
is stepped implicitly in both standalone CanopyModel runs and integrated
SoilCanopyModel runs. Changes include adding a jacobian and implicit
tendency for the CanopyModel, and updating scripts to use ImEx steppers.
We've also added scripts to test convergence behavior of the standalone
and integrated cases, which are output as plots on buildkite.

This change increases the stability of our global
SoilCanopyModel runs, decreasing the number of NaNs we see, and allows
us to take timesteps of 450s.

Notes

There are two things we dont understand about this change:

• there seem to be some times during the simultation where the error is always large, and a smaller dt doesnt help. For example, the 99th percentile error curve dips below the mean error for small dt, which indicates there are outliers that are very large in the top 1% error
  • update 9/18: it's possible this may happen during times in the simulation where temperature is changing a lot, e.g. shortly after a driver update, as opposed to when the temperature is in equilibrium. See plots in comments for more information
• the derivative of SHF with respect to temperature is correct to a few percent, but the derivative of LHF with respect to temeperature is ~30% wrong. This is partly due to d_qsat/d_T errors (but Ive double checked that I entered the formula correct), but seems like it is also due to d_LHF/d_qsat. I think I have followed CLM exactly. Not sure why the error is large.

These two questions will be investigated in a later PR - see #782

Results

With an explicit stepper, RK4, a test script at the Ozark site with the canopy alone permits dt = 60 [ac_canopy = 1e3]. Time to solution = 114 seconds - see kd/canopy_explicit_tests branch. Using dt = 180 fails.
With an implicit stepper ARS111, updated jacobian every newton iteration, max_iters = 6, the same simulation can be run with dt=3600. Time to solution = 16 seconds. Using only a single iteration failed.

Accuracy as a function of timestep (ref solution = dt = 6, implicit solver)
Float 32:

Float64:

In the global long run, on this branch: kd/canopy_explicit_global_nans I just changed the value of ac_canopy but left everything else the same as #main - i.e. using an explicit solver. The resulting map is almost entirely NaN [ look at time 86400]

On the implicit branch (this PR), we have no NaNs after one day. [ look at time 86400]

To-do

• Merge ClimaCore Branch
• Test timestep improvement
• Unit test Jacobian
• Cleanup canopy timestep_test.jl
• Add script to show soil/canopy convergence (analogous to canopy timestep_test.jl) - add to buildkite pipeline
• Rebase
• Check longruns
  • requires add back uniform_vertical_levels ClimaCore.jl#2016 for GPU compatibility
• Review

Content

• Adds canopy temp to the list of implicitly stepped variables
• Adds the canopy jacobian
• Adds the canopy implicit tendency
• updates scripts and some timesteps
• adds a script testing timestep convergence, adds to buildkite
• update docs Manifest to ClimaCore 0.14.11

---

• I have read and checked the items on the review checklist.

[dennisYatunin]

These changes look good overall, but I have some concerns about the reliability of the tests. The current test results indicate that there are significant errors in the Jacobian approximation, but these errors might appear bigger than they actually are if there is a problem with the reference solutions. This issue can be dealt with in future PRs, though; the new infrastructure added here is fine to merge in.

[dennisYatunin]

Did you encounter some sort of issue with the ARS343 timestepper here (and in other experiments), or did you just switch to IMEX Euler for convenience?