Testing updates (#188)

source/Reviewers/howtocommit.rst

@@ -479,6 +479,10 @@ for all affected tests before you commit to the trunk.
 
  python3 ./rose-stem/bin/update_branch_kgos.py -s <suite name> -w <path to working copy>
 
+ .. note::
+ This script requires at least python 3.9. This can be achieved on
+ Met Office machines by running ``module load scitools``
+
 
  .. tab-item:: LFRic Core
 

source/WorkingPractices/TestSuites/lfric_apps.rst

@@ -3,14 +3,23 @@
 Testing LFRic Apps
 ==================
 
-LFRic Apps testing is run with the following commands from a working copy:
+Rose stem:
+ LFRic Apps testing uses rose-stem and is run with the following commands
+ from a working copy:
 
-.. code-block::
+ .. code-block::
+
+ export CYLC_VERSION=8
+ rose stem --group=developer
+ cylc play <working copy name>
+ cylc gui
+
+Local testing:
+ Alternatively, a single application can be built and run locally using
+ `these instructions <https://code.metoffice.gov.uk/trac/lfric_apps/wiki/local_builds>`_
 
- export CYLC_VERSION=8
- rose stem --group=developer
- cylc play <working copy name>
- cylc gui
+ This test does not use rose or cylc and is particularly useful for
+ checking for compile errors while developing.
 
 -----
 
@@ -21,29 +30,25 @@ LFRic Apps testing is run with the following commands from a working copy:
  * If setting the source to an fcm URL, the mirror needs to be used and the revision can either be blank (for latest commit) or any valid revision for that branch.
  * If setting the source to a Working Copy, the hostname needs to be provided (as Hostname:Path) and the revision must be blank.
 
+ For more details, see :ref:`multi-repo_testing`.
+
+
+Rose stem
+---------
 The LFRic Apps rose stem includes a range of tests to exercise all the applications
 stored in this repository, using multiple compilers, and checksum and plot tasks to
 confirm the outputs.
 
 .. tip::
 
- For LFRic Apps it is possible to update the checksum files on your branch to
- aid with testing and development. To do so run the rose-stem suite as described
- to cover all failing tests, then run the following from your working copy
-
- .. code-block::
-
- python3 ./rose-stem/bin/update_branch_kgos.py -s <suite name> -w <path to working copy>
-
- More details on the KGO update proceedures for all repositories can be found
- :ref:`here <kgo_instructions>`.
+ For LFRic Apps it is possible to update the checksum files on your branch as
+ you progress your development to aid with testing. Details on how to do this
+ are on the :ref:`KGO page <kgo>`.
 
 Below is a (by no means comprehensive) set of groups that you may wish to use on
 Met Office systems. Note that there is a lot of overlap between these groups,
 and that you can specify more than one at once, e.g. ``--group=developer,gungho_model``.
 
-
-
 +--------------------+----------------------------------------------------------+
 | Group | Description |
 +====================+==========================================================+

source/WorkingPractices/kgo.rst

@@ -7,26 +7,11 @@ Normally it is to be expected that code changes regress (i.e. all prognostic
 variables **and** diagnostics maintain the same answers with your code included
 but switched off).
 
-The most common reason why a code change does not regress is when a bug is
-discovered and fixed.
-
-.. important::
- Where a change in answers occurs, the developer must contact the configuration
- owners of all affected configurations. See :ref:`approvals` for details.
-
-Configuration owners will review the change and will either accept the change
-as it is, or will request the use of a temporary variable to switch the
-change off. See :ref:`templogicals` for details of this process.
-
-Sometimes, a KGO update may also be required to simply add a new job to the
+Usually, a code change does not regress when either a bug is a
+discovered and fixed, or a science area is introduced or enhanced. Sometimes,
+a KGO update may also be required to simply add a new job to the
 test suite or to port the rose stem suite to new HPC architecture.
 
-.. important::
- If your development includes updates to the KGO, you **must** add the
- ``kgo`` keyword to your ticket so that your CodeSys reviewer knows to
- install the new KGO. Failure to do so may lead to delays in your
- ticket making the project's trunk.
-
 **LFRic** KGO checksums are stored in the repository. As such with LFRic tickets
 the expectation is that you, as the developer, will include updated KGO files
 as part of your branch.
@@ -36,6 +21,62 @@ to this area is restricted to members of the Simulation Systems and Deployment
 Team and they will update the KGO files to include your new answers as part of
 the commit process.
 
+
+KGO Update Process
+------------------
+
+Getting the process right for KGO changing tickets significantly helps get
+such changes onto the trunk. When preparing your change for review:
+
+1. Run the ``all`` rose-stem group in order to make sure that all
+ changes to answers have been found.
+
+ * Include the :ref:`trac.log <traclog>` output from this testing in your ticket summary.
+
+2. Add the ``kgo`` keyword to your ticket.
+
+.. tab-set::
+
+ .. tab-item:: LFRic Apps
+
+ 3. Update the checksum files on your branch. To do so run rose stem and
+ then the following from the head of your working copy
+
+ .. code-block::
+
+ python3 ./rose-stem/bin/update_branch_kgos.py -s <suite name> -w <path to working copy>
+
+ .. note::
+ This script requires at least python 3.9. This can be achieved on
+ Met Office machines by running ``module load scitools``
+
+ 4. The changes in answers should be science reviewed by someone familiar with
+ the failing tests - if unsure then start with the Code Owner for the affected
+ application.
+
+ Once all the above is in place and the science and code reviews have been completed
+ then the Code Reviewer will merge your change to the head of trunk. If there are
+ merge conflicts in the checksum files then the reviewer will repeat step 3
+ to refresh these files. Your change is then committed to trunk.
+
+ .. tab-item:: UM & JULES
+
+ 3. Contact the configuration owners of all affected configurations. A list is
+ provided in the trac.log . See :ref:`approvals` for details.
+
+ Configuration owners will review the change and will either accept the change
+ as it is, or will request the use of a temporary variable to switch the
+ change off. See :ref:`templogicals` for details of this process.
+
+ Once all of the above is in place and the science and code reviews have been
+ completed the Code Reviewer will merge the branch to the head of trunk, run the
+ tests that have changed answers and use those results to install KGO files to
+ the filesystem. Your change is then committed to trunk.
+
+.. tip::
+ More details on the KGO update proceedures for all repositories can be found
+ :ref:`here <kgo_instructions>`.
+
 .. toctree::
  :hidden:
 

source/WorkingPractices/reviews.rst

@@ -31,6 +31,13 @@ this through completely makes a big difference to the review experience.
 Once you are happy that everything is in place then you are ready to continue
 to review.
 
+.. important::
+ If your development changes answers then make sure you have followed the
+ steps on :ref:`preparing a KGO ticket for review.<kgo>`
+
+ If you suspect there may be a change in answers but none have shown up during
+ testing then run the rose-stem ``all`` group to confirm this.
+
 .. Tip::
  Remember to follow all code-related steps and commit all your changes before
  running final testing to avoid needing to re-run.