From 570ddcce3d36535091f273b446c2b569bc2d5f17 Mon Sep 17 00:00:00 2001 From: "Patrick W. Crawford" Date: Thu, 22 Aug 2024 21:43:30 -0700 Subject: [PATCH 1/2] Added base issue template for user acceptance testing Meant to be part of the procedure for hand-validating releases now on. --- .../User-Acceptance-Testing.yml | 66 +++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/User-Acceptance-Testing.yml diff --git a/.github/ISSUE_TEMPLATE/User-Acceptance-Testing.yml b/.github/ISSUE_TEMPLATE/User-Acceptance-Testing.yml new file mode 100644 index 00000000..42318e3b --- /dev/null +++ b/.github/ISSUE_TEMPLATE/User-Acceptance-Testing.yml @@ -0,0 +1,66 @@ +name: User Acceptance Testing +description: Procedure for doing human QA release testing +labels: ["uat"] +body: + - type: markdown + attributes: + value: | + For each test below, check the box if it passes. If it fails, elaborate + and then either way, attach this issue to the final + - type: textarea + id: blender-versions + attributes: + label: Blender version(s) tested + validations: + required: true + - type: checkboxes + id: uat-tests + attributes: + label: Test cases + description: Go line by line and check the boxes for each test passed + options: + - label: Remove MCprep, restart blender, then Install MCprep, enable, disable, then enable again without errors + - label: Open Mineways or jmc2obj button (configure as necessary to make it run the executbale) + - label: Import any exported obj world, and verify that it imported into own collection, correct exporter was auto-selected, and on selecting one of the objects it contains the custome object properties which are appropriate + - label: Go into rendered mode to verify blurry/bad textures, then press prep materials and check they are fixed as expected + - label: Import a world with broken textures, then check that prep materials or Advanced/find missing works + - label: Select a block like lava, and use advanced > animate textures, press play to make sure it's now animated + - label: Go to the materials tab with any primitive object selected, load materials, then try applying lava - again selecting animate textures; ensure on play it's animated + - label: Try running mesh swap on the entire scene, veryfing at least one object is meshswaped (will be world dependant, typically tall grass should at least be there) + - label: Try advanced > combine materials and combine images, mostly making sure it doesn't error out + - label: Try sync materials (more advanced, you might need to make a custom material first in the addon lib folder) + - label: Enter rendered mode, then press Create MC sky, try once with each of the options (changing in the redo last options) in rendered mode and verifying that it reacts e.g. adding/removing shader/mesh moon/sun and adding/removing clouds per tick box + - label: Test pressing render panorama, picking a small resolution to make it fast + - label: Test changing the time of day (after having added one of the dynamic sky options) + - label: Try skin swapping (from file, username, and/or pre-installed list). Check that it applies to all children of a rig when the armature is selected, but only individual meshes if meshes are selected without the rig + - label: Try a mob spawn + - label: Try a (few) block spawns + - label: Try an item spawn (either from list and/or from a file on disk) + - label: Try each type of available effect spawn (weather geo, weather particle, tnt collection at different frames, and some particle) + - label: Try an entity spawn + - label: Try a meshswap spawn, especially animated ones like torches or lighting-based ones like jack_o_lanterns + + - type: textarea + id: test-failures + attributes: + label: Test failures + description: For each test failure, note the number here and describe what exactly failed + validations: + required: false + - type: textarea + id: automated-tests + attributes: + label: Automated test results + description: Paste the output of unit tests here, either local run or the github action result + validations: + required: false + + +My actual UAT results: +- Default imported fire renders black in eevee, should select alpha clip for both materials +- Same for campfires above +- Could not add the sun/moon mesh in 2.80 +- Render pano started with 1/6 done, should have started saying 0/6 done; also is rendeirng panos just broken entirely? Doesn't seem to go past the first one. Should try again in 4.x +- One crash on addin big particle. Reliably crashing blener in 2.80! Working fine in 4.2 though. +- The "furnace on" meshswap has a visible white box (probably for particle spawning) which should be changed to not be visible in render (in 4.2 both eevee and cycles) + From 92c9ebaf614b27e3b99d4ef4b723f26fcc2fa804 Mon Sep 17 00:00:00 2001 From: "Patrick W. Crawford" Date: Thu, 22 Aug 2024 21:57:13 -0700 Subject: [PATCH 2/2] Updated contributing guide for UAT steps and new release script --- CONTRIBUTING.md | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6d4cd52a..1d8e5d3b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -129,21 +129,20 @@ In order to run these tests, **you must ensure your git folder with your MCprep At the moment, only the project lead (TheDuckCow) should ever mint new releases for MCprep. However, the steps are noted here for completeness: -1. All releases need a corresponding milestone, like v3.3.1 -1. Make sure the version is the correct number for the release (should match a corresponding milestone). Make this commit on the dev branch. -1. Create a pull request and merge dev into master (without deleting the dev branch). Close out any remaining corresponding milestones. Add this pull request to the milestone, only merging when all other tasks are completed or moved out of the milestone. -1. Check out master, git pull -1. Make sure that `"dev" = False` in conf.py, so that prod resources are used for reporting (it should always be this way on the master branch without doing anything, even after doing a merge from dev; but *always* double check) -1. Create [a draft release](https://github.com/TheDuckCow/MCprep/releases/new) on GitHub - - Tag is in the form `3.3.1`, no leading `v`. - - The title however is in the form `MCprep v3.3.0 | ShortName` where version has a leading `v`. - - Copy the body fo the description from the prior release, and then update the text and splash screen (if a major release). Edit a prior release without making changes to get the raw markdown code, e.g. [from here](https://github.com/TheDuckCow/MCprep/releases/edit/3.3.0). -1. Run `bpy-addon-build.py` to build the addon -1. Run all tests, ideally on two different operating systems. Use the `./run_tests.sh -all` flag to run on all versions of blender -1. If all tests pass, again DOUBLE CHECK that "dev" = false in conf.py, then -1. Drag and drop the generated updated zip file onto github. -1. Rename so it's in the form including its version number, e.g. `MCprep_addon_v3.3.0.zip` -1. Press publish release + +1. Checkout dev, and commit the correct release version in `MCprep_addon/__init__.py` (should match a corresponding milestone) +1. Create a pull request to merge dev into master. This can be approved and merged without review, since all code is already reviewed - but only TheDuckCow may do this bypass with current repo permissions +1. Locally, check out master and run `git pull` +1. Run all local unit tests using `python run_tests.py -a` + - While we do have remote github unit tests, TheDuckCow has many more versions locally for wider testing to be more comprehensive. But, github action unittests can be used in a standin if necessary. +1. Create a new UAT issue from [issues here](https://github.com/Moo-Ack-Productions/MCprep/issues/new/choose) with the name of the corresponding milestone + - The automated test results above should be pasted into the last section of this UAT form. + - If not all UAT steps pass, consider halting the release and/or updating/creating new issues +1. If all UAT steps pass, the run `./push_latest.sh` in the repo root + - Follow the script's instructions and prompts for ensuring the release completes + - You will likely need to update POT files and the json mapping via `mcprep_data_refresh.py` which gets called from within this script. Make a new PR if appropriate. +1. After this script has finished, go to [github releases](https://github.com/Moo-Ack-Productions/MCprep/releases) and edit the draft release to hand hand-adjusted changelogs +1. Press release 1. **Immediately** download and install an old release MCprep, and install into blender (restart blender too) 1. Make sure that the trigger to update MCprep to the new version is working. 1. If it works, then **immediately update** the https://theduckcow.com/dev/blender/mcprep-download/ page to point to the new number (must be manually updated by TheDuckCow). @@ -152,7 +151,6 @@ At the moment, only the project lead (TheDuckCow) should ever mint new releases 1. git checkout dev, and then upversion the dev branch to a unique incremental version. So if you just released v3.3.1, then the dev branch should be updated to be (3, 3, 1, 1) so that we can tell official releases apart from dev versions. - ## Creating your blender_execs.txt Your `blender_execs.txt` defines where to find the executables used in the automated testing scripts. Only these executables will be used during automated testing, noting that the testing system only supports blender version 2.8+ (sadly, only manual testing is possible in blender 2.7 with the current setup). It could look like: