Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add base UAT issue template #625

Merged
merged 2 commits into from
Aug 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .github/ISSUE_TEMPLATE/User-Acceptance-Testing.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: User Acceptance Testing
description: Procedure for doing human QA release testing
labels: ["uat"]
body:
- type: markdown
attributes:
value: |
For each test below, check the box if it passes. If it fails, elaborate
and then either way, attach this issue to the final
- type: textarea
id: blender-versions
attributes:
label: Blender version(s) tested
validations:
required: true
- type: checkboxes
id: uat-tests
attributes:
label: Test cases
description: Go line by line and check the boxes for each test passed
options:
- label: Remove MCprep, restart blender, then Install MCprep, enable, disable, then enable again without errors
- label: Open Mineways or jmc2obj button (configure as necessary to make it run the executbale)
- label: Import any exported obj world, and verify that it imported into own collection, correct exporter was auto-selected, and on selecting one of the objects it contains the custome object properties which are appropriate
- label: Go into rendered mode to verify blurry/bad textures, then press prep materials and check they are fixed as expected
- label: Import a world with broken textures, then check that prep materials or Advanced/find missing works
- label: Select a block like lava, and use advanced > animate textures, press play to make sure it's now animated
- label: Go to the materials tab with any primitive object selected, load materials, then try applying lava - again selecting animate textures; ensure on play it's animated
- label: Try running mesh swap on the entire scene, veryfing at least one object is meshswaped (will be world dependant, typically tall grass should at least be there)
- label: Try advanced > combine materials and combine images, mostly making sure it doesn't error out
- label: Try sync materials (more advanced, you might need to make a custom material first in the addon lib folder)
- label: Enter rendered mode, then press Create MC sky, try once with each of the options (changing in the redo last options) in rendered mode and verifying that it reacts e.g. adding/removing shader/mesh moon/sun and adding/removing clouds per tick box
- label: Test pressing render panorama, picking a small resolution to make it fast
- label: Test changing the time of day (after having added one of the dynamic sky options)
- label: Try skin swapping (from file, username, and/or pre-installed list). Check that it applies to all children of a rig when the armature is selected, but only individual meshes if meshes are selected without the rig
- label: Try a mob spawn
- label: Try a (few) block spawns
- label: Try an item spawn (either from list and/or from a file on disk)
- label: Try each type of available effect spawn (weather geo, weather particle, tnt collection at different frames, and some particle)
- label: Try an entity spawn
- label: Try a meshswap spawn, especially animated ones like torches or lighting-based ones like jack_o_lanterns

- type: textarea
id: test-failures
attributes:
label: Test failures
description: For each test failure, note the number here and describe what exactly failed
validations:
required: false
- type: textarea
id: automated-tests
attributes:
label: Automated test results
description: Paste the output of unit tests here, either local run or the github action result
validations:
required: false


My actual UAT results:
- Default imported fire renders black in eevee, should select alpha clip for both materials
- Same for campfires above
- Could not add the sun/moon mesh in 2.80
- Render pano started with 1/6 done, should have started saying 0/6 done; also is rendeirng panos just broken entirely? Doesn't seem to go past the first one. Should try again in 4.x
- One crash on addin big particle. Reliably crashing blener in 2.80! Working fine in 4.2 though.
- The "furnace on" meshswap has a visible white box (probably for particle spawning) which should be changed to not be visible in render (in 4.2 both eevee and cycles)

30 changes: 14 additions & 16 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,21 +129,20 @@ In order to run these tests, **you must ensure your git folder with your MCprep

At the moment, only the project lead (TheDuckCow) should ever mint new releases for MCprep. However, the steps are noted here for completeness:

1. All releases need a corresponding milestone, like v3.3.1
1. Make sure the version is the correct number for the release (should match a corresponding milestone). Make this commit on the dev branch.
1. Create a pull request and merge dev into master (without deleting the dev branch). Close out any remaining corresponding milestones. Add this pull request to the milestone, only merging when all other tasks are completed or moved out of the milestone.
1. Check out master, git pull
1. Make sure that `"dev" = False` in conf.py, so that prod resources are used for reporting (it should always be this way on the master branch without doing anything, even after doing a merge from dev; but *always* double check)
1. Create [a draft release](https:/TheDuckCow/MCprep/releases/new) on GitHub
- Tag is in the form `3.3.1`, no leading `v`.
- The title however is in the form `MCprep v3.3.0 | ShortName` where version has a leading `v`.
- Copy the body fo the description from the prior release, and then update the text and splash screen (if a major release). Edit a prior release without making changes to get the raw markdown code, e.g. [from here](https:/TheDuckCow/MCprep/releases/edit/3.3.0).
1. Run `bpy-addon-build.py` to build the addon
1. Run all tests, ideally on two different operating systems. Use the `./run_tests.sh -all` flag to run on all versions of blender
1. If all tests pass, again DOUBLE CHECK that "dev" = false in conf.py, then
1. Drag and drop the generated updated zip file onto github.
1. Rename so it's in the form including its version number, e.g. `MCprep_addon_v3.3.0.zip`
1. Press publish release

1. Checkout dev, and commit the correct release version in `MCprep_addon/__init__.py` (should match a corresponding milestone)
1. Create a pull request to merge dev into master. This can be approved and merged without review, since all code is already reviewed - but only TheDuckCow may do this bypass with current repo permissions
1. Locally, check out master and run `git pull`
1. Run all local unit tests using `python run_tests.py -a`
- While we do have remote github unit tests, TheDuckCow has many more versions locally for wider testing to be more comprehensive. But, github action unittests can be used in a standin if necessary.
1. Create a new UAT issue from [issues here](https:/Moo-Ack-Productions/MCprep/issues/new/choose) with the name of the corresponding milestone
- The automated test results above should be pasted into the last section of this UAT form.
- If not all UAT steps pass, consider halting the release and/or updating/creating new issues
1. If all UAT steps pass, the run `./push_latest.sh` in the repo root
- Follow the script's instructions and prompts for ensuring the release completes
- You will likely need to update POT files and the json mapping via `mcprep_data_refresh.py` which gets called from within this script. Make a new PR if appropriate.
1. After this script has finished, go to [github releases](https:/Moo-Ack-Productions/MCprep/releases) and edit the draft release to hand hand-adjusted changelogs
1. Press release
1. **Immediately** download and install an old release MCprep, and install into blender (restart blender too)
1. Make sure that the trigger to update MCprep to the new version is working.
1. If it works, then **immediately update** the https://theduckcow.com/dev/blender/mcprep-download/ page to point to the new number (must be manually updated by TheDuckCow).
Expand All @@ -152,7 +151,6 @@ At the moment, only the project lead (TheDuckCow) should ever mint new releases
1. git checkout dev, and then upversion the dev branch to a unique incremental version. So if you just released v3.3.1, then the dev branch should be updated to be (3, 3, 1, 1) so that we can tell official releases apart from dev versions.



## Creating your blender_execs.txt

Your `blender_execs.txt` defines where to find the executables used in the automated testing scripts. Only these executables will be used during automated testing, noting that the testing system only supports blender version 2.8+ (sadly, only manual testing is possible in blender 2.7 with the current setup). It could look like:
Expand Down