Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extension of the ReportPortal plugin using API #2305

Merged
merged 7 commits into from
Feb 6, 2024
Merged

Extension of the ReportPortal plugin using API #2305

merged 7 commits into from
Feb 6, 2024

Conversation

4N0body5
Copy link
Collaborator

@4N0body5 4N0body5 commented Sep 4, 2023 •
edited
Loading

PR should cover following use cases:

1.| Create one launch for all plans and create suite per each plan
2.| Additionaly upload new suite/tests to a an existing launch
3.| Prepare launch with empty (idle) tests and additionally report their results (within the same run id)
4.| Rerun and upload new version into new Retry item within the last launch based on mapping via names.
5.| Upload files
6.| Enable using v2

New implemented options:

  • option --launch-per-plan for creating a new launch per each plan (default)
  • option --suite-per-plan for mapping a suite per each plan, all enclosed in one launch (launch uuid is stored in run of the first plan)
  • option --launch-description for providing unified launch description, intended mainly for suite-per-plan mapping
  • option --upload-to-launch LAUNCH_ID to append new plans to an existing launch
  • option --upload-to-suite SUITE_ID to append new tests to an existing suite within launch
  • option --launch-rerun for reruns with 'Retry' item in RP, name-based mapping, only for suite-per-plan structure.
  • option --defect-type for passing the defect type to failing tests, enables report idle tests to be additionally updated.

Additional work behind that:

  • rewritten environment variables to the uniform form TMT_PLUGIN_REPORT_REPORTPORTAL_${option}
  • enabled to report without results
  • get launch attributes from all common ones in plans (suite-per-plan)
  • storing uuids per each launch/suite followed by reusing them (update based on identifiers)
  • analyzed option combinations
  • version v2
  • uploading to suites
  • check stored identifiers
  • restrict problematic option combinations
  • resolve problems with mypy
  • and lot of small debugging
  • schemas, documentation

Yet todo:

  • tests (for new PR)
  • uploading files (for new PR)
  • option for a link to be added into description

happz
happz reviewed Sep 5, 2023
View reviewed changes
tmt/base.py Outdated Show resolved Hide resolved
@happz happz added step | report Stuff related to the report step plugin | reportportal The reportportal report plugin labels Sep 22, 2023
@brakogh
Copy link

brakogh commented Oct 18, 2023

@4N0body5 can you share progress pls?

@psss psss added this to the 1.30 milestone Nov 7, 2023
@4N0body5 4N0body5 removed the request for review from janhavlin November 18, 2023 01:08
@psss
Copy link
Collaborator

psss commented Nov 27, 2023

@4N0body5, is the WIP tag still relevant or is the pull request ready for review?

@4N0body5
Copy link
Collaborator Author

@4N0body5 can you share progress pls?

@brakogh I have updated PR description to inform you about the progress.
Though it is still in WIP state, due to the difficulties adding several functionalities, I am actively working on it and hope to have it ready for a review in two weeks.
For more details I add continuous commits with progress, and document the analysis in the live doc (see for analysis of option combinations, any feedback is welcome).

@4N0body5
Copy link
Collaborator Author

@4N0body5, is the WIP tag still relevant or is the pull request ready for review?

@psss This week, I plan to look into uploading files, setting version, fixing the problem with uploading to suites, and resolving problems due to option combinations and plan to remove the tag afterwards.

@psss
Copy link
Collaborator

psss commented Nov 27, 2023

Thanks for the update, @4N0body5. Nice progress!

PR should cover following use cases:

  1. Create one launch for all plans and create suite per each plan
  2. Additionaly upload new suite/tests to a an existing launch
  3. Prepare launch with empty (idle) tests and additionally report their results (within the same run id)
  4. Rerun and upload new version into new Retry item within the last launch based on mapping via names.
  5. Upload files
  6. Enable using v2

New implemented options:

  • option --launch-per-plan for creating a new launch per each plan (default)
  • option --suite-per-plan for mapping a suite per each plan, all enclosed in one launch (launch uuid is stored in run of the first plan)
  • option --launch-description for providing unified launch description, intended mainly for suite-per-plan mapping
  • option --upload-to-launch LAUNCH_ID to append new plans to an existing launch
  • option --upload-to-suite SUITE_ID to append new tests to an existing suite within launch
  • option --launch-rerun for reruns with 'Retry' item in RP, name-based mapping, only for suite-per-plan structure.
  • option --defect-type for passing the defect type to failing tests, enables report idle tests to be additionally updated.

Sounds like a large bunch of new features! Would it make sense to cover e.g. the upload files in a separate pull request and not block already finished changes on it? Or is everything here too much intertwined that it would not make sense to have smaller pull requests dedicated to a single change / smaller scope?

@4N0body5
Copy link
Collaborator Author

... Would it make sense to cover e.g. the upload files in a separate pull request and not block already finished changes on it? Or is everything here too much intertwined that it would not make sense to have smaller pull requests dedicated to a single change / smaller scope?

@psss It does make sense. I guess I'll make a brief look into the issue; If it is no struggle, I'll try adding it, so it can be effectively reviewed with all of others, but as soon as it shows any sign of difficulties, I'll suggest postponing it into another PR.
E.g. with the version I only hope to add one environment variable as a quick (maybe temporary) solution, but I'm not so sure about the upload files.
Definitely, the main priority is debug those already implemented parts.

@psss psss modified the milestones: 1.30, 1.31 Dec 5, 2023
@4N0body5 4N0body5 changed the title Extension of the ReportPortal plugin using API [WIP] Extension of the ReportPortal plugin using API Dec 12, 2023
@psss psss added the priority | must high priority, must be included in the next release label Dec 13, 2023
@kkaarreell
Copy link
Collaborator

One thing I am struggling with is how to properly use options like --launch-rerun, --upload-to-suite etc for a rerun. An example would be nice.

@kkaarreell
Copy link
Collaborator

kkaarreell commented Jan 13, 2024 •
edited by happz
Loading

One more thing to think about. How a user or an automation will find out URLs of created RP launches. Especially once tests will ve running through TF.

@4N0body5
Copy link
Collaborator Author

One more thing to think about. How a user or an automation will find out URLs of created RP launches. Especially once tests will ve running through TF.

AFAIK I could only provide it via info log that lists the link in terminal. I'm not sure how TF works, let me know if there's any idea to make the url more available.

@thrix
Copy link
Collaborator

thrix commented Feb 5, 2024

One more thing to think about. How a user or an automation will find out URLs of created RP launches. Especially once tests will ve running through TF.

AFAIK I could only provide it via info log that lists the link in terminal. I'm not sure how TF works, let me know if there's any idea to make the url more available.

@kkaarreell @4N0body5 what about this one, leave it for the next round? Or is there anything needed from TF?

@4N0body5 @happz
Extension of the ReportPortal plugin using API [WIP]
268d21c 
First part - grouping plans into one launch

* option `--merge` to create suite per plan and merge them all in one launch
  (stored launch uuid in run of the first plan)
* option '--attributes' for additional attributes,
  but mainly for assignment of attributes to the launch with merged plans
* option `--uuid` to append new plans to an existing launch
* store launch uuid as rp_uuid per merged run, and as launch_uuid per each plan,
  store launch_url per each plan
* rewritten environment variables to the uniform form TMT_PLUGIN_REPORT_REPORTPORTAL_${option}

note: skipped mypy errors unresolved
@4N0body5 @happz
Small updates of ReportPortal plugin extension
e3b578b 
* Fixed the environment variables
* Prepared defect type locator for implementation of continuous update (idle)
* Prepared rerun for implementation of launch update (retry)
@4N0body5 @happz
Rewriting ReportPortal plugin extension [WIP]:
431d5b7 
* mapping according to options --launch-per-plan and --suite-per-plan
* uploading to existing launch/suite with options --upload-to-launch LAUNCH_ID, --upload-to-suite SUITE_ID
* option --launch-description and preparation for --launch-attributes (for suite-per-plan mapping)
* trial option --launch-rerun
* trial option --defect-type
@4N0body5 @happz
Rewriting ReportPortal plugin extension [WIP in its final stages]
7746f71 
* functional --suite-per-plan option that uploads all plans into a launch;
  + reporting common atributes from all plans, closing the launch after last plan
* additional upload of tests/suites into a existing launch
* minor fixing of previously implemented (--defect-type, etc.)
* listed all the points to be done yet so it is ready for review
@4N0body5 @happz
Rewriting ReportPortal plugin extension [WIP in its final stages]
2e99f3b 
* functional idle report and additional report within run id
* functional upload to launch
* debug based on option combinations
@4N0body5 @happz
Rewriting ReportPortal plugin extension
719f564 
* fixed the functonality to report idle tests and additional results
* rewritten code into methods
* added version
* postposing the functionality 'adding files as attanchement' to another PR
@4N0body5 @happz
Updated ReportPortal plugin extension based on review feedbacks
b2be094 
+ resolved mypy conflicts
@lukaszachy
Copy link
Collaborator

lukaszachy commented Feb 6, 2024 •
edited
Loading

/packit test -i full

@lukaszachy
Copy link
Collaborator

/packit test -i provision

@lukaszachy lukaszachy merged commit afe1a56 into main Feb 6, 2024
18 of 19 checks passed
@lukaszachy lukaszachy deleted the reportportalAPI branch February 6, 2024 12:44
@4N0body5 4N0body5 mentioned this pull request Feb 13, 2024
Closed
@4N0body5 4N0body5 mentioned this pull request Jul 4, 2024
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@happz happz happz approved these changes

@lukaszachy lukaszachy lukaszachy approved these changes

@psss psss Awaiting requested review from psss psss is a code owner

@thrix thrix Awaiting requested review from thrix thrix is a code owner

Assignees
No one assigned
Labels
plugin | reportportal The reportportal report plugin priority | must high priority, must be included in the next release step | report Stuff related to the report step
Projects
None yet
Milestone
1.31
Development

Successfully merging this pull request may close these issues.
7 participants
@4N0body5 @brakogh @psss @kkaarreell @thrix @lukaszachy @happz