Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve xfailed management in API performance tests #3665

Closed
Selutario opened this issue Dec 12, 2022 · 2 comments · Fixed by #4657
Closed

Improve xfailed management in API performance tests #3665

Selutario opened this issue Dec 12, 2022 · 2 comments · Fixed by #4657
Assignees
Labels
level/task Task issue qa_known Issues that are already known by the QA team type/bug

Comments

@Selutario
Copy link
Contributor

Target version Related issue Related PR/dev branch
- wazuh/wazuh#15653 -

Description

Currently, the way xfail is used in API performance tests prevents endpoint execution from even being attempted:

# Apply xfails
if test_case['endpoint'] in xfailed_items.keys() and \
test_case['method'] == xfailed_items[test_case['endpoint']]['method']:
pytest.xfail(xfailed_items[test_case['endpoint']]['message'])

This makes it impossible to determine if any of the listed endpoints have worked correctly:

xfailed_items = {
'/active-response': {'message': 'Agent simulator not handling active-response messages: '
'https:/wazuh/wazuh-qa/issues/1266',
'method': 'put'},
'/agents/group': {'message': 'Investigate performance issues with PUT /agents/group API endpoint: '
'https:/wazuh/wazuh/issues/13872',
'method': 'put'}
}

Sometimes, such as for the PUT /agents/group endpoint in the tests run in the issue below, it might be useful to know if it's still failing or not, without explicitly modifying the list:

Configurations

Proposed test cases

Considerations

These tests are usually run inside a performance pipeline.

@Selutario Selutario added team/qa and removed team/qa labels Dec 12, 2022
@davidjiglesias davidjiglesias changed the title Improve xfailed managment in API performance tests Improve xfailed management in API performance tests Mar 3, 2023
@jmv74211 jmv74211 removed the bug label Mar 15, 2023
@damarisg damarisg added the qa_known Issues that are already known by the QA team label Oct 19, 2023
@RamosFe
Copy link
Member

RamosFe commented Nov 3, 2023

Update

Passed to on hold due to Release 4.7.0 Integration tests.

@davidjiglesias
Copy link
Member

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue qa_known Issues that are already known by the QA team type/bug
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

5 participants