Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-49928][PYTHON][TESTS] Refactor plot-related unit tests #48415

Closed
wants to merge 1 commit into from

Conversation

xinrong-meng
Copy link
Member

What changes were proposed in this pull request?

Refactor plot-related unit tests.

Why are the changes needed?

Different plots have different key attributes of the resulting figure to test against. The refactor makes the comparison easier.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Test changes.

Was this patch authored or co-authored using generative AI tooling?

No.

@xinrong-meng
Copy link
Member Author

Irrelevant tests failed, retriggering:

ERROR [3.661s]: test_listener_events (pyspark.sql.tests.streaming.test_streaming_listener.StreamingListenerTests.test_listener_events)

ERROR [0.145s]: test_streaming_progress (pyspark.sql.tests.streaming.test_streaming_listener.StreamingListenerTests.test_streaming_progress)

@xinrong-meng xinrong-meng marked this pull request as ready for review October 11, 2024 04:33
def _check_fig_data(self, fig_data, **kwargs):
for key, expected_value in kwargs.items():
if key in ["x", "y", "labels", "values"]:
converted_values = [v.item() if hasattr(v, "item") else v for v in fig_data[key]]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To accommodate the change in the representation of scalars in NumPy 2, see NumPy 2.0 release notes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, shall we use isinstance(v, np.generic) for this purpose?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern is that might require the numpy dependency. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can use it with has_numpy

@xinrong-meng
Copy link
Member Author

cc @zhengruifeng @HyukjinKwon would you please review thanks!

@xinrong-meng
Copy link
Member Author

We may later port those expected_fig_data dictionaries to a separate JSON file for easier auditing if the number of tests increases

@xinrong-meng
Copy link
Member Author

Merged to master, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants