Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When streaming jobs with weather-mv, should not require URIs exist before launching the streaming #311

Open
floraxue opened this issue Apr 6, 2023 · 0 comments · May be fixed by #343
Open

Comments

@floraxue
Copy link
Collaborator

floraxue commented Apr 6, 2023

When the user launches a weather-mv job with topics (meaning it's watching for streaming events like new objects being created), it is possible that the provided URI pattern doesn't match any existing objects in the bucket yet.

However, we currently always require all URIs exist before we launch the beam job code.

This will cause immediate failures for the user so they are not able to launch the streaming job at all.

The check for all URIs exist should only happen for the case when no topics are created. Similarly, for the first_uri argument, we only use it for the non-streaming case. With topics, we should not create first_uri to be clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant