Quota manager for GCS #2084
Merged
Quota manager for GCS #2084
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bemoody
force-pushed
the
gcs-quota-manager
branch
from
88af46f
to
4152ef0
Compare
|
Late last night I realized that despite how little functional code there is here, I still managed to make a small mistake. That's why we need test cases ;) |
added 3 commits
September 22, 2023 09:40
This class implements the basic QuotaManager API for the Google Cloud Storage backend. (Note the SubmissionInfo.quota_manager method is currently broken for GCS - and always has been, since GCS was introduced.) Note that GCS doesn't (as far as I know) provide any way to limit the storage usage of a bucket or a project, let alone the usage of a particular prefix within a bucket. (I also don't know how Google calculates the cost of storing objects in GCS, but obviously there is some storage used for metadata in addition to the object content.) I also don't know if there is any way to report the storage usage of a bucket or prefix in any meaningful way, other than by actually iterating over all the blobs in the bucket as GCSObject.size does. So GCSQuotaManager calculates bytes_used in the same inefficient way as GCSProjectFiles.active_project_storage_used (which this class is intended to replace.) Currently it does *not* calculate inodes_used (though that should be easy enough to do, it should ideally be done by GCSObject and avoid making more unnecessary requests.) GCSQuotaManager also implements the other QuotaManager methods, such as create_toplevel_directory and check_create_file, in much the same way that DemoQuotaManager does, so that we can eventually use the QuotaManager API in other areas of the code.
Quota management behavior depends on the project file storage. For GCS storage, a GCSQuotaManager is required. (Until now, the SubmissionInfo.quota_manager method has been broken on GCS, and the GCS backend has carefully avoided using this API - which unfortunately has also blocked development of the quota system.) Rather than including this logic in SubmissionInfo, it seemingly makes more sense for quota policy to be part of ProjectFiles; so a new abstract method 'project_quota_manager' is defined.
The 'active_project_storage_used' method is used to report "how much of the storage allowance for the given project submission has been used." (The name suggests that this is somehow similar to 'published_project_storage_used', but this is not the case: *that* method is used to inform visitors "how large will the project files be if I download all of them", which is only tangentially related to the resource usage that a particular submission is responsible for.) Reporting (as well as limiting) resource usage is the task of the QuotaManager. Use the QuotaManager API and remove the redundant ProjectFiles API.
bemoody
force-pushed
the
gcs-quota-manager
branch
from
4152ef0
to
eb87ee5
Compare
|
Thanks Benjamin, looking forward to seeing where this is going! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today's yak-shaving has brought me to the fact that the GCS storage backend never properly implemented the
SubmissionInfo.quota_managermethod.(Want to know what
quota_manageris about? See #1003 (comment) and issue #518)This pull request rearranges logic a bit but shouldn't have any functional changes - storage usage is still measured by calling
GCSObject.size.This is necessary for exciting and wonderful things in the future.