New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] Filecache chunking #43576

Closed

icewind1991 wants to merge 7 commits into master from filecache-chunking

Member

icewind1991 commented Feb 14, 2024 •

edited

Loading

block queries to filecache tables that don't go through the sharding logic
allow getting a query builder for the filecache tables when provided a storage id to select a shard
- probably not going to expose this to apps and leave it internal for the cache handling
provide methods for querying by fileid without known storage

icewind1991 mentioned this pull request

Implement access abstraction to queries #43563

Closed

45 tasks

This comment was marked as resolved.

Sign in to view

icewind1991 force-pushed the filecache-chunking branch from 928fe31 to 367ba8b Compare

February 19, 2024 15:39

skjnldsv added the 2. developing label

icewind1991 force-pushed the filecache-chunking branch from 367ba8b to c77f583 Compare

February 27, 2024 15:49

github-advanced-security bot found potential problems

View reviewed changes

lib/private/DB/QueryBuilder/QueryBuilder.php

@@ @@ -693,6 +717,7 @@ @@
  * @return $this This QueryBuilder instance.
  */
  public function from($from, $alias = null) {
+ $this->checkTableAccess($from);

Check failure

Code scanning / Psalm

ImplicitToStringCast Error

Argument 1 of OC\DB\QueryBuilder\QueryBuilder::checkTableAccess expects string, but OCP\DB\QueryBuilder\IQueryFunction|string provided with a __toString method

lib/private/Files/Cache/CacheDatabase.php

+ }
+ /**
+ * @param List<int> $storages

Check failure

Code scanning / Psalm

UndefinedDocblockClass Error

Docblock-defined class, interface or enum named OC\Files\Cache\List does not exist

lib/private/Files/Cache/CacheDatabase.php

+ }
+ /**
+ * @param List<int> $storages

Check failure

Code scanning / Psalm

MismatchingDocblockParamType Error

Parameter $storages has wrong type 'OC\Files\Cache\List', should be 'array<array-key, mixed>'

lib/private/Files/Cache/CacheDatabase.php

+ /**
+ * @param List<int> $storages
+ * @return array<int, List<int>>

Check failure

Code scanning / Psalm

UndefinedDocblockClass Error

Docblock-defined class, interface or enum named OC\Files\Cache\List does not exist

lib/private/Files/Cache/CacheDatabase.php

+ * The results from the callback will be combined and returned
+ *
+ * @template T
+ * @param List<int> $storages

Check failure

Code scanning / Psalm

UndefinedDocblockClass Error

Docblock-defined class, interface or enum named OC\Files\Cache\List does not exist

lib/private/Files/Cache/QuerySearchHelper.php

- if (in_array('owner', $requestedFields) || in_array('share_with', $requestedFields) || in_array('share_type', $requestedFields)) {
- $this->equipQueryForShares($query);
- }
+ $rawEntries = $this->cacheDatabase->queryStorages($storageIds, function(CacheQueryBuilder $builder, $storages) use ($requestedFields, $searchQuery, $cachesByStorage) {

Check failure

Code scanning / Psalm

InvalidArgument Error

Argument 1 of OC\Files\Cache\CacheDatabase::queryStorages expects OC\Files\Cache\List, but array<T:fn-oc\files\cache\querysearchhelper::searchincaches as array-key, int> provided

lib/private/Files/Cache/QuerySearchHelper.php

- $this->equipQueryForShares($query);
- }
+ $rawEntries = $this->cacheDatabase->queryStorages($storageIds, function(CacheQueryBuilder $builder, $storages) use ($requestedFields, $searchQuery, $cachesByStorage) {
+ $cachesForShard = array_map(fn (int $storage) => $cachesByStorage[$storage], $storages);

Check failure

Code scanning / Psalm

InvalidArgument Error

Argument 2 of array_map expects array<array-key, mixed>, but OC\Files\Cache\List provided

lib/private/Files/Config/UserMountCache.php Fixed Show fixed Hide fixed

lib/private/FilesMetadata/Service/MetadataRequestService.php


		return $list;
		return $this->cacheDatabase->queryStorages($storageIds, function(CacheQueryBuilder $qb, array $storages) use ($fileIdsByStorage) {

Check failure

Code scanning / Psalm

InvalidArgument Error

Argument 1 of OC\Files\Cache\CacheDatabase::queryStorages expects OC\Files\Cache\List, but array<array-key, mixed> provided

lib/private/FilesMetadata/Service/MetadataRequestService.php Outdated

Comment on lines 128 to 149

+ return $this->cacheDatabase->queryStorages($storageIds, function(CacheQueryBuilder $qb, array $storages) use ($fileIdsByStorage) {
+ $fileIds = [];
+ foreach ($storages as $storage) {
+ $fileIds += $fileIdsByStorage[$storage];
+ }
+ $qb->select('file_id', 'json', 'sync_token')->from(self::TABLE_METADATA);
+ $qb->where(
+ $qb->expr()->in('file_id', $qb->createNamedParameter($fileIds, IQueryBuilder::PARAM_INT_ARRAY))
+ );
+ $list = [];
+ $result = $qb->executeQuery();
+ while ($data = $result->fetch()) {
+ $fileId = (int) $data['file_id'];
+ $metadata = new FilesMetadata($fileId);
+ try {
+ $metadata->importFromDatabase($data);
+ } catch (FilesMetadataNotFoundException) {
+ continue;
+ }
+ $list[$fileId] = $metadata;
+ }
+ return $list;
+ });

Check failure

Code scanning / Psalm

InvalidArgument Error

Argument 2 of OC\Files\Cache\CacheDatabase::queryStorages expects callable(OC\Files\Cache\CacheQueryBuilder, OC\Files\Cache\List):array<array-key, OC\FilesMetadata\Model\FilesMetadata>, but impure-Closure(OC\Files\Cache\CacheQueryBuilder, array<array-key, mixed>):array<int, OC\FilesMetadata\Model\FilesMetadata> provided

icewind1991 force-pushed the filecache-chunking branch 7 times, most recently from f06b5b8 to 6d4e9c0 Compare

March 6, 2024 13:54

github-advanced-security bot found potential problems

View reviewed changes

lib/private/FilesMetadata/Service/MetadataRequestService.php

@@ @@ -53,7 +56,8 @@ @@
  * @throws Exception
  */
  public function store(IFilesMetadata $filesMetadata): void {
- $qb = $this->dbConnection->getQueryBuilder();
+ $file = $this->cacheDatabase->getByFileId($filesMetadata->getFileId());

Check failure

Code scanning / Psalm

UndefinedMethod Error

Method OC\Files\Cache\CacheDatabase::getByFileId does not exist

lib/private/FilesMetadata/Service/MetadataRequestService.php

@@ @@ -72,7 +76,12 @@ @@
  */
  public function getMetadataFromFileId(int $fileId): IFilesMetadata {
  try {
- $qb = $this->dbConnection->getQueryBuilder();
+ $file = $this->cacheDatabase->getByFileId($fileId);

Check failure

Code scanning / Psalm

UndefinedMethod Error

Method OC\Files\Cache\CacheDatabase::getByFileId does not exist

lib/private/FilesMetadata/Service/MetadataRequestService.php

- $list[$fileId] = $metadata;
+ public function getMetadataFromFileIds(array $fileIds, array $storageIds = []): array {
+ if (!$storageIds) {
+ $files = $this->cacheDatabase->getByFileIds($fileIds);

Check failure

Code scanning / Psalm

UndefinedMethod Error

Method OC\Files\Cache\CacheDatabase::getByFileIds does not exist

lib/private/FilesMetadata/Service/MetadataRequestService.php

- $qb->delete(self::TABLE_METADATA)
- ->where($qb->expr()->eq('file_id', $qb->createNamedParameter($fileId, IQueryBuilder::PARAM_INT)));
- $qb->executeStatement();
+ $file = $this->cacheDatabase->getByFileId($fileId);

Check failure

Code scanning / Psalm

UndefinedMethod Error

Method OC\Files\Cache\CacheDatabase::getByFileId does not exist

lib/private/FilesMetadata/Service/MetadataRequestService.php

@@ @@ -155,7 +183,8 @@ @@
  * @throws Exception
  */
  public function updateMetadata(IFilesMetadata $filesMetadata): int {
- $qb = $this->dbConnection->getQueryBuilder();
+ $file = $this->cacheDatabase->getByFileId($filesMetadata->getFileId());

Check failure

Code scanning / Psalm

UndefinedMethod Error

Method OC\Files\Cache\CacheDatabase::getByFileId does not exist

icewind1991 mentioned this pull request

Add interface to allow "more direct" access to the filecache without direct db #44044

Closed

icewind1991 force-pushed the filecache-chunking branch from 6d4e9c0 to a5d7410 Compare

March 7, 2024 09:31

github-advanced-security bot found potential problems

View reviewed changes

apps/files_sharing/lib/Updater.php Fixed Show fixed Hide fixed

apps/files_sharing/lib/Updater.php Fixed Show fixed Hide fixed

lib/private/Files/Config/UserMountCache.php Fixed Show fixed Hide fixed

lib/private/Files/Cache/CacheAccess.php Fixed Show fixed Hide fixed

susnux added this to the Nextcloud 30 milestone

icewind1991 mentioned this pull request

expose escapeLikeParameter trough query builder #45457

Merged

icewind1991 added 3 commits

May 22, 2024 16:56


feat: add class for (future) shared access to the filecache

216a92e

Signed-off-by: Robin Appelman <[email protected]>


expose escapeLikeParameter trough query builder

647e64d

Signed-off-by: Robin Appelman <[email protected]>


block access to the filecache tables by default

b3b0c14

Signed-off-by: Robin Appelman <[email protected]>

icewind1991 force-pushed the filecache-chunking branch 2 times, most recently from 8aa41af to adc3d9c Compare

May 22, 2024 15:11

icewind1991 added 4 commits

May 22, 2024 17:12


adjust share provider wip

Signed-off-by: Robin Appelman <[email protected]>


refactor: don't join on filecache in usermountcache

f481a4a

Signed-off-by: Robin Appelman <[email protected]>


transform favorite, tag and systemtag search queries to fileid lists

e7c8840

not as performant but compatible with sharding

Signed-off-by: Robin Appelman <[email protected]>


register DI alias for IFileAccess

4c6aabf

Signed-off-by: Robin Appelman <[email protected]>

icewind1991 force-pushed the filecache-chunking branch from adc3d9c to 4c6aabf Compare

May 22, 2024 15:12

github-advanced-security bot found potential problems

View reviewed changes

lib/private/Files/Cache/Cache.php

@@ @@ -723,26 +718,26 @@ @@
  $retryLimit = 4;
  for ($i = 1; $i <= $retryLimit; $i++) {
  try {
- $this->connection->beginTransaction();
+ $this->cacheDb->beginTransaction();

Check failure

Code scanning / Psalm

TooFewArguments Error

Too few arguments for OC\Files\Cache\CacheDatabase::beginTransaction - expecting storageId to be passed

lib/private/Files/Cache/Cache.php

@@ @@ -723,26 +718,26 @@ @@
  $retryLimit = 4;
  for ($i = 1; $i <= $retryLimit; $i++) {
  try {
- $this->connection->beginTransaction();
+ $this->cacheDb->beginTransaction();

Check failure

Code scanning / Psalm

TooFewArguments Error

Too few arguments for method OC\Files\Cache\CacheDatabase::begintransaction saw 0

lib/private/Files/Cache/Cache.php

  throw $e;
  } catch (RetryableException $e) {
  // Simply throw if we already retried 4 times.
  if ($i === $retryLimit) {
  throw $e;
  }
- $this->connection->rollBack();
+ $this->cacheDb->rollBack();

Check failure

Code scanning / Psalm

TooFewArguments Error

Too few arguments for OC\Files\Cache\CacheDatabase::rollBack - expecting storageId to be passed

lib/private/Files/Cache/Cache.php

  throw $e;
  } catch (RetryableException $e) {
  // Simply throw if we already retried 4 times.
  if ($i === $retryLimit) {
  throw $e;
  }
- $this->connection->rollBack();
+ $this->cacheDb->rollBack();

Check failure

Code scanning / Psalm

TooFewArguments Error

Too few arguments for method OC\Files\Cache\CacheDatabase::rollback saw 0

lib/private/Files/Cache/Cache.php

  // Sleep a bit to give some time to the other transaction to finish.
  usleep(100 * 1000 * $i);
  }
  }
  } else {
- $this->connection->beginTransaction();
+ $this->cacheDb->beginTransaction();

Check failure

Code scanning / Psalm

TooFewArguments Error

Too few arguments for OC\Files\Cache\CacheDatabase::beginTransaction - expecting storageId to be passed

lib/private/Files/Cache/Cache.php

  // Sleep a bit to give some time to the other transaction to finish.
  usleep(100 * 1000 * $i);
  }
  }
  } else {
- $this->connection->beginTransaction();
+ $this->cacheDb->beginTransaction();

Check failure

Code scanning / Psalm

TooFewArguments Error

Too few arguments for method OC\Files\Cache\CacheDatabase::begintransaction saw 0

lib/private/Share20/DefaultShareProvider.php

+ foreach ($shares as $share) {
+ if (isset($files[$share->getNodeId()])) {
+ $cacheItem = $files[$share->getNodeId()];
+ if ($this->isAccessibleResult($cacheItem->getData())) {

Check failure

Code scanning / Psalm

UndefinedInterfaceMethod Error

Method OCP\Files\Cache\ICacheEntry::getData does not exist

skjnldsv modified the milestones: Nextcloud 30, Nextcloud 31

icewind1991 closed this

AndyScherzinger removed this from the Nextcloud 31 milestone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment