Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache functionality #875

Merged
merged 51 commits into from
Nov 23, 2022
Merged

Cache functionality #875

Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
db38852
Cache functionality added. Main tests pass
MehmedGIT May 25, 2022
60e1a60
Fixes and cache tests added
MehmedGIT May 30, 2022
d8c2f50
Fix typo
MehmedGIT May 31, 2022
ef1c757
Applying the changes suggested by kba
MehmedGIT Jun 1, 2022
2414e63
page_cache and fptr_cache added
MehmedGIT Jun 3, 2022
d7d196e
Add the missed page_cache in clearCache
MehmedGIT Jun 3, 2022
0058823
Fixing some bugs
MehmedGIT Jun 7, 2022
ce3ffc8
Extend tests for 200 pages
MehmedGIT Jun 8, 2022
39bdf5e
Comment out test case for 200 pages - takes too long
MehmedGIT Jun 8, 2022
f8d3ac2
No change. Trigger scrutinizer again.
MehmedGIT Jun 15, 2022
b423d1d
Include extreme example benchmarking tests
MehmedGIT Jun 28, 2022
dc6e387
Extreme benchmark test for 750 files per page (5000 pages)
MehmedGIT Oct 4, 2022
e86b8c2
clean the changes
MehmedGIT Oct 12, 2022
f1e6597
To keep the cache_functionality branch up-to-date
MehmedGIT Oct 12, 2022
9ff4d26
Cache functionality after master merge
MehmedGIT Oct 12, 2022
2326a94
Test for 500 pages and 1500 files per page
MehmedGIT Oct 12, 2022
90212ea
Fix the test case
MehmedGIT Oct 12, 2022
50e0f95
Remove the conflicting file
MehmedGIT Oct 24, 2022
afa0162
Return back ocrd_mets
MehmedGIT Oct 24, 2022
6ed7c07
Merge branch 'master' into cache_functionality
MehmedGIT Oct 24, 2022
f997e5a
Cache functionality added again
MehmedGIT Oct 24, 2022
bdf5741
Fix missing parts
MehmedGIT Oct 24, 2022
ffe97cc
Fix the returned constructor with caching flag
MehmedGIT Oct 24, 2022
335d8d2
Fix cache tests
MehmedGIT Oct 24, 2022
a70bf58
test_ocrd_mets_cache: adapt to changed behavior because of caching
kba Nov 3, 2022
2206ccd
remove unnecessary else indent
kba Nov 3, 2022
d095fa6
use log.debug instead print
kba Nov 3, 2022
3f9348a
mets caching: iterate only over actual fileGrp elements
kba Nov 3, 2022
06d22af
mets caching: iterate only over mets:div[@TYPE="page"]
kba Nov 3, 2022
bd04777
mets caching: use fileGrp cache for OcrdMets.file_groups
kba Nov 3, 2022
78715a7
Merge branch 'master' into cache_functionality-kba
kba Nov 3, 2022
ae75e37
Merge pull request #944 from OCR-D/cache_functionality-kba
MehmedGIT Nov 4, 2022
5b50ca0
OcrdMets.find_files: allow mixing regex, range and literal multi-valu…
kba Nov 17, 2022
6fd0220
generate_range: raise ValueError if start == end
kba Nov 17, 2022
9cf0d9c
generate_range: choose the last number in a string
kba Nov 17, 2022
016a370
Merge branch 'master' into cache_functionality
kba Nov 20, 2022
c9e1180
separate targets benchmark{,-extreme} for the METS benchmarks
kba Nov 20, 2022
6522e54
test_ocrd_mets: combine cachinig and non-caching tests
kba Nov 20, 2022
f7a0f5b
merge #955
kba Nov 20, 2022
a6656da
Add fileGrp parameter to remove function
MehmedGIT Nov 21, 2022
4e4b3ee
OcrdMets.__str__: also provide cached/non-cached status
kba Nov 21, 2022
82b3e4f
OcrdMets.__str__: fix it and str test
MehmedGIT Nov 22, 2022
27b6c86
OcrdMets: Don't defend against inconsistency cache vs XML
kba Nov 22, 2022
1e8ff90
OcrdMets: remove outdated comment
kba Nov 22, 2022
ffcd89f
OcrdMets.set_physical_page_for_file: pageId is always a str
kba Nov 22, 2022
4da45f6
OcrdMets: Don't defend against inconsistency cache vs XML
kba Nov 22, 2022
7724191
docstring for OcrdMets.remove_one_file
kba Nov 22, 2022
384b4ac
Merge branch 'cache_functionality' of https:/OCR-D/core i…
kba Nov 22, 2022
2fad30b
revert 4da45f6b3 (el_pagediv can be legitimately None here )
kba Nov 22, 2022
3c5ac1e
enable caching by setting OCRD_METS_CACHING=true env var
kba Nov 22, 2022
f21a33a
readme: add a stub section on configuration
kba Nov 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
- checkout
- run: HOMEBREW_NO_AUTO_UPDATE=1 brew install imagemagick geos
- run: make deps-test install PIP=pip3
- run: make test PYTHON=python3
- run: make test benchmark PYTHON=python3

test-python36:
docker:
Expand All @@ -24,7 +24,7 @@ jobs:
- checkout
- run: apt-get -y update
- run: make deps-ubuntu deps-test install
- run: make test
- run: make test benchmark

test-python37:
docker:
Expand All @@ -34,7 +34,7 @@ jobs:
- checkout
- run: apt-get -y update
- run: make deps-ubuntu deps-test install
- run: make test
- run: make test benchmark

test-python38:
docker:
Expand All @@ -44,7 +44,7 @@ jobs:
- checkout
- run: apt-get -y update
- run: make deps-ubuntu deps-test install
- run: make test
- run: make test benchmark

test-python39:
docker:
Expand All @@ -54,7 +54,7 @@ jobs:
- checkout
- run: apt-get -y update
- run: make deps-ubuntu deps-test install
- run: make test
- run: make test benchmark

deploy:
docker:
Expand Down
8 changes: 7 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,13 @@ assets: repo/assets
test: assets
HOME=$(CURDIR)/ocrd_utils $(PYTHON) -m pytest --continue-on-collection-errors -k TestLogging $(TESTDIR)
HOME=$(CURDIR) $(PYTHON) -m pytest --continue-on-collection-errors -k TestLogging $(TESTDIR)
$(PYTHON) -m pytest --continue-on-collection-errors --durations=10 --ignore=$(TESTDIR)/test_logging.py $(TESTDIR)
$(PYTHON) -m pytest --continue-on-collection-errors --durations=10 --ignore=$(TESTDIR)/test_logging.py --ignore-glob="$(TESTDIR)/**/*bench*.py" $(TESTDIR)

benchmark:
$(PYTHON) -m pytest $(TESTDIR)/model/test_ocrd_mets_bench.py

benchmark-extreme:
$(PYTHON) -m pytest $(TESTDIR)/model/*bench*.py

test-profile:
$(PYTHON) -m cProfile -o profile $$(which pytest)
Expand Down
1 change: 0 additions & 1 deletion ocrd_models/ocrd_models/ocrd_file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,6 @@ def pageId(self, pageId):
raise Exception("OcrdFile %s has no member 'mets' pointing to parent OcrdMets" % self)
self.mets.set_physical_page_for_file(pageId, self)


@property
def loctype(self):
"""
Expand Down
Loading