Pass manager to data loader #1280
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamgayoso
reviewed
Nov 29, 2021
| adata_manager = self.adata_manager | ||
| else: | ||
| adata_manager = self.get_anndata_manager(adata) | ||
| if adata_manager is None: |
There was a problem hiding this comment.
should we proceed here with transfer? This function is typically run after validation. See get latent rep for example...
There was a problem hiding this comment.
perhaps we raise an error to go run validate?
There was a problem hiding this comment.
i see, that sounds good
justjhong
added a commit
that referenced
this pull request
Jan 16, 2022
* pass manager to data loader * address comment
justjhong
added a commit
that referenced
this pull request
Feb 2, 2022
* pass manager to data loader * address comment
justjhong
added a commit
that referenced
this pull request
Feb 5, 2022
* placeholder release notes for base * AnnData Manager refactor (#1238) * move anndata file into anndata subfolder * mock out anndata manager and fields for categorical obs fields only * add layerfield * fix imports * fix imports in models * add temporary test and fix bugs in manager code * manager store on model class, refactor what needs to be defined for a model * clean up some existing code * restructure fields file, clean up more view anndata setup code, add more skeleton * add transfer_field for obs and layer fields * naive implementation of transfer_setup in anndata manager * implement backwards compat with setup dict on load * add docstrings to all new methods * address comments: change to registry_key, change fields to set in manager * add tests for transfer and manager from setup * remove anndata_fields method * address pr comment * Implement NonCategoricalJointObsField (#1247) * implement continuous obsm field * make class specific to joint obs fields * make fields public * address comments * faster concat by subsetting obs * Implement CategoricalJointObsField (#1248) * add categorical joint obs field * fix _make_obsm_categorical * add line in test * address pr comment * Pass manager to data loader (#1280) * pass manager to data loader * address comment * Move and reorganize setup dict out of AnnData into Manager (#1285) * new base field API for setup dict * adapt manager to new registry * backwards compatibility implementation * train works * adapt get_from_registry function * simplify needs transfer logic, working test_scvi * change _REGISTRY_KEYS back to _CONSTANTS * address comments, fix compat test * codacy * Add anndata registration classes to developer API (#1292) * add anndata registration classes to developer API * move manager to anndata submodule to fix docs * Save/Load with saved setup inputs (#1294) * skeleton for new save load, missing field logic * save load with saved setup inputs * add docstring for setup_inputs * codacy * fix documentation * additional pr comments * fix manager imports * Adapt TotalVI to new setup (#1298) * obsm field * adapt totalVI to new setup * fix additional totalvi test * make field registration ordered * fix totalvi protein priors logic * address pr comments * address pr comments * Fix tests for SCVI, TOTALVI, generic AnnData utils (#1299) * adapt totalVI to new setup * fix additional totalvi test * remove run_setup_anndata kwarg, reassign uuid on view copy, fix some tests * remove run_setup_anndata comments * fix transfer anndata test * fix test data format test * fix setup anndata test * fix most pyro tests * fix most scarches tests * remove state dict from field * Adapt all internal models to new setup (#1301) * adapt LDA * adapt linearscvi * remove _get_var_names_from_setup_anndata * adapt peakvi * adapt autozi * adapt scanvi * fix scanvi test * fix totalvi test * fix dataloader tests * fix multiple cov tests * adapt condscvi * adapt destvi * adapt multivi * fix setup compat test * remove get_from_registry util * fix scanvi and peakvi scarches tests * fix backwards compat tests and default missing summary stat in models * address comment * Adapt all external models to new setup (#1302) * adapt cellassign * adapt gimvi * adapt solo model * adapt stereoscope * codacy * rename _CONSTANTS to REGISTRY_KEYS (#1306) * Remove deprecated functions (#1308) * remove unused utils * remove references to removed functions in docs * Cleanup Manager register reference with AttrDict implementation (#1307) * implement attrdict and add it to properties of manager * use attrdict feature model code * address pr comments * codacy * view_anndata_setup new implementation (#1313) * implement attrdict and add it to properties of manager * use attrdict feature model code * address pr comments * view anndata setup implementation * fix bugs and tweak text * Field specific `view_anndata_setup` (#1315) * implement attrdict and add it to properties of manager * use attrdict feature model code * address pr comments * view anndata setup implementation * fix bugs and tweak text * field specific view setup * rich links in docstrings * update get_totalvi_protein_priors code w/ master * Fix docs (#1330) * codacy * address first round of comments * comments in constants file * add view registry tests * fix compat function * Update _layer_field.py * Update _obsm_field.py Co-authored-by: Adam Gayoso <[email protected]>
nrclaudio
pushed a commit
to nrclaudio/scvi-tools-tune
that referenced
this pull request
Jun 21, 2022
* placeholder release notes for base * AnnData Manager refactor (scverse#1238) * move anndata file into anndata subfolder * mock out anndata manager and fields for categorical obs fields only * add layerfield * fix imports * fix imports in models * add temporary test and fix bugs in manager code * manager store on model class, refactor what needs to be defined for a model * clean up some existing code * restructure fields file, clean up more view anndata setup code, add more skeleton * add transfer_field for obs and layer fields * naive implementation of transfer_setup in anndata manager * implement backwards compat with setup dict on load * add docstrings to all new methods * address comments: change to registry_key, change fields to set in manager * add tests for transfer and manager from setup * remove anndata_fields method * address pr comment * Implement NonCategoricalJointObsField (scverse#1247) * implement continuous obsm field * make class specific to joint obs fields * make fields public * address comments * faster concat by subsetting obs * Implement CategoricalJointObsField (scverse#1248) * add categorical joint obs field * fix _make_obsm_categorical * add line in test * address pr comment * Pass manager to data loader (scverse#1280) * pass manager to data loader * address comment * Move and reorganize setup dict out of AnnData into Manager (scverse#1285) * new base field API for setup dict * adapt manager to new registry * backwards compatibility implementation * train works * adapt get_from_registry function * simplify needs transfer logic, working test_scvi * change _REGISTRY_KEYS back to _CONSTANTS * address comments, fix compat test * codacy * Add anndata registration classes to developer API (scverse#1292) * add anndata registration classes to developer API * move manager to anndata submodule to fix docs * Save/Load with saved setup inputs (scverse#1294) * skeleton for new save load, missing field logic * save load with saved setup inputs * add docstring for setup_inputs * codacy * fix documentation * additional pr comments * fix manager imports * Adapt TotalVI to new setup (scverse#1298) * obsm field * adapt totalVI to new setup * fix additional totalvi test * make field registration ordered * fix totalvi protein priors logic * address pr comments * address pr comments * Fix tests for SCVI, TOTALVI, generic AnnData utils (scverse#1299) * adapt totalVI to new setup * fix additional totalvi test * remove run_setup_anndata kwarg, reassign uuid on view copy, fix some tests * remove run_setup_anndata comments * fix transfer anndata test * fix test data format test * fix setup anndata test * fix most pyro tests * fix most scarches tests * remove state dict from field * Adapt all internal models to new setup (scverse#1301) * adapt LDA * adapt linearscvi * remove _get_var_names_from_setup_anndata * adapt peakvi * adapt autozi * adapt scanvi * fix scanvi test * fix totalvi test * fix dataloader tests * fix multiple cov tests * adapt condscvi * adapt destvi * adapt multivi * fix setup compat test * remove get_from_registry util * fix scanvi and peakvi scarches tests * fix backwards compat tests and default missing summary stat in models * address comment * Adapt all external models to new setup (scverse#1302) * adapt cellassign * adapt gimvi * adapt solo model * adapt stereoscope * codacy * rename _CONSTANTS to REGISTRY_KEYS (scverse#1306) * Remove deprecated functions (scverse#1308) * remove unused utils * remove references to removed functions in docs * Cleanup Manager register reference with AttrDict implementation (scverse#1307) * implement attrdict and add it to properties of manager * use attrdict feature model code * address pr comments * codacy * view_anndata_setup new implementation (scverse#1313) * implement attrdict and add it to properties of manager * use attrdict feature model code * address pr comments * view anndata setup implementation * fix bugs and tweak text * Field specific `view_anndata_setup` (scverse#1315) * implement attrdict and add it to properties of manager * use attrdict feature model code * address pr comments * view anndata setup implementation * fix bugs and tweak text * field specific view setup * rich links in docstrings * update get_totalvi_protein_priors code w/ master * Fix docs (scverse#1330) * codacy * address first round of comments * comments in constants file * add view registry tests * fix compat function * Update _layer_field.py * Update _obsm_field.py Co-authored-by: Adam Gayoso <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Passes AnnDataManager object to data loaders and splitters. This will enable the following in coming PRs: