Skip to content
This repository has been archived by the owner on Oct 19, 2024. It is now read-only.

Create database on the fly for inference #73

Open
jonasteuwen opened this issue Mar 28, 2024 · 2 comments · May be fixed by #87
Open

Create database on the fly for inference #73

jonasteuwen opened this issue Mar 28, 2024 · 2 comments · May be fixed by #87
Assignees
@YoniSchirris
Labels
enhancement New feature or request

Comments

@jonasteuwen
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Right now you need to create a database to do inference

Describe the solution you'd like
On the fly creation (e.g. in memory) for inference.
@jonasteuwen jonasteuwen added the enhancement New feature or request label Mar 28, 2024
@jonasteuwen jonasteuwen changed the title Create database on the fly for Create database on the fly for inference Mar 28, 2024
@YoniSchirris YoniSchirris self-assigned this May 29, 2024
@YoniSchirris
Copy link

My plan is:

  • create a new DataDescription class, namely OnTheFlyDataDescription (or something that sounds nicer), that includes only the parameters required for an in-memory database
class DataDescription(BaseModel):
    !! REMOVE -- here to generate a mask !! mask_label: Optional[str] = None
    !! REMOVE -- not training, just inferring!! mask_threshold: Optional[float] = None 
    !! REMOVE -- only input is WSI !! roi_name: Optional[str] = None
    !! REMOVE -- not sure what this is? !! num_classes: NonNegativeInt
    == KEEP  -- need a root dir for WSIs == data_dir: Path 
    !! REMOVE -- we'll generate this on the fly !!manifest_database_uri: str
    ?? NOT SURE -- i dont know what a manifest is ?? manifest_name: str
    !! REMOVE -- not using splits, just do everything !! split_version: str
    !! REMOVE -- notu sing any, just generating !! annotations_dir: Path
    ?? NOT SURE? Remove? not training? ?? training_grid: GridDescription
    == KEEP -- we need to know how to infer == inference_grid: GridDescription
    !! REMOVE !! index_map: Optional[Dict[str, int]]
    !! REMOVE !!remap_labels: Optional[Dict[str, str]] = None
    !! REMOVE !! use_class_weights: Optional[bool] = False
    !! REMOVE !! convert_mask_to_rois: bool = True
    !! SET TO FALSE !! use_roi: bool = True
    ?? NOT SURE? ??apply_color_profile: bool = False

So i guess we get something like

class OnTheFlyDataDescription(BaseModel):
    data_dir: Path # root dir for WSIs
    wsi_glob_pattern: str # add a glob pattern to find all WSIs
    inference_grid: GridDescription # Inference details

Then, in inference.py, before running inference, there'll be a check for the __target__ of the data_description config. If it's something on-the-fly, we'll generate a minimal database in memory, set the config properly, and do inference.

I need to check how to do the in-memory db, but if you agree with this i can start this tomorrow.

@YoniSchirris
Copy link

See 4a524bc

@YoniSchirris YoniSchirris linked a pull request Jun 6, 2024 that will close this issue
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@YoniSchirris YoniSchirris

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat/on-the-fly inference NKI-AI/ahcore
2 participants
@jonasteuwen @YoniSchirris