Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor batch item file management #51

Merged
merged 8 commits into from
Jun 20, 2022
Merged

Conversation

kushalkolar
Copy link
Collaborator

@kushalkolar kushalkolar commented Jun 20, 2022

Refactor all algos and extensions so that output data that corresponds to a batch item are stored within their respective subdirs within the batch dir (which is the same dir as a batch file). Raw input movies remain where they are.

Created PathsDataFrameExtension and PathsSeriesExtension in batch_utils module to simplify resolving or splitting paths of input and output files for batch items.

The following extensions will automatically resolve the full path, first searching within the batch dir and then within the parent dir of the current parent_raw_data_path.
df.paths.resolve(<path>), also works for Series for example: df.iloc[0].resolve(<path>

This functionality require the current main branch of pandas as of June 19, 2022. Specifically it requires that the dict pd.DataFrame.attrs gets propagated to child pd.Series.attrs. This functionality will apparently be released in pandas v1.5.0 on June 30, 2022. For more info see: pandas-dev/pandas#46101

Example:

load_batch(<batch_path.pickle>)
batch_item = df.iloc[-1]
input_movie_path = batch_item.params["input_movie_path"]
full_input_movie_path = batch_item.paths.resolve(input_movie_path)

also df.caiman.run() now uses subprocess backend by default and doesn't required the batch_path as a kwarg, it gets it from df.paths.get_batch_path()

@kushalkolar
Copy link
Collaborator Author

WIP, tests pass on my end for mcorr, working on CNMF and CNMFE

@kushalkolar kushalkolar mentioned this pull request Jun 20, 2022
11 tasks
@kushalkolar kushalkolar merged commit e7c2f15 into master Jun 20, 2022
@kushalkolar kushalkolar added this to the v0.1 milestone Jul 3, 2022
@kushalkolar kushalkolar deleted the refactor-batch-filepaths branch November 8, 2022 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant