You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi there,
First of I want to thank the tsai library it has been a tremendous help in every way, although I need some help now...
So I used the MiniRocketClassifier() model to classify movements of an EMG data. The first dataset worked perfectly fine but now I have a bigger dataset and I always get Memory errors all over the place.
So I have read the tutorial with using mmaps to not allocate to much memory on the RAM but for some reason it is not working properly. Here is what I have done so far:
`
X_train = np.load(f'{save}/X_train.npy', mmap_mode='r') ...
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
`
and then trained the model. Before I saved and loaded the variables with 'r' and 'c' but it did not make a difference. I have tried using batches as well to reduce the memory allocation but this is just bad for the minirocket model in general because mini-rocket does not work that way.
My Question here would be:
How can I use the mmaps and dataloader correctly to train a large dataset in tsai minirocket?
Any help is appreciated.
Thank you so much in advance and have a good day!
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi there,
First of I want to thank the tsai library it has been a tremendous help in every way, although I need some help now...
So I used the MiniRocketClassifier() model to classify movements of an EMG data. The first dataset worked perfectly fine but now I have a bigger dataset and I always get Memory errors all over the place.
So I have read the tutorial with using mmaps to not allocate to much memory on the RAM but for some reason it is not working properly. Here is what I have done so far:
`
X_train = np.load(f'{save}/X_train.npy', mmap_mode='r') ...
Convert numpy arrays to torch tensors
X_train_tensor = torch.tensor(X_train, dtype=torch.float32)
y_train_tensor = torch.tensor(y_train, dtype=torch.long)
X_test_tensor = torch.tensor(X_test, dtype=torch.float32)
y_test_tensor = torch.tensor(y_test, dtype=torch.long)
Create datasets and dataloaders
train_dataset = TensorDataset(X_train_tensor, y_train_tensor)
test_dataset = TensorDataset(X_test_tensor, y_test_tensor)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
`
and then trained the model. Before I saved and loaded the variables with 'r' and 'c' but it did not make a difference. I have tried using batches as well to reduce the memory allocation but this is just bad for the minirocket model in general because mini-rocket does not work that way.
My Question here would be:
How can I use the mmaps and dataloader correctly to train a large dataset in tsai minirocket?
Any help is appreciated.
Thank you so much in advance and have a good day!
Best regards,
A desperate student
Beta Was this translation helpful? Give feedback.
All reactions