How to evalute tsai models using cross-validation

[kiview]

The answer might be obvious for folks familiar with different deep learning libraries, but for me it is not and I want to avoid falling into any potential traps.

From scikit-learn, I know about their cross-validate function, which is extremely easy (and safe?) to use. What is the easiest way to perform this using tsai?

I've seen different examples for splitting the data for cross-validation in the tsai docs, so I assume it's easy once I understand the API better. Or maybe there is even an example somewhere in the tutorials, which I've missed.

[vrodriguezf]

I think the easiest way to get the classic K-fold validation would be to create multiple splits of the data calling get_splits with n_splits = K and shuffle=False, and then iterate on the resulting list. Learner-wise I do not know if fastai has something to do it directly, although I doubt it, since it's not something that Jeremy applies often in practical deep learning.

Best!

[vrodriguezf]

It depends on how much data you have in my opinion. In classic ml with scarce data, K-fold validation allows to use every single item for training and that used to be critical. In deep learning scenarios with lots of data, the most important thing is that the validation set is similar to the test set, that is, similar to the situation in which your model is deployed in the real world. In you want to put a +/- symbol of the performance of your model in a paper, you can always run the training with different seeds and see how stable the metrics are.

Btw, you may want to have a look at this nice article on validation sets by Rachel Thomas from fastai:
https://www.fast.ai/2017/11/13/validation-sets/

[kiview]

For the data I am having (body point coordinate data basically), I was thinking of using something like:

splits = get_splits(y, n_splits=5, shuffle=False, stratify=True)

Shuffling leads to very wrong (unrealistically good actually) results, because I suppose there is a tendency of similar data in train and validation set after shuffling (which could potentially be mitigated by bigger step size for the windows). If I would just select a distinct validation set (which theoretically contains enough data), I thought this would be a very arbitrary selection, while the cross-validation should give a better indication of general performance.

[oguiza]

Something that may help you make a more informed decision whether to shuffle data or not is to use something like this:

y = np.random.permutation(np.concatenate([np.zeros(100), np.ones(1000)]))
print(np.mean(y))
splits = get_splits(y, n_splits=5, shuffle=False, stratify=True)
for s in splits:
    print(np.mean(y[s[0]]), np.mean(y[s[1]]))
splits

y = np.random.permutation(np.concatenate([np.zeros(100), np.ones(1000)]))
print(np.mean(y))
splits = get_splits(y, n_splits=5, shuffle=True, stratify=True)
for s in splits:
    print(np.mean(y[s[0]]), np.mean(y[s[1]]))
splits

get_splits has a built-in function to check no leakage between train, val (and test) exists. If there was any leakage it would raise a warning.
One aspect to take into account if sliding window is that you don't want to use valid or test splits that occur before any training sample. If this is the case, you should consider using a walk-forward method, which is not yet implemented in tsai.

[kiview]

I am now doing the following, which in general does what I want:

def train_and_evaluate_it(X, y, splits, bs, lr, epochs):
    tfms  = [None, [Categorize()]]
    dsets = TSDatasets(X, y, tfms=tfms, splits=splits, inplace=True)
    dls = TSDataLoaders.from_dsets(dsets.train, dsets.valid, bs=bs, batch_tfms=[TSStandardize()], num_workers=0)
    model = InceptionTime(dls.vars, dls.c)
    learn.fit_one_cycle(epochs, lr_max=lr)
    # [...] omitted rest of the code

splits = get_splits(y, n_splits=5, shuffle=False, stratify=True, check_splits=True)
for s in splits:
    train_and_evaluate_it(X, y, s, 512, 0.001, 1)

I just wonder, if there is a more elegant way to use TSDatasets, since I am constructing it for every fold and this also takes some time.

In my case, walk-forward is not really important for my data splitting, but group seperation, as given in this scikit-learn example, probably are.

[xcvil]

Does it need to load the weights?

splits = get_splits(y, n_splits=5, balance=True, stratify=True, random_state=23, shuffle=True)
tfms = [None, [Categorize()]]
for split in splits:
    dsets = TSDatasets(X, y, tfms=tfms, splits=split, inplace=True)
    dls = TSDataLoaders.from_dsets(dsets.train, dsets.valid, bs=[64, 128], batch_tfms=[TSStandardize(by_var=True)], num_workers=0)
    model = TCN(dls.vars, dls.c)
    learn = Learner(dls, model, metrics=[Precision(), Recall(), accuracy])
    try:
        learn.load('stage')
        print(f'loaded weights')
    except FileNotFoundError: 
        pass
    learn.fit_one_cycle(5, lr_max=1e-3)
    learn.save('stage')