progress indicator/bar for "by" operations

[eantonya]

I frequently run by operations that take minutes and sometimes hours to complete. To deal with the uncertainty of what's going on, I often resort to printing .BY in the i-expression:

dt = data.table(a = 1:5)
dt[, {print(.BY)
      Sys.sleep(10)
      5}
   , by = a]

Would be great to have an automatic progress bar for these, similar to fread, if the expected run time is greater than e.g. 30s. Expected run time can be estimated from actual average time spent running the i-expression for each by item so far * number of remaining by items.

[franknarf1]

Dupe of closed #1409

I manually add progress-printers when I see an operation going slowly, but it'd be nice if it were automatic whenever time > 30 s as you suggest.

[mattdowle]

Related: #3050

[jangorecki]

@franknarf1 agree with Frank, but lets print it only for interactive().

[eantonya]

I've been thinking about this, and ideally I'd like to see the following information:

• number of groups processed
• total number of groups, or groups remaining
• time elapsed
• total predicted time, or time remaining
• maybe current group (but I'm not sure how you'd do this without creating a mess)

[MichaelChirico]

last part can be triggered if there are less than n groups... sounds great overall

…

On Wed, Sep 26, 2018, 5:00 AM eduard ***@***.***> wrote: I've been thinking about this and ideally I'd like to see the following information: - number of groups processed - total number of groups or groups remaining - time elapsed - total predicted time or time remaining - maybe current group (but I'm not sure how you'd do this without creating a mess) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3060 (comment)>, or mute the thread <https:/notifications/unsubscribe-auth/AHQQdXIQHM64Jhzg-TKGjYwOWtQU0SDOks5uepmGgaJpZM4W0VV6> .

[Kodiologist]

I too would like this as an opt-in feature of data.table. In the meantime, here's a code snippet to manually add a progress bar with a percentage, given a data table d. Replace GROUP_EXPRESSION and J_EXPRESSION to taste.

bar = txtProgressBar(style = 3, min = 0,
    max = nrow(unique(d[, GROUP_EXPRESSION))))
d[, by = GROUP_EXPRESSION,
   {setTxtProgressBar(bar, .GRP)
    J_EXPRESSION}]
close(bar)

[jangorecki]

if the expected run time is greater than e.g. 30s. Expected run time can be estimated from actual average time spent running the i-expression for each by item so far * number of remaining by items.

i is not run for each item in by, even if it would, then still such estimation would be quite inaccurate because it doesn't say anything about j, to make that automatically we would have to measure time of iterations over progressing groups evaluated in j, and then decide to print at some threshold. This unfortunatelly would impose an overhead, it may be better to enable this feature explicitly rather than automatically.

[MichaelChirico]

the same can be said of the progress bar for fread/fwrite right? at least w.r.t. the overhead concern

[Kodiologist]

You could use pbapply as a dependency, since it already implements time estimation. In my example above, you'd replace txtProgressBar with startpb and setTxtProgressBar with setpb.

[jangorecki]

Time estimation, once we start to evaluate j, is not something we should be worried about. Previous comment was about time estimation before we start evaluating j. Recent PR by @MichaelChirico provides efficient way to know how many groups we have being inside j, so .GRP/.NGRP is not a problem. Problem is that to handle that automatically, when time is longer than a threshold, we have to sum timings always, for all iterations, and this will add overhead. Just R C API impose quite big overhead there already so it doesn't look fine for me to add this one, by default.

[Kodiologist]

Having to turn it on explicitly with an argument, like progress = TRUE, makes sense to me.

[joshhwuu]

What would be the best way to opt-in to this feature? I was wondering what something like showProgress = TRUE would look like within the square bracket syntax.

[Kodiologist]

I imagine a typical call would look like dt[, by = ..., progress = TRUE, ...], where the last ... is j.