[R-Forge #5417] keyby doesn't work in a presence of a key and a filtering condition

[arunsrinivasan]

Submitted by: Eduard Antonyan; Assigned to: Nobody; R-Forge link

This bug has been introduced some time between 1.8.11 and 1.9.2:

dt = data.table(a = 1:3, c = c("X", "P", "X"), d = 1:3, key = 'a')
dt[TRUE, sum(d), keyby = c]
#   c V1
#1: X  4
#2: P  2

When not filtering, it works correctly, but the filtering screws something up:

dt[, sum(d), keyby = c]
#   c V1
#1: P  2
#2: X  4

A couple more demos of the problem:

key(dt[TRUE, sum(d), keyby = c])
# [1] "c"

setkey(dt[TRUE, sum(d), keyby = c], c)
# Warning message:
# In setkeyv(x, cols, verbose = verbose) :
#   Already keyed by this key but had invalid row order, key rebuilt. If you didn't go #under the hood please let datatable-help know so the root cause can be fixed.