Need long-vector support

[renkun-ken]

Currently, data.table does not support long vectors as columns. Long vector support was not requested much (related: #2751, #2340). However, I do need handle data with long vectors more frequently as I step more into the area of high frequency data.

I'm not sure how difficult it is to support long vectors but it would be very nice and much appreciated to put this on the roadmap.

[2005m]

Would anyone know why data.frame has a limit in rows in the first place? I couldn't find an answer on R-help. Just wondering why they implemented it this way?

[renkun-ken]

Please see https://stat.ethz.ch/R-manual/R-devel/library/base/html/LongVectors.html.

[MichaelChirico]

@renkun-ken I know we have built long vector support in a few places, it might be helpful to make a prioritized list of where it's still not implemented?

[renkun-ken]

@MichaelChirico Do we have a list of places where long vectors are supported or could be supported?

[jangorecki]

I don't think it make sense to start with anything else than forder (which is neded for grouping and joining). AFAIK Matt has that on his radar. Once we will have forder working for long vectors, then the rest can follow. Just speculating now... but we may ended up making an abstraction layer to handle int32/int64 length, because supporting long vectors always may impose performance penalty even for int32 length vectors. It all needs benchmarking to make proper decisions. My suggestion is to hold on with this issue for @mattdowle.