Proposal to load subset of data for Intake Datasets

[talebzeghmi]

Issue:

hvPlot by default loads all partitions into memory, see process_intake(). This is an issue for very large datasets.

Proposed solution:

Two extra parameters (in addition to use_dask) sent to hvPlot.process_intake(): read_partition, dask_sample.

• dask_sample: value would be between 0 and 1.
• read_partition: would indicate which partition to read.

The contents of process_intake can be updated to:

    # todo: argument validation
    if dask_sample:
        data = data.to_dask().sample(frac=float(dask_sample), replace=True) 
    elif use_dask:
        data = data.to_dask()
    elif read_partition:
        data = data.read_partition(read_partition)
    else:
        data = data.read()
    return data

Related

• Parent issue from Intake Large dataset Plot loads all partitions intake/intake#409
• intake select a subset intake select a subset #72