Addition of cuDF and HoloViews for GPU acceleration #485
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This implementation of LUX adds the option of utilizing NVIDIA GPUs, with RAPIDS cuDF and HoloViews as the plotting engine. It is capable of a speed up 3-10X compared to the original LUX, and avoids browser memory issues when dealing with datasets in the millions+ rows (measured on the NVIDIA RTX A3000 Laptop GPU).
HoloViews
HoloViews does not require the creation of a JSON file which for larger datasets is both memory and time expensive. It is able to show magnitudes of more data points on its curve without being time consuming, also removing the constraint of having to fall back to heatmaps rather than scatter plots, when the number of rows is too high. In this implementation we have not relied on the LUX widget for displaying the charts, for simplicity in viewing.
To Run
To run the cuDF + HoloViews implementation, simply do the following
To plot the HoloViews curves, run
df.maintain_recs()
rather thandf
in a different cell.Example Output
A brief output is shown below
Next Steps
This implementation is a proof of concept demonstrating the acceleration that RAPIDS can bring to LUX. It also shows the benefits of adding HoloViews as an additional option for plotting. @exactlyallan, @AjayThorve and I would like to discuss if and how an integration like this might proceed, @dorisjlee?