-
Features
-
Enhancements
-
Bug Fixes
- Order scatter plot data by usage instead of efficiency statistic. (#323)
- Add
LEAST()function to SQL that gets thewall_time_accuracyvalue. (#324) - Use different endpoint for filter values on efficiency tab filter store freeform search. (#325)
- Fix bugs in resource specification queries for the internal dashboard. (#366)
- Sanitize NAN values from timeseries data (#353)
-
Maintenance
-
Miscellaneous
-
Bug Fixes
- Updates to mitigate php warning seen using php 7.2 (Rocky 8)
-
Features
- Updated default dataset mapping filename to remove the datasource name. The same mapping file can be used with both PCP and Prometheus data sources.
- Bug Fixes
- Fix bug in Efficiency Tab drilldown. This bug is only seen when using php 7.2 (Rocky 8).
-
Features
- Added new "Efficiency" tab that provides reporting and analysis of HPC job efficiency.
- Added new statistics to the Usage and Metric Explorer. These are wall time accuracy, total GPU usage, Homogeneity and Averge max memory. See the online help in XDMoD for full details about these new statistics.
- Added new dimensions to the Usage and Metric Explorer. These are homogeneity rank, total GPU usage and wall time accuracy value See the online help in XDMoD for full details about these new dimensions.
-
Miscellaneous
- Updated nodejs dependency version to nodejs >= 16.13
-
Bug Fixes
- Fix database timeout exception that could occur when running the aggregation for a large number of jobs (such as reaggregating the whole datawarehouse or using an I/O bound database server).
-
Features
- added configuration parameter to
aggregate_supremm.shscript to control whether database table analysis runs.
- added configuration parameter to
-
Changes
- The application classifier now ignores the
sleepprogram when searching for candidate scientific codes. - Update to latest version of the nodejs mongo driver.
- The application classifier now ignores the
-
Miscellaneous
- Updates to internal logging code for compatibility with Open XDMoD 9.5
-
Bug Fixes
- fix numerical error in the calculation for "Walltime per Job' and 'Requested Wall time per Job' metrics in aggregate plot mode. Previously the calculation would ignore jobs that were running, but did not end in the time interval that was plotted.
- add missing category information for Share mode, Resource and Queue search filters in the Job Viewer advanced search interface.
-
Features
-
Changes
- Update internal configuration files to work with the new Open XDMoD datawarehouse Group By and Statistics configuration mechanism. (#218)
- Update internal configuration files to work with the new Open XDMoD raw data configuration mechanism (#238)
- Update the display order of the 'Exit Status' filter to be deterministic (#235)
- Add extra fields to support integration with Open On Demand (#241)
-
Miscellaneous
-
Bug Fixes
- Update the application category for the pegasus workflow software. Previously it was incorrectly marked as having a proprietary license. (#227)
-
Features
- Add Job Efficiency reporting capability. This includes classification of jobs based on performance metrics and components for the new Dashboard tab that show the efficiency metrics by user.
- Add ability to export Job Performance data via the new Data Export tab.
- Added extra statistics for mounted filesystems
/home,/projectsand/util. The default source data for these statistics is from the nfs mounted filesystems. - Added ability to configure which devices are used for the various I/O metrics.
-
Miscellaneous
- Various updates to the module required to support XDMoD 8.5.
- The automated CI testing now confirms that the software works with a password protected MongoDB database.
- Add more automated CI tests.
-
Bug Fixes
- Jobs listed in the advance search results in the job viewer and in the show raw data dialog in the metric explorer are now guaranteed to be ordered based on the job end time.
-
Features
- Add support for GPU metrics. If available, the GPU usage and GPU memory usage for job are shown in the Job Viewer. It is now possible to group and filter by GPU usage in the Metric and Usage Explorer tabs.
- Add support for energy metrics. If available, energy metrics for a job are shown in the Job Viewer. Energy metrics are not available in the Metric or Usage Explorer tabs.
- Improve the application identification algorithm and add more community applications to the database.
- Add more command line options to the sharedjobs script to control which resources are scanned and the scan time range.
- Improve performance of the sharedjobs script when processing large amounts of data (~millions of jobs at a time).
-
Bug Fixes
- The data mapping for InfiniBand metrics previously would only use data for the hardcoded mlx0 device. The mapping has been updated to default to the first available InfiniBand device.
- Add job end time as an additional unique constraint on the jobhosts table.
- Updated label for GPU usage displayed in the job viewer.
-
Miscellaneous
- Several updates required by internal API changes in xdmod 8.1. This includes updates to the internal API for the Job Viewer search and updates for the internal configuration file API.
- The nodejs library dependencies are now packaged in the main xdmod and no longer need to be installed/updated as a separate step. Removed code associated with this install step.
- The dynamically generated MySQL tables are now managed via the ETLv2 framework.
- Updates to the continuous integration (CI) scripts to add more tests.
- Update mongodb driver version
-
Features
- Improved performance of aggregation process by switching to the ETLv2 framework.
- Improved performance of shared jobs analysis script.
-
Bug Fixes
- Changed the database table that stores job scripts so that it can support job arrays and fixed missing unique key that resulted in redundant data storage.
- Only show enabled resources in the Internal Dashboard dataflow diagram.
-
Miscellaneous
- Updated documentation and added troubleshooting information.
- Added a
xdmod-supremm-jobinfoscript that prints information about individual jobs. This is intended to be used for troubleshooting purposes.
- Bug Fixes
- Added acl-config call to the database setup
- Features
- Added PDF export support to the Job Viewer
- Bug Fixes
- Fix erroneous error message seen when running the ingest process for resources where the exit status is not reported by the resource manager
- Miscellaneous
- Added tests
- Additions to the application categorization database
- Features
- Bug Fixes
- Fixed issue that allowed incompatible versions of XDMoD and this module to be installed when installing via RPM (#67)
- Miscellaneous
- Updated for compatibility with Open XDMoD 7.0.0 (#51)
- Moved Node.js ETL framework to Open XDMoD repository (#40)
- Performed work in anticipation of federated instances (#48)
- Improved development workflow (#41)
- Improved quality assurance (#42, #49, #50, #55, #56, #58)
- Improved documentation (#61, #67, #68)
- Features
- Bug Fixes
- Miscellaneous
Important Note: This update adds a dependency to npm. If you are updating an existing installation via RPM, you will need to reinstall npm dependencies afterward. To do this, run the commands below.
# Assuming XDMoD's share directory is RPM default "/usr/share/xdmod"
cd /usr/share/xdmod/etl/js
npm install- Features
- General
- Added peak memory usage metric.
- Improved application identification data.
- Added aggregation data removal to the data reset script.
- Added ability to track metrics for a "projects" filesystem.
- Job Viewer
- Added a count column to the detailed metrics pane to show how many data points were used to calculate the metrics.
- Node ETL
- Added support for uppercase auto-generated labels.
- General
- Bug Fixes
- Job Viewer
- Fixed "Show Raw Data" button in Metric Explorer not filtering results correctly when using some combinations of drilldowns and filters.
- Fixed single-point datasets not appearing in exported charts.
- Fixed Search History tree sorting nodes that should not be sorted.
- Fixed "Show Raw Data" window in Metric Explorer staying active after the chart underneath it changes (for example, when a new chart is imported from Usage).
- Improved handling of raw data specified in kilobytes and megabytes in Detailed Metrics pane.
- Fixed handling of Search History entries that have numeric names.
- Node ETL
- Fixed ingestion process hanging indefinitely if it failed to connect to the Mongo database at certain points.
- Fixed SQL statement queue sometimes batching more statements together in one MySQL driver call than the driver can handle.
- Job Viewer
- Refactors and Miscellaneous
- Spun this module out from the Open XDMoD repository.
- Note that although the Job Viewer and Node ETL is part of Open XDMoD, changes will continued to be tracked as part of SUPReMM as long as it is the only user of both.
- Also note that the Job Viewer is included with Open XDMoD install packages, whereas the Node ETL is included with SUPReMM packages.
- Moved to custom option parser that supports long options and
multi-character short options.
- This replaces minimist and removes it as a dependency.
- Spun this module out from the Open XDMoD repository.
- Features
- General
- Added ability to redact specific job-level values for some users.
- Job Viewer
- Added sort options to search history panel.
- Organized advanced search filters into categories.
- Modified timeseries charts to use the timezone of a job's resource instead of the timezone used by the web browser.
- Modified analytics pane to always be present and explain why missing data is missing.
- Modified byte units to use IEC prefixes instead of SI ones.
- Allowed some metrics to be displayed in multiple tabs.
- Added tooltips to advanced search filters.
- Added tooltips to detailed metrics.
- Added help sections to tabs that didn't have any previously.
- Added a help button to the at-a-glance analytics.
- Added a loading message to charts that are loading.
- General
- Bug Fixes
- Job Viewer
- Fixed a number of cases where editing a search did not work as expected.
- Fixed case where timeseries chart drilldowns stopped working after leaving the Job Viewer and returning.
- Fixed case where a top-level timeseries chart was exported instead of the current, drilled-down chart.
- Fixed case where timeseries chart drilldowns performed on a chart were not reflected in the navigation tree.
- Fixed case where selecting a low-level timeseries chart in the navigation tree opened a top-level chart instead.
- Fixed basic search resource list loading immediately on page load.
- Job Viewer
- New Features
- Configuration
- Switched to URL-based method for specifying Mongo databases.
- This adds support for Mongo databases that require authentication.
- Improved setup process to be more user-friendly.
- The interactive setup script now generates the required configuration files.
- Improved configuration file structure.
- Switched to URL-based method for specifying Mongo databases.
- Data Processing
- Added ability to transfer ingested/aggregated data between databases.
- This allows SUPReMM data to be reprocessed in a secondary database before deploying a new version of the SUPReMM ingestor in the main XDMoD instance.
- Improved logging for ingestion and aggregation scripts.
- Added ability to transfer ingested/aggregated data between databases.
- Job Viewer
- Added ability to edit searches.
- Improved layout of search window.
- Added ability to export timeseries plots as images or CSV data.
- Configuration
- Bug Fixes
- Job Viewer
- Added error dialog box for if Quick Job Lookup's resource list fails to load instead of silently failing.
- Fixed existing searches breaking after performing a re-ingest of SUPReMM data.
- Fixed charts sometimes not resizing properly.
- Fixed memory leak in search history right-click menu.
- Job Viewer
- Initial public release