Skip to content

Commit

Permalink
Centos8 changes (#321)
Browse files Browse the repository at this point in the history 
* Fixing php7.2 pass-by-ref changes

In PHP7.2 only variables can be passed by reference as opposed to the result of
a function.

* Adding Initial Centos8 & RockyLinux Dockerfiles

* Additional Rocky8 updates

* Re-adding Centos7 specific requires

* Fixing variable reference

* removing unnecessary assignment by reference

* Removing unnecessary assignment by reference

* Appeasing the style gods

* Fixing a previous refactor

These should have referenced `data` not `stats` as `stats` is an object.

* StopGap so that I can continue with migration effort

Added a function that will ultimately end up in `TestHarness\Utilities` so that
I can continue work on getting supremm to work on Rocky8.

* blah

* Adding centos/rocky8 executors to circleci

* fixing fat fingered image name

* lsdjkf

* asdkfj

* lksjdflkd

* slkdjf

* updating composer.json

* Adding mongodb deps for xdmod

* asdlkfj

* Updating spec file

* Updating circleci config

* fixing circleci validation errors

* blah

* blaskdjf

* lskjdfds

* lskjdfds

* slkdjf

* slkjfd

* slkdjf

* slkdjf

* lskdjf

* sldkjf

* sldkjf

* sldkjf

* updating el7 requires

* updating el7 requires

* Updating required version of XDMoD

* Updating version of XDMoD

* Differentiating running the UI tests on rocky8 vs centos7

* fixing circleci validation errors

* Installing UI test dependencies for CentOS/Rocky8

* Adding mongo shim layer and updating php versions

- Added a small shim layer to SupremmDBInterface that will detect
  which version of mongo driver is installed and use the correct classes
  / functions. This should be removed once we move solely to PHP7.2+.
- Updated the php versions that supremm can be installed w/ to include
  both 5.4 & 7.2.

* Changing up where php versions are required

* Appeasing the style gods

* Updating to reflect changes in composer.json

* Update xdmod-supremm.spec.in

Co-authored-by: Jeffrey T. Palmer <[email protected]>

* Making the phpunit dir explicit

* Removing unneeded step in circleci config

The conditions that required this step are now different and so it can
be removed.

The conditions were that originally xdmod10.0's composer.json file
didn't have the `composer/installers` and
`simplesamlphp/composer-module-installer` in the `allow-plugins` config
so the build would hang as it would interactively prompt to allow these
plugins.

* Updating CircleCI's config

* YAML syntax is hard

* Naming things is hard apparently

* Appeasing style gods

* le sigh

* Adding some regex slashify-ing per @jtpalmer

@jtpalmer caught that the regex being passed to the old MongoRegex class
was not valid, as it had been updated to what is required for the new
Mongo\BSON\Regex class, and throwing exceptions which was causing the
integration tests to fail.

This commit adds a helper function that will
add a trailing / leading slash if one is missing from the regex to make
it compliant with what \MongoRegex expects.

* Update classes/DataWarehouse/Query/SUPREMM/SupremmDbInterface.php

Saying goodbye to my beloved `slashifyRegex` function.

Co-authored-by: Jeffrey T. Palmer <[email protected]>

* Updates per @jtpalmer code review

Just some minor cleanups to the Dockerfiles & the spec file.

* Removing unnecessary comment

* Updating authors of base images for consistency

* Updating authors of base images for consistency

* Re-Adding Centos8 build

* Re-removing centos8 build until the docker image is pushed

* Re-adding Centos8 build

* Really re-adding the Centos8 build

Co-authored-by: Jeffrey T. Palmer <[email protected]>
  • Loading branch information
@ryanrath @jtpalmer
ryanrath and jtpalmer authored Jun 23, 2022
1 parent c6b2fbf commit 330c2f7
Show file tree
Hide file tree
Showing 11 changed files with 392 additions and 76 deletions.
105 changes: 82 additions & 23 deletions .circleci/config.yml
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
# PHP CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-php/ for more details
#
version: 2.1
executors:
centos7: &centos7-executor
docker:
- image: tools-ext-01.ccr.xdmod.org/xdmod-job_performance-9.5.0:centos7.9-0.1
centos8: &centos8-executor
docker:
- image: tools-ext-01.ccr.xdmod.org/xdmod-job_performance-10.0.0:centos8.4-0.1
rocky8: &rocky8-executor
docker:
- image: tools-ext-01.ccr.xdmod.org/xdmod-job_performance-10.0.0:rockylinux8-0.1
jobs:
build:
parameters:
os:
type: executor
install-type:
type: string
docker:
# The first image entry here will be used as the image for the parent container.
- image: tools-ext-01.ccr.xdmod.org/xdmod-job_performance-9.5.0:centos7.9-0.1
executor: << parameters.os >>
environment:
COMPOSER_ALLOW_SUPERUSER: 1
XDMOD_REALMS: 'jobs,storage,cloud,supremm,jobefficiency'
Expand All @@ -30,18 +36,56 @@ jobs:
name: Link the SUPREMM module into the XDMoD Source Directory
command: |
ln -s `pwd` $XDMOD_SRC_DIR/open_xdmod/modules/supremm
- when:
condition:
or:
- equal: [ *rocky8-executor, << parameters.os >> ]
- equal: [ *centos8-executor, << parameters.os >> ]
steps:
- run:
name: Remove composer lock files
command: rm -f "$XDMOD_SRC_DIR/composer.lock" && rm -f composer.lock
- run:
name: Update XDMoD's composer config.platform
command: |
sed -i 's/"php": "5.4"/"php": "7.2"/' $XDMOD_SRC_DIR/composer.json
- run:
name: Move composer.json so we can modify it
command: |
mv "${XDMOD_SRC_DIR}/composer.json" "${XDMOD_SRC_DIR}/composer.json.orig"
- run:
name: Update XDMoD's config to trust addons
command: |
jq '.config += {"allow-plugins": {"composer/installers": true, "simplesamlphp/composer-module-installer": true}}' "$XDMOD_SRC_DIR/composer.json.orig" > "$XDMOD_SRC_DIR/composer.json"
- run:
name: Add mongodb/mongodb to XDMoD via composer
command: composer require -d "$XDMOD_SRC_DIR" mongodb/mongodb
- run:
name: Install XDMoD UI Test Dependencices
command: |
pushd "$XDMOD_SRC_DIR/tests/ui" || exit
npm install
popd || exit
# We need to update our acct before we can enable docker layer caching.
#- setup_remote_docker:
# docker_layer_caching: true
# Download and cache dependencies
- run:
name: temporarily toast composer.lock
command: rm -f composer.lock
- run:
name: Install SUPREMM Dependencies
command: |
composer install --no-progress
- run:
name: Install SUPREMM RPM build dependencies
command: |
./tests/ci/scripts/install_dependencies.sh
- when:
condition:
equal: [ *centos7-executor, << parameters.os >> ]
steps:
- run:
name: Install SUPREMM RPM build dependencies
command: |
./tests/ci/scripts/install_dependencies.sh
- run:
name: Install XDMoD Dependencies
command: |
Expand All @@ -50,8 +94,8 @@ jobs:
- run:
name: Create Test Artifact Directories
command: |
mkdir ~/phpunit
mkdir /tmp/screenshots
mkdir -p /tmp/phpunit
mkdir -p /tmp/screenshots
- run:
name: Setup Environment Variables
command: |
Expand All @@ -60,28 +104,24 @@ jobs:
- run:
name: Build RPMS
command: |
pwd
pushd $XDMOD_SRC_DIR
pwd
~/bin/buildrpm xdmod supremm
popd
pwd
- run:
name: Install / Upgrade XDMoD from RPM
command: |
echo `pwd`
./tests/integration_tests/scripts/bootstrap.sh
- run:
name: Validate the newly installed / Upgraded XDMoD
command: ./tests/integration_tests/scripts/validate.sh
- run:
name: Make sure that the Composer Test Depedencies are installed
name: Make sure that the Composer Test Dependencies are installed
command: composer install -d $XDMOD_SRC_DIR --no-progress
- run:
name: Setup & Run QA Tests
command: ./tests/ci/scripts/qa-test-setup.sh
- run:
name: Make sure that the Composer Test Depedencies are installed
name: Make sure that the Composer Test Dependencies are installed
command: composer install -d $XDMOD_SRC_DIR --no-progress
- run:
name: Updating XDMOD_INSTALL_DIR env variable
Expand All @@ -92,9 +132,22 @@ jobs:
- run:
name: Run Integration Tests
command: ./tests/integration_tests/runtests.sh --log-junit ~/phpunit
- run:
name: Run UI Tests
command: scl enable rh-nodejs6 "$XDMOD_SRC_DIR/tests/ui/runtests.sh --headless --log-junit ~/phpunit"
- when:
condition:
equal: [ *centos7-executor, << parameters.os >> ]
steps:
- run:
name: Run UI Tests
command: scl enable rh-nodejs6 "$XDMOD_SRC_DIR/tests/ui/runtests.sh --headless --log-junit ~/phpunit"
- when:
condition:
or:
- equal: [ *rocky8-executor, << parameters.os >> ]
- equal: [ *centos8-executor, << parameters.os >> ]
steps:
- run:
name: Run UI Tests
command: "$XDMOD_SRC_DIR/tests/ui/runtests.sh --headless --log-junit ~/phpunit"
- run:
name: Run Regression Tests
command: ./tests/regression_tests/runtests.sh --log-junit ~/phpunit
Expand All @@ -103,12 +156,18 @@ jobs:
- store_artifacts:
path: /var/log/xdmod
- store_test_results:
path: ~/phpunit
path: /tmp/phpunit

workflows:
full-build:
jobs:
- build:
matrix:
parameters:
os: [centos7, rocky8, centos8]
install-type: ["fresh_install", "upgrade"]
exclude:
- os: rocky8
install-type: "upgrade"
- os: centos8
install-type: "upgrade"
28 changes: 12 additions & 16 deletions classes/DataWarehouse/Query/SUPREMM/JobMetadata.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,9 @@ public function getJobSummary(XDUser $user, $jobid)
}
$jobdata = $this->getjobdata($job['resource_id'], $job['local_job_id'], $job['end_time_ts']);
$jversion = isset($jobdata['summary_version']) ? $jobdata['summary_version'] : $jobdata['summarization']['version'];

$schema = $this->getsummaryschema($job['resource_id'], $jversion);
if ($schema !== null) {
return $this->arrayMergeRecursiveWildcard($jobdata, $schema['definitions']);
return $this->arrayMergeRecursiveWildcard(json_decode(json_encode($jobdata), true), json_decode(json_encode($schema['definitions']), true));
} else {
return $jobdata;
}
Expand Down Expand Up @@ -321,9 +320,8 @@ private function lookupJob($user, $jobid)

private function getsummaryschema($resource_id, $summary_version)
{

$resconf =& $this->supremmDbInterface->getResourceConfig($resource_id);

$resourceConfig = $this->supremmDbInterface->getResourceConfig($resource_id);
$resconf = $resourceConfig;
if ($resconf === null) {
return null;
}
Expand All @@ -333,15 +331,15 @@ private function getsummaryschema($resource_id, $summary_version)

private function gettimeseries($resource_id, $jobid, $end_time_ts, $filter = null)
{

$resconf =& $this->supremmDbInterface->getResourceConfig($resource_id);
$resourceConfig = $this->supremmDbInterface->getResourceConfig($resource_id);
$resconf = $resourceConfig;

if ($resconf === null) {
return null;
}

$collection = $resconf['handle']->selectCollection('timeseries-'.$resconf['collection']);
$query = array( "_id" => new \MongoRegex("/^$jobid-.*$end_time_ts/") );
$timeseriesCollection = 'timeseries-'.$resconf['collection'];
$collection = $this->supremmDbInterface->getCollection($resconf['handle'], $timeseriesCollection);
$query = array( "_id" => $this->supremmDbInterface->getRegex("^$jobid-.*$end_time_ts"));

if ($filter === null) {
$doc = $collection->findOne($query);
Expand Down Expand Up @@ -376,19 +374,17 @@ private function getrawdata($resource_id, $jobid, $end_time_ts, $metric, $nodeid

private function getjobdata($resource_id, $jobid, $end_time_ts)
{

$resconf =& $this->supremmDbInterface->getResourceConfig($resource_id);
$resourceConfig = $this->supremmDbInterface->getResourceConfig($resource_id);
$resconf = $resourceConfig;

if ($resconf === null) {
return null;
}
$collection = $this->supremmDbInterface->getCollection($resconf['handle'], $resconf['collection']);

$collection = $resconf['handle']->selectCollection($resconf['collection']);

$query = array( "_id" => new \MongoRegex("/^$jobid-.*$end_time_ts/") );
$query = array( "_id" => $this->supremmDbInterface->getRegex("^$jobid-.*$end_time_ts"));

$res = $collection->findOne($query);

if ($res !== null) {
ksort($res);
}
Expand Down
Loading

0 comments on commit 330c2f7

Please sign in to comment.