Refactoring of ETL/DbEntity into ETL/DbModel #138
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
smgallo
added
enhancement
plessbd
reviewed
May 9, 2017
| @@ -1,3 +1,5 @@ | |||
|
|
|||
There was a problem hiding this comment.
Seems like some extra lines ended up here you dont need.
plessbd
suggested changes
May 12, 2017
plessbd
approved these changes
May 15, 2017
|
This enhancement breaks existing tests that are found in open_xdmod/modules/xdmod/tests/lib/ETL. These tests should be replaced or fixed. |
|
@jsperhac Did you run |
|
Ah, sounds like my repo scripts are not accounting for the test artifacts. Let me try... |
chakrabortyr
pushed a commit
to chakrabortyr/xdmod
that referenced
this pull request
Oct 17, 2017
* Added tests for DbEntity\Table * Moved namespace ETL\DbEntity to ETL\DbModel * Refactoring of ETL/DbModel to remove old code and support upcoming features * Updated commit for xdmod-test-artifacts * Fix record formula verification and saving of overseer restriction value * Add @plessbd blacklist filters * Ignore case where temp directory already exists for tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors the existing ETL/DbEntity code into a new namespace (ETL/DbModel) and also removes cruft that was not needed and makes it easier to extend. This work is in preparation for supporting improvements such as better subquery support, programmatic changes to queries and tables during ETL, and renaming columns as opposed to dropping/adding them.
See also PR ubccr/xdmod-xsede#32 for additional required changes.
Note that commit
533b2f3clearly shows several files as renamed and0969e8fshows many of the changes to those files but the code shown at the end of the PR shows them deleted and recreated. I'm not sure why this happens.Description
The
ETL\DbEntitycode was largely untouched since inception 1.5 years ago and needed to be updated to support desirable features such as subqueries, change column, and improved programmatic access and modification of tables and queries during ETL (for batch aggregation speed improvements, for example).The newly refactored
ETL/DbModelcode has been updated to continue to support a lightweight representation of database queries and tables.The following changes have been made:
ETL/DBModelprotectedproperties array inDbEntity.phpand accessed via__get(),__set(), and__isset(). This means that classes no longer need to implement getters in most cases and only need to extend__set()for complex data types such as arrays of objects where a particular type of object must be created.stdClassobject that can then be fed back into itself as the configuration object. This allows for programmatic modification of an object on the fly during ETL.Table::discover()method no longer needed to be static.DbEntityand classes that extend this class now only implement the definition, verification, and setting of the properties that they define.stdClassobjectsstdClassrepresentation, and feeding this back to ensure the same statement is generatedMotivation and Context
General code cleanup and preparation for new features. Also added tests.
Tests performed
Component tests were run (
PHPUnit) and theresource-allocationsandxdcdb-jobspipelines were run with their data being compared to the a baseline from v6.6 data. Baseline commands are shown here for reference and were the same options as branch 6.7 executions.Test
resource-allocationsandxdcdb-jobspipelinesVerify data in generated tables. Note that the default values for
submission_venue_id,job_record_type_id, andjob_task_type_idhave changed between 6.6 and 6.7 so these columns are ignored in the jobs data (we do not currently populate them other than the default).Types of changes
Checklist: