Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lower Impact KBs have on Database #2020

Open
originalname51 opened this issue Feb 24, 2021 · 0 comments
Open

Lower Impact KBs have on Database #2020

originalname51 opened this issue Feb 24, 2021 · 0 comments

Comments

@originalname51
Copy link
Contributor

As a database administer I want to have smaller queries requested if possible.

Please check to see if saving the binary KB and rule drl to the database.

originalname51 added a commit that referenced this issue Mar 12, 2021
The KB from the database is not required to be saved to function.

This will save a large data-call to the database as the rule blob is
saved as a large text file.

#2020
originalname51 added a commit that referenced this issue Mar 12, 2021
originalname51 added a commit that referenced this issue May 14, 2021
* Update the rule engine to never save the KB blob to the database.

The KB from the database is not required to be saved to function.

This will save a large data-call to the database as the rule blob is
saved as a large text file.

#2020

* Slight cleanup for #2020

* Update rule runner for modular process and read/write separation.

** Update knowledge bases to be broken into sets of 10,000
    * Add a way to re-balance rules. When a rule or watchlist item is
added the rules will re-balance. This will page through all rule and
watchlist items and assign them to knowledge bases.
** Update rule runner to accept multiple KBs of watchlist or UDRs
(instead of just one of each)
** Break rule runner scheduler into several stepped parts to separate
out fact gathering, kb processes, hit detail generation, and hit detail
persistence.
    * Separate out updated rule runner parts to only read from the
database with no modifications.
** Update KB generation to not hook in individual rules to the KB, but
instead use the UDR

* Update rule runner to go horizontal.

These code changes are to assist in horizontal drools rules. Instead of
using vertical scaling threads the rule engine for drools switch over to
a queue based system.

Hits are also always persisted in an async manner.

This allows the rule engine to process messages on a queue instead of a
scheduled.

Some  infrastructure changes were also done in order to support the new
rule runing application

* Add the gtas-rule-runner to the GTAS project.

Break out rule runner from the rule engine. This changes from a
scheduled task to a queue based system. There are no write executions on
the gtas-rule-runner which allow for scaling horizontally and
vertically.

* Docker updates:

Update ETL job dockerfile reference (was older reference which was
preventing builds

Add local properties for rule runner

Remove HTTP-PROXY as it has been superceded by the gtas-ui.

* Update local deployment

*to reference gtas-rule-runner instead of rule-runner.

* Update ConditionalOnProperty for Notificaiton Services

To use conditional on property to use email.hit.notification instead of
the manual enable.email.notification", name = "service was incorrectly
erroring when running the rules due to not instantiating the
HitEmailNotificationService.

* Upload loader jar to rule-runner

* Upload gtas-information-share jar

* Update gtas-commons

* Update gtas-parsers

* Move GTAS-Rule-Runner to gtas-parent

Gtas-rule-runner was very tightly coupled with gtas-parent and so it
made the most sense to add it as a module instead of a separate project.

* update docker compose and local docker compose

* Add gtas-information-share

* Updates to Rule Runner

Updated rule runner to return a list of hit details instead of raw
results that would need to be converted to hit details.

Updated the fuzzy matcher to return a list of hit details instead of
persisting the list. This tightens the logic on what happens on the
final step of the queue.

Updated hit detail's hitmakerid to not be json ignored. This allows the
hit maker id to be used by the rule engine.

Updated notification service instantiation to be agnostic to whether the
email service was toggled on or off.

* Add scripts for db change.

* Fix test

* Updates for deployment

Control memory
control dev deployment
control local deployment

* Update main.yml

* Minor adjustments and addendums to docker related stuff

* memory

Co-authored-by: simbam1 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant