Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add env var feature flag to enable queueing library #1004

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+149 −98
Open

Add env var feature flag to enable queueing library #1004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bf574e5
Add env var feature flag to enable queueing library
carlpartridge Oct 16, 2024
130b32d
Quick adjustments
carlpartridge Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions bcda/api/requests.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -632,6 +632,7 @@ func (h *Handler) bulkRequest(w http.ResponseWriter, r *http.Request, reqType se
sinceParam := (!rp.Since.IsZero() || conditions.ReqType == service.RetrieveNewBeneHistData)
jobPriority := h.Svc.GetJobPriority(conditions.CMSID, j.ResourceType, sinceParam) // first argument is the CMS ID, not the ACO uuid

logger.Infof("Adding jobs using %T", h.Enq)
Copy link
Contributor Author

@carlpartridge carlpartridge Oct 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sanity check logging that we are are using the proper queueing library. I dont totally understand why we seem to have different loggers and different access to them?

if err = h.Enq.AddJob(*j, int(jobPriority)); err != nil {
logger.Error(err)
h.RespWriter.Exception(r.Context(), w, http.StatusInternalServerError, responseutils.InternalErr, "")
Expand Down
19 changes: 18 additions & 1 deletion bcdaworker/queueing/enqueue.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (

"github.com/CMSgov/bcda-app/bcda/database"
"github.com/CMSgov/bcda-app/bcda/models"
"github.com/CMSgov/bcda-app/conf"
"github.com/bgentry/que-go"
"github.com/ccoveille/go-safecast"
)
Expand All @@ -20,7 +21,13 @@ type Enqueuer interface {
}

func NewEnqueuer() Enqueuer {
return queEnqueuer{que.NewClient(database.QueueConnection)}
if conf.GetEnv("QUEUE_LIBRARY") == "que-go" {
return queEnqueuer{que.NewClient(database.QueueConnection)}
} else if conf.GetEnv("QUEUE_LIBRARY") == "river" {
return riverEnqueuer{}
} else {
panic("NO QUEUE LIBRARY SET")
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if panic-ing is better than just defaulting to a specific library?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're going to a green-blue solution for the libraries I'd just default to que-go.

}

type queEnqueuer struct {
Expand Down Expand Up @@ -67,3 +74,13 @@ func (q queEnqueuer) AddAlrJob(job models.JobAlrEnqueueArgs, priority int) error

return q.Enqueue(j)
}

type riverEnqueuer struct{}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dummy type and interface for now.


func (q riverEnqueuer) AddJob(job models.JobEnqueueArgs, priority int) error {
return nil
}

func (q riverEnqueuer) AddAlrJob(job models.JobAlrEnqueueArgs, priority int) error {
return nil
}
28 changes: 28 additions & 0 deletions bcdaworker/queueing/enqueue_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (

"github.com/CMSgov/bcda-app/bcda/database"
"github.com/CMSgov/bcda-app/bcda/models"
"github.com/CMSgov/bcda-app/conf"
"github.com/huandu/go-sqlbuilder"
_ "github.com/jackc/pgx"
"github.com/pborman/uuid"
Expand Down Expand Up @@ -51,3 +52,30 @@ func TestQueEnqueuer_Integration(t *testing.T) {
_, err := db.Exec(query, args...)
assert.NoError(t, err)
}

func TestNewEnqueuer(t *testing.T) {
origEnqueuer := conf.GetEnv("QUEUE_LIBRARY")

// Test que-go implementation
conf.SetEnv(t, "QUEUE_LIBRARY", "que-go")
enq := NewEnqueuer()
var expectedEnq queEnqueuer
assert.IsType(t, expectedEnq, enq)

// Test river implementation
conf.SetEnv(t, "QUEUE_LIBRARY", "river")
enq = NewEnqueuer()
var expectedRiverEnq riverEnqueuer
assert.IsType(t, expectedRiverEnq, enq)

// If unmatched name: panic
conf.SetEnv(t, "QUEUE_LIBRARY", "bad implementation name")
assert.Panics(t, func() { enq = NewEnqueuer() }, "NO QUEUE LIBRARY SET")

// If unset: panic
conf.UnsetEnv(t, "QUEUE_LIBRARY")
assert.Panics(t, func() { enq = NewEnqueuer() }, "NO QUEUE LIBRARY SET")

// Reset env var
conf.SetEnv(t, "QUEUE_LIBRARY", origEnqueuer)
}
1 change: 1 addition & 0 deletions conf/configs/dev.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ BCDA_WORKER_ERROR_LOG=/var/log/worker/error.log
DEBUG=true
DEPLOYMENT_TARGET=dev
EXPIRED_THRESHOLD_HR=24
QUEUE_LIBRARY=que-go
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These files are the non-sensitive env vars used for each environment?

RUNOUT_CLAIM_THRU_DATE=2023-12-31
SSAS_TIMEOUT_MS=5000
SSAS_USE_TLS=true
Expand Down
1 change: 1 addition & 0 deletions conf/configs/opensbx.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ BCDA_WORKER_ERROR_LOG=/var/log/worker/error.log
DEBUG=false
DEPLOYMENT_TARGET=opensbx
EXPIRED_THRESHOLD_HR=24
QUEUE_LIBRARY=que-go
RUNOUT_CLAIM_THRU_DATE=2023-12-31
SSAS_TIMEOUT_MS=5000
SSAS_USE_TLS=true
Expand Down
1 change: 1 addition & 0 deletions conf/configs/prod.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ BCDA_WORKER_ERROR_LOG=/var/log/worker/error.log
DEBUG=false
DEPLOYMENT_TARGET=prod
EXPIRED_THRESHOLD_HR=24
QUEUE_LIBRARY=que-go
RUNOUT_CLAIM_THRU_DATE=2023-12-31
SSAS_TIMEOUT_MS=5000
SSAS_USE_TLS=true
Expand Down
1 change: 1 addition & 0 deletions conf/configs/test.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ BCDA_WORKER_ERROR_LOG=/var/log/worker/error.log
DEBUG=true
DEPLOYMENT_TARGET=test
EXPIRED_THRESHOLD_HR=24
QUEUE_LIBRARY=que-go
RUNOUT_CLAIM_THRU_DATE=2023-12-31
SSAS_TIMEOUT_MS=5000
SSAS_USE_TLS=true
Expand Down
195 changes: 98 additions & 97 deletions shared_files/encrypted/local.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,98 +1,99 @@
$ANSIBLE_VAULT;1.1;AES256
31623366616135336561313133316264353263633634363334643861386266323764636136363839
3435383136303763623138616331653036663535346337350a303738623865626335393439316165
32666261646539623934306636653935633837653235356434323930303639353738636631346233
3037316334313732640a373766336464373132393965313231656262366434313731313730616533
31303366363835326666313539336233353139656231303636656431383433633439623230303439
37333438636138343863313939353534373835336163643333636334356238376435363836323831
37666634636238303437306563393034373134643762383237373934326439643063316535353230
37623962666166393630623564643866343665643435306534346564396236383063393163363332
37616231323639326565383066656432633934663066316438336266333335643031383866376138
32353963393038346230663936343663343936393339393663396436656136623261336665393862
62383937636630373161363533346263306363613731626166633462386332383832653732626637
30313766333039343438623563383461633736616632626461616562666632343762643732653832
66336264386264373938643162316135623261306439313233653636386266616433633836663834
36376538303836303030666232656132346136656338326262303335646261333562653037313037
34313237613234326166616132313232306265316333666630363837306233633664373164313536
30643534363062306634656666393932636561393963613964626430383637393636336261376564
38363064343566663730343864373331306363386330396165613539313838323934316363646362
37333235336135393964623834383038626162646232646131613363386162303933623337333232
30613036306566656635653330396638626164316433303061363932653537343539653334313863
31626361613964376333303939643663363538356637376331393230333463336330363761373230
66303365626139633462366637626437656431396635323962313433383232396533363538306532
31373835643063356631303035343161626466336631386636396133656238386265653762656337
61313362666537306437366365643066313031396239313535323062343063326364353035653762
31663264646130653633336165656331613636306633313232626562623336633765663939343636
62613238343462636238323033366135356563396631343639343166663635666361613031373661
66336462636663626566316136656436373062326362326363626566616332623863376266653366
34633739343431383538373437323761366139653162663234653836636435383835363661643535
35373831313737663632333435643037656335663537326534336331636236346536386637653966
32366462323263386631616666646461623434343036303631306633636632383966333631376438
31313864643361353665333663323738333031613864613636353139663736343638356632656337
34336530333330633837393633383030623365326235366132333633346465373166343039396333
65336462666438303961393264393238383465356364326261363261633630666138373337313139
35323435333037333935633331663563653133623136613838343137643230343037386332343966
35393730346133303234326135303035386462333738396265623132616439353361643564313734
38643632333863616632386564643035653136313663663330333561663137333634303864346337
31613832656538623361393039376638373435643234383738366439383363663666376630383239
35386632356431356465303037346638333038353464353935646561643864613839323839386533
36306433306532333630613666616636346331306637643734646430636634366130393561346132
31323434646231653534663366396238323234313630333064623834336437313861313636366363
65323265363636306634346436613065336639383362313364313736313238616434353263356466
32393464306234363064656435316239343331316130373238656162336364356266326265633361
63333037333637323332306261636462313734373062653538656630393837653837613963616632
61343236316639326432636265396563376337343235336439343661373031303331336165373439
32323763396265356462623438323333333266653634393935353733373666643637346436303664
39646463623936356430303961626131393363636263663133623534323139383939636363313565
31386130353662386339656535393730393564356530356361323730663366396366643730393835
62303861343964363035366538623139393735396562356465333130306465306639346231653432
39623161323831623239613433396264663431306134346637353137613831393536346164333032
39363137333865663762326634656363336138323662623035316262393031336462346164376635
33396338313335343139393635386436333465373030386663386463353736373465393061636236
32616534346166326363646337393035393530323066656338333964656237666462393263346364
32316162383763363030376634643835643338333966323230346232363230326338626364376463
39343238343438373565323133653838333266653534633966343135663061353733333064633437
62326162316636383637626164393133303764643637386336353734626561383432336266356531
32316263356238356632626338386162356663333561333938343839643732313039626665623766
38323735323339303535616138623732376536366166356165623535643432393662616463353130
65646633316533383366326362613934346262326165386239376264656365306637303237356537
39366138383736353364393435616637326635663664383462386162653461623464336365306438
30626232383461336464653830366565663463643562616263323935323635393037653938613863
33326131663736386239633763653963663064323965633564663737386635343630303536633163
36303932376464346266626565613939386636343566366432393664373330373465356337633537
38353135353939633438653464656361363164343835353639336662616363636134356462393333
38636162303662393061633461366437393161363966396434653236313232616266633434386334
63376430666666346335376661383831346565633862303936386536656137303734666436343130
34653262613936643533353530303066316235336563323339653932623063393861323630373465
65303736663065336163306634643933616663313631313937306532623737366531393139666637
37383532353832643561323433616366633539323664383634366638343630376438616131623434
38633739636430353432663431613832396462363836323566653537636334353966323135306666
37313431383264663536333934643734323433373939653037643464386564393634303231396338
30336334336265373138616664623961636639376235373330303766663665613365636465393533
63623163353533393363373932316138353964613265306136376433376562376664386430663034
63643537643565353662383265663364633135323735663338333735396638303163373339633365
39376234333461396134653661613466356134323439326136386562313166366434376561633935
65356633343336383638316331633863346639383039373437346366303633306565393862383734
37366261346430623235336336633137653464316361376361623439306139633861613463373237
63383738326566643930636237656331353035333265333236363464323637313635313063656133
61326464626636623162393339363264643235653137663538316461306665313163353964353438
61346261336366343461376263613232653061363736363037656562613931663739643062613237
35623137333237323731643061323739656333316162363664366439363364643166363833636666
64636331336135616330383638666330633066616163663430623432303837313635636636646161
34396339623463383833623930346662343761376162356637376131336238616663663464333564
36363066666134666230383832636535386432643166613032613334323937623164626632373261
62376135363436306532303533373134303835353638653762656535353466336639303634623231
31313035316439386264626438383062613664356264643565346638353936373036393235653939
65373035633030363133303232363031333331373739373734323539323562626366343561643936
38663136353835663463383233656264663438306564653430663835643565353764386464313736
63643938383332613462336532646533326262333537353733333931393661386539666632663130
30306137633363353965353165383366613436666664613734343234306133633335316363386439
64363735613962353638336134373837343830623132376161383334353364336237653435626537
65346436326364643637323763623130393731326630646263643138626161346436326131366436
36623438646633336131356130646536316333646663316535396666616639396131393835346435
36616463393065396230663161326364373165316236623335656165316539356664626131383261
30646536643539313430303034646139316234323064316463383830346166303061646362663063
66613765636163353438666264363931663838303534623131626434346436613062656639373266
37663531333036303334373633353365373361626363313530666237643131393938393335643233
38343331313630616633333434613733393732643039306532633233643931353363613563646465
6637
64393162666637316433323462366335663639363038333430626131343832343538333434313437
6632313234343362623434663938386366376136636261630a346364393535323134626632626234
34616264616335646339376665643063353930643331643232353231373136396637353434323732
3132396561323235660a353137383063316661306633373162393461346636383965616135663364
36643039643065636265646234356563316137616634663038653830323532376134326363363632
37656130646332386533376139313066303636373530623663616563343134623737376462333437
36376462333737656563656134343638623433383566666637383438326565366366653463333237
61653635663738393539303762616632333839616361393264353962656431366538616264613633
64373334626362366636366464393433313133386130393961323035303633316434393838623063
38623138633836383637613962363962626365656465343938643136383831633038306338656663
63353439666138313733343136383934616333613964353465323661366138643537643731313637
35373336643266373434666263666134636331313761646530343561386537653939316434393064
35393930373062376561653363646638396631386461643838616435623332643462323535363539
33643763316536343733623631353230653231656430366166336134613364646638313434386134
66393164326639626635313133653234346131333539636532373535656538363033333337633434
39363961333365613133646265316231373732356162383232383333383265373562356666336435
38616431313432313231373935383339643666373165326238643534323332313964373436623062
32643765396662303334346631376532656535323533343134323966303962623037663937313061
62656563663464386364393064343139336638323032303930303664393639356434356166333231
35316337613261623861613939366432616536326437376663373139326264356338636330313062
37316333313431623666306339333933626631656530646437663637613737666132663762396165
64376531306662303965623135636134346265306533376330303735303566373936363266363234
34366334663564363962343261383733373339326337626464386461623730336334363030653639
63616232313337636438313739303761326130646337626162366137633438353039633565346362
34656365633935383933343635316339316261376330386332363266653737353437643661333961
64613736656337363937306662363638356435363564393835366334323633363636313365363661
30633564306430366662393866366532313531353061663661663033616530643963333539643239
31346564373635613839383535333032633630623061383736346435303534343366313933646130
31653239373733656239326238343361633761633265393931643732336539313762316135653163
32366339646535383934663663363733653838623931663861626565386238353839323633323966
31623633353231643739366131393437613562353063386538666166636636633730316164326330
64646334623236323133326436663237303564356239636162303365633666393061353166663436
66316534616566636563396330343762623236333761613632666238316535376330346633633738
33343963643466333134383166656265303565656539663165396438313163316433316533353634
32626536376338343335303763663335346431653937663233386338396433646538613266333466
64653061393934316165336662626535643339656161396135393666316666613764623835616661
33653433363266316235643138346364316231633462393965333037313365366263353762623537
34626563323363306235346230336363326633383331353531343339616664646231646465633735
35613437366233643334316332303630653865363262343034626639633533626561613938623332
34646565303362303432376636633638666638303936663463643066306433636630613232373861
31326431656536393438383463666361306362373333393365366331326162323636633331653437
62393132306238303037393763336562353335613564343562656139343438656666356563663230
62333736616332626331613936653761663737323039363330373731666534346166376566663936
37326535656262653033383638626630333933356137613863336632633938373431393262656264
34623331376433303331636561363064373532636566356335373034376532663664373836366232
30353336656339386366666537633635666332386263626532346234633335383338303635373934
66323061623033303933373234623362323432656166343865633933313034613863373039376634
39386333663965653838363734393839323032306366333962353163303033633336373265646432
37346538356164323935343430366635333530373962353733383034303231383963643765346432
34393433623330323533303865306364303935313233343533366566383531376665656431623162
62636134383733366331386537373837626232663865333864623333346232653239363335363763
37323764643662303430373261306531353436363834653166386335363038656637323630646436
31363730613961633763316365653566353963383862353336373863383132393663333036373963
61626638643961666238643765646239356238373036633165313032346663616437326462303936
66323136363636373362316561383435326365373938306533643530313631333762626631353262
30306137626637633131353430623062326666383531313539323164343539393961383564356534
35633236666236663361636434313962373863363733646531633533383039386233336364386530
35393030623535323662333864323166303461366166646163333039386164323638653134316233
35653966313861393763383833326335613433353937303364616137646635663737373561636539
34333562393539303765366134343936363964353261343636393634646162326635616138373361
34613033626430663834613935326535373261396530303165663539396135633430393562333238
30623435653733613464313738323531366233323662363537376432623535326166643537613138
39643030376134623832613565393035303631353036613032356137613732343034326462643964
32393433626632303966393734663235353662353232636238653264303539313934323837346338
35383434616136376137363536396630353236313739623261363630616230333233366435326534
36326163376533316363363332386231626437386264336262626561623432326164376434653566
62303237316363313838326532303063646362373664663363333433316562376534356138366232
38353163663937333465633061313564316338643338656331636361653066613862653566633135
63636630656564646634333331383861383534643035353430643564663064633963363335353561
34316264373765376539383663383861303336666433663137643732333531656231353930346632
63636230623238646334386163643631393830393761613537633862636361383264626331326536
66626332656433653361646434656330386638333635643534666334353135383536373562313934
61303530323737613461653531613033346231646238616337633636383436333166363732396437
35326632303062323165303338373965363564623764343539656239613437353030393535336565
30376465373630356339383632363863643337313537336136363965646238303765323461646535
61626131303830643037636431366536353133353563383961313866393966616663323330653132
31653561303738303233386433303835623966316534386133373139353531613835653466383231
30336238666163393532326165323939626565353635396130316631616636326630306638616630
30633064323339386166616135313430333036393837383961653438613061363431613866336435
37346436333139623636383630383930343862316663636337383861303636393331613537613932
32363663643430346161666266613632326639363537353535623530323334653031353837633965
34303539303837356261363833393836663237383230613131343433363530643136323262333363
34303537373362396336653937333866323065663930363564356639393264363261336337343261
35303136353039323335346534373537333134666439616332333936303537303135623231653564
65306239633734353235313062623463343461303733313564353132646538663537373034303062
33623164396538666166373763666133623230393532363762303163333631353134656238616637
65393836306538396663346366306233393733336136396134636166363164366438366130633339
38656233336331353030303930653462633630633166373132303064633635313433616365653238
38363032303031653538623062333463366532343061366665316432663234613935373438633832
62636163303235333362303235373539623438333763383338333030333233653866326139633662
32366236613165663439383666636363386430366332396663386663633063303462386164636231
37313332346137393061373430316639313934336336326437356337333162356165646532653662
38353863363937396637303035653832363837366237323666636462326261623632643534376565
36376465643562386337653539353135396537336432343035366564613939373837363835623430
30356634646339613332393230616230386364616137366462396465653462643532653633306438
39613734626661356232613833643137616163373635396632326137326566336133653061613433
62353663396530653836623834343165656262326230366630353234663564613132343661383739
3164653465646439343538626362316662626332333466343930
Loading