Feature/bssh icav2 fastq copy manager

[alexiswl]

Summary

The bssh manager runs performs the following logic:

Reads the bssh_output.json file from the BCLConvert run output folder

Collects the basespace run id from the bssh output json

Collects the run id from the run info xml

Collects the fastq list csv file from the BCLConvert run output folder

Read and return the fastq list rows as a list of dictionaries (gzip compressed / b64 encoded)

Collects the samplesheet path from the BCLConvert run output folder (as we might need this to go into the cttso directories)

Read and return the samplesheet as a dictionary (via the v2-samplesheet-maker package) (gzip compressed / b64 encoded)

Return a manifest of icav2 uris where each key represents a ICAv2 data uri (file) and the value is a list of icav2 uris where each uri is a destination for a given file.

Dependent PRs

• Added feature icav2 copy batch state machine #143

TODO List

- [ ] Implement stateful service to track fastq copy For another day

• Implement event service to raise event when all data has been copied over

[victorskl]

LGTM, too.!

[victorskl]

Put this branch into draft mode until we finalise

[victorskl]

reviewing.. but we merge this after #254 fyi

[alexiswl]

Yes will also need to update the snake_case attributes to camelCase

[victorskl]

LGTM all in all. Minor comments. Let us see this first-cut prototype work.

I skip TypeScript case things as we discussed in #186 convo for refactor PR, later.

[victorskl]

(No blocker / no action need for this PR. Just something to tinker down track.)

• I think, this makes dependency on ICAv2CopyBatchUtilityStack (deployment aspect) and, service that stack creates (app aspect) - this is okay, expected.
• Wonder, whether we could do CloudMap service discovery with Step Function as a Step and, handle when/what-if upstream service go down/no-avail scenario.
• This would also defer deployment stack dependency to application service dependency at runtime.

See discussion and reference pointers:
https://umccr.slack.com/archives/C03ABJTSN7J/p1714381660323079

[victorskl]

Alternatively.

We should send manifest_b64gz back into MainBus channel; let ICAv2CopyBatchUtility service pick it up and, do its job.

That'd create an event schema, say, ICAv2CopyBatchUtilityRequest for example.

In this case, we just have to watch out on EventBridge event message size and number of characters limit.

• https://www.google.com/search?q=eventbridge+message+size+limit
• https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_Target.html
• https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html

[alexiswl]

I think, this makes dependency on ICAv2CopyBatchUtilityStack (deployment aspect) and, service that stack creates (app aspect) - this is okay, expected.

Just using the name as defined in the constants, rather than relying on an SSM parameter path, but yes the lookup based on the state machine name will fail if it doesn't exist. Not sure if this is a bad thing given the dependency.

We should send manifest_b64gz back into MainBus channel; let ICAv2CopyBatchUtility service pick it up and, do its job.

Hmm, depends how far down the events rabbit hole we want to go. This might work in this case because the bssh fastq copy manager's last step is calling the copy batch utility. Completion of the copy batch utility could then raise an internal event that lets bssh fastq copy know it can now raise its own completion event.

However, we use this same service for the cttsov2 pipeline. It takes about 3 seconds for the tso fastqs to transfer to their new directory, the analysis is then started after the fastq transfers are complete. If we went with an event based structure for this, the cttso step function would need to submit the event, pause the workflow and then resume for the analysis submission once it hears back from the copy batch utility service. I get the desire to go with the event-based architecture, but I don't think it completely decouples things.

In hindsight, the icav2 copy batch utility service could probably have been part of the infrastructure repository and a static service that can be called on, like the credentials rotation stack

[reisingerf]

We learn...

I can throw another option into the mix: create your own (internal) ICAv2CopyBatchUtility exclusively for this stack's purpose. No dependency on existing/external stacks.

[victorskl]

😭

[reisingerf]

HeHe, not saying we should refactor. Just keeping in mind that not all duplication is bad... it can have it's advantages.

[victorskl]

Fair a bit of logic here and some from utils package. We should test this when we have more time a bit, down track. I can help with this. No action for now; early day.

[reisingerf]

LGTM

[reisingerf]

We learn...

I can throw another option into the mix: create your own (internal) ICAv2CopyBatchUtility exclusively for this stack's purpose. No dependency on existing/external stacks.

[alexiswl]

Can we have one last review of this before there's another refactor on the main branch pls?

[victorskl]

sure, going through...

[victorskl]

Looking good, Alexis. Merge as you see fit. Thanks.

[alexiswl]

Thanks! One last synth check