[BUG]: CI Test stage is hitting the one hour limit

[dagardner-nv]

Version

24.10

Which installation method(s) does this occur on?

Docker

Describe the bug.

We're running into CI issues with our tests where we're hitting the 1 hour limit.
Initializing the containers takes ~15 minutes, leaving ~45 minutes for tests.
Test time looks like:

• Update conda env 6 minutes
• Build 2 minutes
• Downloading lfs files <1 min
• C++ tests 40s
• Python tests > 35 min

Minimum reproducible example

No response

Relevant log output

Click here to see error details

[Paste the error here, it will be hidden by default]

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

Slowest test by module:

Runtimes by class:
tests.morpheus.stages.test_kafka_source_stage_pipe: 220.6 seconds
tests.morpheus_llm.services.test_milvus_vector_db_service: 181.3 seconds
tests.morpheus.common.test_http_server: 131.1 seconds
tests.morpheus_llm.llm.test_rag_standalone_pipe: 67.0 seconds
tests.morpheus.stages.test_multi_processing_stage: 38.8 seconds
tests.morpheus.apps.test_abp_kafka: 31.5 seconds
tests.morpheus.apps.test_phishing_kafka: 30.6 seconds
tests.morpheus.stages.test_filter_detections_stage_pipe: 28.6 seconds
tests.morpheus_llm.stages.test_milvus_write_to_vector_db_stage_pipe: 19.3 seconds
tests.morpheus.utils.test_shared_process_pool: 18.4 seconds
tests.morpheus.apps.test_sid_kafka: 17.2 seconds
tests.morpheus.pipeline.test_pipe_viz: 16.7 seconds
tests.morpheus.stages.test_write_to_kafka_stage_pipe: 16.0 seconds
tests.morpheus_dfp.test_dfp_kafka: 15.1 seconds
tests.morpheus_llm.llm.nodes.test_llm_retriever_node_pipe: 13.7 seconds
tests.morpheus.pipeline.test_file_in_out: 13.4 seconds
tests.morpheus.stages.test_http_server_sink_stage: 9.2 seconds
tests.morpheus.dfencoder.test_dfencoder_distributed_e2e: 8.6 seconds
tests.morpheus.stages.test_filter_detections_stage_column_pipe: 7.9 seconds
tests.morpheus.stages.test_add_scores_stage_pipe: 6.6 seconds

Script to extract test times from the junit xml files:

from collections import defaultdict
from xml.etree import ElementTree as ET

tree = ET.parse('report_pytest.xml')
root = tree.getroot()

times_by_class = defaultdict(float)

for testcase in root.iter('testcase'):
    classname = testcase.attrib['classname']
    runtime = float(testcase.attrib['time'])
    times_by_class[classname] += runtime

runtimes = sorted(times_by_class.items(), key=lambda x: x[1], reverse=True)
print("Runtimes by class:")
for runtime in runtimes:
    if runtime[1] > 5:
        print(f"{runtime[0]}: {round(runtime[1], 1)} seconds")