BFD-3331: Represents Data Dictionary in BFD Model along FHIR Resources & pulls in descriptions from codebook

[meliGuzman]

JIRA Ticket:
BFD-3331

User Story or Bug Summary:

This draft PR shows the prototype for the BFD-Metamodel.

---

What Does This PR Do?

Note: the PR is large mainly because of the json files that I moved to a different location.
Note: disregard the code comments / naming. Just worked fast to get through checkstyle so i could test

• Created a new mojo generatedatadictionaryfromdslmojo.java under bfd-model-dsl-codegen.
• Moved over and reused some existing code that generates the current data dictionary from bfd-server-docs.
• The general idea is to convert the json files into a FhirElementBean then iterate through mapping files (Beneficiary.yaml. Inpatient.yaml, etc) and per file you filter the FhirElementBeans for the fhir elements that correspond to the mapping file (based on the mapping id, bfd table type, and the appliesTo which is a field in the json file). Once you have the corresponding FhirElementBeans for the mapping file you can then modify the FhirElementBean with existing data like updating the name of the bfd table, bfd db type, column name, java field name, and ccw mapping. This is done by extracting these fields from the ColumnBean and TableBean associated to the MappingBean (which is produced when iterating through those mapping files for example a MappingBean for Beneficiary.yaml). Next you'd group all those FhirElementBeans by resource so Patient, ExplanationOfBenefit, and Coverage while adding a new field mappingId to FhirElementBean so we can know the relationship between FhirElementBean and which mapping file it's associated with. With this grouping we can then generate the R2 FHIR Resource yaml files by adding the file as a compile source root.
• Pulls variable descriptions from codebook by reading in code book xml files. A map that maps variable id to description is stored to later be used to inject the descriptions in the correct rows in the new data dictionary
• Basic fhirPath construction: "Core FHIR elements" are obtained from HAPI FHIR models. Since the expressions under the search params on the specification are the fhir paths, I take the element as the FHIR path. If the element is an extension, then I am not sure how to handle this and just default to what's written in the json files. Future spike work on this.

Questions / Issues to Address:

• How to construct FHIR paths for all variables especially for extensions?
• Almost all R4 elements are covered in the new DD, however, there are still a few missing due to bugs in the json files like mismatching bfdTableType and appliesTo. Also, some files that do not map to any mapping yaml file like patient resource profile, profile, Contract Identifier, Contract Reference, a random place holder file, etc
• Should the ccw mapping, data type, and data size columns still be included in the data dictionary?
• Right now, this process relies on the json files existing in order to generate the FHIR resource mapping files: ExplanationOfBenefits.yaml, Patient.yaml, Coverage.yaml. Their values are also used as a default when we're unable to map to our existing data like the extensions and other fields. However, later we'll want to completely remove these json files.
• How to add PAC data?

See this slack thread for more context: https://cmsgov.slack.com/archives/C010WDXAZFZ/p1709661972292489

What Should Reviewers Watch For?

If you're reviewing this PR, please check for these things in particular:

• Verify all PR security questions and checklists have been completed and addressed.

What Security Implications Does This PR Have?

Submitters should complete the following questionnaire:

• If the answer to any of the questions below is Yes, then you must supply a link to the associated Security Impact Assessment (SIA), security checklist, or other similar document in Confluence here: N/A

  • Does this PR add any new software dependencies?
    • Yes
    • No
  • Does this PR modify or invalidate any of our security controls?
    • Yes
    • No
  • Does this PR store or transmit data that was not stored or transmitted before?
    • Yes
    • No

• If the answer to any of the questions below is Yes, then please add @StewGoin as a reviewer, and note that this PR should not be merged unless/until he also approves it.

  • Do you think this PR requires additional review of its security implications for other reasons?
    • Yes
    • No

What Needs to Be Merged and Deployed Before this PR?

This PR cannot be either merged or deployed until the following prerequisite changes have been fully deployed:

• N/A

Submitter Checklist

I have gone through and verified that...:

• I have named this PR and branch so they are automatically linked to the (most) relevant Jira issue. Ie: BFD-123: Adds foo
• This PR is reasonably limited in scope, to help ensure that:
  1. It doesn't unnecessarily tie a bunch of disparate features, fixes, refactorings, etc. together.
  2. There isn't too much of a burden on reviewers.
  3. Any problems it causes have a small "blast radius".
  4. It'll be easier to rollback if that becomes necessary.
• This PR includes any required documentation changes, including README updates and changelog / release notes entries.
• The data dictionary has been updated with any field mapping changes, if any were made.
• All new and modified code is appropriately commented, such that the what and why of its design would be reasonably clear to engineers, preferably ones unfamiliar with the project.
• All tech debt and/or shortcomings introduced by this PR are detailed in TODO and/or FIXME comments, which include a JIRA ticket ID for any items that require urgent attention.
• Reviews are requested from both:
  • At least two other engineers on this project, at least one of whom is a senior engineer or owns the relevant component(s) here.
  • Any relevant engineers on other projects (e.g. DC GEO, BB2, etc.).
• Any deviations from the other policies in the DASG Engineering Standards are specifically called out in this PR, above.
  • Please review the standards every few months to ensure you're familiar with them.

[JFU-NAVA-PBC]

just checked out and run into below at bfd-model-rif:

[INFO] bfd-model-rda ...................................... SUCCESS [ 4.318 s]
[INFO] bfd-model-rif ...................................... FAILURE [ 0.635 s]
[INFO] bfd-model-rif-samples .............................. SKIPPED
[INFO] bfd-pipeline-parent ................................ SKIPPED
[INFO] bfd-pipeline-shared-utils ..............

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal gov.cms.bfd:bfd-model-dsl-codegen-plugin:2.110.0-SNAPSHOT:dataDictionary (default) on project bfd-model-rif: Execution default of goal gov.cms.bfd:bfd-model-dsl-codegen-plugin:2.110.0-SNAPSHOT:dataDictionary failed: java.io.FileNotFoundException: /Users/jimfuqian-mac-pro/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps/bfd-model/bfd-model-rif/../../../dist-temp/data-dictionary-2.110.0-SNAPSHOT.xlsx (No such file or directory) -> [Help 1]
[ERROR]

did I miss something?
thanks

[JFU-NAVA-PBC]

just checked out and run into below at bfd-model-rif:

[INFO] bfd-model-rda ...................................... SUCCESS [ 4.318 s] [INFO] bfd-model-rif ...................................... FAILURE [ 0.635 s] [INFO] bfd-model-rif-samples .............................. SKIPPED [INFO] bfd-pipeline-parent ................................ SKIPPED [INFO] bfd-pipeline-shared-utils ..............

[INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal gov.cms.bfd:bfd-model-dsl-codegen-plugin:2.110.0-SNAPSHOT:dataDictionary (default) on project bfd-model-rif: Execution default of goal gov.cms.bfd:bfd-model-dsl-codegen-plugin:2.110.0-SNAPSHOT:dataDictionary failed: java.io.FileNotFoundException: /Users/jimfuqian-mac-pro/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps/bfd-model/bfd-model-rif/../../../dist-temp/data-dictionary-2.110.0-SNAPSHOT.xlsx (No such file or directory) -> [Help 1] [ERROR]

did I miss something? thanks

thanks Melissa, it went through

[JFU-NAVA-PBC]

stepped through the execution path and try to see what are generated:

inputs:

./apps/bfd-model/bfd-model-rif/fhir-mapping-path/V2/*.json
plus: descriptions extracted from ffs xml (extracted from codebook pdf)

outputs:

1. fhir entities yaml:
   ~/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps$ ls -al ./bfd-model/bfd-model-rif/target/generated-sources/fhirEntities
   total 2352
   drwxr-xr-x 6 jimfuqian-mac-pro staff 192 Mar 18 22:36 .
   drwxr-xr-x 6 jimfuqian-mac-pro staff 192 Mar 18 18:16 ..
   -rw-r--r-- 1 jimfuqian-mac-pro staff 13130 Mar 18 22:32 .yaml
   -rw-r--r-- 1 jimfuqian-mac-pro staff 200612 Mar 18 22:32 Coverage.yaml
   -rw-r--r-- 1 jimfuqian-mac-pro staff 922575 Mar 18 22:36 ExplanationOfBenefit.yaml
   -rw-r--r-- 1 jimfuqian-mac-pro staff 47500 Mar 18 22:32 Patient.yaml
2. DD xlsx at:
   /Users/jimfuqian-mac-pro/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps/bfd-model/bfd-model-rif/../../../dist-temp/data-dictionary-2.110.0-SNAPSHOT.xlsx

Would like to confirm that:

1. We want to use the 629 json files from ./apps/bfd-model/bfd-model-rif/fhir-mapping-path/V2/*.json as source of true, and generate the DD xlsx
2. Who consume the fhirEntities generated, i.e. Patient.yaml, Coverage.yaml, EOB.yaml, and there is a .yaml contains claim line items.

I only stepped through the mojo i.e. doing this and attach:
~/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps/bfd-model$ mvnDebug clean install

Is there any JPA code gen hapening? may be I missed something...

[JFU-NAVA-PBC]

stepped through the execution path and try to see what are generated:

inputs:

./apps/bfd-model/bfd-model-rif/fhir-mapping-path/V2/*.json
plus: descriptions extracted from ffs xml (extracted from codebook pdf)

outputs:

fhir entities yaml:
~/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps$ ls -al ./bfd-model/bfd-model-rif/target/generated-sources/fhirEntities
total 2352
drwxr-xr-x 6 jimfuqian-mac-pro staff 192 Mar 18 22:36 .
drwxr-xr-x 6 jimfuqian-mac-pro staff 192 Mar 18 18:16 ..
-rw-r--r-- 1 jimfuqian-mac-pro staff 13130 Mar 18 22:32 .yaml
-rw-r--r-- 1 jimfuqian-mac-pro staff 200612 Mar 18 22:32 Coverage.yaml
-rw-r--r-- 1 jimfuqian-mac-pro staff 922575 Mar 18 22:36 ExplanationOfBenefit.yaml
-rw-r--r-- 1 jimfuqian-mac-pro staff 47500 Mar 18 22:32 Patient.yaml
DD xlsx at:
/Users/jimfuqian-mac-pro/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps/bfd-model/bfd-model-rif/../../../dist-temp/data-dictionary-2.110.0-SNAPSHOT.xlsx
Would like to confirm that:

We want to use the 629 json files from ./apps/bfd-model/bfd-model-rif/fhir-mapping-path/V2/*.json as source of true, and generate the DD xlsx
Who consume the fhirEntities generated, i.e. Patient.yaml, Coverage.yaml, EOB.yaml, and there is a .yaml contains claim line items.
I only stepped through the mojo i.e. doing this and attach:
~/CMS_PROJECTS/BFD-3331-pull-dd-from-codebook/apps/bfd-model$ mvnDebug clean install

Is there any JPA code gen hapening? may be I missed something...