Vs 1416 modify ingest to correctly handle ploidy differences in dragen 3 7 8 samples #8994

koncheto-broad · 2024-10-09T13:58:11Z

Three major changes here.

Added in logic to create the ploidy table during ingest (with necessary supporting class) and use it during extract automatically as part of the default joint workflow. Also removed a column that we won't need when creating it automatically.
Rearranged the PAR checking logic to consolidate it in its own class (PloidyUtils.java).

Successful run against tiny sample set "PLOIDY_TEST" in echo callset project:

https://app.terra.bio/#workspaces/allofus-drc-wgs-dev/GVS%20AoU%20WGS%20Echo%20Callset%20v2/job_history/a93aa2ef-9cef-451d-8cf8-b31f1c6a8407

You'll need your aou credentials to see the results.

…andle-ploidy-differences-in-dragen-3-7-8-samples

gatk-bot · 2024-10-09T14:31:37Z

Github actions tests reported job failures from actions build 11256533054
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
integration	17.0.6+10	11256533054.11	logs
integration	17.0.6+10	11256533054.0	logs

RoriCremer · 2024-10-10T20:38:36Z

src/main/java/org/broadinstitute/hellbender/tools/gvs/common/PloidyUtils.java

+     * https://github.com/broadinstitute/picard/blob/master/src/main/java/picard/vcf/MendelianViolations/FindMendelianViolations.java#L203
+     * Wikipedia data: https://en.wikipedia.org/wiki/Pseudoautosomal_region
+     */
+    private static final Set<String> PSEUDO_AUTOSOMAL_REGIONS_DEFINITION = CollectionUtil .makeSet("X:60001-2699520", "X:154931044-155260560", "chrX:10001-2781479", "chrX:155701383-156030895", "Y:10001-2649520", "Y:59034050-59363566", "chrY:10001-2781479", "chrY:56887903-57217415");


I'm a little surprised this doesn't exist already somewhere in GATK

So was I, to be honest. I grepped a bunch of terms related to PARs, and the best I could find is that FindMendelianViolations tool that I linked in the comments. Strangely, that one doesn't even include the regions on the y chromosome, for whatever reason. AND its definition and implementation was private, so I couldn't even just import it here.

It's possible I missed something, but at the very least I can say that anything that may exist is pretty non-obvious.

ask in the methods channel perhaps?

Maybe it would be defined in the reference fasta itself? I've seen some with XY or PAR as the contig

mcovarr · 2024-10-16T22:18:47Z

scripts/variantstore/wdl/GvsUtils.wdl

@@ -74,7 +74,7 @@ task GetToolVersions {
    String cloud_sdk_slim_docker = "gcr.io/google.com/cloudsdktool/cloud-sdk:435.0.0-slim"
    String variants_docker = "us-central1-docker.pkg.dev/broad-dsde-methods/gvs/variants:2024-09-30-alpine-27b8708f5808"
    String variants_nirvana_docker = "us.gcr.io/broad-dsde-methods/variantstore:nirvana_2022_10_19"
-    String gatk_docker = "us-central1-docker.pkg.dev/broad-dsde-methods/gvs/gatk:2024_08_19-gatkbase-cd5b6b7821b2"
+    String gatk_docker = "us-central1-docker.pkg.dev/broad-dsde-methods/gvs/gatk:2024-10-08-gatkbase-2e8764db86c3"


just a heads up you and @gbggrant appear to be ⚔️ here with your GATK Docker changes

src/main/java/org/broadinstitute/hellbender/tools/gvs/common/PloidyUtils.java

mcovarr · 2024-10-16T22:25:20Z

src/main/java/org/broadinstitute/hellbender/tools/gvs/common/PloidyUtils.java

+     * https://github.com/broadinstitute/picard/blob/master/src/main/java/picard/vcf/MendelianViolations/FindMendelianViolations.java#L203
+     * Wikipedia data: https://en.wikipedia.org/wiki/Pseudoautosomal_region
+     */
+    private static final Set<String> PSEUDO_AUTOSOMAL_REGIONS_DEFINITION = CollectionUtil .makeSet("X:60001-2699520", "X:154931044-155260560", "chrX:10001-2781479", "chrX:155701383-156030895", "Y:10001-2649520", "Y:59034050-59363566", "chrY:10001-2781479", "chrY:56887903-57217415");


ask in the methods channel perhaps?

src/main/java/org/broadinstitute/hellbender/tools/gvs/common/PloidyUtils.java

src/main/java/org/broadinstitute/hellbender/tools/gvs/extract/ExtractCohortEngine.java

src/main/java/org/broadinstitute/hellbender/tools/gvs/ingest/CreateVariantIngestFiles.java

src/main/java/org/broadinstitute/hellbender/tools/gvs/ingest/SamplePloidyCreator.java

gbggrant

Looks good to me. Thumbs up, conditional on:

Miguel's comments are addressed.
Passing integration test (ideally on ALL chromosomes).

gbggrant · 2024-10-18T14:50:42Z

scripts/variantstore/wdl/GvsAssignIds.wdl

+      datatype = "sample_chromosome_ploidy",
+      schema_json = sample_chromosome_ploidy_schema_json,
+      max_table_id = 1,
+      superpartitioned = "false",


Suggested change

superpartitioned = "false",

superpartitioned = false,

gbggrant · 2024-10-18T14:50:54Z

scripts/variantstore/wdl/GvsAssignIds.wdl

+      schema_json = sample_chromosome_ploidy_schema_json,
+      max_table_id = 1,
+      superpartitioned = "false",
+      partitioned = "false",


Suggested change

partitioned = "false",

partitioned = false,

…andle-ploidy-differences-in-dragen-3-7-8-samples

gatk-bot · 2024-10-24T16:22:51Z

Github actions tests reported job failures from actions build 11502878660
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
integration	17.0.6+10	11502878660.11	logs
integration	17.0.6+10	11502878660.0	logs

gatk-bot · 2024-10-24T20:27:38Z

Github actions tests reported job failures from actions build 11506462963
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
integration	17.0.6+10	11506462963.11	logs
integration	17.0.6+10	11506462963.0	logs

gatk-bot · 2024-10-24T20:30:37Z

Github actions tests reported job failures from actions build 11506500757
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
integration	17.0.6+10	11506500757.11	logs
integration	17.0.6+10	11506500757.0	logs

gatk-bot · 2024-10-25T14:36:16Z

Github actions tests reported job failures from actions build 11519677920
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
integration	17.0.6+10	11519677920.11	logs
integration	17.0.6+10	11519677920.0	logs

…iscovered as part of ticket

gatk-bot · 2024-10-29T19:09:59Z

Github actions tests reported job failures from actions build 11580320242
Failures in the following jobs:

Test Type	JDK	Job ID	Logs
integration	17.0.6+10	11580320242.11	logs
integration	17.0.6+10	11580320242.0	logs

gbggrant · 2024-10-29T22:33:41Z

CreateVariantIngestFilesIntegrationTest test is failing now.

koncheto-broad added 9 commits August 27, 2024 13:51

Pushing naive implementation to test for correctness

0ed9e22

stupid dockstore error

9fb61c3

Updates to track down weird bugs

34ce046

Updating fix to only scan reference blocks in RefCreator

44f8c89

Finally verified the successful ploidy run based on this code

7a41da6

Merge branch 'ah_var_store' into VS-1416-modify-ingest-to-correctly-h…

e442dd6

…andle-ploidy-differences-in-dragen-3-7-8-samples

updating gatk version

3d724f7

cleaning things up a bit

d697019

cleaning things up a bit v2

bd0a99c

RoriCremer approved these changes Oct 10, 2024

View reviewed changes

RoriCremer reviewed Oct 10, 2024

View reviewed changes

koncheto-broad requested a review from mcovarr October 16, 2024 14:41

mcovarr reviewed Oct 16, 2024

View reviewed changes

mcovarr reviewed Oct 17, 2024

View reviewed changes

gbggrant approved these changes Oct 18, 2024

View reviewed changes

koncheto-broad added 3 commits October 24, 2024 09:53

PR feedback

c685c88

Merge branch 'ah_var_store' into VS-1416-modify-ingest-to-correctly-h…

c875267

…andle-ploidy-differences-in-dragen-3-7-8-samples

Updating the docker image

9366d91

koncheto-broad added 2 commits October 24, 2024 15:53

dockstore

fef7938

dockstore

f8450df

updating the CORRECT image this time

187c194

Updating quickstart expected info to account for haploid bge sample d…

4b1256e

…iscovered as part of ticket

gbggrant self-requested a review October 29, 2024 22:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vs 1416 modify ingest to correctly handle ploidy differences in dragen 3 7 8 samples #8994

Vs 1416 modify ingest to correctly handle ploidy differences in dragen 3 7 8 samples #8994

koncheto-broad commented Oct 9, 2024

gatk-bot commented Oct 9, 2024 •

edited

Loading

RoriCremer Oct 10, 2024

koncheto-broad Oct 10, 2024

mcovarr Oct 16, 2024

gbggrant Oct 18, 2024

mcovarr Oct 16, 2024

mcovarr Oct 16, 2024

gbggrant left a comment

gbggrant Oct 18, 2024

gbggrant Oct 18, 2024

gatk-bot commented Oct 24, 2024 •

edited

Loading

gatk-bot commented Oct 24, 2024 •

edited

Loading

gatk-bot commented Oct 24, 2024 •

edited

Loading

gatk-bot commented Oct 25, 2024 •

edited

Loading

gatk-bot commented Oct 29, 2024 •

edited

Loading

gbggrant commented Oct 29, 2024

Vs 1416 modify ingest to correctly handle ploidy differences in dragen 3 7 8 samples #8994

Are you sure you want to change the base?

Vs 1416 modify ingest to correctly handle ploidy differences in dragen 3 7 8 samples #8994

Conversation

koncheto-broad commented Oct 9, 2024

gatk-bot commented Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gbggrant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatk-bot commented Oct 24, 2024 • edited Loading

gatk-bot commented Oct 24, 2024 • edited Loading

gatk-bot commented Oct 24, 2024 • edited Loading

gatk-bot commented Oct 25, 2024 • edited Loading

gatk-bot commented Oct 29, 2024 • edited Loading

gbggrant commented Oct 29, 2024

gatk-bot commented Oct 9, 2024 •

edited

Loading

gatk-bot commented Oct 24, 2024 •

edited

Loading

gatk-bot commented Oct 24, 2024 •

edited

Loading

gatk-bot commented Oct 24, 2024 •

edited

Loading

gatk-bot commented Oct 25, 2024 •

edited

Loading

gatk-bot commented Oct 29, 2024 •

edited

Loading