Releases: broadinstitute/gatk
4.2.2.0
Download release: gatk-4.2.2.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.2.2.0 release:
-
The
ReblockGVCF
tool is now out of beta with several important improvements. This tool can be used to postprocessHaplotypeCaller
GVCFs to decrease filesize. -
FilterMutectCalls
now has a--microbial-mode
argument that sets filters to defaults appropriate for microbial calling -
Important bug fixes to
CalibrateDragstrModel
andFuncotator
Full list of changes:
-
New Tools
ShiftFasta
: create a fasta with the bases shifted by an offset (#6694)
-
ReblockGVCF
ReblockGVCF
is now out of beta (#7419)- Improved
ReblockGVCF
output to eliminate overlapping reference blocks and reference gaps following trimmed deletions (#7122) - Fixed bugs associated with input no-call genotypes and fixed an off-by-one error at contig starts (#7404)
- Fixed an error on ref blocks with missing DPs (if
--floor-blocks
arg is not provided); fixed rare cases where spanning deletion (*) allele is incorrectly modified (#7400)
-
Mutect2
FilterMutectCalls
: added a--microbial-mode
argument that sets filters to defaults appropriate for microbial calling (#6694)
-
ValidateVariants
- Added an optional argument to check for GVCF reference blocks overlapping variants or other reference blocks (#7405)
-
DRAGEN-GATK
-
Funcotator
- Fixed an issue where the
Match_Norm_Seq_Allele1
andMatch_Norm_Seq_Allele2
fields were not being populated in MAF output (#7422)
- Fixed an issue where the
-
Mitochondrial pipeline
- Removed calls to
FilterNuMTs
andFilterLowHetSites
, which are no longer being used (#7325)
- Removed calls to
-
CNV Calling
- Fixed a bug resulting from prefix strings of less than 3 characters when creating temporary files in
GermlineCNVCaller
and improved documentation of corresponding utility methods. (#7411)
- Fixed a bug resulting from prefix strings of less than 3 characters when creating temporary files in
-
Documentation
4.2.1.0
Download release: gatk-4.2.1.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.2.1.0 release:
-
Several important fixes to HaplotypeCaller and the new DRAGEN-GATK code introduced in GATK 4.2.0.0
-
Started laying the groundwork in
Mutect2
forMutect3
, which will be more machine learning focused -
LocalAssembler
: a new tool that performs local assembly of small regions to discover structural variants (#6989) -
Support for multi-sample segmentation in
ModelSegments
-
Major speed improvements and several important fixes to
Funcotator
-
A new version of the Intel Genomics Kernel Library (GKL), with many important fixes and improvements
-
A new version of GenomicsDB, with improved cloud support
-
A GATK-wide option to shard VCFs on output, which is often useful for pipelining
-
GATK support for block compressed interval (
.bci
) files, which is useful when working with extremely large interval lists
Full list of changes:
-
New Tools
LocalAssembler
: a new tool that performs local assembly of small regions to discover structural variants (#6989)
-
HaplotypeCaller
- Fixed a rare edge case in DRAGEN mode that could result in negative GQs when
USE_POSTERIOR_PROBABILITIES
is set (#7120) - Fixed a rare edge case (mainly affecting DRAGEN mode) that could cause the PL arrays to be deleted when genotyping in
HaplotypeCaller
(#7148) - Fixed a bug in the
AlleleLikelihoods
that could result in new evidence X being assigned arbitrary likelihoods left over from previous evidence (#7154) - Fixed a "Padded span must contain active span" error caused by invalid feature file intervals that weren't being checked for validity against the sequence dictionary (#7295)
- Do not add the artificial haplotype read group to the bamout file when
--bam-writer-type NO_HAPLOTYPES
is specified (#7141) - Suppressed excessive log output related to
JumboAnnotation
warnings inHaplotypeCaller
(#7358)
- Fixed a rare edge case in DRAGEN mode that could result in negative GQs when
-
DRAGEN-GATK
-
Mutect2
- Added a training data mode (
--training-data-mode
) toMutect2
to prepare forMutect3
(#7109)- Training data mode collects data on variant- and artifact-supporting read sets for fitting a deep learning filtering model
- Better error bars for samples with small contamination in
CalculateContamination
(#7003)
- Added a training data mode (
-
Funcotator
- Greatly improved
Funcotator
performance by optimizing the VCF sanitization code (#7370)- In our tests, this change appears to speed up the tool by roughly 2x
- Updated the Gencode GTF Codec to be more permissive with transcript and gene types (#7166)
- Now the Gencode GTF Codec no longer restricts
transcriptType
andgeneType
to a limited set of values. These fields are now each stored as a String. This allows for arbitrary values in these fields and will help to future-proof (and species-proof) the GTF parser. - Fixes "IndexFeatureFile Error to Run Funcotator with Mouse Ensembl GTF" (#7054)
- Now the Gencode GTF Codec no longer restricts
- Now can decode codons containing IUPAC bases into amino acids. (#7188)
- Updated the tool to allow for protein changes with N / IUPAC bases. (#6778)
- Added the ability to have IUPAC bases in either the ref/alt alleles OR in the reference when calculating the amino acid sequence. In this case, the code will no longer throw a user exception, but will log a warning and will produce ? amino acids in the case that they cannot be decoded from the amino acid table. Currently this will happen any time an N or IUPAC base is in the region to be coded into amino acids.
- Added AminoAcid.UNDECODABLE as a placeholder for any unknown / undecodable amino acid (such as in the case of an ambiguous IUPAC base).
Funcotator
now checks whether the input has already been annotated, and by default throws an error in that case.- We also added a
--reannotate-vcf
override argument to explicitly allow reannotation (#7349)
- We also added a
- Greatly improved
-
CNV Calling
-
SV Calling
- Added
LocalAssembler
, a new tool that performs local assembly of small regions to discover structural variants (#6989)
- Added
-
The Genomics Kernel Library (GKL)
- Updated to GKL version 0.8.8, and remove the FPGA PairHMM as an option (#7203)
- This is a significant update to the GKL that comes with many fixes and improvements:
- Update ISAL and OTC Zlib libraries to latest version (Q1 2021)
- Fixed 3 reproducible issues and retested out of 4 more in GKL
- Updated build for Centos 7 and Current Mac.
- Ran valgrind on limited C unit tests (passed)
- Major improvements to input validation
- Major updates to Error handling and propagation.
- Added Negative space unit testing coverage
- Regular Static Code Scanning
- Good overall quality of life improvement for the software
- This is a significant update to the GKL that comes with many fixes and improvements:
- Updated to GKL version 0.8.8, and remove the FPGA PairHMM as an option (#7203)
-
GenomicsDB
- Moved to GenomicsDB 1.4.1, and add a toggle between the GCS Connector and native GCS support (#7224)
- This release allows for the direct use of the native GCS C++ client instead of the GCS Cloud Connector via HDFS. The GCS Cloud Connector can still be used with GenomicsDB via the ``--genomicsdb-use-gcs-hdfs-connector option`
- Using the native client with GCS allows for GenomicsDB to use the standard paradigms to help with authentication, retries with exponential backoff, configuring credentials, etc., and also helps with performance issues with GCS. See #7070.
- Allow specifying S3 and Azure blob storage uri's to GenomicsDB in addition to GCS and HDFS (#7271)
- Fixes related to the GenomicsDB upgrade (#7257)
- Improved the error message in
GenomicsDBImport
when failing to open aFeatureReader
(#7375)
- Moved to GenomicsDB 1.4.1, and add a toggle between the GCS Connector and native GCS support (#7224)
-
Mitochondrial pipeline
- Added median coverage metric to the mitochondrial pipeline (#7253)
-
Notable Enhancements
- Added a GATK-wide option (
--max-variants-per-shard
) to shard VCFs on output (#6959)- Sharded output is often extremely useful for pipelining
- Added GATK support for block compressed interval (
.bci
) files (#7142) - Added an
AlleleDepthPseudoCounts
(DD) genotype annotation. (#7303)- Similar to AD, the new annotation (DD) captures the depth of each allele's supporting evidence or reads, however it does so by following a variational Bayes approach looking into the likelihoods rather than applying a fixed threshold. This turns out to be more robust in some instances.
- To get the new non-standard annotation in
HaplotypeCaller
you need to add-A AllelePseudoDepth
- We now track the source of variants in
MultiVariantWalkers
, which is important for some tools such asVariantEval
(#7219)
- Added a GATK-wide option (
-
Bug Fixes
- Fixed key ordering bugs in the implementations of
Histogram.median()
andCompressedDataList.iterator()
(#7131)- These bugs could result in incorrect RankSumTest annotations in some cases
- Fixed the
DepthPerSampleHC
andStrandBiasBySample
annotations to not spam the logs with "Annotation will not be calculated" warnings (#7357) VariantEval
: fixed contig stratification to defer to user-defined intervals (#7238)
- Fixed key ordering bugs in the implementations of
-
Miscellaneous Changes
- The
ProgressMeter
can now be completely disabled for all tools / traversals by overridingGATKTool.disableProgressMeter()
(#7354) - We now authenticate with Dockerhub in our Travis builds, to help avoid tests failing due to quota issues (#7204) (#7256)
- Migrated
VariantEval
to be aMultiVariantWalkerGroupedOnStart
(#6973) VariantEval
: added an argument to specify thePedigreeValidationType
(#7240)- Converted
InfoFieldAnnotation
/GenotypeAnnotation
into interfaces. (#7041) - Allow
MultiVariantWalkerGroupedOnStart
subclasses to view/setignoreIntervalsOutsideStart
(#7301) PedigreeAnnotation
: consolidate code, provide getters, and allowPedigreeValidationType
to be set (#7277)ASEReadCounter
: added a warning for variants lacking GT fields (#7326)- Added filters to
dockstore.yml
so that only the master branch and the releases get synced to Dockstore (#7217) - Fixed a compatibility issue between Java 11 and
log4j2
(#7339) - We now update the gcloud package signing key at the start of every docker build (#7180)
- Updated our Artifactory key (#7208)
- Disabled some Spark dataproc tests because of dependency issues. (#7170)
- Removed some embedded licenses from scripts (#7340)
- The
-
Documentation
4.2.0.0
Download release: gatk-4.2.0.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.2.0.0 release:
-
We've worked closely with Illumina to port a number of significant innovations for germline short variant calling from their DRAGEN pipeline to GATK. These improvements will form the basis of the upcoming open-source implementation of the DRAGEN pipeline which we're calling DRAGEN-GATK
-
A number of other fixes and improvements to
HaplotypeCaller
to improve the phasing of variant calls and to fix edge cases with indels and spanning deletions -
A new pipeline for gCNV exome joint calling
Full list of changes:
-
- With this release we've worked closely with Illumina to make improvements to the GATK
HaplotypeCaller
to allow it to output germline short variant calls that are functionally equivalent to the calls made by their DRAGEN 3.4.12 pipeline. See our blog post on DRAGEN-GATK for more details on these improvements. A fullDRAGEN-GATK
pipeline that leverages these new features will be released in the near future as a WDL workflow script in the WARP repo on GitHub as well as a featured workspace in Terra. - Below is a summary of the improvements we've ported from DRAGEN in this release. We recommend that most users wait until the complete
DRAGEN-GATK
pipeline is released as a WDL workflow before evaluating these features, though advanced users comfortable with building their own pipelines are welcome to try them out now:- DragSTR: a port of DRAGEN's model for STRs (Short Tandem Repeats) that adjusts HMM indel priors based on empirical reference contexts for better indel calling.
- Using DragSTR involves running two new tools prior to the
HaplotypeCaller
:ComposeSTRTableFile
: scans a reference for STR sites and outputs a table file with a subsample of the available STR sites across the genome.CalibrateDragstrModel
: given the STR table for a reference produced byComposeSTRTableFile
and the reads for a specific sample, generates a model for potential sequencing errors for STR sites of various sizes for that sample.
- After running these tools, you then run
HaplotypeCaller
with the--dragstr-params-path
argument to pass it the DragSTR model generated byCalibrateDragstrModel
.
- Using DragSTR involves running two new tools prior to the
- BQD (Base Quality Dropout) and FRD (Foreign Read Detection): two new genotyper error models ported from DRAGEN
- The
Base Quality Dropout (BQD)
model penalizes variants with low average base quality scores and high average sequencing cycle counts among genotyped reads and reads that were otherwise excluded from the genotyper to model read-context dependent sequencing errors. - The
Foreign Read Detection (FRD)
model uses an adjusted mapping quality score as well as read strandedness information to penalize reads that are likely to have originated from somewhere else on the genome or from contamination. - To activate the BQD and FRD models, run
HaplotypeCaller
with the--dragen-mode
argument.
- The
- Added a new variant QUAL score model that reports the variant QUAL score as the posterior of the reference genotype based on the sample-dependent DRAGEN STR and flat SNP priors.
- DragSTR: a port of DRAGEN's model for STRs (Short Tandem Repeats) that adjusts HMM indel priors based on empirical reference contexts for better indel calling.
- With this release we've worked closely with Illumina to make improvements to the GATK
-
HaplotypeCaller
- We now add physical phasing information (PGT/PID/PS attributes) to genotypes with spanning deletion alleles (#6937)
- Fixed two phasing bugs (#7019)
- Fixed quality score calculation for sites with spanning deletions (#6859)
- This fixes a bug in the AlleleFrequencyCalculator that was causing quality to be overestimated for sites with * alleles representing spanning deletions.
- Added the ability for indels to be recovered from dangling heads in the assembly graph, and a new
--num-matching-bases-in-dangling-end-to-recover
argument for filtering dangling ends (#6113) (#7086) - Improved handling of indels/spanning deletions in the cigar base quality adjustment code. (#6886)
- This aims to better handle the edge cases that come up when mates have mismatching numbers of bases at the start or end of the reads relative to each-other.
- Fixed a bug where overlapping reads in subsequent assembly regions could have invalid base qualities (#6943)
- Convert non-ACGT IUPAC bases to N in HaplotypeCaller prior to assembly to prevent a crash (#6868)
- Renamed the
--mapping-quality-threshold
argument to--mapping-quality-threshold-for-genotyping
, and updated its documentation to be less confusing (#7036) - Added an option for
HaplotypeCaller
andMutect2
to produce a bamout without artificial haplotypes (#6991) - Updated the
--debug-graph-transformations
argument to emit the assembly graph both before and after chain pruning (#7049)
-
Mutect2
- Fixed the
--dont-use-soft-clipped-bases
argument inMutect2
to actually work as intended (#6823)- Due to a bug, this option did nothing because a copy of the original reads was modified. By deleting the unnecessary mapping quality filtering (this is totally redundant with the M2 read filter), we finalize (and thereby discard soft clips if requested) an assembly region made from the original reads, not a copy.
- Fixed a bug in the
Mutect2
engine active region code that could affect the ability to call tumor alts when the normal has a different alt at the same site (#6908) - Removed an obsolete cram to bam conversion step in the
Mutect2
WDL (#6970) - Updated the
Mutect2
whitepaper indocs/mutect/mutect.pdf
to accurately reflect current filter names, and updated the section onFilterAlignmentArtifacts
(#6967)
- Fixed the
-
CNV Calling
- A new pipeline for gCNV exome joint calling (#6554)
- Added a new tool (
JointGermlineCNVSegmentation
) and associated workflow (scripts/cnv_wdl/germline/joint_call_exome_cnvs.wdl
) to combine gCNV segments and calls across samples JointGermlineCNVSegmentation
segments and genotypes CNV calls from the germline CNV pipeline jointly across multiple samples.- The workflow in
scripts/cnv_wdl/germline/joint_call_exome_cnvs.wdl
produces a joint, multi-sample genotyped VCF. - For whole genomes, we recommend CNVs as part of a full SV callset with https://github.com/broadinstitute/gatk-sv (soon to be added to Terra)
- Added a new tool (
GermlineCNVCaller
now restarts inference once with a new random seed when inference diverges. Also added a new entry point to PythonScriptExecutor that returnes ProcessOutput. (#6866)- This is intended to alleviate transient issues with GermlineCNVCaller inference in which the ELBO converges to a NaN value, by calling the python gCNV code with an updated random seed input.
CreateReadCountPanelOfNormals
: fixed a bug in the logic for filtering zero-coverage samples and intervals (#6624)FilterIntervals
: fixed a bug in the tool logic when filtering on annotations and -XL is used to exclude intervals (#7046)
- A new pipeline for gCNV exome joint calling (#6554)
-
SV Calling
PrintSVEvidence
: a new tool that prints any of the Structural Variation evidence file types: read count (RD), discordant pair (PE), split-read (SR), or B-allele frequency (BAF) (#7026)- This tool is used frequently in the GATK-SV pipeline for retrieving subsets of evidence records from a bucket over specific intervals. Evidence file formats comply with the current specifications in the existing GATK-SV pipeline.
-
GenomicsDB
- Introduced a new feature for
GenomicsDBImport
that allows merging multiple contigs into fewer GenomicsDB partitions (#6681)- Controlled via the new
--merge-contigs-into-num-partitions
argument toGenomicsDBImport
- This should produce a huge performance boost in cases where users have a very large number of contigs. Prior to this change, GenomicsDB would create a separate folder/partition for each contig, which slowed down import to a crawl when there were many contigs.
- Controlled via the new
- Introduced a new feature for
-
Funcotator
- Added sorting by strand order for transcript subcomponents (#7065)
- This fixes an issue where the coding sequence, protein prediction, and other annotations could be incorrect for the hg19 version of Gencode, due to the individual elements of each transcript appearing in numerical order, rather than the order in which they appear in the transcript at transcription time.
- Updated the Funcotator tutorial link in the tool documentation. (#6920) (#6925)
- Added sorting by strand order for transcript subcomponents (#7065)
-
Mitochondrial pipeline
-
Notable Enhancements
4.1.9.0
Download release: gatk-4.1.9.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.1.9.0 release:
-
A major update to
Funcotator
, bringing in the latest Gencode release, fixing compatibility issues with dbSNP, and more! -
Two new tools,
GeneExpressionEvaluation
andReferenceBlockConcordance
-
Significant performance improvements to
DepthOfCoverage
andSelectVariants
-
Some important bug fixes:
- Fixed a bug in
HaplotypeCaller
andMutect2
where we were losing insertion events that immediately followed a deletion - A fix for the "CreateSomaticPanelOfNormals output PoN has much less variants in 4.1.8.0 than before" issue reported in #6744
- A fix for a frequently-encountered
NullPointerException
in theAS_StrandBiasTest
annotation when runningCombineGVCFs
reported in #6766
- Fixed a bug in
Full list of changes:
-
New Tools
-
GeneExpressionEvaluation
: a tool for evaluating gene expression from RNA-seq reads aligned to whole genome (#6602)- This tool counts fragments to evaluate gene expression from RNA-seq reads aligned to the genome. Features to evaluate expression over are defined in an input annotation file in gff3 fomat. Output is a tsv listing sense and antisense expression for all stranded grouping features, and expression (labeled as sense) for all unstranded grouping features.
-
ReferenceBlockConcordance
: a new tool to evaluate concordance of reference blocks in GVCF files (#6802)- This tool compares the reference blocks of two GVCF files against each other and produces three histograms:
- Truth block histogram: Indicates the number of occurrences of reference blocks with a given confidence score and length in the truth GVCF
- Eval block histogram: Indicates the number of occurrences of reference blocks with a given confidence score and length in the eval GVCF
- Confidence concordance histogram: Reflects the confidence scores of bases in reference blocks in the truth and eval VCF, respectively. An entry of 10 at bin "80,90" means that there are 10 bases which simultaneously have a reference confidence of 80 in the truth GVCF and a reference confidence of 90 in the eval GVCF.
- This tool compares the reference blocks of two GVCF files against each other and produces three histograms:
-
-
HaplotypeCaller/Mutect2
- Fixed a bug in
HaplotypeCaller
andMutect2
where we were losing insertion events that immediately followed a deletion (#6696) - Added a workaround for an issue with multiallelics in the
CreateSomaticPanelOfNormals
pipeline (#6871)- This fixes the "CreateSomaticPanelOfNormals output PoN has much less variants in 4.1.8.0 than before" issue reported in #6744
- Made improvements to the
Mutect2
active region detection code that resulted in recovering some low-AF calls that we were missing (#6821) - Made the
HaplotypeCaller
/Mutect2
adaptive pruner smarter in complex graphs, resulting in modest improvements to indel sensitivity when using the adaptive pruning option (#6520) - Fixed a bug in variation event detection code that could sometimes lead to mistreating indel assembly windows as SNP assembly windows (#6661)
- Fixed a bug in
FragmentUtils
where insertion quals were used instead of deletion quals when adjusting base qualities for two overlapping reads from the same fragment (#6815) - Fixed a concurrent modification exception error for local runs of
HaplotypeCallerSpark
(#6741) - Marked the
--linked-de-bruijn-graph
argument as Advanced rather than Hidden (#6737) - Made a small tweak to
Mutect2
's callable sites count (#6791) - Added a "requester pays" option to
Mutect2
WDL tasks that access bams for use with Google Cloud "requester pays" buckets (#6879)
- Fixed a bug in
-
Funcotator
- A major set of updates to
Funcotator
(#6660)- Updated to the latest Gencode release
- Fixed the contig naming compatibility issue with dbSNP reported in #6564 ("hg38 dbSNP has incorrect contig names")
- Now both hg19 and hg38 have the contig names translated to "chr__"
- Added 'lncRNA' to GeneTranscriptType.
- Added "TAGENE" gene tag.
- Added the MANE_SELECT tag to FeatureTag.
- Added the STOP_CODON_READTHROUGH tag to FeatureTag.
- Updated the GTF versions that are parseable.
- Fixed a parsing error with new versions of gencode and the remap positions (for liftover files).
- Added test for indexing new lifted over gencode GTF.
- Added Gencode_34 entries to MAF output map.
- Pointed data source downloader at new data sources URL.
- Minor updates to workflows to point at new data sources.
- Updated retrieval scripts for dbSNP and Gencode.
- Added required field to gencode config file generation.
- Now gencode retrieval script enforces double hash comments at top of gencode GTF files.
- Fixed an erroneous trailing tab in MAF file output reported in #6693
- Added a maximum version number for data sources in
Funcotator
(#6807) - Added a "requester pays" option to the
Funcotator
WDL for use with Google Cloud "requester pays" buckets (#6874) FuncotateSegments
: fixed an issue with the default value of --alias-to-key-mapping being set to an immutable value (#6700)
- A major set of updates to
-
GenomicsDB
- Updated to GenomicsDB Version 1.3.2, which brings better propagation of errors messages from the GenomicsDB library (#6852)
- Using the GATK option GATK_STACKTRACE_ON_USER_EXCEPTION will now also output a limited C/C++ stacktrace
- Updated to GenomicsDB Version 1.3.2, which brings better propagation of errors messages from the GenomicsDB library (#6852)
-
CNV Tools
- Fixed a bug in the
KernelSegmenter
: the minimal data to calculate the segmentation cost should be2 * windowSize
, rather thanwindowSize
(#6835) - Germline CNV WDL improvements for WGS (#6607)
- Modified gCNV WDLs to improve Cromwell performance when running on a large number of intervals, as in WGS
- Added optional disabled_read_filters input to CollectCounts
- Enabled GCS streaming for CollectCounts and CollectAllelicCounts
- Added a "requester pays" option to the germline and somatic CNV WDLs for use with Google Cloud "requester pays" buckets (#6870)
- Fixed a bug in the
-
Mitochondrial Pipeline
-
Notable Enhancements
- Significantly improved the performance of
DepthOfCoverage
by removing slow string formatting calls (#6740)- In a test run with default arguments locally the runtime for a WGS full chr15 drops from ~8.9 minutes to ~4.7 minutes after this patch
- Significantly improved the performance of
SelectVariants
with large numbers of samples by changing an operation to scale linearly instead of quadratically with the number of samples (#6729)- On one example with several thousand samples there was a speed up from ~5 minutes to 0.1 minutes
- WDL generation: made several improvements to automatic WDL generation, annotated additional tools for WDL generation, and added a section to the README with instructions on generating WDLs for GATK tools (#6800)
- Added a suite of utility methods for working with Google BigQuery:
BigQueryUtils
(#6759) (#6861) - The GATK docker image can now be built with a simple
docker build .
command (no extra arguments needed) (#6764) (#6842) (#6782) - Added a Dockstore yml file with workflow descriptions for the WDLs in the GATK repo, to facilitate automatic publication to Dockstore (#6770)
- Significantly improved the performance of
-
Bug Fixes
- Fixed a
NullPointerException
in theAS_StrandBiasTest
annotation reported in #6766 (#6847) - Fixed a bug with soft clips in
LeftAlignIndels
(#6792) VariantRecalibrator
: uniquify annotations to fix the error reported in #2221 (#6723)- Fixed an issue where
ContextCovariate
inBaseRecalibrator
mistakenly assumed that all non-ACGT bases in the read are N (#6625) - Fixed a crash in
CountBasesSpark
when using the-L
option (#6767)
- Fixed a
-
Miscellaneous Changes
- Significant refactoring of the SV discovery classes (#6652)
FilterVariantTranches
: report more info when the ref alleles don't match (#6723)- We now report the target url in exceptions thrown by
HtsgetReader
(#6799) - Added more information to error messages in
AssemblyRegion
for contigs not in the reference dictionary (#6781) - Improved an error message in
GATKRead.setMatePosition()
(#6779) - Updated the Barclay WDL template for compatibility with the Debian distribution (#6841)
- Temporarily disabled
HtsgetReader
tests to work around issues caused by a server-side upgrade. (#6804) - Re-enabled an
IndexFeatureFile
test for uncompressed BCF. (#6716)
-
Documentation
- Marked
LearnReadOrientationModel
as aDocumentedFeature
(#6726) - Added a gentle warning about loss of True Positives with the default
FilterIntervals
params (#6751) - Updated the README to mention that the conda environment is not officially supported on macOS at this time. (#6788)
- Fixed a typo in the example command for
SplitIntervals
(#6869) - Fixed a typo in the `--tmp-dir...
- Marked
4.1.8.1
Download release: gatk-4.1.8.1.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.1.8.1 release:
-
This is a minor point release intended primarily to push out a needed enhancement to the
Mutect2
pipeline. -
This release also introduces a new framework for the auto-generation of WDLs for GATK/Picard tools. Over the next several GATK releases, we intend to hook GATK/Picard tools up to the new WDL generator, with the ultimate goal of having WDLs automatically published for all tools with each release.
Full list of changes:
-
Mutect2
- We now allow for the passing of additional arguments to
GetPileupSummaries
from theMutect2
WDL (#6713)
- We now allow for the passing of additional arguments to
-
GATK Engine
- Added a new framework for the auto-generation of WDLs for GATK/Picard tools (#6504)
- Over the next several GATK releases, we intend to hook GATK/Picard tools up to the new WDL generator, with the ultimate goal of having WDLs automatically published for all tools with each release
- Added a new framework for the auto-generation of WDLs for GATK/Picard tools (#6504)
-
Bug Fixes
-
Miscellaneous Changes
- Removed a few GATK classes that are redundant with Picard classes. (#6678)
-
Documentation
-
Dependencies
- Updated HTSJDK to
2.23.0
(#6702)
- Updated HTSJDK to
4.1.8.0
Download release: gatk-4.1.8.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.1.8.0 release:
-
A major new release of
GenomicsDB
(1.3.0), with enhanced support for shared filesystems such as NFS and Lustre, support for MNVs, and better compression leading to a roughly 50% reduction in workspace size in our tests. This also includes a fix for an error inGenotypeGVCFs
that several users were encountering when reading from GenomicsDB. -
A major overhaul of the
PathSeq
microbial detection pipeline containing many improvements -
Initial/prototype support for reading from HTSGET services in GATK
- Over the next several releases, we intend for HTSGET support to propagate to more tools in the GATK
-
Fixes for a couple of frequently-reported errors in
HaplotypeCaller
andMutect2
(#6586 and #6516) -
Significant updates to our Python/R library dependencies and Docker image
Full list of changes:
-
New Tools
HtsgetReader
: an experimental tool to localize files from an HTSGET service (#6611)- Over the next several releases, we intend for HTSGET support to propagate to more tools in the GATK
ReadAnonymizer
: a tool to anonymize reads with information from the reference (#6653)- This tool is useful in the case where you want to use data for analysis, but cannot publish the data without anonymizing the sequence information.
-
HaplotypeCaller/Mutect2
- Fixed an "evidence provided is not in sample" error in
HaplotypeCaller
when performing contamination downsampling (#6593)- This fixes the issue reported in #6586
- Fixed a "String index out of range" error in the
TandemRepeat
annotation withHaplotypeCaller
andMutect2
(#6583)- This addresses an edge case reported in #6516 where an alt haplotype starts with an indel, and hence the variant start is one base before the assembly region due to padding a leading matching base
- Better documentation for
FilterAlignmentArtifacts
(#6638) - Updated the
CreateSomaticPanelOfNormals
documentation (#6584) - Improved the tests for
NuMTFilterTool
(#6569)
- Fixed an "evidence provided is not in sample" error in
-
PathSeq
- Major overhaul of the PathSeq WDLs (#6536)
- This new PathSeq WDL redesigns the workflow for improved performance in the cloud.
- Downsampling can be applied to BAMs with high microbial content (ie >10M reads) that normally cause performance issues.
- Removed microbial fasta input, as only the sequence dictionary is needed.
- Broke pipeline down to into smaller tasks. This helps reduce costs by a) provisioning fewer resources at the filter and score phases of the pipeline and b) reducing job wall time to minimize the likelihood of VM preemption.
- Filter-only option, which can be used to cheaply estimate the number of microbial reads in the sample.
- Metrics are now parsed so they can be fed as output to the Terra data model.
- CRAM-to-BAM capability
- Updated WDL readme
- Deleted unneeded WDL json configuration, as the configuration can be provided in Terra
- Added an
--ignore-alignment-contigs
argument toPathSeq
filtering that lets users specify any contigs that should be ignored. (#6537)- This is useful for BAMs aligned to hg38, which contains the Epstein-Barr virus (chrEBV)
- Major overhaul of the PathSeq WDLs (#6536)
-
GenomicsDB
- Upgraded to
GenomicsDB
version 1.3.0 (#6654)- Added a new argument
--genomicsdb-shared-posixfs-optimizations
to help with shared POSIX filesystems like NFS and Lustre. This turns on disable file locking and for GenomicsDB import it minimizes writes to disks. The performance on some of the gatk datasets for the import of about 10 samples went from 23.72m to 6.34m on NFS which was comparable to importing to a local filesystem. Hopefully this helps with Issue #6487 and #6627. Also, fixes Issue #6519. - This version of GenomicsDB also uses pre-compression filters for offset and compression files for new workspaces and genomicsdb arrays. The total sizes for a GenomicsDB workspace using the same dataset as above and the 10 samples went from 313MB to 170MB with no change in import and query times. Smaller GenomicsDB arrays also help with performance on distributed and cloud file systems.
- This version has added support to handle MNVs similar to deletions as described in Issue #6500.
- There is added support in
GenomicsDBImport
to have multiple contigs in the same GenomicsDB partition/array. This will hopefully help import times in cases where users have many thousands of contigs. Changes are still needed from the GATK side to make use of this support. - Logging has been improved somewhat with the native C/C++ code using spdlog and fmt and the Java layer using apache log4j and log4j.properties provided by the application. Also, info messages like No valid combination operation found for INFO field AA - the field will NOT be part of INFO fields in the generated VCF records will only be output once for the operation.
- Added a new argument
- Made
VCFCodec
the default for query streams fromGenomicsDB
(#6675)- This fixes the frequently-reported
NullPointerException
inGenotypeGVCFs
when reading from GenomicsDB (see #6667) - Added a
--genomicsdb-use-bcf-codec
argument to opt back in to using the BCFCodec, which is faster but prone to the above error on certain datasets
- This fixes the frequently-reported
- Upgraded to
-
CNV Tools
-
Docker/Conda Overhaul (#5026)
- Our docker image is now built off of Ubuntu 18.04 instead of 16.04
- This brings in newer versions of several important packages such as
samtools
- This brings in newer versions of several important packages such as
- Updated many of the Python libraries installed via our conda environment and included in our Docker image to newer versions, resolving several outstanding issues in the process
- R dependencies are now installed via conda in our Docker build instead of the now-removed
install_R_packages.R
script- Due to this change, we recommend that tools that use R packages (e.g., to create plots) should now be run using the GATK docker image or the conda environment.
- NOTE: significant updates and changes to the Ubuntu version, native packages, and R/python packages may result in corresponding numerical changes in results.
- Our docker image is now built off of Ubuntu 18.04 instead of 16.04
-
Mitochondrial Pipeline
- Minor updates to the mitochondrial pipeline WDLs (#6597)
-
Notable Enhancements
RevertSamSpark
now supports CRAMs (#6641)- Fixed a
VariantAnnotator
performance issue that could cause the tool to run very slowly on certain inputs (#6672) - More flexible matching of dbSNP variants during variant annotation (#6626)
- Add all dbsnp id's which match a particular variant to the variant's id, instead of just the first one found in the dbsnp vcf.
- Be less brittle to variant normalization issues, and match differing variant representations of the same underlying variant. This is implemented by splitting and trimming multiallelics before checking for a match, which I suspect are the predominant cause of these types of matching failures.
- Added a
--min-num-bases-for-segment-funcotation
argument toFuncotateSegments
(#6577)- This will allow for segments of length less than 150 bases to be annotated if given at run time (defaults to 150 bases to preserve the previous behavior).
SplitIntervals
can now handle more than 10,000 shards (#6587)
-
Bug Fixes
- Fixed interval summary files being empty in
DepthOfCoverage
(#6609) - Fixed a crash in the BQSR R script with newer versions of R (#6677)
- Fix crash when reporting error when trying to build GATK with a JRE (#6676)
- Fixed an issue where
ReadsSourceSpark.getHeader()
wasn't propagating the reference at all when a CRAM file input resides on GCS, so it always resulted in a "no reference was provided" error, even when a reference was provided. (#6517) - Fixed an issue where
ReadsSourceSpark.checkCramReference()
always tried to create a Hadoop Path object for the reference no matter what file system it lives on, which fails when using a reference on GCS. (#6517) - Fixed an issue where the tab completion integration tests weren't emitting any output (#6647)
- Fixed interval summary files being empty in
-
Miscellaneous Changes
- Created a new
ReadsDataSource
interface (#6633) - Migrated read arguments and downstream code to
GATKPath
(#6561) - Renamed
GATKPathSpecifier
toGATKPath
. (#6632) - Add a read/write roundtrip Spark integration test for a CRAM and reference on HDFS. (#6618)
- Deleted redundant methods in
SVCigarUtils
, and rewrote and moved the rest toCigarUtils
(#6481) - Re-enabled tests for HTSGET now that the reference server is back to a stable version (#6668)
- Disabled
SortSamSparkIntegrationTest.testSortBAMsSharded()
(#6635) - Fixed a typo in a
SortSamSpark
log message. (#6636) - Removed incorrect logger from
DepthOfCoverage
. (#6622)
- Created a new
-
Documentation
4.1.7.0
Download release: gatk-4.1.7.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.1.7.0 release:
-
Added allele-specific filtering to the mitochondrial pipeline.
- Allele-specific filtering is important for mitochondrial calling because there are many more multi-allelic sites than in the germline autosome.
-
A fix for the frequently-encountered "Smith-Waterman alignment failure" error in
HaplotypeCaller
andMutect2
-
Initial support for http(s) paths for BAM inputs, including signed urls
-
A new tool,
DownsampleByDuplicateSet
, to randomly sample a fraction of duplicate sets from an input bam sorted by UMI
Full list of changes:
-
New Tools
DownsampleByDuplicateSet
: a new tool to randomly sample a fraction of an input bam sorted by UMI. (#6512)- Given a bam grouped by unique molecular identifier (UMI), this tool drops a specified fraction of duplicate sets and returns a new bam.
- A duplicate set refers to a group of reads whose fragments start and end at the same genomic coordinate and share the same UMI.
- The input bam must first be sorted by UMI using FGBio GroupReadsByUmi.
- Use this tool to create, for instance, an insilico mixture of duplex-sequenced samples to simulate tumor subclones.
-
HaplotypeCaller/Mutect2
- Fixed a regression in
HaplotypeCaller
andMutect2
where alt haplotypes with a deletion at the end of the padded region caused exceptions (#6544)- This bug produced error messages like the following: "Smith-Waterman alignment failure. Cigar = 275M with reference length 275 but expecting reference length of 303"
- Fixed an
ArrayIndexOutOfBoundsException
inGenotypeUtils.computeDiploidGenotypeCounts()
caused by mistakenly assuming ploidy two for no-calls (#6563) - Added more control over scattering in the
Mutect2
PON WDL to allow arbitrarily fine scattering, reducing the memory required for downstream runs ofGenomicsDBImport
(#6527) - Invert
--correct-overlapping-quality
argument inHaplotypeCaller
to--do-not-correct-overlapping-quality
(#6528)
- Fixed a regression in
-
Mitochondrial Pipeline
- Added allele-specific filtering to the mitochondrial pipeline (#6399)
- Allele-specific filtering is important for mitochondria because there are many more multi-allelic sites than in the germline autosome and therefore, downstream tools have access to more of the good allele data.
- These Mutect2 filters used in the MT pipeline are now allele-specific:
weak_evidence
,base_qual
,map_qual
,duplicate
,strand_bias
,strand_artifact
,position
,contamination
, andlow_allele_frac
. - They are added to the
AS_FilterStatus
annotation in the INFO field. - The
numt_chimera
andnumt_novel
filters have been replaced by thepossible_numt
filter. - Two new filtering tools have been added:
NuMTFilterTool
for thepossible_numt
filter andMTLowHeteroplasmyFilterTool
for themt_many_low_hets
filter, both of which are allele-specific. - The
--split-multi-allelics
option of theLeftAlignAndTrimVariants
tool now splits the annotations in the FORMAT and INFO fields that are of type A and R (allele-specific, and allele-specific with reference). - The
VariantFiltration
tool now has an--apply-allele-specific-filters
option that will apply masks at the allele level. Before this addition, sites that should not be masked, but had deletions that spanned a masked site would have been masked. Now, if this option is specified, only the alleles spanning the masked site will be masked.
- Added allele-specific filtering to the mitochondrial pipeline (#6399)
-
GATK Engine
- Added initial support for http(s) paths for BAM inputs, including signed urls (#6526)
-
Miscellaneous Changes
- Exposed maximum copy ratio and point size for CNV plotting tools (#6482)
- Decreased an epsilon value in
VariantRecalibrator
so that our production exome joint genotyping tests pass (#6534) - Migrated reference arguments and downstream code to
GATKPathSpecifier
(#6524) - Removed obsolete
isCompatibleWithSparkBroadcast()
method. (#6523)
-
Documentation
- Cleaned up the handling of some missing values in auto-generated GATK tool documentation (#6565)
- Now docs won't include null, "", or [] in the default value list.
- Added a README for the CNN variant scoring workflow, and added an input JSON for
Mutect2
workflow files located in GCS buckets (#6542) - Fixed a typo in a ploidy prior example in the docs for
DetermineGermlineContigPloidy
(#6531)
- Cleaned up the handling of some missing values in auto-generated GATK tool documentation (#6565)
4.1.6.0
Download release: gatk-4.1.6.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.1.6.0 release:
-
Funcotator
now supports ENSEMBL GTF files (and non-human species) -
A beta port of the GATK3 tool
DepthOfCoverage
, a tool to assess sequence coverage by a wide array of metrics, partitioned by sample, read group, library, or gene (#5913) -
Several important bug fixes and enhancements to
HaplotypeCaller
andMutect2
, including:- A fix for an often-reported issue where
HaplotypeCaller
could produce reads starting with deletions during the realignment step and error out. - A fix for another often-reported issue where
Mutect2
could emit MNPs despite--max-mnp-distance
being 0, causing downstream errors inGenomicsDB
about MNPs not being supported.
- A fix for an often-reported issue where
Full list of changes:
-
New Tools
- A beta port of the GATK3 tool
DepthOfCoverage
, a tool to assess sequence coverage by a wide array of metrics, partitioned by sample, read group, library, or gene (#5913)- This port fixes several bugs and changes some behavior present in the GATK3 version:
- Fixed a longstanding bug in GATK3 DepthOfCoverage where using multiple partition types results in column header and body lines having mismatching ordering causing incorrect output.
- The old version used to merge adjacent and overlapping intervals when generating interval summary files. This is no longer the case as in GATK4 adjacent and overlapping intervals are tabulated as separate lines in the output (This also applies to gene lists which would previously have been merged as well).
- Changed the behavior of gene list coverage to no longer count introns when generating interval summaries for gene lists.
- Added support for RefSeqGeneList files as optional gene list input.
- This port fixes several bugs and changes some behavior present in the GATK3 version:
- A beta port of the GATK3 tool
-
HaplotypeCaller
- Fixed a bug where single-base intervals led to no calls (#6507)
- This fixes the issue reported in #6495 "HaplotypeCaller doesn't detect alternate alleles with 1 bp intervals"
- Clean leading deletions from reads realigned to best haplotypes (#6498)
- This fixes the issue reported in #6490 "HaplotypeCaller might be producing bogus reads with deletions at their alignment start during realignment to best haplotype step"
- Fixed an edge case when haplotypes have leading insertion after trimming (#6518)
- Fixed a bug where single-base intervals led to no calls (#6507)
-
Mutect2
Mutect2
can now filter MNVs with orientation bias (#6486)- Added an experimental pileup-based read error corrector, which in our evaluations reduces false positives and improves speed at no cost to sensitivity (#6470)
- Switched CigarBuilder's order for adjacent indels to be deletion first (#6510)
- Fixes #6473 "Mutect2 (GATK 4.1.5.0) emitting MNPs despite max-mnp-distance 0"
- This also resolves downstream errors in
GenomicsDB
about not supporting MNPs
- Fixed several bugs involving
getReadCoordinateForReferenceCoordinate()
(#6485)- Fixes #6342 "Mutect2 occasionally writes nonsense / invalid values for MPOS info tag"
- Fixes #6314 "GATK4.1.3.0 Mutect2 enable-all-annotations option error"
- Fixes #6294 "ReadPosRankSumTest with leading insertions"
- Fixes #5492 "ReadPosRankSumTest doesn't work for two deletions with one base in between"
-
Funcotator
Funcotator
now supports ENSEMBL GTF files (and non-human species) (#6477) (#6492)- Users can now create datasources for any species for which ENSEMBL has an annotated GTF file and the corresponding coding region FASTA file
- When creating new data sources, the user must still use
gencode
as the parent folder for the GTF data source subfolders. For example, for E. coli MG1655:- DATASOURCES
- gencode
- ASM584v2
- Escherichia_coli_str_k_12_substr_mg1655.ASM584v2.44.gtf
- Escherichia_coli_str_k_12_substr_mg1655.ASM584v2.cds.all.fa
- …
- gencode.config
- ASM584v2
- gencode
- DATASOURCES
- For more information on creating data sources see the Funcotator tutorial on the GATK Forums.
- An example datasource for E. coli MG1655 can be found in the large test files for Funcotator
- For ENSEMBL datasources for vertebrates: ftp://ftp.ensembl.org/pub/
- For ENSEMBL datasources for other species: ftp://ftp.ensemblgenomes.org/pub/
-
CNV Calling
-
Miscellaneous Changes
- Simplified cigar and clipping code; added tests and fixed a few bugs including #6130 (#6403)
- Refactored and enhanced ArgumentsBuilder (#6474)
- Allow all GATKSparkTools to set the SBI index granularity (#6458)
- Delete NioBam and related classes (#6479)
- Clean up old interval code (#6465)
- Remove duplicate copy of the NIO prefetching code (#6464)
- Fix ignored test in GATKReadAdaptersUnitTest (#6471)
- Fix alternate spellings of De Bruijn in the codebase (#6472)
-
Documentation
- Fix a broken set of javadoc references in FeatureDataSource (#6478)
4.1.5.0
Download release: gatk-4.1.5.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.1.5.0 release:
-
A new, improved version of the
--linked-de-bruijn-graph
mode forHaplotypeCaller
andMutect2
that has better sensitivity compared to the previous linked DeBruijn graph implementation (#6394) -
A new version of
GenomicsDB
that fixes many frequently-reported issues -
LeftAlignIndels
now works for multiple indels -
VariantAnnotator
andConcordance
are now out of beta -
A significant number of bug fixes to major tools like
GenotypeGVCFs
andSelectVariants
Full list of changes:
-
HaplotypeCaller
- New, improved version of the
--linked-de-bruijn-graph
mode forHaplotypeCaller
andMutect2
that has better sensitivity compared to the previous linked DeBruijn graph implementation (#6394)- Running
HaplotypeCaller
in this mode will reduce the number of erroneous haplotypes discovered which can improve genotyping, phasing, and runtime. - Changed the haplotype recovery step to check that it covers all paths through the graph even if there are poorly supported paths in the JunctionTrees. Added the argument
--disable-artificial-haplotype-recovery
to disable this behavior. - Added the ability to expand graph kmer size after haplotype recovery in the event that there was a failure due to overcomplicated assembly graphs.
- Added code to squeeze extra sensitivity out of the junction trees by tolerating SNP errors when threading the junction trees themselves
- Running
- Realigning to best haplotype handles indels better (#6461)
- Fixed issue #5434 on inconsistent selection of reads for the PL, AD, and DP calculations. (#6055)
- Fixed bug where SNP and indel pseudocounts were swapped in the
AlleleFrequencyCalculator
(#6401) - The qual used in
HaplotypeCaller
'sisActive()
method now matches that ofGenotypeGVCFs
. That is, they both now use the new qual. (#6343) - Skip non-nucleotide alleles in force-calling mode, fixing bug (#6405)
- Fixed the hidden/experimental
--error-correct-reads
argument to actually correct the bases and qualities (#6366) - Removed the deprecated and obsolete
--use-new-qual-calculator
argument (#6398) - Refactored code related to windows and padding for assembly and genotyping, with slight changes to HMM padding for indels (#6358)
- New, improved version of the
-
Mutect2
- Improved
SomaticClusteringModel
(#6337) - Sped up Mutect2 reference confidence model with fast likelihoods model (#6457)
- Modified Fragment creation for Mutect2 to not fail for supplementary reads (#6327)
- Uniqify PG IDs in
FilterAlignmentArtifacts
(#6304) - Fixed error in RealignmentEngine due to converting from exclusive to inclusive interval ends (#6404)
- Added an error message for no callable sites in Mutect2 (#6445)
- Changed filter reporting in Mutect2 (#6288)
- Fixed force-calling mode in M2 mito WDL (#6359)
- Pass the reference to the realignment filter in the Mutect2 WDL (#6360)
- Deleted the old orientation bias filter (#6408)
- Made callable sites a Long to avoid integer overflow (#6303)
- Improved
-
GenomicsDB
- Move to
GenomicsDB
1.2.0 (#6305)- Fixes an issue with
GenomicsDBImport
erroring out due to duplicate fields in the Info, Format, and/or Filter fields. (#6158) - Fixes an issue with
GenomicsDBImport
not completing for mixed ploidy samples (#6275) - This version uses a 64-bit htslib to workaround overflow issues when computed annotation sizes exceed the 32-bit integer space
- Fixes an issue with
- Move to
-
Joint Calling
GenotypeGVCFs
: improved checking for upstream deletions in theGenotypingEngine
(#6429)- Fixes rare cases where
GenotypeGVCFs
could emit a variant with a spanned allele (*), and a genotype that references the spanned allele, but fail to emit the upstream spanning variant.
- Fixes rare cases where
GenotypeGVCFs
: Don't call the NON_REF allele in genotypes or ADs (#6437)- Parse combined
AS_QUALapprox
values from older reblocked GVCFs properly (#6442) - Added a force output sites argument to
GenotypeGVCFs
(#6263) - Remove extraneous alleles in GenotypeGVCFs force-output mode (#6406)
-
CNV Calling
- Copy temporary files early in gcnvkernel to avoid inadvertent temporary directory cleanup. (#6297)
- Enabled streaming of counts.tsv/counts.tsv.gz files in gCNV CLIs. (#6266)
- Fixed shard index in PostprocessGermlineCNVCalls log message. (#6313)
- gCNV vcf cleanup (#6352)
- Index output VCFs for GCNV postprocessing (#6330)
-
Notable Enhancements
VariantAnnotator
is now out of beta (#6402)Concordance
is out of beta (#6397)LeftAlignIndels
now works for multiple indels (#6427)FilterVariantTranches
can now handle cases where there are only SNPs or only indels, and not both (#6411)- Added new read filters for
NotProperlyPaired
and forMateDistant
(#6295) - Made the
.git
directory optional during build (#6450)
-
Bug Fixes
- Handle zero-weight Gaussians correctly in
VariantRecalibrator
(#6425) - Fixed the
--invalidate-previous-filters
argument inVariantFiltration
to work as intended (ie., roll back all variants to unfiltered status) (#6412) - Fixed a bug where
SelectVariants
takes forever on many-allelic somatic samples (#6446) - Make sure
SelectVariants
outputs variants in correct order (assuming input vcf is correctly sorted) (#6444) - Fixed a NPE crash in
VariantEval
when run with no intervals/reference (#6283) - Fixed a NPE crash in
FastaReferenceMaker
(#6435) - Fixed an out-of-bounds error in
CountNs
annotation (#6355) - Fixed a bug in hardClipCigar function that caused incorrect cigar calculation (#6280)
AnalyzeSaturationMutagenesis
: fixed bug in codon calling for in-frame inserts (#6332)
- Handle zero-weight Gaussians correctly in
-
Miscellaneous Changes
- Collect split read and paired end evidence files for GATK-SV pipeline (#6356)
- Add "PASS" filter line for
ApplyVQSR
andFilterMutectCalls
(#6436) - Added engine functionality for accessing the user defined intervals without merging them (#5887)
- Trim intervals loaded from interval files. (#6375)
- Propagate read group filters in
ReadGroupBlackListReadFilter
. (#6300) - Modified ANDed read filter output message for readability (#6315)
- Clearly label the number of reads processed in the
BaseRecalibrator
log output (#6447) - Clearly label the
CountReads
tool output (#6449) - Improved the error messages for missing contigs in the reference (#6469)
- Avoid a copy and reverse operation in
CigarUtils.isGood()
(#6439) - Fixed
GenotypeAlleleCount
's toString() method (#6376) - Minor Funcotator WDL updates. (#6326)
- Added a
getPairOrientation()
method toGATKRead
(#6420) - Merged
GATKProtectedVariantContextUtils
methods into other classes (#6409) - Deleted a lot of unused VCF constants (#6361)
- Deleted some unused genotyping code (#6354)
- Fixed incoherent unit test cases in allele subsetting utils (#6448)
- Add Python script executor error message for SIGKILL exit code 137. (#6414)
- Pip install pinned numpy. (#6413)
- Do not install R on travis, and only run the R tests on the Docker. (#6454)
- Fixes for
IndexFeatureFile
error reporting. (#6367) - Temporarily remove dead Berkeley mirror to unblock builds. (#6422)
- Disable CNNVariantPipelineTest.testTrainingReadModel until failures are resolved. (#6331)
- Delete unused JsonSerializer (#6415)
- Delete empty file SparkToggleCommandLineProgram.java. (#6311)
-
Documentation
- Clarify the definition of the
NON_REF
allele (#6431) - Clarify behavior of
SplitIntervals
for lists of adjacent intervals (#6423) - Update docs to reflect the fact that
TandemRepeat
works withHaplotypeCaller
(#5943) - Update LeftAlignIndels documentation (#6177)
- Update hyperlink to new GATK forum page in the README (#6381)
- Add minValue/minRecommended value to ApplyBQSRArgumentCollection (#6438)
- Small README fixes (#6451)
- Fix some GATK doc issues (#6318)
- Update copyright date in LICENSE.TXT (#6383)
- Clarify the definition of the
-
Dependencies
4.1.4.1
Download release: gatk-4.1.4.1.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/
Highlights of the 4.1.4.1 release:
- New experimental
HaplotypeCaller
assembly mode which improves phasing, reduces false positives, improves calling at complex sites, and has 15-20% speedup vs the current assembler. It is enabled with option--linked-de-bruijn-graph
. This mode is still experimental and not recommended for production use yet. IndexFeatureFile
improvements:- now cloud enabled
- changed controversial
F
argument toI
instead.
- Bug fixes and improvements in
GenomicsDB
,Mutect2
, variant annotation, and more!
Full list of changes:
-
New Tools
PrintReadsHeader
: a new tool to print a BAM/SAM/CRAM header to a file (#6153)
-
HaplotypeCaller
-
Mutect2
Mutect2
now warns but does not fail when three or more reads have the same name. (#6240)- Fixed the random seed at the beginning of
FilterMutectCalls
(#6208) GetSampleName
andGetPileupSummaries
in the M2 pipeline are no longer beta. (#6215)- Increase number of iterations in
CalculateContamination
to 30. (#6282) - Handled an edge case with high scatter count in M2 WDL. (#6216)
- Use ArgumentsBuilder in M2 tests. (#6219)
-
Joint Calling
-
CNV Calling
- Fixed model parameter assignment typo in gCNV ploidy model (#6285)
- Added docker option to the gcnv QC tasks. (#6185)
- Added epsilons to overdispersion in gCNV models to avoid NaNs. (#6245) #4824 #6226 #6227
- Assert that ELBO did not become NaN during each step of inference of gCNV. (#6186)
- Added ability to override
THEANO_FLAGS
environment variable in gCNV tools. (#6244) #6235 - Removed erroneous short argument names in R scripts for CNV plotting. (#6197)
-
GenomicsDB
- Allow GATK to configure annotation processing instead of hardcoding values in GenomicsDB GDB-39
- High ploidy sites with many genotypes no longer causes an overflow error. GDB-54
- Add missing libcurl in the native GenomicsDB library. #6122 GDB-66
- No longer crashes when vcfbufferstream from htslib appears to be invalid. GDB-67
- Propagated native GenomicsDB exceptions as java IOExceptions. GDB-68
- Fix issue when using vid protobuf interface and there is more than 1 config. GDB-70
- Cleanup GenomicsDB vid combine protobuf mapping overrides. #6190
-
Miscellaneous Changes
- Cloud-enable
IndexFeatureFile
and change input arg name from -F to -I. (#6246) #6161 - WDL to run
ReadsPipelineSpark
on a multicore machine (#6213) - Replace
TwoPassReadWalker
with more generalMultiplePassReadWalker
. (#6154) - Abolish unfilled likelihoods and revamp
VariantAnnotator
. (#6172) - Improve exception message in
ValidateVariants
. (#6076) - Fix Syntax Warning when running GATK with python 3.8 (#6231)
- Cloud-enable
-
Developer / Testing
-
Documentation
-
Dependencies