pass dest project id #9023
pass dest project id #9023
Conversation
| @@ -78,7 +78,8 @@ workflow GvsExtractAvroFilesForHail { | |||
|
|
|||
| call Utils.IsUsingCompressedReferences { | |||
| input: | |||
| project_id = project_id, | |||
| query_project_id = project_id, | |||
| dest_project_id = project_id, | |||
There was a problem hiding this comment.
I did not see Hail contains dest project and I don't know the use case. So I just pass the same project
There was a problem hiding this comment.
Can we make one of the project_ids optional then? I dont imagine we will ever extract avro files into another project
There was a problem hiding this comment.
actually this is fine--let's just get it working!
| @@ -811,7 +812,7 @@ task IsUsingCompressedReferences { | |||
| SELECT | |||
| column_name | |||
| FROM | |||
| `~{dataset_name}.INFORMATION_SCHEMA.COLUMNS` | |||
| `~{dest_project_id}.~{dataset_name}.INFORMATION_SCHEMA.COLUMNS` | |||
There was a problem hiding this comment.
actually I don't know wdl. Is this syntax correct?
There was a problem hiding this comment.
Why is query_project_id being introduced here if it's never used? Possibly related, line 811 is still referencing project_id which no longer exists.
womtool validate will check the syntactical correctness of WDL for you, and is among the actions that execute automatically against ah_var_store PRs in this repo. I'm not sure why it didn't flag this error. 🤔
There was a problem hiding this comment.
Ah, good finding! It is line 811
| @@ -811,7 +812,7 @@ task IsUsingCompressedReferences { | |||
| SELECT | |||
| column_name | |||
| FROM | |||
| `~{dataset_name}.INFORMATION_SCHEMA.COLUMNS` | |||
| `~{dest_project_id}.~{dataset_name}.INFORMATION_SCHEMA.COLUMNS` | |||
There was a problem hiding this comment.
Why is query_project_id being introduced here if it's never used? Possibly related, line 811 is still referencing project_id which no longer exists.
womtool validate will check the syntactical correctness of WDL for you, and is among the actions that execute automatically against ah_var_store PRs in this repo. I'm not sure why it didn't flag this error. 🤔
SG! I also pushed this to Agora and will test it out using our genomic extraction workflow |
The current
IsUsingCompressedReferencesIt does not pass project id that hosted dataset.
When the project_id is missing in a BigQuery SQL query, the bq command will use the --project_id flag specified in the command as the default project for resolving dataset and table references.
Add additional parameter to allow passing dest project.
In our case
Error we saw in GCP console:
terra-vpc-sc-dev-7ee328ad:1kg_wgs_2022q1.INFORMATION_SCHEMA.COLUMNSis wrong -terra-vpc-sc-dev-7ee328adis the user workspaceIt should be
fc-aou-cdr-synth-test-2.1kg_wgs_2022q1-fc-aou-cdr-synth-test-2is the project that contains CDR data.