[BUG] ParquetToArrowDecodingHandler fails to resolve s3 paths correctly

[yherin]

Describe the bug

Due to (I believe) a downstream bug in pyarrow, when ParquetToArrowDecodingHandler tries to resolve a path to a parquet file, it fails with a FileNotFoundError.

It seems that the s3:// prefix is not passed to the final step of reading the parquet file, which results in pyarrow trying to load the file from a local path which does not exist.

apache/arrow#31812
The issue contains a description of where in pyarrow the issue occurs.

User workaround:
Use another type that interfaces well with parquet e.g. Polars and its flyte plugin.

Expected behavior

Parquet files can be decoded successfully with ParquetToArrowDecodingHandler even from s3

Additional context to reproduce

Depedencies:

flyteidl==1.13.4 ; python_full_version == "3.10.11"
flytekit==1.13.8 ; python_full_version == "3.10.11"
fs==2.4.16 ; python_full_version == "3.10.11"
fsspec==2024.9.0 ; python_full_version == "3.10.11"
fsspec[s3fs]==2024.9.0 ; python_full_version == "3.10.11"
pyarrow==16.1.0 ; python_full_version == "3.10.11"

1. Use an s3 backend in flyte
2. Define task A which returns a pyarrow.Table object
3. Define task B which accepts that same object as an input
4. Run the workflow, observe that a FileNotFoundError occurs.

Remark: In the error message, the s3:// prefix is missing from the erroneous path

Screenshots

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠