-
Notifications
You must be signed in to change notification settings - Fork 550
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Description
Use metadata check to replace. This metadata check will contains all the parquet metadata check.
val PARQUET_UNEXPECTED_METADATA_FALLBACK_ENABLED =
buildConf("spark.gluten.sql.fallbackUnexpectedMetadataParquet")
.doc("If enabled, Gluten will not offload scan when unexpected metadata is detected.")
.booleanConf
.createWithDefault(false)
val PARQUET_UNEXPECTED_METADATA_FALLBACK_FILE_LIMIT =
buildConf("spark.gluten.sql.fallbackUnexpectedMetadataParquet.limit")
.doc("If supplied, metadata of `limit` number of Parquet files will be checked to" +
" determine whether to fall back to java scan.")
.intConf
.checkValue(_ > 0, s"must be positive.")
.createWithDefault(10)
And the fileLimit config is for the FileSourceScanExec now it is rootPaths * fileLimit config, fix to global check
Gluten version
None
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request