Skip to content

Deprecate parquet encrypted check config #11239

@jinchengchenghh

Description

@jinchengchenghh

Description

Use metadata check to replace. This metadata check will contains all the parquet metadata check.

  val PARQUET_UNEXPECTED_METADATA_FALLBACK_ENABLED =
    buildConf("spark.gluten.sql.fallbackUnexpectedMetadataParquet")
      .doc("If enabled, Gluten will not offload scan when unexpected metadata is detected.")
      .booleanConf
      .createWithDefault(false)

  val PARQUET_UNEXPECTED_METADATA_FALLBACK_FILE_LIMIT =
    buildConf("spark.gluten.sql.fallbackUnexpectedMetadataParquet.limit")
      .doc("If supplied, metadata of `limit` number of Parquet files will be checked to" +
        " determine whether to fall back to java scan.")
      .intConf
      .checkValue(_ > 0, s"must be positive.")
      .createWithDefault(10)

And the fileLimit config is for the FileSourceScanExec now it is rootPaths * fileLimit config, fix to global check

Gluten version

None

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions