FlavorFungibility: replace FlavorFungibilityImplicitPreferenceDefault feature gate with API

[vladikkuzn]

What type of PR is this?

/kind api-change

What this PR does / why we need it:

Which issue(s) this PR fixes:

Part of #6184

Special notes for your reviewer:

Does this PR introduce a user-facing change?

FlavorFungibility: introduce the ClusterQueue's API for flavorFungibility: `.spec.flavorFungibility.preference` to indicate
the user's preference for borrowing or preemption when there is no flavor which avoids both. This new field is a replacement for the alpha feature gate FlavorFungibilityImplicitPreferenceDefault which will be removed in 0.16.

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	d669ce0
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/6915d3bcd3fe2e0008e78266

[vladikkuzn]

/assign

[vladikkuzn]

/ok-to-test

[vladikkuzn]

/retest

[mimowo]

Thank you 👍
/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 7a3eb9372dcc5a1d80bcdbe17810b5caa4a96606

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mimowo, vladikkuzn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mimowo]

/hold

[mimowo]

I synced with @vladikkuzn that there is one suspicious test which causes endless loop when the FG is enabled. @vladikkuzn please share more details.

[mimowo]

/test pull-kueue-test-e2e-multikueue-main
this is unrelated, opened: #7650

[vladikkuzn]

Here's the test

With FlavorFungibilityImplicitPreferenceDefault enabled, the scheduler keeps trying flavor f2 instead (logs repeatedly show “insufficient unused quota for cpu in flavor f2 … Pending the preemption of 1 workload”), so the expected admission on f1 never arrives and the Eventually times out.

Steps to reproduce: enable FlavorFungibilityImplicitPreferenceDefault feature gate and run make test-integration GINKGO_ARGS="--focus \"finds correct flavor by discarding the first one in which preemption is not possible\""

Now I think this is expected behaviour, so we can proceed without creating an issue

[mimowo]

cc @pajakd wdyt?

[pajakd]

This is correct.

If FlavorFungibilityImplicitPreferenceDefault would be enabled, the test "finds correct flavor by discarding the first one in which preemption is not possible" should fail because this high-priority workload would choose f2 instead of f1 as we expect in the test:

kueue/test/integration/singlecluster/scheduler/scheduler_test.go

Lines 2670 to 2674 in 47397f8

	cq2HighPriority := createWorkloadWithPriority("cq1", "1", 9999)
	{
	admission := utiltestingapi.MakeAdmission("cq1").PodSets(utiltestingapi.MakePodSetAssignment(kueue.DefaultPodSetName).Assignment(corev1.ResourceCPU, "f1", "1").Obj()).Obj()
	util.ExpectWorkloadToBeAdmittedAs(ctx, k8sClient, cq2HighPriority, admission)
	}

[mimowo]

/unhold
lets see if there are still no conflicts

[vladikkuzn]

/retest-required

[mimowo]

/remove-kind documentation
/kind feature

[mimowo]

proposal:
/release-note-edit

FlavorFungibility: introduce the ClusterQueue's API for flavorFungibility: `.spec.flavorFungability.preference` to indicate
the user's preference for borrowing or preemption when there is no flavor which avoids both. This new field is a replacement for the alpha feature gate FlavorFungibilityImplicitPreferenceDefault which will be removed in 0.16.

[mbobrovskyi]

/release-note-edit

FlavorFungibility: introduce the ClusterQueue's API for flavorFungibility: `.spec.flavorFungibility.preference` to indicate
the user's preference for borrowing or preemption when there is no flavor which avoids both. This new field is a replacement for the alpha feature gate FlavorFungibilityImplicitPreferenceDefault which will be removed in 0.16.

[mbobrovskyi]

/retitle FlavorFungibility: replace FlavorFungibilityImplicitPreferenceDefault feature gate with API