Skip to content

Conversation

@mbobrovskyi
Copy link
Contributor

@mbobrovskyi mbobrovskyi commented Oct 16, 2025

What type of PR is this?

/kind bug

What this PR does / why we need it:

Update PriorityClass only for WorkloadPriorityClassSource.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fixed a bug that Kueue would keep sending empty updates to a Workload, along with sending the "UpdatedWorkload" event, even if the Workload didn't change. This would happen for Workloads using any other mechanism for setting
the priority than the WorkloadPriorityClass, eg. for Workloads for PodGroups.

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. labels Oct 16, 2025
@netlify
Copy link

netlify bot commented Oct 16, 2025

Deploy Preview for kubernetes-sigs-kueue canceled.

Name Link
🔨 Latest commit 7359d08
🔍 Latest deploy log https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/68f120d12c214300084de0fe

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 16, 2025
@mbobrovskyi mbobrovskyi force-pushed the fix/dont-update-workload-for-priority-class branch from 39276ef to 30790cd Compare October 16, 2025 16:30
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 16, 2025
@mbobrovskyi mbobrovskyi force-pushed the fix/dont-update-workload-for-priority-class branch from 30790cd to ce58bf5 Compare October 16, 2025 16:31
@mbobrovskyi
Copy link
Contributor Author

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 16, 2025
@mbobrovskyi mbobrovskyi force-pushed the fix/dont-update-workload-for-priority-class branch from ce58bf5 to 7359d08 Compare October 16, 2025 16:43
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Oct 16, 2025
@mbobrovskyi
Copy link
Contributor Author

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 16, 2025
@mbobrovskyi
Copy link
Contributor Author

/cc @mimowo

// update workload priority if job's label changed
if WorkloadPriorityClassName(object) != wl.Spec.PriorityClassName {
// Update workload priority if job's label changed.
if wl.Spec.PriorityClassSource == constants.WorkloadPriorityClassSource && WorkloadPriorityClassName(object) != wl.Spec.PriorityClassName {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't wl.Spec.PriorityClassSource == constants.WorkloadPriorityClassSource implied? How would the WorkloadPriorityClassName(object) change if WorkloadPriorityClassSource wasn't used?

Do you mean a user manually changing the Workload object, or it could happen somehow by mutating the Job?

I'm asking cause I want to make the scenario clear in the release notes. For now it seems like impossible scenario for a user to face (at least not clear to me).

Copy link
Contributor Author

@mbobrovskyi mbobrovskyi Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it happens even if nothing has changed. We’re checking whether WorkloadPriorityClassName(object) != wl.Spec.PriorityClassName. But what if we have PodPriorityClassSource? In that case, "" != "test-priority-class", which evaluates to true every time, so the workload gets updated on each reconciliation.

You can check the unit test I created — if we revert this change, we’ll see an event with the UpdatedWorkload reason during every reconciliation.

image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, so this is a serious bug. I think for users using WorkloadPriorityClass it is safe, it is actually problematic for Pod's priority users.

We should reflect that better in the release note for sure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for users using WorkloadPriorityClass it is safe

Yes, you’re right — this bug affects only Pod priority users.

@mimowo
Copy link
Contributor

mimowo commented Oct 16, 2025

proposal:
/release-note-edit

Fixed a bug that Kueue would keep sending empty updates to a Workload, along with sending the "UpdatedWorkload" event, even if the Workload didn't change. This would happen for Workloads using any other mechanism for setting
the priority than the WorkloadPriorityClass, eg. for Workloads for PodGroups.

When was that introduced, probably only 0.14 is affected?
/cherrypick release-0.14
cc @amy

@k8s-infra-cherrypick-robot
Copy link
Contributor

@mimowo: once the present PR merges, I will cherry-pick it on top of release-0.14 in a new PR and assign it to you.

In response to this:

proposal:
/release-note-edit

Fixed a bug that Kueue would keep sending empty updates to a Workload, along with sending the "UpdatedWorkload" event, even if the Workload didn't change. This would happen for Workloads using any other mechanism for setting
the priority than the WorkloadPriorityClass, eg. for Workloads for PodGroups.

When was that introduced, probably only 0.14 is affected?
/cherrypick release-0.14
cc @amy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@mbobrovskyi
Copy link
Contributor Author

mbobrovskyi commented Oct 16, 2025

When was that introduced, probably only 0.14 is affected?

It was introduced in #5197. I think we should also cherry-pick it to release-0.13.

@mbobrovskyi
Copy link
Contributor Author

/cherrypick release-0.13

@k8s-infra-cherrypick-robot
Copy link
Contributor

@mbobrovskyi: once the present PR merges, I will cherry-pick it on top of release-0.13 in a new PR and assign it to you.

In response to this:

/cherrypick release-0.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@mbobrovskyi
Copy link
Contributor Author

mbobrovskyi commented Oct 16, 2025

It was updated in #5241, but the issue exists in both PRs.

/cc @IrvingMg

@k8s-ci-robot k8s-ci-robot requested a review from IrvingMg October 16, 2025 17:46
@amy
Copy link
Contributor

amy commented Oct 16, 2025

@mbobrovskyi Can I also ask how you noticed this bug / how we can reproed it for older versions without the fix?

Checking which priority classes we use.

@mbobrovskyi
Copy link
Contributor Author

@mbobrovskyi Can I also ask how you noticed this bug / how we can reproed it for versions without the fix?

Just create a job with a Pod PriorityClass. I also added a unit test — you can check it. It doesn’t require any changes. From the user’s perspective, they will only see new events, which could be confusing since nothing actually changed. We are updating the same workload with the same values.

@mimowo
Copy link
Contributor

mimowo commented Oct 17, 2025

Thanks 👍
/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 17, 2025
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 10374a1f3dc510c9cc03dd28a31937f2ab21b21e

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mbobrovskyi, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 17, 2025
@k8s-ci-robot k8s-ci-robot merged commit bb2b22f into kubernetes-sigs:main Oct 17, 2025
23 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v0.15 milestone Oct 17, 2025
@k8s-infra-cherrypick-robot
Copy link
Contributor

@mimowo: new pull request created: #7302

In response to this:

proposal:
/release-note-edit

Fixed a bug that Kueue would keep sending empty updates to a Workload, along with sending the "UpdatedWorkload" event, even if the Workload didn't change. This would happen for Workloads using any other mechanism for setting
the priority than the WorkloadPriorityClass, eg. for Workloads for PodGroups.

When was that introduced, probably only 0.14 is affected?
/cherrypick release-0.14
cc @amy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-infra-cherrypick-robot
Copy link
Contributor

@mbobrovskyi: new pull request created: #7303

In response to this:

/cherrypick release-0.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@mimowo
Copy link
Contributor

mimowo commented Oct 17, 2025

The cherry picks need to be prepared manually

@mbobrovskyi mbobrovskyi deleted the fix/dont-update-workload-for-priority-class branch October 17, 2025 07:19
k8s-ci-robot pushed a commit that referenced this pull request Oct 17, 2025
…dPriorityClassSource. (#7306)

* Update PriorityClass only for WorkloadPriorityClassSource.

* Fix after cherry-pick.
k8s-ci-robot pushed a commit that referenced this pull request Oct 17, 2025
…dPriorityClassSource. (#7305)

* Update PriorityClass only for WorkloadPriorityClassSource.

* Fix after cherry-pick.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants