Higher-priority workloads are blocked while a “sticky” low-priority workload is pinned to the head of the queue

[ichekrygin]

Related issues and PRs:

• Root cause discussion: #6929
• Sticky workload introduction: #7157

Summary

After the “sticky pending workload” fix introduced in #7157, higher-priority workloads may become blocked behind a lower-priority workload that is pinned to the head of the queue.

The intent of the sticky behavior was to prevent starvation by keeping the workload that initiated preemption at the queue head until its preemption sequence completes. However, this also prevents newly added, higher-priority workloads from being scheduled while the sticky workload is still waiting for preemption to finish.

Setup

The setup is similar to #6929.

ClusterQueues:

• cq1:

  • nominal CPU = 3

  • borrowingLimit = 0

  • preemption:

    preemption:
      reclaimWithinCohort: Any
      withinClusterQueue: LowerPriority

• cq2: nominal CPU = 0, borrowingLimit = 3

LocalQueues:

• lq1 and lq2, corresponding to cq1 and cq2

PriorityClasses:

• high = 10000
• low = 1000

Jobs:

Job	CPU	Priority	ClusterQueue
j01	2	(default)	cq2
j02	4	low	cq1
j03	2	low	cq1
j04	2	high	cq1

---

Repro Steps

1. Submit j01 → admitted
2. Submit j02 → pending (exceeds available CPU)
3. Submit j03, then j04 in order

Observed behavior:

• j03 enters scheduling cycle first and triggers preemption of j01
• Due to sticky logic, j03 is pinned at the head of the queue until preemption completes
• Meanwhile, j04 (higher priority) cannot proceed
• After several scheduling cycles j03 is eventually admitted
• When j04 runs next, it preempts j03

Expected behavior

When a higher-priority workload (j04) enters the queue, it should preempt or take precedence over lower-priority workloads, even if one of them is currently “sticky.” The sticky behavior should not override priority-based scheduling order.

Impact

This can lead to:

• Unintended scheduling delays for higher-priority workloads
• Inversion of expected priority ordering
• Reduced responsiveness of the scheduler under preemption-heavy workloads
• Unordered scheduling may result in preemption churn, as illustrated in the example where j03 blocks j04, with eventual j04 preemption of j03

[amy]

cc/ @mimowo @gabesaba @mwysokin

[amy]

@ichekrygin Mainly concerned about this part: "Unordered scheduling may result in preemption churn, as illustrated in the example where j03 blocks j04, with eventual j04 preemption of j03"

To clarify, do you see rapid preemption churn in your repro? 🙏 thank you for further validating the original fix.

[ichekrygin]

To clarify, do you see rapid preemption churn in your repro? 🙏 Thank you for further validating the original fix.

The key point here is that if the scheduler correctly respects priority order, then j04 should precede j03 and therefore should not require an additional preemption in this specific example.

It’s important to recognize that if our choices are limited to either:
a. a “head-of-the-queue” blocker, or
b. suboptimal scheduling as a result of “sticky” workloads,
I would take (b) over (a) any time.

That said, I think we should explore potential scheduling and queuing designs that avoid being constrained to these two options altogether.

[mimowo]

Thank you for reporting, I believe there is indeed some potential for improvement, but I think it requires more careful investigation, so that we do not to re-introduce the bugs fixed by the "sticky" head approach.

If I remember correctly the main issue with considering other workloads during preemption is that it may lead to re-admitting the just preempted workloads, because they can be winning in terms of DRS with the "new head" even though they were losing in terms of DRS with the previous head. This scenario of losing DRS to one "head" workload, but winning DRS to another "head" workload was the root cause for infinite preemption loops.

I assume this follow up issue is less severe than the original ones, but let me know if you think the situation is worsened on your setup?

[ichekrygin]

I assume this follow up issue is less severe than the original ones,

💯 correct (IMHO).

Update: less severe

[mimowo]

cc @mwysokin who may also help to prioritize

[mimowo]

Sorry, correct that it is worse or correct that less severe?

EDIT: ah, ok I see update that it is "less severe" . This is great, but we should still think how to coverage on something better.

[amy]

The key point here is that if the scheduler correctly respects priority order, then j04 should precede j03 and therefore should not require an additional preemption in this specific example.

To clarify, you notice this resulting in what looks like 1 extra preemption. But does not result in: infinite preemption loops.

[ichekrygin]

To clarify, you notice this resulting in what looks like 1 extra preemption. But does not result in: infinite preemption loops.

That is correct.