Skip to content

Application Controller higher CPU in 3.X upgrade #25517

@rumstead

Description

@rumstead

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

Related issues #24167, #17277, #23965

I am noticing a performance regression in v3.x after upgrading from 2.14, where syncs are taking minutes instead of seconds in Argo CD 3.x. CPU profiling shows 62% of time spent in GetDestinationCluster.

Image Image

#21189 replaced ValidateDestination with GetDestinationCluster.

ValidateDestination was pretty lightweight, but with GetDestinationCluster, it's parsing secrets, decoding JSON, etc. Every 10 seconds, for each cluster, we check every app to count how many belong to that cluster. This results in N (apps) × M (clusters) calls to GetDestinationCluster.

Even though the db uses informer caching, GetDestinationCluster has significant overhead when called thousands of times per cycle.

To Reproduce

Expected behavior

Screenshots

Version

❯ argocd version
argocd: v3.2.1+8c4ab63
  BuildDate: 2025-12-01T09:30:10Z
  GitCommit: 8c4ab63a9c72b31d96c6360514cda6254e7e6629
  GitTreeState: clean
  GoVersion: go1.25.4
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v3.2.1+8c4ab63
  BuildDate: 2025-11-30T11:48:14Z
  GitCommit: 8c4ab63a9c72b31d96c6360514cda6254e7e6629
  GitTreeState: clean
  GoVersion: go1.25.0
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.7.0 2025-06-28T07:01:13Z
  Helm Version: v3.18.4+gd80839c
  Kubectl Version: v0.34.0
  Jsonnet Version: v0.21.0

Logs

Paste any relevant application logs here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcomponent:application-controllerIssues related to the Application Controller componenttriage/pendingThis issue needs further triage to be correctly classifiedtype:scalabilityIssues related to scalability and performance related issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions