Skip to content

hydrator: hydrator.metadata git note can prevent necessary Application hydration #26154

@epini-dw

Description

@epini-dw

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

Newly-added Applications which share the same DRY source and hydration target with another Application will not be hydrated if the other Application has already hydrated the latest DRY commit. This is a regression caused by 6cd30d3 introducing a change to the commit-server where it will skip creating the commit when the DRY commit's SHA matches the one recorded in the hydrator.metadata git note.

When using a single repo with deployments that share the same DRY source and sync source, this can cause an unexpected behavior where newly-added Applications will appear to have been successfully hydrated but will fail to sync because their generated manifests haven't been committed.

As an aside, we also run a fairly unique setup where the hydrator.metadata git note causes an additional issue:

  • We run 2 Argo CD instances, each targeting a subset of clusters (e.g. staging-argo for staging, prod-argo for prod). This is intentional partitioning.
  • Both instances share the same drySource (repo + main branch) and syncSource (manifests branch).
  • When the hydrator is enabled, if staging-argo hydrates a DRY commit, it prevents prod-argo from correctly hydrating its own apps.
  • The hydration statuses will show as updated for the prod-argo-managed apps, but the manifests will be stale.

To Reproduce

It's most easily reproduced manually:

  1. In the DRY source (main branch), create a commit containing a Helm chart, app-1-values.yaml, and app-2-values.yaml
  2. Separately, apply App/one configured with drySource=main, syncSource=hydrated. The hydrator will run and create a new commit on the hydrated branch.
  3. Apply another App/two also configured with drySource=main, syncSource=hydrated. The hydrator will run, but no new commit will be created and App/two's sync will fail.

It can also happen as the result of a race when using an ApplicationSet or the app-of-apps pattern:

  • If a commit or series of commits is made and App/one is hydrated (e.g. by a user) before the applicationset-controller creates App/two.
  • If the app-of-apps also uses the Source Hydrator in a way where the commit has to be hydrated before the App/two can be created and synced.

To demonstrate, I created a repo with some concrete examples: https://github.com/epini-dw/argocd-bug-repro-1

Expected behavior

When a new Application is added, Argo CD will hydrate the new app's manifests even if the DRY commit was already hydrated for a different app.

Screenshots

N/A

Version

argocd: v3.3.0-rc3
  BuildDate: 2025-12-18T23:36:38Z
  GitCommit: 51b595b1ee6b3bfc04ca0a648f94a7d69579faee
  GitTreeState: clean
  GitTag: v3.3.0-rc3
  GoVersion: go1.25.3
  Compiler: gc
  Platform: linux/arm64

Logs

logs.txt

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcomponent:hydratorIssue related to the Source Hydrator and commit serverregressionBug is a regression, should be handled with high priority

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions