Skip to content

Conversation

@yuzhichang
Copy link
Contributor

@yuzhichang yuzhichang commented Dec 2, 2025

When pushing images to a remote host, especially with newer versions of containerd (e.g., v2.0+ or Docker 27+), concurrent writes to the containerd content store can result in locking errors. Users may see errors like:
'rpc error: code = Unavailable desc = ref ... locked'

This commit introduces a mutex to serialize all write operations to the containerd writer. To maintain performance, 'ReadFrom' is implemented manually to perform network reads outside the lock, ensuring that data transfer remains parallel while the final write to disk/containerd is serialized. This prevents the locking contention while keeping upload speeds high.

Related container issues:

containerd error log I saw:

2025-12-02T09:45:26.645000+08:00 inf28 containerd[890]: time="2025-12-02T09:45:26.644700618+08:00" level=error msg="(*service).Write failed" error="rpc error: code = Unavailable desc = ref moby/1/upload-a76d0865-75d4-4334-b425-24525cf04bf5 locked for 2.351445767s (since 2025-12-02 09:45:24.293229297 +0800 CST m=+950707.661078481): unavailable" ref=upload-a76d0865-75d4-4334-b425-24525cf04bf5
2025-12-02T09:45:28.452469+08:00 inf28 containerd[890]: time="2025-12-02T09:45:28.452187783+08:00" level=error msg="(*service).Write failed" error="rpc error: code = Canceled desc = context canceled" ref=upload-90eb9238-2b2a-45d0-9ba6-5307aeeabff1

When pushing images to a remote host, especially with newer versions of
containerd (e.g., v2.0+ or Docker 27+), concurrent writes to the
containerd content store can result in locking errors. Users may see
errors like:
'rpc error: code = Unavailable desc = ref ... locked'

This commit introduces a mutex to serialize all write operations to the
containerd writer. To maintain performance, 'ReadFrom' is implemented
manually to perform network reads outside the lock, ensuring that data
transfer remains parallel while the final write to disk/containerd is
serialized. This prevents the locking contention while keeping upload
speeds high.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant