Copy benchmark extremely inaccurate, varies radically at default 5s.

[icscript]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

[2nd UPDATE - December 4 PM]: Root Cause Identified - Insufficient Memory Benchmark Duration
After systematic testing, I've identified the root cause: the default 5-second memory benchmark duration is inadequate and produces unreliable results.
Key Finding: Extending --memory-duration from 5 to 10+ seconds completely eliminates the variance and failures.
Test Results (using --memory-duration X --disk-duration 1 --hash-duration 1 --verify-duration 1):
=== Memory Duration: 5s ===
| Memory | Copy | 17.10 GiBs | 11.49 GiBs | ✅ Pass (148.8 %) |
| Memory | Copy | 4.94 GiBs | 11.49 GiBs | ❌ Fail ( 43.0 %) |
| Memory | Copy | 5.14 GiBs | 11.49 GiBs | ❌ Fail ( 44.7 %) |

=== Memory Duration: 10s ===
| Memory | Copy | 17.37 GiBs | 11.49 GiBs | ✅ Pass (151.2 %) |
| Memory | Copy | 17.26 GiBs | 11.49 GiBs | ✅ Pass (150.2 %) |
| Memory | Copy | 17.11 GiBs | 11.49 GiBs | ✅ Pass (148.9 %) |

=== Memory Duration: 20s ===
| Memory | Copy | 16.52 GiBs | 11.49 GiBs | ✅ Pass (143.8 %) |
| Memory | Copy | 17.10 GiBs | 11.49 GiBs | ✅ Pass (148.8 %) |
| Memory | Copy | 17.14 GiBs | 11.49 GiBs | ✅ Pass (149.1 %) |

=== Memory Duration: 30s ===
| Memory | Copy | 17.13 GiBs | 11.49 GiBs | ✅ Pass (149.0 %) |
| Memory | Copy | 17.21 GiBs | 11.49 GiBs | ✅ Pass (149.8 %) |
| Memory | Copy | 16.69 GiBs | 11.49 GiBs | ✅ Pass (145.2 %) |

Statistics:
5-second duration testing (50+ total runs across multiple sessions):

• Overall failure rate: ~25-30%

• Failures show 3.5-4x variance (4.47-18.78 GiBs range)

• Pass results are consistent: 16-18 GiBs

• Representative samples:

• • Session 1: 67% failure rate (2/3 failed)
• • Session 2: 40% failure rate (2/5 failed)
• • Session 3: 20% failure rate (3/15 failed)
• • Session 4: 20% failure rate (2/10 failed)

10+ second duration testing:

• 100% pass rate (9/9 passed)
• Consistent results: 16.52-17.37 GiBs (5% variance)
• No failures observed across any test session

Conclusion:
The hardware consistently delivers 16-17 GiBs memory copy performance when given adequate benchmark time. The 5-second default is insufficient, likely not allowing enough time for the test to stabilize past transient kernel activity or cache/memory subsystem warmup effects.
Recommendation:
The default --memory-duration should be increased from 5 to at least 10 seconds to ensure reliable results. The current default incorrectly flags capable hardware as inadequate.
Note: The startup benchmark appears to use the same 5-second default and does not accept custom duration parameters, which explains why validator nodes with adequate hardware are experiencing spurious Copy benchmark failures at startup.

Original issue details preserved below for reference. Initial hypothesis about startup process contention was reasonable given symptoms, but systematic testing revealed the actual issue is insufficient benchmark duration.

[1st UPDATE - Dec 4]: Initial disk benchmarks were comparing ramdisk (/tmp) to NVMe. After running benchmark machine --base-path against actual storage, Rnd Write aligns with startup values (~402 MiBs). However, this confirms the Copy (memory) benchmark shows significant non-deterministic behavior during startup, corroborated by community reports below.

Issue occurs on multiple nodes of different hardware spec -- all very fast specs far exceeding minimum threshold on 'benchmark machine'. Issue occurs during reboots of the server as well as simple restarts of the polkadot service (eliminating external server startup contention as a possibility).

It's possible the issue is due to concurrency/contention between the startup benchmark and other polkadot startup processes. I imagine the intention was for the startup benchmark to run with nothing else in parallel, however perhaps with the evolution of the code and the many processes involved, that's no longer the case.

I believe this because:

A) The polkadot log flow at startup shows log output from startup processes proceeding the benchmark, meaning polkadot is doing other work before/during the benchmark begins. Example:

23:59:23.259  INFO main sc_cli::runner: 👤 Role: AUTHORITY
23:59:23.259  INFO main sc_cli::runner: 💾 Database: ParityDb at /var/lib/polkadot/chains/polkadot/paritydb/full
23:59:24.431 DEBUG main sc_client_db: Initializing shared trie cache with size 1073741824 bytes, 0.80070156685443% of total memory
23:59:24.607  INFO main txpool: Creating transaction pool txpool_type=ForkAware ready=Limit { count: 8192, total_bytes: 20971520 } future=Limit { count: 819, total_bytes: 2097152 }
23:59:25.071  INFO main sc_service::builder: 📦 Highest known block at #28805690
23:59:25.072  WARN main polkadot_service::builder: ⚠️  The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 5.58 GiBs),  for role 'Authority' find out more at:

You can see things starting and posting to the log prior to the benchmark posting its results. If the benchmark is to be accurate, nothing whatsoever should start until after the benchmark is complete and has posted its result. In reality it seems that amongst other things, the database trie cache is initializing before the benchmark, which could imply the database is started before the benchmark is ran.

B) The most logical explanation is concurrency and contention with other processes given the variance between the startup benchmark and 'benchmark machine' including when ran back to back. Data and examples:

Benchmark Machine:

Benchmark Machine (with --base-path=/var/lib/polkadot):
| Memory   | Copy                  | 16.45 GiBs  | 11.49 GiBs   | ✅ Pass (143.1 %) |

Startup benchmark examples:

Kernel:
⚠️ The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 7.83 GiBs), for role 'Authority'

⚠️ The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 5.58 GiBs), for role 'Authority' find out more at:

⚠️ The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 8.88 GiBs), for role 'Authority'

**

C) The startup benchmark appears to apply stricter pass/fail criteria than 'benchmark machine'. 'benchmark machine' explicitly applies a "10% fault tolerance" and would pass results within 10% which the startup benchmark would fail. This inconsistency suggests the startup benchmark may be using different thresholds or lacks the same tolerance logic, which could result in spurious failures even on hardware that genuinely meets requirements.

Additional Data:

Command line options:

ExecStart=/usr/local/bin/polkadot \
    --validator \
    --database=ParityDb \
    --state-pruning=256 \
    --blocks-pruning=256 \
    --telemetry-url 'wss://telemetry-backend.w3f.community/submit 1' \
    --telemetry-url 'wss://telemetry.polkadot.io/submit 0' \
    --no-mdns \
    --no-private-ip \
    --db-cache 49152 \
    --base-path=/var/lib/polkadot \
    --network-backend litep2p \
    --sync=warp \
    --log=warn,sc_cli=info,txpool=info \

Steps to reproduce

Stop polkadot. Run 'polkadot benchmark machine' several times and observe significantly different results from the Memory Copy test. Example of testing loop:

for i in {1..10}; do
  echo "Run $i:"
  polkadot benchmark machine \
    --base-path=/var/lib/polkadot \
    --memory-duration 5 \
    --disk-duration 1 \
    --hash-duration 1 \
    --verify-duration 1 \
    2>&1 | grep "Memory.*Copy"
  sleep 10
done

• tests we dont need are reduced to one second. Memory duration set at 5 seconds, the same as the default.

Start/Restart polkadot to catch a startup benchmark failure of 'Memory Copy', ideally more than once, note the statistics.

[kukabi]

I confirm that the benchmark at node startup is inaccurate between runs, and varies from benchmark machine.

I see that you are having the inaccuracy in both disk and memory benchmarks. I have had it for memory only, and the modules on the devices are 5600 MHz DDR5 ECC, so it's pretty unrealistic for these to fail the benchmark.

The machines passed benchmark machine consistently, but only once in a while I used to get (still get it sometimes) a memory fail with a very similar result as the one you posted, and the fails mostly happened at the node startup.

As a measure I upgraded all the nodes from Debian 12 to 13, installed sysfsutils, and appended these to /etc/sysfutils.service to improve memory conditions:

kernel/mm/transparent_hugepage/enabled = madvise
kernel/mm/transparent_hugepage/defrag = defer+madvise
kernel/mm/transparent_hugepage/khugepaged/defrag = 0
kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs = 600000
kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs = 600000

As a further measure I run mbw -n 32 2048 between node restarts, somehow seems to reduce the memory benchmark issue at node startup.

But these are all educated guesswork. I have not been able to pinpoint the exact issue.

In summary, the benchmark does not seem to be deterministic between runs.

[florentina57]

I've seen something similar with a new server just last week, however it's not Debian, it's Rocky Linux 10 in this case (kernel 6.12).

Only Memory copy was failing to pass at startup of the node and only sometimes, even though the machine benchmark passes copy with result: 220%

2025-11-29 11:57:19 ⚠️   The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 4.28 GiBs),  for role 'Authority' find out more at:
2025-11-29 11:59:11 ⚠️   The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 6.64 GiBs),  for role 'Authority' find out more at:

$ sudo -u polkadot /usr/local/bin/polkadot benchmark machine -d /srv/tmp
2025-11-28 15:43:36 Running machine benchmarks...
2025-11-28 15:44:06
+----------+-----------------------+-------------+--------------+-------------------+
| Category | Function              | Score       | Minimum      | Result            |
+===================================================================================+
| CPU      | BLAKE2-256            | 1.54 GiBs   | 1000.00 MiBs | ✅ Pass (157.9 %) |
|----------+-----------------------+-------------+--------------+-------------------|
| CPU      | BLAKE2-256-Parallel-8 | 1.58 GiBs   | 1000.00 MiBs | ✅ Pass (162.2 %) |
|----------+-----------------------+-------------+--------------+-------------------|
| CPU      | SR25519-Verify        | 1.15 MiBs   | 637.62 KiBs  | ✅ Pass (184.7 %) |
|----------+-----------------------+-------------+--------------+-------------------|
| Memory   | Copy                  | 25.35 GiBs  | 11.49 GiBs   | ✅ Pass (220.5 %) |
|----------+-----------------------+-------------+--------------+-------------------|
| Disk     | Seq Write             | 4.99 GiBs   | 950.00 MiBs  | ✅ Pass (538.2 %) |
|----------+-----------------------+-------------+--------------+-------------------|
| Disk     | Rnd Write             | 962.41 MiBs | 420.00 MiBs  | ✅ Pass (229.1 %) |
+----------+-----------------------+-------------+--------------+-------------------+
From 6 benchmarks in total, 6 passed and 0 failed (10% fault tolerance).
2025-11-28 15:44:06 The hardware meets the requirements

[ingusw3f]

@icscript
Note that polkadot benchmark machine without extra parameters, performs disk benchmark in /tmp, which, depending on OS, could even be RAM disk, while startup performs it in the actual base-path folder.

Could you repeat your manual benchmark with the proper base path specified?

[icscript]

@icscript Note that polkadot benchmark machine without extra parameters, performs disk benchmark in /tmp, which, depending on OS, could even be RAM disk, while startup performs it in the actual base-path folder.

Could you repeat your manual benchmark with the proper base path specified?

Thank you for catching this! You're absolutely right - my benchmark machine was using /tmp (ramdisk). After running with --base-path pointing to the actual NVMe, the Rnd Write results now align with startup (402 MiBs vs 399-418 MiBs).
However, this actually strengthens the case for a bug in the Copy benchmark specifically. The corrected benchmark shows 16.45 GiBs, but startup varies wildly from 4.16 GiBs (75% lower) to 8.35 GiBs (49% lower). The two other reports in comments confirm similar Copy-specific non-determinism on different hardware/OSes.
I'll update the issue to focus on the Copy benchmark variance and possibly Seq Write (33% lower at startup).

[icscript]

My conclusion is that the Copy benchmark is rendered highly inconsistent when it is limited to only 5 seconds. When it is 10 seconds or longer, it has consistent results 100% of the time.

Solution ideas:

• Make the startup benchmark 10 seconds.
• Allow the user to specify the length of time of startup benchmarks via command line switches similar to the switches in 'benchmark machine'.
• Default 'benchmark machine' to 10 seconds for the Copy benchmark.

When time permits I'm going to try and test with different linux kernel versions.

[infrachris]

90 second wait in between 5s Copy tests. Below represents the typical pattern:

Run 1:
| Memory | Copy | 18.25 GiBs | 11.49 GiBs | ✅ Pass (158.8 %) |
Run 2:
| Memory | Copy | 18.44 GiBs | 11.49 GiBs | ✅ Pass (160.4 %) |
Run 3:
| Memory | Copy | 6.73 GiBs | 11.49 GiBs | ❌ Fail ( 58.6 %) |
Run 4:
| Memory | Copy | 18.11 GiBs | 11.49 GiBs | ✅ Pass (157.6 %) |
Run 5:
| Memory | Copy | 18.29 GiBs | 11.49 GiBs | ✅ Pass (159.1 %) |
Run 6:
| Memory | Copy | 17.91 GiBs | 11.49 GiBs | ✅ Pass (155.9 %) |
Run 7:
| Memory | Copy | 18.32 GiBs | 11.49 GiBs | ✅ Pass (159.4 %) |
Run 8:
| Memory | Copy | 8.98 GiBs | 11.49 GiBs | ❌ Fail ( 78.1 %) |
Run 9:
| Memory | Copy | 9.49 GiBs | 11.49 GiBs | ❌ Fail ( 82.6 %) |
Run 10:
| Memory | Copy | 18.35 GiBs | 11.49 GiBs | ✅ Pass (159.7 %) |
Run 11:
| Memory | Copy | 9.98 GiBs | 11.49 GiBs | ❌ Fail ( 86.9 %) |
Run 12:
| Memory | Copy | 18.37 GiBs | 11.49 GiBs | ✅ Pass (159.8 %) |
Run 13:
| Memory | Copy | 18.36 GiBs | 11.49 GiBs | ✅ Pass (159.7 %) |
Run 14:
| Memory | Copy | 18.46 GiBs | 11.49 GiBs | ✅ Pass (160.6 %) |
Run 15:
| Memory | Copy | 18.46 GiBs | 11.49 GiBs | ✅ Pass (160.6 %) |
Run 16:
| Memory | Copy | 6.70 GiBs | 11.49 GiBs | ❌ Fail ( 58.3 %) |
Run 17:
| Memory | Copy | 18.05 GiBs | 11.49 GiBs | ✅ Pass (157.0 %) |
Run 18:
| Memory | Copy | 5.57 GiBs | 11.49 GiBs | ❌ Fail ( 48.5 %) |
Run 19:
| Memory | Copy | 17.97 GiBs | 11.49 GiBs | ✅ Pass (156.4 %) |
Run 20:
| Memory | Copy | 6.61 GiBs | 11.49 GiBs | ❌ Fail ( 57.5 %) |

35% outlier results

System:
Ryzen 9 7950X
4 x 32GB DDR5
Kernel 6.17.9
polkadot 1.20.2