Benchmarks
The benchmarks demonstrate the loading performance and sizes of raw CRDT documents (Automerge & Yjs) compared to Secsync snapshot and updates.
The secsync version use encryption/decryption and don't do any of the optional signature checks. The raw CRDT documents are base64 encoded which adds about 30% to the size of the document compared to storing a binary document. This was chosen since often this data is transferred over the network and encoded using base64.
Yjs
10k Changes
This benchmark uses a document with 10k changes and the results show 3 different scenarios:
- The whole document as separate changes
- 9k changes as a snapshot and 1k changes as a separate changes
- The whole document as a snapshot
The results demonstrate that only encrypting changes increases the loading times of a Yjs document and also due the lost compression the size of the document increases. The loading time of a snapshot is significantly faster than loading changes and the size of the document is smaller.
When introducing Secsync the the impact is even larger. When combining Snapshots with Updates it's possible to get to acceptable loading times and sizes.
| Loading time (seconds) | Size (KB) | |
|---|---|---|
| Yjs (Changes) | 63 | 421 |
| Secsync + Yjs (Changes) | 1130 | 4451 |
| Yjs (Snapshot of 9k changes + 1k changes) | 250 | 55 |
| Secsync + Yjs (Snapshot of 9k changes + 1k changes) | 370 | 459 |
| Yjs (Snapshot) | 0.8 | 12.9 |
| Secsync + Yjs (Snapshot) | 1.5 | 13.4 |
Raw Data
Recorded on an Macbook Pro M1, 2020, 16 GB Memory
Snapshot + Updates Decryption: 116.48012506961823 milliseconds.
Snapshot + Updates Doc: 253.57566595077515 milliseconds.
Snapshot + Updates Decryption + Doc: 370.0557910203934 milliseconds.
Changes Decryption: 1072.7890410423279 milliseconds.
Changes Decryption + Doc: 57.17595899105072 milliseconds.
Changes Decryption + Doc: 1129.9650000333786 milliseconds.
Snapshot Decryption: 0.4355419874191284 milliseconds.
Snapshot Doc: 1.0230000019073486 milliseconds.
Snapshot Decryption + Doc: 1.458541989326477 milliseconds.
---
Snapshot with 1000 Changes Base64: 8.481458067893982 milliseconds.
Snapshot with 1000 Changes Doc: 250.2777919769287 milliseconds.
Changes Base64: 27.473582983016968 milliseconds.
Changes Doc: 62.76724994182587 milliseconds.
Snapshot Base64: 1.2406660318374634 milliseconds.
Snapshot Doc: 0.8804590702056885 milliseconds.250k Changes
| Loading time (seconds) | Size (KB) | |
|---|---|---|
| Yjs (Changes) | 1615 | 11234 |
| Secsync + Yjs (Changes) | 29476 | 116317 |
| Yjs (Snapshot of 249k changes + 1k changes) | 151 | 252 |
| Secsync + Yjs (Snapshot of 249k changes + 1k changes) | 277 | 659 |
| Yjs (Snapshot) | 18 | 213 |
| Secsync + Yjs (Snapshot) | 35 | 214 |
Raw Data
Recorded on an Macbook Pro M1, 2020, 16 GB Memory
> node loadYjs2.js
Snapshot with 1000 Changes Base64: 23.62625002861023 milliseconds.
Snapshot with 1000 Changes Doc: 151.11641597747803 milliseconds.
Changes Base64: 718.525083065033 milliseconds.
Changes Doc: 1615.3544169664383 milliseconds.
Snapshot Base64: 15.500166058540344 milliseconds.
Snapshot Doc: 18.124833941459656 milliseconds.
> node loadSecsyncYjs2.js
Snapshot + Updates Decryption: 125.45704102516174 milliseconds.
Snapshot + Updates Doc: 151.64654195308685 milliseconds.
Snapshot + Updates Decryption + Doc: 277.1035829782486 milliseconds.
Changes Decryption: 27888.64362502098 milliseconds.
Changes Decryption + Doc: 1587.4889999628067 milliseconds.
Changes Decryption + Doc: 29476.132624983788 milliseconds.
Snapshot Decryption: 3.7959580421447754 milliseconds.
Snapshot Doc: 31.969749927520752 milliseconds.
Snapshot Decryption + Doc: 35.76570796966553 milliseconds.Automerge
10k Changes
This benchmark uses a document with 10k changes and the results show 3 different scenarios:
- The whole document as separate changes
- 9k changes as a snapshot and 1k changes as a separate changes
- The whole document as a snapshot
The results are similar to the Yjs benchmarks. The loading time of a snapshot is significantly faster than loading changes and the size of the document is smaller.
| Loading time (seconds) | Size (KB) | |
|---|---|---|
| Automerge (Changes) | 68 | 1442 |
| Secsync + Automerge (Changes) | 1162 | 5471 |
| Automerge (Snapshot of 9k changes + 1k changes) | 72 | 152 |
| Secsync + Automerge (Snapshot of 9k changes + 1k changes) | 188 | 557 |
| Automerge (Snapshot) | 39.9 | 9.3 |
| Secsync + Automerge (Snapshot) | 38.2 | 9.8 |
Raw Data
Recorded on an Macbook Pro M1, 2020, 16 GB Memory
Snapshot + Updates Decryption: 120.0347501039505 milliseconds.
Snapshot + Updates Doc: 68.33024990558624 milliseconds.
Snapshot + Updates Decryption + Doc: 188.36500000953674 milliseconds.
Changes Decryption: 1095.1798330545425 milliseconds.
Changes Decryption + Doc: 67.0505839586258 milliseconds.
Changes Decryption + Doc: 1162.2304170131683 milliseconds.
Snapshot Decryption: 0.36016595363616943 milliseconds.
Snapshot Doc: 39.89745903015137 milliseconds.
Snapshot Decryption + Doc: 40.25762498378754 milliseconds.
---
Snapshot with 1000 Changes Base64: 0.6732079982757568 milliseconds.
Snapshot with 1000 Changes Doc: 71.84425008296967 milliseconds.
Changes Base64: 4.018791913986206 milliseconds.
Changes Doc: 67.62387502193451 milliseconds.
Snapshot Base64: 0.031916022300720215 milliseconds.
Snapshot Doc: 38.19574999809265 milliseconds.250k Changes
| Loading time (seconds) | Size (KB) | |
|---|---|---|
| Automerge (Changes) | 1806 | 38725 |
| Secsync + Automerge (Changes) | 31119 | 143835 |
| Automerge (Snapshot of 9k changes + 1k changes) | 1115 | 323 |
| Secsync + Automerge (Snapshot of 9k changes + 1k changes) | 1233 | 729 |
| Automerge (Snapshot) | 1026 | 172 |
| Secsync + Automerge (Snapshot) | 1014 | 173 |
Raw Data
Recorded on an Macbook Pro M1, 2020, 16 GB Memory
> node loadAutomerge.js
Snapshot with 1000 Changes Base64: 0.8246250152587891 milliseconds.
Snapshot with 1000 Changes Doc: 1115.5433340072632 milliseconds.
Changes Base64: 67.34670794010162 milliseconds.
Changes Doc: 1806.1128751039505 milliseconds.
Snapshot Base64: 0.13020896911621094 milliseconds.
Snapshot Doc: 1026.205708026886 milliseconds.
> node loadSecsyncAutomerge.js
Snapshot + Updates Decryption: 127.65812504291534 milliseconds.
Snapshot + Updates Doc: 1105.7897909879684 milliseconds.
Snapshot + Updates Decryption + Doc: 1233.4479160308838 milliseconds.
Changes Decryption: 29083.603749990463 milliseconds.
Changes Decryption + Doc: 2035.4658749103546 milliseconds.
Changes Decryption + Doc: 31119.069624900818 milliseconds.
Snapshot Decryption: 3.142209053039551 milliseconds.
Snapshot Doc: 1011.0410829782486 milliseconds.
Snapshot Decryption + Doc: 1014.1832920312881 milliseconds.Conclusion
The benchmarks show that the impact of Secsync on loading times and sizes of a document is insignificant when using snapshots. The more CRDT document changes are stored as separate Secsync updates the larger the impact by the CRDT libraries Yjs and Automerge, but even more so by Secsync.
Having up to 1000 changes per snapshot still lead to reasonable numbers for loading times and sizes, but a lower number like 200-500 changes would be even better and is recommended.
Running the benchmarks
The source code for the benchmarks is available at https://github.com/serenity-kit/secsync/tree/main/benchmarks/snapshots (opens in a new tab)
The README.md (opens in a new tab) in this directory contains further details on the setup and how to run the benchmarks yourself.
Further ideas for Benchmarking
- Benchmarking the Secsync sync engine and analyse the impact of the state machine
- Benchmarking a real editor integration e.g. Prosemirror (loading the document, write performance)