fastsimdjson Hits 65.7 GB/s on 64 Threads, but Only on Free-Threaded Python
Software / news
fastsimdjson Hits 65.7 GB/s on 64 Threads, but Only on Free-Threaded Python
Daniel Lemire's Oct. 12 benchmark counts GitHub events in 0.12 seconds, and the lazy mode that wins it avoids building most Python objects.
The fastest result in Daniel Lemire's Oct. 12, 2026 benchmark is 65.7 GB/s: fastsimdjson 0.6.0, in its lazy mode, on 64 threads, reading 7.6 GB of GitHub event data in about 0.12 seconds. The standard library's json module managed 3.17 GB/s on the same 64 threads.
The test machine had two Intel Xeon Gold 6548N processors with 64 cores in total, and Python 3.14.6 in its free-threaded build, the variant with the global interpreter lock removed. The comparison libraries were orjson 3.13.0 and msgspec 0.22.0. Lemire's question in the post was whether JSON can be parsed at tens of gigabytes a second in Python. His answer: "Yes."
The task and the single-core numbers
The workload is eight hours of GH Archive data from March 13, 2024: 7.6 GB decompressed, 2.2 million events. The script counts events by type and counts pull requests opened per repository language. It found 65,269 opened pull requests, led by TypeScript at 11,565, JavaScript at 7,778, Python at 7,057 and Java at 4,668.
On one core, the table shows how much of the gap is object creation rather than parsing.
| Single core | GB/s |
|---|---|
| json (standard library) | 0.18 |
| orjson | 0.42 |
fastsimdjson map_many | 0.63 |
| msgspec, typed structs | 1.29 |
fastsimdjson map_many, lazy | 2.13 |
The standard library needed about 42 seconds; the lazy fastsimdjson run needed about 3.6. Lemire's explanation for the spread is that "most of the time goes into creating Python objects that we discard right away." A script that reads a few fields per event pays for every field it never looks at, unless the parser is lazy.
What 64 threads buy
With threads added, msgspec reached 24.6 GB/s and the eager fastsimdjson run 29.8 GB/s. The lazy run reached 65.7 GB/s, roughly 31 times its own single-core rate.
- json (standard library)3.17 GB/s
- msgspec24.6 GB/s
- fastsimdjson map_many29.8 GB/s
- fastsimdjson map_many, lazy65.7 GB/s
Source: Daniel Lemire's blog, Oct. 12, 2026
Lemire wrote that "nothing in the counters is shared, so the threads do not wait for each other." That is the condition that makes the scaling possible, and it describes a counting job, not every job.
Caveats the post states
Timing excludes decompression, which happens in memory before the clock starts. The fast variants are the best of seven runs, but the single-thread parsers and the standard library at 64 threads ran once. msgspec was handed an advantage, because its schema declares only the fields the script needs. A commenter asked for ssrJSON to be added; the post does not include it, and the comment says it has not been tested on the free-threaded build.
The lazy mode is not a drop-in swap for json.loads. The result is a view over the document, so the speed depends on the script touching few fields, and the post itself says other access patterns may differ.
What the project's own README adds
The fastsimdjson repository describes the library as "a drop-in replacement for the json module's loads, load, dumps and dump." Its own benchmarks, run on one Xeon Gold 6548N with Python 3.14.6, report a geometric-mean loads speed of 0.78 GB/s against 0.22 GB/s for json, and dumps about 4.6 times faster than json.dumps. Those README figures cite fastsimdjson 0.3.0 and simdjson 5.0.2, while Lemire's post tests 0.6.0, so the two sets of numbers are from different versions.
The README says the threads=N option is meant for free-threaded Python 3.14t and newer, and that with the global interpreter lock it has no effect on speed. Wheels cover Python 3.10 to 3.14. At the time of reading, the repository showed four stars and 36 commits, and the README text does not state a licence for the project itself, which matters to anyone planning to ship it.
For readers keeping score of recent open-source attention, REA's 130,000 GitHub stars and this four-star repository sit at opposite ends. Another release filed this week, Microsoft's MXC agent sandbox, reached general availability. The check for fastsimdjson is concrete: rerun Lemire's script from his repository on a free-threaded 3.14 build with your own event data.
Sources
More in Software
- 01Rust Borrow Checker Bug 11 Years Old Lets Safe Code Hide a Short-Lived Reference in a 'static ValueIssue #164250 shows a Drop impl smuggling a stack reference past the compiler on stable releases back to 1.85.0, and commenters trace variants to Rust 1.4.
- 02REA, a Reverse-Engineering Tool for AI Agents, Gained 52,186 GitHub Stars in a Day. It Has 653 Watchers.Three trackers report three different star totals for morluto/rea, and the 15 newest issues and pull requests show an installer still being patched.
- 03QGIS Merges a 'Closed Door' on AI-Written Code 13 Days After Proposing ItQEP 428 bars LLM, AI and agent-assisted code from qgis/qgis after a vote that closed October 7, with two follow-up proposals still open.
- 04Anthropic Moved Its Monorepo Off Git to Scion as Daily Merges Grew From 2,500 to 30,000A seven-person team built Scion in five months. The post is candid about merge-queue conflicts and says the system is not a git replacement for general use.