@wruza 10d
The decision they made to store potentially multigigabyte-sized backups into a single json is just idiotic, to begin with.
@taspeotis 10d
.NET has this built in with Utf8JsonReader [1].

> Utf8JsonReader is a high-performance, low allocation, forward-only reader for UTF-8 encoded JSON text, read from a ReadOnlySpan<byte> or ReadOnlySequence<byte>

Although it's a bit cumbersome to use with a stream [2].

[1] https://learn.microsoft.com/en-us/dotnet/standard/serializat...

[2] https://learn.microsoft.com/en-us/dotnet/standard/serializat...

@simonw 10d
I've used ijson in Python for this kind of thing in the past, it's pretty effective: https://pypi.org/project/ijson/
@Liron 9d
I downloaded a huge Firebase backup looking for a particular record.

I ended up using the “split” shell command to get a bunch of 1gb files, then grepping for which file had the record I was looking for, then using my own custom script to scan outward from the position of matched text until it detected a valid parsable JSON object within the larger unparseable file, and return that.

@mlhpdx 10d
Back in the bad old days when XML consumers hit similar problems we’d use and event based parser like SAX. I’m a little shocked there isn’t a mainstream equivalent for JSON — is there something I’ve missed?