The decision they made to store potentially multigigabyte-sized backups into a single json is just idiotic, to begin with.

.NET has this built in with Utf8JsonReader [1].

> Utf8JsonReader is a high-performance, low allocation, forward-only reader for UTF-8 encoded JSON text, read from a ReadOnlySpan<byte> or ReadOnlySequence<byte>

Although it's a bit cumbersome to use with a stream [2].

[1] https://learn.microsoft.com/en-us/dotnet/standard/serializat...

[2] https://learn.microsoft.com/en-us/dotnet/standard/serializat...

I've used ijson in Python for this kind of thing in the past, it's pretty effective: https://pypi.org/project/ijson/

I downloaded a huge Firebase backup looking for a particular record.

I ended up using the “split” shell command to get a bunch of 1gb files, then grepping for which file had the record I was looking for, then using my own custom script to scan outward from the position of matched text until it detected a valid parsable JSON object within the larger unparseable file, and return that.

Back in the bad old days when XML consumers hit similar problems we’d use and event based parser like SAX. I’m a little shocked there isn’t a mainstream equivalent for JSON — is there something I’ve missed?