By Ugorji Nwoke   Tue, 16 Dec 2014 12:00:00 -0700   /blog   technology go-codec

Yet Another JSON Library? But Why?

View articles in the go-codec series, source at http://github.com/ugorji/go

And let me answer …

The encoding/json library bundled with the standard library is a GREAT piece of software.

The reasons why I decided to write a json library are:

  1. decode from a multi-part stream.
    encoding/json would read many bytes from the stream into an internal buffer, and then incrementally decode from that internal buffer.

    This makes it hard to parse some json string from a stream, and then continue reading other values in other formats from the stream. A potential use-case here is a multipart stream which contains some text in json, some images in gif format, etc. I had a usecase similar to this.

    encoding/json has a kludge where it exposes a Buffered() method so users can gain access to the bytes it read but didn’t use. That is hard to work with.

    The go-codec json library would only read the data it needs from the stream.

  2. Encode/Decode from/to chan type (for iterative streaming support).
    A use-case involves a json array containing millions of values. For efficiency, they should be processed concurrently as the values are being decoded. The idiomatic way is to send the values into a chan, and process them one-by-one in a different goroutine.

    encoding/json does not support chan types. go-codec allows you easily encode a chan as an array in json, or decode from an array in json into a chan natively, elegantly solving this use-case.

  3. ability to decode a number as either a float64, int64 or uint64.
    Currently, encoding/json only decodes as float64, and you need to use the hatch “UseNumber()” to keep it as json.Number.

    This made things hard to work with, as you have to do a second parse to convert the json.Number to an appropriate value. It would be nice if the decoder could determine the best “type” to use e.g. decode 789 as uint64, -789 as int64, and -78.9 as float64.

  4. avoid allocation when parsing numbers.
    encoding/json uses strconv.ParseFloat when decoding a number.

    go-codec parses the number while reading the stream, and not as a subsequent out-of-band action.

  5. leverage all the performance enhancements of the go-codec library.
    The go-codec library has some very impressive performance attributes. We wanted to leverage it for json encoding and decoding.

    See why go-codec has stellar performance

  6. Support for fully-featured code generation for more performance.
    encoding/json only works with runtime introspection (reflection). However, it is shown that you could get over a 2X performance improvement by static encoding/decoding when the structure of the types are known at compile time.

    See code generation support from go-codec

  7. Drop-in replacement for encoding/json. json: key in struct tag supported.
    encoding/json uses the json: key in the struct tag value to configure how the struct is encoded. go-codec will use the json key as a fallback, if the codec key is unavailable in the struct tag value.

    This allows go-codec to be used as a drop-in replacement for encoding/json without having to make changes to the structs.

The performance numbers speak for themselves. When we introduce code generation support, the numbers are even more stellar.

Raw Data:

ENCODE (RUNTIME)
Benchmark__Std_Json___Encode	    5000	    124477 ns/op	   16313 B/op	     207 allocs/op
Benchmark__Json_______Encode	   10000	    108092 ns/op	    9267 B/op	      70 allocs/op

ENCODE (CODE GENERATION)
Benchmark__Json_______Encode	   20000	     33873 ns/op	     304 B/op	       3 allocs/op

DECODE (RUNTIME)
Benchmark__Std_Json___Decode	    2000	    359771 ns/op	   17992 B/op	     629 allocs/op
Benchmark__Json_______Decode	    3000	    205337 ns/op	   15344 B/op	     455 allocs/op

DECODE (CODE GENERATION)
Benchmark__Json_______Decode	    5000	    103326 ns/op	    8680 B/op	     210 allocs/op

The table below shows how much more time, memory allocated and number of memory allocations is used by encoding/json compared to the go-codec json library. The improvements are clear:

Time Memory Allocations
Encoding (runtime) 1.2 X 1.8 X 3.0 X
Encoding (code generation) 3.5 X 53.0 X 69.0 X
Decoding (runtime) 1.7 X 1.1 X 1.4 X
Decoding (code generation) 3.1 X 2.0 X 3.0 X

The beauty of this is that encoding and decoding is just as feature-rich as the encoding/json package. go-codec builds more features and performance beyond what is provided by the encoding/json package.

Tags: technology go-codec


Subscribe: Technology
© Ugorji Nwoke