By Ugorji Nwoke   Tue, 16 Dec 2014 18:00:00 -0700   /blog   technology go-codec

Benchmarks!!! Serialization in Go!!!

View articles in the go-codec series, source at http://github.com/ugorji/go

Let’s have some fun with some numbers.

In the serialization in go article, we discussed a number of types of encoding formats and their libraries in go.

In this article, we will compare them on these metrics:

  • speed: clock time
  • memory usage: amount of bytes allocated
  • memory allocations: number of allocations

We will first compare the benchmark results in visual charts and explain the data. Thereafter, we will show the raw data, which includes amount of memory allocated and number of allocations.

Show me the numbers

For each encoding library, we will encode and decode a TestStruc value. This value is shown in detail in a follow up section. The encoded lengths using each library is shown below:

Below, we will compare the encoding and decoding speed of each library, in 3 different scenarios:

  • Runtime Introspection with interfaces
  • Runtime Introspection
  • Code Generation (for libraries which support code generation)

Some libraries have very weak support for interfaces, only supporting a limited set of types and explicitly using a type switch to detect those types. Performing benchmarks against them of types with dynamic interface support results in invalid benchmarks. These include:

  • Msgp
  • Bsongen
  • VMsgpack
  • Xdr
  • Sereal

When a struct has fields which never reference interfaces, encoding and decoding is straightforward. Once interfaces come into the picture, the runtime introspection has to do more work. It is here that we see the performance numbers diverge more.

Code Generation has no way to handle interfaces (as interfaces by nature are dynamic and the concrete type is only known at runtime). Robust encodings will transparently fallback to runtime introspection when interfaces are detected. Only the go-codec libraries do this.

Let us look at the encoding speed of the different libraries below.

Let us also look at the decoding speed of the different libraries below.

Requirement to participate: How do libraries make the cut?

To be a candidate, a library must be able to encode and decode the TestStruc type reproduced below:

type AnonInTestStruc struct {
	AS        string
	AI64      int64
	AI16      int16
	AUi64     uint64
	ASslice   []string
	AI64slice []int64
	AF64slice []float64
	AMSU16    map[string]uint16
}

type AnonInTestStrucIntf struct {
	Islice []interface{}
	Ms     map[string]interface{}
	Nintf  interface{} //don't set this, so we can test for nil
	T      time.Time
}

type TestStruc struct {
	_struct struct{} `codec:",omitempty"` //set omitempty for every field

	S    string
	I64  int64
	I16  int16
	Ui64 uint64
	Ui8  uint8
	B    bool
	By   uint8 // byte: msgp doesn't like byte

	Sslice    []string
	I64slice  []int64
	I16slice  []int16
	Ui64slice []uint64
	Ui8slice  []uint8
	Bslice    []bool
	Byslice   []byte

	Iptrslice []*int64

	AnonInTestStruc

	//M map[interface{}]interface{}  `json:"-",bson:"-"`
	Msi64 map[string]int64

	// make this a ptr, so that it could be set or not.
	// for comparison (e.g. with msgp), give it a struct tag (so it is not inlined),
	// make this one omitempty (so it is included if nil).
	*AnonInTestStrucIntf `codec:",omitempty"`

	Nmap       map[string]bool //don't set this, so we can test for nil
	Nslice     []byte          //don't set this, so we can test for nil
	Nint64     *int64          //don't set this, so we can test for nil
	Mtsptr     map[string]*TestStruc
	Mts        map[string]TestStruc
	Its        []*TestStruc
	Nteststruc *TestStruc
}

TestStruc is designed to test out parsing a fully representative struct:

  • Large enough, with adequate representation of basic types, including strings containing utf8 characters in BMP and SMP planes and floats whose mantissa cannot fit into a uint64.
  • Can be expanded by having self-references e.g. type A has a field which contains multiple *A
  • has interfaces which can be optionally turned off
  • uses anonymous fields, including anonymous values and pointer fields e.g. type A struct { Anon1; *Anon2 } where Anon1 and Anon2 are named types.

Benchmark parameters

The benchmark takes a few parameters. Some important one are described below:

  • -bd: bench depth.
    When more than 1, we recursively create more instances of *TestStruc and put some maps, slices and fields of itself. This effectively creates a TestStruc that contains other TestStrucs and is thus larger.
  • -bf: skip interfaces.
    Interfaces make encoders do more work. It is only easiest to “completely” do interface encoding or decoding at runtime, without limiting the types you support statically via a typeswitch. For comparison, we will test performance when the encoded/decoded value contains interfaces or not. *AnonInTestStrucIntf is a pointer, and all interfaces hide behind it.
  • -bv: verify. This flag causes us to take the baseline *TestStruc, encode it to []byte, and then decode a new(TestStruc) from the encoded []byte, and use reflect.DeepEqual to ensure that they are the same value.
  • -brw: use io.Reader/io.Writer
    Many libraries have the ability to encode/decode from/to a []byte, or from/to io.Reader/io.Writer. This flag tells the benchmark to use either of the flavors. Typically, using []byte will support some limited zero-copy functionality that we can test performance of.

Running the benchmark myself: Libraries/Formats compared

We have built a go-codec-bench repository which contains the following:

Provided by go-codec:

  1. msgpack: http://github.com/msgpack/msgpack
  2. binc: http://github.com/ugorji/binc
  3. cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
  4. simple:
  5. json: http://json.org http://tools.ietf.org/html/rfc7159

Other codecs compared:

  1. gopkg.in/mgo.v2/bson
  2. https://github.com/philhofer/msgp

Codecs which made the cut, albeit with a question mark:

  1. github.com/vmihailenco/msgpack
    This only worked without interfaces. This package doesn’t handle different maps and slices of interfaces fully.

Other libraries considered, which did not make the cut:

To run the benchmarks, you need to do some ensure you have all the libraries benchmarked against:

  go get -u -t -tags=x github.com/ugorji/go-codec-bench
  go get -u github.com/ugorji/go/codec/codecgen
  go get -u github.com/philhofer/msgp

Thereafter, you can run the benchmarks. Use the -Z parameter to see a listing of all parameters supported:

go test -Z

  -bd=1: Bench Depth
  -bf=false: Skip Interfaces
  -bg=false: Bench Debug
  -bi=false: Run Bench Init
  -bm=false: Use Must(En|De)code
  -brw=false: Use I/O Reader/Writer, not directly to bytes
  -bs=false: Bench use maps with string keys only
  -bu=false: Show Unscientific Results during Benchmark
  -bv=false: Verify Decoded Value during Benchmark

Sample test execution, including setup for codecgen and execution:

# If you want to run the benchmarks against code generated values.
# Then first generate the code generated values from values_test.go named typed.
# we cannot normally read a _test.go file, so temporarily copy it into a readable file.
cp values_test.go values_temp.go
msgp -tests=false -pkg=codec -o=values_msgp.go -file=values_temp.go
codecgen -rt codecgen -t 'x,codecgen,!unsafe' -o values_codecgen_test.go values_temp.go
codecgen -u -rt codecgen -t 'x,codecgen,unsafe' -o values_codecgen_unsafe_test.go values_temp.go
# remove the temp file
rm -f values_temp.go
# Run the tests, using only runtime introspection support (normal mode)
go test -bm -benchmem -bi '-bench=_.*De' -tags=x
# Run the tests using the codegeneration.
# This involves passing the tags which enable the appropriate files to be run.
go test -bm -benchmem -bi -bf '-bench=_.*De' '-tags=x codecgen unsafe'

Benchmark Raw Data

This benchmark gathers results over the following axes:

  • runtime introspection + interfaces
  • runtime introspection
  • code generation (for libraries which support code generation)
  • code generation vs runtime introspection (for libraries which support both modes)

Encoded Lengths

---- WITH INTERFACES
	ApproxDeepSize Of benchmark Struct: 8259 bytes

	   msgpack: len: 2086 bytes
	binc-nosym: len: 2102 bytes
	  binc-sym: len: 1733 bytes
	    simple: len: 2402 bytes
	      cbor: len: 2102 bytes
	      json: len: 3126 bytes
	  std-json: len: 3470 bytes
	       gob: len: 2756 bytes
	 v-msgpack: len: 2408 bytes
	      bson: len: 3997 bytes
	      msgp: len: 2456 bytes
          
---- WITHOUT INTERFACES
  	ApproxDeepSize Of benchmark Struct: 5811 bytes

	   msgpack: len: 1718 bytes
	binc-nosym: len: 1730 bytes
	  binc-sym: len: 1467 bytes
	    simple: len: 1974 bytes
	      cbor: len: 1723 bytes
	      json: len: 2554 bytes
	  std-json: len: 2962 bytes
	       gob: len: 2196 bytes
	 v-msgpack: len: 2120 bytes
	      bson: len: 3505 bytes
	      msgp: len: 2132 bytes
        

Runtime Introspection Libraries (without interfaces)

The following libraries are supported:

  1. go-codec: will all its supported formats (msgpack, cbor, binc, json)
  2. standard library provided: gob, json
  3. github.com/vmihailenco/msgpack
  4. gopkg.in/mgo.v2/bson
ENCODING

Benchmark__Msgpack____Encode	   10000	     66858 ns/op	    8816 B/op	      58 allocs/op
Benchmark__Binc_NoSym_Encode	   10000	     66061 ns/op	    8848 B/op	      58 allocs/op
Benchmark__Simple_____Encode	   10000	     67579 ns/op	    8816 B/op	      58 allocs/op
Benchmark__Cbor_______Encode	   10000	     66168 ns/op	    8816 B/op	      58 allocs/op
Benchmark__Json_______Encode	   10000	     86525 ns/op	    8896 B/op	      58 allocs/op
Benchmark__Std_Json___Encode	   10000	     91267 ns/op	   14136 B/op	     123 allocs/op
Benchmark__Gob________Encode	    5000	    139903 ns/op	   10162 B/op	     222 allocs/op
Benchmark__Bson_______Encode	    5000	    164735 ns/op	   31120 B/op	     728 allocs/op
Benchmark__VMsgpack___Encode	   10000	     68462 ns/op	   11280 B/op	     224 allocs/op
Benchmark__Msgp_______Encode	   50000	     17359 ns/op	    1920 B/op	       4 allocs/op

DECODING

Benchmark__Msgpack____Decode	   10000	     87545 ns/op	    9616 B/op	     253 allocs/op
Benchmark__Binc_NoSym_Decode	   10000	     92794 ns/op	    9648 B/op	     253 allocs/op
Benchmark__Simple_____Decode	   10000	     91825 ns/op	    9616 B/op	     253 allocs/op
Benchmark__Cbor_______Decode	   10000	     88502 ns/op	    9616 B/op	     253 allocs/op
Benchmark__Json_______Decode	    5000	    152253 ns/op	   11648 B/op	     330 allocs/op
Benchmark__Std_Json___Decode	    2000	    288132 ns/op	   13784 B/op	     493 allocs/op
Benchmark__Gob________Decode	    2000	    420523 ns/op	   70171 B/op	    1745 allocs/op
Benchmark__Bson_______Decode	    5000	    177946 ns/op	   15928 B/op	    1046 allocs/op
Benchmark__VMsgpack___Decode	    5000	    121566 ns/op	   13904 B/op	     420 allocs/op
Benchmark__Msgp_______Decode	   20000	     37965 ns/op	    6488 B/op	     132 allocs/op

Runtime Introspection Libraries (with interfaces)

The same libraries above are now benchmarked with interface support.

ENCODING

Benchmark__Msgpack____Encode	   10000	     82517 ns/op	    9219 B/op	      70 allocs/op
Benchmark__Binc_NoSym_Encode	   10000	     81686 ns/op	    9379 B/op	      74 allocs/op
Benchmark__Simple_____Encode	   10000	     84284 ns/op	    9219 B/op	      70 allocs/op
Benchmark__Cbor_______Encode	   10000	     83221 ns/op	    9187 B/op	      70 allocs/op
Benchmark__Json_______Encode	   10000	    108092 ns/op	    9267 B/op	      70 allocs/op
Benchmark__Std_Json___Encode	    5000	    124477 ns/op	   16313 B/op	     207 allocs/op
Benchmark__Gob________Encode	    3000	    213726 ns/op	   16083 B/op	     333 allocs/op
Benchmark__Bson_______Encode	    5000	    195656 ns/op	   34224 B/op	     852 allocs/op
Benchmark__VMsgpack___Encode	   10000	     94876 ns/op	   17425 B/op	     281 allocs/op
Benchmark__Msgp_______Encode	   20000	     28631 ns/op	    1984 B/op	       8 allocs/op

DECODING

Benchmark__Msgpack____Decode	    5000	    130962 ns/op	   12880 B/op	     370 allocs/op
Benchmark__Binc_NoSym_Decode	    5000	    133424 ns/op	   12736 B/op	     358 allocs/op
Benchmark__Simple_____Decode	    5000	    130351 ns/op	   12688 B/op	     358 allocs/op
Benchmark__Cbor_______Decode	    5000	    131491 ns/op	   12800 B/op	     366 allocs/op
Benchmark__Json_______Decode	    3000	    205337 ns/op	   15344 B/op	     455 allocs/op
Benchmark__Std_Json___Decode	    2000	    359771 ns/op	   17992 B/op	     629 allocs/op
Benchmark__Gob________Decode	    2000	    499588 ns/op	   79764 B/op	    2041 allocs/op
Benchmark__Bson_______Decode	    3000	    217165 ns/op	   19992 B/op	    1246 allocs/op
Benchmark__VMsgpack___Decode	    5000	    152247 ns/op	   17760 B/op	     548 allocs/op
Benchmark__Msgp_______Decode	   20000	     50759 ns/op	    8904 B/op	     200 allocs/op

Code Generation Libraries

The following libraries support code generation:

  1. go-codec: will all its supported formats (msgpack, cbor, binc, json)
  2. https://github.com/philhofer/msgp

msgp uses unsafe when type-switching on struct field names, and does not support interfaces fully, as full support for interfaces requires falling back to reflection.

We compare using the unsafe option to codecgen, and run the benchmark with the interface field (aka *AnonInTestStrucIntf) set to nil. This gives like setup for both.

ENCODING

Benchmark__Msgpack____Encode	   50000	     18374 ns/op	     224 B/op	       3 allocs/op
Benchmark__Binc_NoSym_Encode	   50000	     18342 ns/op	     256 B/op	       3 allocs/op
Benchmark__Simple_____Encode	   50000	     19485 ns/op	     224 B/op	       3 allocs/op
Benchmark__Cbor_______Encode	   50000	     16696 ns/op	     224 B/op	       3 allocs/op
Benchmark__Json_______Encode	   20000	     33873 ns/op	     304 B/op	       3 allocs/op
Benchmark__Msgp_______Encode	   50000	     17605 ns/op	    1920 B/op	       4 allocs/op

DECODING

Benchmark__Msgpack____Decode	   20000	     42610 ns/op	    6808 B/op	     134 allocs/op
Benchmark__Binc_NoSym_Decode	   20000	     45270 ns/op	    6840 B/op	     134 allocs/op
Benchmark__Simple_____Decode	   20000	     44621 ns/op	    6792 B/op	     134 allocs/op
Benchmark__Cbor_______Decode	   20000	     44329 ns/op	    6792 B/op	     134 allocs/op
Benchmark__Json_______Decode	    5000	    103326 ns/op	    8680 B/op	     210 allocs/op
Benchmark__Msgp_______Decode	   20000	     37672 ns/op	    6488 B/op	     132 allocs/op

Code Generation vs Runtime Reflection

Here, we take libraries which support both runtime reflection and code generation.

The following libraries fit this bill:

  1. go-codec: will all its supported formats (msgpack, cbor, binc, json)

This has been done before. More information is available in the article on codecgen.

Tags: technology go-codec


Subscribe: Technology
© Ugorji Nwoke