Protocol Buffers, Part 3 — JSON Format
Protobuf’s binary format options are arguably fast to serialize / deserialize, and the binary representation is fairly compact. However, sometimes we want to trade the speed with a human readable text format, or we interface with a system that for some reason cannot use the Protobuf library.
For these reasons, Protobuf also has the option to serialize / deserialize messages to / from JSON format. This post aims to provide a good overview of how to use the JSON format, as well as to cover areas that are less documented in the Protobuf JSON format.
This is the third post in my series on Protocol Buffers. If you haven’t read the previous posts yet, then I recommend you check them out here & here.
The full code example can be found at — https://github.com/rotomer/protobuf-blogpost-3
The basics
In Java, the classes for serializing / deserializing Protobuf messages to / from JSON format are located in the package protobuf-java-util — so be sure to add it as a dependency.
The frame story is the same as in the previous posts, so let’s assume you are building a backend for a cloud platform (much like my team does at CloudShare).
Message definitions
A (bit simplified) command message to provision a VM would look like this in Protobuf IDL:
Protobuf ← → JSON conversion
Now, let’s define a VmProvisioningService that will receive a JSON serialized command via some messaging system (Message Queue or Pub/Sub), deserialize it, and will “provision a VM” accordingly.
Next, we’ll create a test to act as the client code that invokes the VmProvisioningService by creating a ProvisionVmCommand, serializing it and sending it to the service (in reality the call will be made via message passing transport rather than in-process method call).
Simple isn’t it? Unfortunately, things get trickier with the Any message type.
JSON Format and the “Any” message type
It’s recommended to refresh upon the usage of the Any message type. The previous post covers any prerequisites.
JSON formatting performs deep deserialization of Any packed messages. Consequently, the JsonFormat.Parser and JsonFormat.Printer require passing a TypeRegistry that maps from type URL to the generated message class for each type packed into Any.
Protobuf ← → JSON conversion — now with “Any” message types
A VmProvisioningService that accepts a VmCommandEnvelope as JSON with an Any inner message type would look like so:
Accordingly, a test to act as the client code that invokes the VmProvisioningService would look like so:
JSON serialization breaks “Any” message semantic contract
As mentioned in the end of the previous post, the deep JSON deserialization of Any packed messages has a profound effect on the use cases of Any. i.e. it forces the very first component that deserializes the envelope message to have message type mappings for each of the possible types that can be packed into the Any field.
In my opinion, this breaks the semantic contract of the Any message type since binary format serialization is shallow with regards to the Any typed fields.
There maybe valid reasons for choosing to do so in JSON. For example, this seems like a good choice of API semantics for the use case of interacting with another system that doesn’t use the Protobuf library.
Depending on your use case, you may be fine with the new contract. In case you require the same semantics for Any in JSON format as in the binary format then here’s the solution that I came up with.
Note: Like many non-basic use cases of Protobuf — I found this use case to be “uncharted territory”, with very few preexisting resources. Please feel free to comment and share your experience with using Any and / or JSON format.
“AnyJson” type for shallow JSON serialization
Given that the Any type is pretty simple, we can workaround the deep serialization issue by packing the inner messages into string-typed JSON fields with our own custom (☹ ) “Any” message type which we will call AnyJson.
Here’s the message definition:
Some utility classes
Now, we’ll create the AnyJsonPacker & AnyJsonUnpacker classes for packing / unpacking messages as JSON into / from the AnyJson message:
As we can see, the AnyJsonUnpacker is given a MessageTypeLookup as a dependency. As its name suggests, this class maps from the type URL to the corresponding message builder for the types that are packed into the AnyJson message.
Unfortunately, I couldn’t find any way to use the preexisting JsonFormat.TypeRegistry to perform this role — so this custom mapping is needed.
The population of this mapping can either be done explicitly, or by reflection over the packages that hold the generated messages.
Use case — revisited with shallow deserialization
Finally, let’s see how the previous use case looks like — now with the AnyJson type.
A VmProvisioningService that accepts a VmJsonEnvelope as JSON with an AnyJson inner message type would look like so:
Accordingly, a test to act as the client code that invokes the VmProvisioningService would look like so:
Recap
We’ve shown 3 example usages of JSON formatting in Protobuf:
- Plain simple JSON serialization.
- Serialization of
Anymessage type using the default, deep serialization. - Serialization of an alternative
AnyJsonmessage type in case shallow serialization of of the packed message is warranted.
Further reading
- There’s an excellent blog post from the Envoy team which describes another methodology for dealing with dynamic extensibility in Protobuf, and JSON formatting.
- Alexander Moses wrote a blog post on converting Protobuf messages to JSON and back with Gson using the Protobuf default JSON formatting.
Call for comments
I hope this post helped in shedding some light into how one can go about using the Protobuf JSON formatting, and its relation to the Any message type. Please feel free to comment and share your experience with using the JSON formatting / Any message type, or if you found other techniques to be useful :)
