testing the schema compliance of an applications graphql queries — why not do it automatically?

medium recommends an image to capture the audiences attention, i’d like to think my audience is smarter than that, but i think medium is smarter than me since i’m just an idiot who refuses to use uppercase letters and is in the middle of writing a terrible run on sentence to describe a picture of text and not even really to describe just to make fun of what is probably a helpful tip from medium.

let me just get this right out of the way, i love graphql, it clicked for me as soon i read the first blog post about it.

at the time, i was primarily working on the front end of a large application, and the idea that i could have query syntax in the client with the ability to specify the exact shape of the data i was going to get back was amazing to me. and even better the schema on the server was typed so i wasn’t only going to know the shape of the data i was getting back, i was also going to be able to know the types 😍.

eventually i got the chance to use graphql on a project and it was everything i hoped for, using apollo on the client it was very clear what data every component needed at a glance, i no longer had to dig through internal api docs to find out what i was or wasn’t going to get back.

if everything was always rainbows and butterflies (shout out to adam levine) though i probably wouldn’t be writing this blog post. with multiple developers working on a project it can sometimes be impossible to keep track of everything. it’s very easy to change the schema on the server without making sure the schema change won’t cause errors to any of the queries on the client. i think someone said something once about humans making errors and it makes you human or something, i don’t know, graphql prevents a lot of those errors but it can’t really prevent the situation described above.

my coworker katherine thompson and i have been exploring how we can prevent the situations above and have come up with some solutions that i think are interesting enough to share with the larger community as they have definitely helped us deploy our applications safely and continuously.

the root of the issue is simple, when we deploy (or when we commit, or at 1:19 am on a sunday night when something is just eating away at us because it doesn’t feel quite right and we probably have others issues but right now we’re just thinking about graphql schema so get off our back ok) we want to be able to know that every query in our client app can be validly resolved against our server schema. to that end we’ve created a collection of packages that can either recurse over nested directories of graphql queries in .graphql files or over directories containing javascript files with inline queries written using the graphql-tag library (it will extract these queries first using a small utility library we’ve written — gql-extract — to first parse out the queries from the tagged template strings) and then auto generate tests for these queries against a provided application schema.

we’re providing this tooling in a package that allows for use as either a javascript api or a command line interface and you can find it here: graphql-test-generator.

we personally utilize this in a pre-test step to generate tests that are then immediately run in npm test stage. this gives us the comfort of being able to deploy knowing all the queries our application could potentially attempt are schema safe. here’s an excerpt from a package.json file illustrating our use:

{
  "pretest": "gqlTestExtraction -e ./src -o ./test -s    ../server/schema",    
  "test": "npm run lint && tape test/*.js"
}

and here is an example generated tests:

const { parse } = require("graphql/language");
const { validate } = require("graphql/validation");
const schema = require("../../tests/schema");
const query = `
    query getUserId {
        user {
            id
        }
    }
`;
const queryAST = parse(query);
const errors = validate(schema, queryAST);
const isValid = !errors.length;
const test = require("tape");
test("getUserId query adheres to application schema", assert => {
 assert.ok(isValid, "getUserId contains no schema errors");
 assert.end();
});;

by default a test using the tape library is generated but you can use an optional parameter to point to your own test generation function, an example of that can be found here — https://github.com/socialtables/graphql-test-generator/blob/master/tests/test-for-failure.js — in which we are testing that a query is not valid in order to try to provide some proof that the library avoids false positives. in the future if there’s an interest we may provide defaults for several frameworks instead of forcing the admittedly clunky interface for generating a custom test.

graphql and apollo have been extremely productive for us as a company (and me an individual) and i’m hopeful these tools can offer as much help for others as they have for my team.

the library definitely still has some warts and edge cases so i’d be super appreciative if you submitted any issues you found here: https://github.com/socialtables/graphql-test-generator/issues