Testing framework

My coworkers have put together a testing framework which does some pretty fancy things. It’s able to record test runs against live systems, and then we can check in the results to use for future tests. That lets them work around not having development servers for a variety of different services. Record a run against prod, and save that for future tests.

However, to do this they need to send in a custom data format into their wrapper telling it what to do. The wrapper runs a series of functions to actually build and run the tests they want. Today I got to look at how this wrapper works and they’ve built a state machine to implement recursive descent parsing! So often developers don’t realize they are creating a parser, and don’t think about the tooling as building a language for the problem space.

So instead, we’ve got the raw input data structure, acting as a quasi pre-parsed AST to recurse down. Of course, you have to check the contents of various keys each step of the way to decide which function to run next, and what input keys to grab. And if you don’t have the keys? Boom, return an error for what’s expected “next”. Almost like expecting a noun before a verb in a sentence, with both required.

So none of this code is easily modified, nor is the resulting language obvious. And the only benefit they really get is that they can skip lexing a string and just feed in a raw data structure that…they…wrote… manually. Sometimes this job makes me sad.