For most of my career I’ve been producing and consuming APIs. This inevitably leads to working with OpenAPI, which is the industry standard for describing APIs. Every time I have to either produce or consume an OpenAPI spec, I feel like I am constantly hitting rough edges and inconsistencies of the standard and the tooling. And I think that I am not alone.
I’ve been thinking about this a lot recently and I think I’ve boiled the problems of OpenAPI - and the tooling connected with it - into two underlying issues:
OpenAPI is a huge spec. If you were to print it out, you would need 195 pages1. 195! And that’s without the RFCs that it links to (and there are many of them).
It has that many pages because it supports everything you can think of. It supports JSON, XML, HTML, form data, enums (oneOf), unions (allOf), something in between (anyOf), nulls, undefined, and the list goes on and on.
OpenAPI doesn’t concern itself with tooling. There is no official tooling or reference implementation for it. All the tooling is third party. OpenAPI is at the end of the day just a spec of how a giant JSON or YAML file should be structured. How that file is created, edited, viewed or consumed is none of its concern.
On its own that wouldn’t be a problem. But because the spec is so huge this becomes a problem. It makes consuming and producing OpenAPI correctly difficult.
Consuming the spec is usually done with code generation. You have a spec and you want to generate code for the API client. Sometimes you also want to generate the stubs for server implementation. To keep it short let’s focus on the most common problem with code generation.
When defining an OpenAPI object, you can define some properties as required and some as optional. Additionally,
some properties can be nullable.
In practice that means that OpenAPI has a distinction between null and undefined. In some instances this is actually
useful for defining a spec. But when you want to implement such a spec in a language that isn’t JavaScript or TypeScript
this becomes a problem.
Most languages that are commonly used for working with APIs (Go, Rust, Java, Python, …) don’t actually differentiate
between the two - they just have some form of nil/None/null. To handle the OpenAPI null/undefined separation,
you need to wrap all the properties in additional wrapper type with custom JSON (de)serialization logic. Obviously,
such a custom wrapper type is not the most pleasant thing to work with. Most tooling takes the easy way out by using a
“language default” JSON library which parses both undefined and null as whatever version of null the language has.
This means that you can have a completely valid OpenAPI spec and the server stubs generated in one language won’t be compatible with the client code generated in a different language.
There are many more examples of OpenAPI supporting types that are difficult to work with correctly. Most of them are a side effect of polymorphism. You can easily define a type that is sometimes a string and sometimes a float. I even had a privilege of working with such an API myself!
The problem with this is that the tooling constantly reaches for simplifications - like representing
both null and undefined as whatever version of null the language has. And these simplifications are usually fine.
Until they aren’t. Then you have a bad day because you have to fight both the spec and the tooling. That’s usually
the part where I start asking myself how OpenAPI has been an industry standard for more than a decade and the tooling
around it still kind of sucks.
Producing the spec can be done either by hand writing it, or by auto generating it from your code. When auto generating
it from code, you’ll hit the same kind of type inconsistency problems that you hit when consuming the spec. If your
FastAPI is returning a None, is this an optional field or a nullable field?
In this section I want to focus more on the problems with writing the spec by hand. This is usually done by writing a giant YAML file, which you can already imagine that it’s not a pleasant experience. If the file becomes too large, you might get a genius idea that you can split it into multiple files! OpenAPI supports it. But then you try to do something with that spec and quickly realize that a lot of tooling (linting, code generation, …) doesn’t support splitting the spec into multiple files. So you scratch that idea.
You will want to use the components section to keep the spec a bit more organized and compact. Being a programmer used
to modern code editors, you might expect that you can go to definition of a reusable component. Or maybe find all of
it’s references. Or perhaps rename it. But you would be mistaken. Editor support for OpenAPI schemas is not really any
better2 than editor support for a generic YAML file.
All of this means that if you want to produce a correct specification, you most probably want to write it by hand. And the editor support is basically non existent, apart from some very basic schema validation. So producing a spec is just like consuming it - death by a thousand paper cuts.
If OpenAPI is so tedious to work with, surely someone made something better? Yes and no. There are efforts like TypeSpec and Smithy, that try to make writing the spec easier. They do so with an additional syntax that then gets converted into an OpenAPI spec. To some extent TypeSpec and Smithy are to OpenAPI what TypeScript is to JavaScript.
The problem with that is that the two big underlying problems remain. To get to the actual client code (or server stubs), you have to go through OpenAPI. Which is still a huge spec with no official tooling.
My personal opinion is, that the solution needs to forget about OpenAPI. It needs to be something completely different. Something that doesn’t try to be compatible with OpenAPI.
I imagine the solution as a declarative language that has all the tooling (linter, formatter, LSP, code generation, …) built in. It should be highly opinionated and have a lot less features than OpenAPI. It should focus on the fact that code which follows the spec eventually has to be implemented.
Similarly to how Terraform saved us from using YAML in IaC scenario, this solution should save us from using YAML in API spec.
I have a clear idea of how such a language should look like, because I am currently working on it. It’s not public yet, because it’s still in a very early phase. And since I have a full time job, a newborn and a toddler to take care of, it will probably stay this way for a little longer. Hopefully I will have something interesting to show soon-ish!