You can send 4xx errors yourself too. If the client needs to change something about the request, that’s a 4xx, like 400 Bad Request. If the server has an error and it’s not the client’s fault, that’s a 5xx like 502 Bad Gateway.
The wikipedia listing of all HTTP codes is quite helpful. People also forget you can send a custom response body with a 4xx or 5xx error. So you can still make a custom JSON error message.
It’s easy to think of libraries as evil if they’ve never stepped foot in one.