requests library is a very popular one for Python, when dealing with sending HTTP requests. No wonder, it’s simple and yet gives a lot of options when You really need them. There are however some popular misusages, that lead into unexpected trouble. Believe me, I’ve been there.
The initial way
This is what we can find in tutorials and what generally works:
This oversimplified example just works. First we send a GET request synchronously and then we check the status_code. What could possibly go wrong? Notice that I won’t be getting into the details such as headers, payload etc., because it’s not the point of this article.
DNS name resolution
Let’s say, that for some reason, the library is not able to resolve the hostname. We’ll get a long stacktrace, but basically things should move forward. Unfortunately I stumbled upon a problem, when the server was not available and I still tried to access r.status_code, which again - hung.
Target is not responding
It’s unusual nowadays that a certain service is down, but that surely happens from time to time. We expect the library to use a regular TCP timeout and just exit with an error. Nope, not this time. The function will hang indefinetely as well, because the default timeout is ‘unlimited’. Ouch.
HTTP status code
For some of the calls, we may need to be strict when checking the HTTP status code and differentiate between 204 and 200, or 403 and 404 to act accordingly - that makes sense. Other times however it’s enough that the reponse code is in the 2xx,3xx range to mark that the request succeeded and raise an exception otherwise. This can easily be achieved with the requests library as well.
The right way
Let’s tweak the code a little bit, so that it’s production ready now. I will follow with the same example, but add some more code and meaningful comments.
- When using requests, always set a timeout value and catch it in an exception handler. Also a retry of the call makes sense in such a case.
- To simplify HTTP status handling, You can call
raise_for_status()and handle it in the
HTTPErrorexception - it’s cleaner.
ConnectionErrorshould be caught in case of network related problems.
- Catching a generic
Exceptionmakes also sense in case there is an other exception raised in the block or some other problem.
- If any exception was raised, try not to make any calls to the original requests object (
rin my example)