Talking HTTP with Python

python-for-hackers ended with a script that could store data in a dict, walk a list with a for loop, and branch on a condition. Useful, and pointed at nothing. Most security work points at a web server, and a web server speaks HTTP. This lesson joins the two ends: it turns those loops and dicts into something that can question a live server, read what it says back, and decide what to do next.

Consider a login form. Trying one password means typing it and clicking a button. Trying five thousand means writing a loop, and a loop needs a way to send each attempt as a real HTTP request and read the real answer. That way is the requests library.

requests is not built in

Python does ship with an HTTP module called urllib. Almost nobody reaches for it directly, because it's clumsy for anything past the simplest request. The requests library wraps the same job in something readable, and it's the near-universal choice. It is not part of the standard library, so it has to be installed once before any import requests will find it. The installer is pip, Python's package manager, which downloads the library and puts it where the interpreter looks.

$pip3 install requests

With that done, one GET request is three lines:

import requests

r = requests.get("https://httpbin.org/get", timeout=5)

requests.get sends a single HTTP GET to that URL and waits for the answer. What comes back is assigned to r: a response object. Not the page text, the whole response. The status line, every header the server sent, the body, and a record of the request that produced it all hang off that one object as attributes.

Four attributes carry most of the value. r.status_code is the number the server put on its answer, as an integer, so it compares with == and <. r.headers holds the response headers as a dictionary-like object, keyed by name like Content-Type or Server. r.text is the body decoded into a string. r.url is the final URL, after any redirects the server sent along the way.

import requests

r = requests.get("https://httpbin.org/get", timeout=5)
print(r.status_code)
print(r.headers["Content-Type"])
print(r.url)

That timeout=5 is not optional in any script that matters. Without it, requests waits forever for a server that accepts the connection and then goes silent. One dead host with no timeout freezes the entire loop, with no error and no clue which host stalled it. Five seconds is a reasonable default.

Here is the trap that catches everyone once. A 404 or a 500 is not an error to requests. It's a normal return. A request to https://httpbin.org/status/404 comes back with status_code set to 404 and raises nothing at all, because the request succeeded and the answer was "not found." An exception only fires when the request itself fails: no route to the host, a DNS failure, or the timeout expiring. So the script has to check the number itself.

r = requests.get("https://httpbin.org/status/404", timeout=5)
print(r.status_code)   # prints 404, and no exception was raised

If a bad status should stop the script, r.raise_for_status() turns any 4xx or 5xx into an exception on demand. More often the script branches on the number, because different codes mean different next moves: 401 means the credentials were wrong, 429 means slow down, 200 means in.

Parameters and headers come from dicts

Plenty of GET requests carry parameters, the part of a URL after the ?, like ?q=admin&page=2. Building that string by hand invites bugs, because characters such as spaces and ampersands have to be percent-encoded or the server reads them wrong. requests builds it from a dict instead. Pass params= and it assembles the query string and encodes each value.

r = requests.get("https://httpbin.org/get",
                 params={"q": "admin user", "page": 2},
                 timeout=5)
print(r.url)

The space in admin user became a +, and the integer 2 became the string 2. The dict is the source of truth and the URL is generated from it, which means a loop can change one value and let requests rebuild the URL correctly every time.

Headers work the same way, passed as a dict to headers=. The one most worth setting is User-Agent, the string that tells the server what client is calling. By default requests announces itself as something like python-requests/2.31.0 (the version tracks the installed library). In a server's access log that line reads as "a script, not a browser, was here." Overriding it is a single argument.

r = requests.get("https://httpbin.org/headers",
                 headers={"User-Agent": "Mozilla/5.0"},
                 timeout=5)

Setting a header does not hide anything by itself. It only changes what the request advertises. That's still useful when a server serves different content, or blocks outright, based on the User-Agent it sees.

POST sends data, in one of two shapes

A GET asks for something. A POST sends something: a login, a form submission, a new record. What requests puts in the request body depends on which argument carries the data, and getting that wrong produces an error that looks like a credentials problem but isn't one.

Two shapes cover almost everything. data= sends the dict as a URL-encoded form, the exact format an HTML <form> submits, and sets the header Content-Type: application/x-www-form-urlencoded.

r = requests.post("https://httpbin.org/post",
                  data={"username": "admin", "password": "hunter2"},
                  timeout=5)

json= serializes the dict to a JSON string and sets Content-Type: application/json, which is what most APIs expect.

r = requests.post("https://httpbin.org/post",
                  json={"username": "admin", "password": "hunter2"},
                  timeout=5)

Send data= to an endpoint that wants JSON, or json= to one that wants a form, and the server rejects the body with a 400 or a 415 that says nothing about the username and password being wrong. The fix is never in the credentials. It's matching the shape the endpoint documents.

Reading JSON back

When a server answers with JSON, r.text holds it as one long string, which is painful to pick apart by hand. r.json() parses that string into a real Python object, usually a dict, so the fields become plain dict lookups.

r = requests.post("https://httpbin.org/post",
                  json={"user": "admin"},
                  timeout=5)
data = r.json()
print(data["json"]["user"])   # prints admin

httpbin echoes the JSON body it received back under a json key, so data["json"]["user"] reads out admin. Once the response is a dict, everything from the last lesson applies: index it, loop its keys, pull a token out of one response and feed it into the next request.

r.json() only works when the body actually is JSON. Call it on an HTML error page and it raises a JSONDecodeError, because there's nothing there to parse. A 500 that returns an HTML stack trace is the usual way to trip this. Checking r.status_code, or reading r.headers["Content-Type"] first, sidesteps the crash before it happens.

A Session carries the login

Each call to requests.get or requests.post is amnesiac. It opens a connection, sends one request, reads one response, and forgets everything, including any cookie the server just set. That breaks the instant a login is involved. Logging in works by the server returning a Set-Cookie header, and every later request has to send that cookie back to prove it's the same visitor. A fresh requests.post to log in, followed by a fresh requests.get, never sends the cookie, so the second request arrives as a stranger and gets a 401 or a bounce back to the login page.

A Session object fixes this. It's created once and used in place of the top-level functions. It keeps a cookie jar: any cookie a response sets is stored and sent automatically on every following request made through that same session.

s = requests.Session()

s.post("https://target.example/login",
       data={"username": "admin", "password": "hunter2"},
       timeout=5)

r = s.get("https://target.example/dashboard", timeout=5)
print(r.status_code)

The target.example host there is a stand-in for a real target. The login response's cookie is captured by s and attached to the dashboard request with no code written to do it. That is the backbone of almost every real web script: authenticate once, then walk the pages behind the login in a loop, each request carrying the session on its own.

A session also holds headers set on it, so a User-Agent or an auth token set once rides on every request through that session.

s = requests.Session()
s.headers.update({"User-Agent": "Mozilla/5.0"})

What requests is doing underneath

Everything here sits on a lower layer. To send that first GET, requests resolves the hostname to an IP address, opens a TCP connection to port 443, writes the raw request text (the method, the path, the headers) into the connection, reads the bytes that come back, and parses them into the response object. All of that is hidden. A Session goes one step further and reuses the open connection across requests instead of paying to set up a new one each time, which is why a session is faster over many calls, not only more convenient.

Real lab and CTF targets often present a TLS certificate requests refuses to trust, which raises an SSLError before any response arrives. verify=False turns that check off so the request goes through. It also turns off the protection that stops a man-in-the-middle from impersonating the server, so it belongs on throwaway targets and nowhere near a host whose answer has to be trusted.

import requests
r = requests.get("https://httpbin.org/get", params={"q": "test"}, timeout=5)
print(r.status_code, r.url)

requests turns a web server into an object because HTTP is a request-and-response protocol that fits that shape cleanly. Plenty of services don't. A raw TCP service, a custom binary protocol, or a port that just streams bytes has no .get and no .json(), and reaching one means opening the connection by hand. The next lesson, sockets-and-networking, drops one layer down to the socket module: connecting a TCP socket, sending and receiving raw bytes, setting timeouts, and how all of it maps onto the TCP concepts from Networking Foundations.