pwn.college - Talking Web - Making HTTP Requests

Main ideas / lessons

• Web browsing works by sending HTTP requests and receiving HTTP responses.

  • When you type a URL in a browser and hit Enter, the browser sends an HTTP request to a server.
  • A server responds with an HTTP response, often containing HTML that the browser renders.

• You can inspect or manually create HTTP traffic.

  • The video shows a local web server (listening on 127.0.0.1:80) and demonstrates what an incoming HTTP request looks like (method line like GET / HTTP/1.1, plus headers).
  • It also shows responding by manually writing an HTTP/1.0 200 OK response including headers (e.g., Content-Type: text/html) and an HTML body.

• HTTP requests can be made from the command line using curl.

  • curl takes a URL and performs an HTTP request.
  • By default, it uses GET.
  • The server/user-agent observed in requests reflects the client:
    • Example: Firefox user-agent vs curl user-agent.
  • You can customize requests via options/flags:
    • Send POST requests (e.g., with a --post-style flag).
    • Set custom headers like User-Agent.

• Documentation is the key skill for learning unfamiliar tools/protocols.

  • The presenter emphasizes reading:
    • curl’s man page
    • requests documentation
    • HTTP specifications like RFC 1945
  • Options/behavior aren’t “pulled from thin air”—they’re learned by checking documentation.

• You can view what you actually sent using curl -v (verbose).

  • -v shows:
    • request details
    • response headers and raw protocol exchange

• You can send raw HTTP over TCP using netcat.

  • netcat can connect to an HTTP server (e.g., example.com on port 80).
  • Then you type raw HTTP bytes yourself.
  • Important detail: you must format properly, including using two newlines to end the HTTP headers and trigger the request.
  • The video demonstrates:
    • GET / returning the normal HTML content
    • requesting /test returning 404 Not Found

• You can interact with web servers programmatically (example: Python requests).

  • Use requests.get(url) to send an HTTP request.
  • It returns a response object.
  • You can inspect:
    • response.text (response body)
    • response.headers (iterate key/value pairs)
  • You can also:
    • send POST requests with requests.post(...)
    • provide headers via a headers= dictionary

• Example programming task: extract links from HTML.

  • HTML links are commonly represented with anchor tags (<a href="...">...</a>).
  • A simple approach shown:
    • fetch HTML with requests.get
    • split response text into lines
    • find lines containing <a
    • extract the URL by splitting the line on double quotes and taking the appropriate field

---

Methodologies / instruction lists (detailed)

1) Manually crafting an HTTP response (server-side demo)

• Start from an incoming HTTP request observed by a local server.
• Send a valid HTTP response:
  • Write a status line like: HTTP/1.0 200 OK
  • Include headers, e.g.:
    • Content-Type: text/html
  • Leave a blank line to separate headers from body
  • Provide an HTML body, e.g.:
    • Hello <b>world</b>

2) Making HTTP requests from the terminal with curl

• Basic usage:
  • curl <url>
  • Default HTTP method is GET
• POST requests:
  • Use the appropriate curl option for sending POST (the video references using flags discovered via the man page).
  • Example behavior shown:
    • curl --post ... <url> results in a POST being received by the server.
• Custom headers:
  • Use curl options to set header values, such as setting:
    • User-Agent: Connor (so the server sees that header in the request)
• Inspecting request/response details:
  • Use verbose mode:
    • curl -v <url>
  • This prints raw request/response headers and details.

3) Sending raw HTTP with netcat

• Connect to an HTTP server over TCP:
  • nc <host> <port> (e.g., example.com 80)
• Manually type a request in raw HTTP format:
  • Example GET request line:
    • GET / HTTP/1.0
  • Include a Host: header:
    • Host: example.com
• Terminate headers properly:
  • Press Enter after headers, then add an extra blank line (two newlines total) to indicate end of headers.
• Read the server’s raw HTTP response until it ends (including status like 200 OK or 404 Not Found).

4) HTTP requests in Python with the requests library

• Import and GET:
  • import requests
  • response = requests.get("http://example.com")
• Use the response object:
  • response.text for the body
  • response.headers to inspect headers (iterate key/value pairs)
• POST:
  • response = requests.post("http://example.com", ...) (the video shows the concept)
• Custom headers:
  • Create a headers dictionary, e.g.:
    • headers = {"User-Agent": "Connor"}
  • Pass it to get/post via headers=...

5) Extracting links from HTML (simple parsing logic shown)

• Fetch HTML:
  • response = requests.get("http://example.com")
• Get text:
  • response_text = response.text (implied by use of response.text / response.ext in subtitles)
• Split into lines:
  • lines = response_text.splitlines()
• For each line:
  • If the line contains an anchor tag marker (e.g., "<a"):
    • Print the line containing the anchor
    • Extract the URL by splitting the line on double quotes ("):
      • The video’s approach: line.split('"')[1] (first quoted item after splitting)

---

Speakers / sources featured (as named in the subtitles)

• Speaker/Instructor: “pwn.college” (unnamed presenter; subtitles refer to a user-agent name “Connor” as a custom header value)

• Software/library sources:

  • HTTP
  • curl (including its man page)
  • netcat
  • Python requests (documentation at requests.readthedocs.io)

• Standards source:

  • RFC 1945 (referenced as HTTP documentation to read)

• Demo site mentioned:

  • example.com (illustrative HTTP server)