{"slug":"cracking-private-apis","title":"Reverse Engineering Private APIs: SaaS Endpoint Discovery and Replay","tags":["api","reverse-engineering","playwright","httpx","private-api","automation"],"agent_summary":"Pattern for discovering and replaying private SaaS API endpoints — network tab capture, auth extraction, Playwright session state, httpx replay, schema mapping, token refresh, and rate limiting.","trigger_phrases":["reverse engineer API","private API","undocumented API","crack API","SaaS private endpoint","backend API","network tab capture"],"runnable":false,"markdown":"\n## Overview\n\nWhen a SaaS public API is missing features the UI clearly supports, the UI is usually talking to a private API on a different subdomain. This pattern captures how to discover, authenticate, and replay those calls.\n\n## When to Use\n\n- Public API docs say \"not supported\" for something the UI does\n- You need bulk or admin operations at scale (50+ sub-accounts)\n- You need to replicate web app behavior from a script\n\n## Legal Check First\n\nVerify the platform's ToS does not explicitly forbid reverse engineering. Do not use private API results in client-facing production without a fallback — private APIs change without notice.\n\n## Step 1: Identify the Endpoint\n\n1. Open Chrome DevTools → Network → XHR/Fetch filter\n2. Perform the action in the UI\n3. Note the request domain — if different from the public API base, that's the private one\n4. Capture: auth header name, token format, required custom headers (`channel`, `source`, `version`, tenant ID, etc.)\n\n**Example (GoHighLevel):**\nPublic API: `api.gohighlevel.com`\nPrivate API: `backend.leadconnectorhq.com`\nCustom headers: `channel: APP`, `source: WEB_USER`, `version: 2021-07-28`\n\n## Step 2: Capture Authenticated Session\n\nUse Playwright to log in manually and persist storage state:\n\n```python\nfrom playwright.sync_api import sync_playwright\n\nwith sync_playwright() as p:\n browser = p.chromium.launch(headless=False)\n ctx = browser.new_context()\n page = ctx.new_page()\n page.goto(\"https://app.example.com/login\")\n input(\"Log in manually, then press Enter...\")\n ctx.storage_state(path=\"auth.json\")\n browser.close()\n```\n\n`auth.json` contains cookies + localStorage. Most private APIs authenticate via one of these.\n\n## Step 3: Extract the Token\n\n```python\nimport json\n\nauth = json.load(open(\"auth.json\"))\n\n# Find token in localStorage\ntoken = next(\n item[\"value\"]\n for origin in auth[\"origins\"]\n for item in origin[\"localStorage\"]\n if item[\"name\"] == \"token-id\" # adjust key name per platform\n)\nprint(f\"Token: {token[:20]}...\")\n```\n\nCommon localStorage key names: `token-id`, `access_token`, `jwt`, `authToken`, `session`.\n\n## Step 4: Replay with httpx\n\nCopy a UI request as cURL from DevTools (right-click → Copy as cURL). Convert to Python:\n\n```python\nimport httpx\n\nPRIVATE_BASE = \"https://backend.leadconnectorhq.com\"\nLOCATION_ID = \"loc_abc123\"\n\nheaders = {\n \"Authorization\": f\"Bearer {token}\",\n \"channel\": \"APP\",\n \"source\": \"WEB_USER\",\n \"version\": \"2021-07-28\",\n \"content-type\": \"application/json\",\n}\n\nr = httpx.get(\n f\"{PRIVATE_BASE}/contacts/\",\n headers=headers,\n params={\"locationId\": LOCATION_ID, \"limit\": 100},\n)\nr.raise_for_status()\nprint(r.json())\n```\n\n## Step 5: Map the Schema\n\nTo learn payload shape for POST/PUT, intercept a UI save action:\n\n1. Perform the create/update in the UI while capturing network traffic\n2. Copy the request body — it's your payload template\n3. Start from the full known-good payload\n4. Remove optional fields one at a time to find the minimum required set\n\n```python\n# Template from captured UI request\npayload = {\n \"firstName\": \"Test\",\n \"lastName\": \"User\",\n \"email\": \"test@example.com\",\n \"locationId\": LOCATION_ID,\n \"source\": \"API\",\n # ... other fields\n}\n\nr = httpx.post(f\"{PRIVATE_BASE}/contacts/\", headers=headers, json=payload)\n```\n\n## Step 6: Handle Token Refresh\n\nTokens expire. On 401:\n\n```python\ndef refresh_token():\n with sync_playwright() as p:\n browser = p.chromium.launch(headless=True)\n ctx = browser.new_context(storage_state=\"auth.json\")\n page = ctx.new_page()\n page.goto(\"https://app.example.com/dashboard\")\n page.wait_for_load_state(\"networkidle\")\n ctx.storage_state(path=\"auth.json\")\n\ndef request_with_retry(url, headers, params=None, max_retries=2):\n for attempt in range(max_retries):\n r = httpx.get(url, headers=headers, params=params)\n if r.status_code == 401 and attempt < max_retries - 1:\n refresh_token()\n # reload token and update headers\n continue\n r.raise_for_status()\n return r.json()\n```\n\n## Step 7: Throttle and Build Resilience\n\n```python\nimport time\nimport logging\n\ndef safe_request(client, url, headers, params=None, delay=1.0):\n time.sleep(delay) # start at 1 req/sec, back off on 429\n try:\n r = client.get(url, headers=headers, params=params, timeout=30)\n if r.status_code == 429:\n retry_after = int(r.headers.get(\"Retry-After\", 60))\n logging.warning(f\"Rate limited. Waiting {retry_after}s...\")\n time.sleep(retry_after)\n return safe_request(client, url, headers, params, delay)\n r.raise_for_status()\n return r.json()\n except Exception as e:\n logging.error(f\"Request failed: {e}\")\n raise\n```\n\n## Gotchas\n\n- **Headers change**: Private APIs often check custom headers strictly. Missing one returns 400 or silent empty results.\n- **Pagination**: Private APIs rarely match public API pagination. Common patterns: `page`+`limit`, `offset`+`limit`, cursor-based.\n- **Response shape**: Private responses often differ from public API docs. Always log raw responses during schema mapping.\n- **Breaking changes**: Private APIs are versioned internally. A platform update can break your integration with no notice.\n","html":"

Overview

\n

When a SaaS public API is missing features the UI clearly supports, the UI is usually talking to a private API on a different subdomain. This pattern captures how to discover, authenticate, and replay those calls.

\n

When to Use

\n\n

Legal Check First

\n

Verify the platform's ToS does not explicitly forbid reverse engineering. Do not use private API results in client-facing production without a fallback — private APIs change without notice.

\n

Step 1: Identify the Endpoint

\n
    \n
  1. Open Chrome DevTools → Network → XHR/Fetch filter
    2. \n
  2. Perform the action in the UI
    3. \n
  3. Note the request domain — if different from the public API base, that's the private one
    4. \n
  4. Capture: auth header name, token format, required custom headers (channel, source, version, tenant ID, etc.)
    5. \n
\n

Example (GoHighLevel):\nPublic API: api.gohighlevel.com\nPrivate API: backend.leadconnectorhq.com\nCustom headers: channel: APP, source: WEB_USER, version: 2021-07-28

\n

Step 2: Capture Authenticated Session

\n

Use Playwright to log in manually and persist storage state:

\n
from playwright.sync_api import sync_playwright\n\nwith sync_playwright() as p:\n    browser = p.chromium.launch(headless=False)\n    ctx = browser.new_context()\n    page = ctx.new_page()\n    page.goto(\"https://app.example.com/login\")\n    input(\"Log in manually, then press Enter...\")\n    ctx.storage_state(path=\"auth.json\")\n    browser.close()\n
\n

auth.json contains cookies + localStorage. Most private APIs authenticate via one of these.

\n

Step 3: Extract the Token

\n
import json\n\nauth = json.load(open(\"auth.json\"))\n\n# Find token in localStorage\ntoken = next(\n    item[\"value\"]\n    for origin in auth[\"origins\"]\n    for item in origin[\"localStorage\"]\n    if item[\"name\"] == \"token-id\"  # adjust key name per platform\n)\nprint(f\"Token: {token[:20]}...\")\n
\n

Common localStorage key names: token-id, access_token, jwt, authToken, session.

\n

Step 4: Replay with httpx

\n

Copy a UI request as cURL from DevTools (right-click → Copy as cURL). Convert to Python:

\n
import httpx\n\nPRIVATE_BASE = \"https://backend.leadconnectorhq.com\"\nLOCATION_ID = \"loc_abc123\"\n\nheaders = {\n    \"Authorization\": f\"Bearer {token}\",\n    \"channel\": \"APP\",\n    \"source\": \"WEB_USER\",\n    \"version\": \"2021-07-28\",\n    \"content-type\": \"application/json\",\n}\n\nr = httpx.get(\n    f\"{PRIVATE_BASE}/contacts/\",\n    headers=headers,\n    params={\"locationId\": LOCATION_ID, \"limit\": 100},\n)\nr.raise_for_status()\nprint(r.json())\n
\n

Step 5: Map the Schema

\n

To learn payload shape for POST/PUT, intercept a UI save action:

\n
    \n
  1. Perform the create/update in the UI while capturing network traffic
    2. \n
  2. Copy the request body — it's your payload template
    3. \n
  3. Start from the full known-good payload
    4. \n
  4. Remove optional fields one at a time to find the minimum required set
    5. \n
\n
# Template from captured UI request\npayload = {\n    \"firstName\": \"Test\",\n    \"lastName\": \"User\",\n    \"email\": \"test@example.com\",\n    \"locationId\": LOCATION_ID,\n    \"source\": \"API\",\n    # ... other fields\n}\n\nr = httpx.post(f\"{PRIVATE_BASE}/contacts/\", headers=headers, json=payload)\n
\n

Step 6: Handle Token Refresh

\n

Tokens expire. On 401:

\n
def refresh_token():\n    with sync_playwright() as p:\n        browser = p.chromium.launch(headless=True)\n        ctx = browser.new_context(storage_state=\"auth.json\")\n        page = ctx.new_page()\n        page.goto(\"https://app.example.com/dashboard\")\n        page.wait_for_load_state(\"networkidle\")\n        ctx.storage_state(path=\"auth.json\")\n\ndef request_with_retry(url, headers, params=None, max_retries=2):\n    for attempt in range(max_retries):\n        r = httpx.get(url, headers=headers, params=params)\n        if r.status_code == 401 and attempt < max_retries - 1:\n            refresh_token()\n            # reload token and update headers\n            continue\n        r.raise_for_status()\n        return r.json()\n
\n

Step 7: Throttle and Build Resilience

\n
import time\nimport logging\n\ndef safe_request(client, url, headers, params=None, delay=1.0):\n    time.sleep(delay)  # start at 1 req/sec, back off on 429\n    try:\n        r = client.get(url, headers=headers, params=params, timeout=30)\n        if r.status_code == 429:\n            retry_after = int(r.headers.get(\"Retry-After\", 60))\n            logging.warning(f\"Rate limited. Waiting {retry_after}s...\")\n            time.sleep(retry_after)\n            return safe_request(client, url, headers, params, delay)\n        r.raise_for_status()\n        return r.json()\n    except Exception as e:\n        logging.error(f\"Request failed: {e}\")\n        raise\n
\n

Gotchas

\n\n"}