Skip to content

Commit fed8979

Browse files
committed
fix: disable HTTP/2 to eliminate CONNECT-tunnel churn under concurrency
When the same twikit Client is shared across high-concurrency tasks (or when many clients share an upstream CONNECT proxy), ``http2=True`` produces two failure modes that surface as ``ConnectError`` / ``[SSL: RECORD_LAYER_FAILURE]`` and 502s in the calling service: 1. httpx opens a new CONNECT tunnel per concurrent task at race time, then closes it once another task wins the multiplexing race on an already-open h2 connection. In production, 3proxy logs ~67% of tunnels with error code 21 (client closed before sending TLS data), leaving only ~26% of tunnels actually carrying traffic. 2. ALPN-negotiated h2 through CONNECT exposes a Cloudflare bot-detection RST path mid-handshake that surfaces as ``RECORD_LAYER_FAILURE`` for ~10% of requests originating from datacenter IPs. HTTP/1.1 with keep-alive avoids the negotiation entirely and Cloudflare lets it through. X.com's GraphQL is HTTP/1.1-friendly, and the perf cost of dropping h2 is negligible at this concurrency profile (the keep-alive pool was the bottleneck, not stream multiplexing). Apply the same change to the guest client so unauthenticated calls share the same transport behaviour.
1 parent 699fb17 commit fed8979

2 files changed

Lines changed: 16 additions & 4 deletions

File tree

twikit/client/client.py

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,8 +111,18 @@ def __init__(
111111
)
112112
warnings.warn(message)
113113

114-
# Initialize with an isolated cookie jar to prevent cookie conflicts
115-
self.http = AsyncClient(proxy=proxy, cookies=Cookies(), http2=True, **kwargs)
114+
# Initialize with an isolated cookie jar to prevent cookie conflicts.
115+
#
116+
# ``http2=False`` is intentional: when scraping at high concurrency
117+
# through a CONNECT proxy, httpx with HTTP/2 opens speculative
118+
# tunnels per concurrent task, then closes them un-used as
119+
# multiplexed streams settle on existing connections. Upstream
120+
# 3proxy logs this as ``00021`` (client-closed-early) on ~67% of
121+
# tunnels in production. ALPN-negotiated h2 through CONNECT also
122+
# exposes a Cloudflare bot-detection RST path that surfaces as
123+
# ``[SSL: RECORD_LAYER_FAILURE]`` for ~10% of requests; HTTP/1.1
124+
# with keep-alive sidesteps both.
125+
self.http = AsyncClient(proxy=proxy, cookies=Cookies(), http2=False, **kwargs)
116126
self.language = language
117127
self.proxy = proxy
118128
self.captcha_solver = captcha_solver

twikit/guest/client.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,8 +86,10 @@ def __init__(
8686
)
8787
warnings.warn(message)
8888

89-
# Initialize with an isolated cookie jar to prevent cookie conflicts
90-
self.http = AsyncClient(proxy=proxy, cookies=Cookies(), http2=True, **kwargs)
89+
# Initialize with an isolated cookie jar to prevent cookie conflicts.
90+
# ``http2=False`` mirrors the authenticated client — see comment in
91+
# ``twikit/client/client.py``.
92+
self.http = AsyncClient(proxy=proxy, cookies=Cookies(), http2=False, **kwargs)
9193
self.language = language
9294
self.proxy = proxy
9395

0 commit comments

Comments
 (0)