You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

HTTP timeouts: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Effie Mouzeli
No edit summary
imported>Vgutierrez
(Add connect timeout on ats-tls && ats-backend)
Line 7: Line 7:
!TLS termination layer
!TLS termination layer
!SSL handshake timeout
!SSL handshake timeout
!connect timeout (origin server)
!TTFB (origin server)
!TTFB (origin server)
!successive reads (origin server)
!successive reads (origin server)
!Keepalive timeout (client)
!Keepalive timeout (client)
|-
|-
|nginx
|nginx (deprecated)
|60 seconds (nginx default value)
|60 seconds (nginx default value)
|10 seconds (nginx default value)
|[https://github.com/wikimedia/puppet/blob/91c1a976955b0b8e16d808aa2371f3f66c1e8f3e/modules/tlsproxy/manifests/localssl.pp#L103 180 seconds]
|[https://github.com/wikimedia/puppet/blob/91c1a976955b0b8e16d808aa2371f3f66c1e8f3e/modules/tlsproxy/manifests/localssl.pp#L103 180 seconds]
|180 seconds (same config parameter as TTFB)
|180 seconds (same config parameter as TTFB)
Line 19: Line 21:
|ats-tls
|ats-tls
|[https://github.com/wikimedia/puppet/blob/91c1a976955b0b8e16d808aa2371f3f66c1e8f3e/hieradata/common/profile/trafficserver/tls.yaml#L33 60 seconds]
|[https://github.com/wikimedia/puppet/blob/91c1a976955b0b8e16d808aa2371f3f66c1e8f3e/hieradata/common/profile/trafficserver/tls.yaml#L33 60 seconds]
|[https://github.com/wikimedia/puppet/blob/production/hieradata/common/profile/trafficserver/tls.yaml#L139 180 seconds]
|[https://github.com/wikimedia/puppet/blob/765d39f66320a4def7adccaa8a63fc970e278eb0/hieradata/common/profile/trafficserver/tls.yaml#L140 3 seconds]
|[https://github.com/wikimedia/puppet/blob/production/hieradata/common/profile/trafficserver/tls.yaml#L144 200 seconds]
|[https://github.com/wikimedia/puppet/blob/765d39f66320a4def7adccaa8a63fc970e278eb0/hieradata/common/profile/trafficserver/tls.yaml#L145 180 seconds]
|[https://github.com/wikimedia/puppet/blob/765d39f66320a4def7adccaa8a63fc970e278eb0/hieradata/common/profile/trafficserver/tls.yaml#L145 180 seconds]
|[https://github.com/wikimedia/puppet/blob/1410c8aa6043d002aaf32ca49cdc4bd4c3434927/hieradata/common/profile/trafficserver/tls.yaml#L140 120 seconds]
|[https://github.com/wikimedia/puppet/blob/1410c8aa6043d002aaf32ca49cdc4bd4c3434927/hieradata/common/profile/trafficserver/tls.yaml#L140 120 seconds]
|}
|}
Line 37: Line 40:
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/role/common/cache/text.yaml#L414 65 seconds]<sup>text</sup> / [https://github.com/wikimedia/puppet/blob/1410c8aa6043d002aaf32ca49cdc4bd4c3434927/hieradata/role/common/cache/upload.yaml#L34 35 seconds]<sup>upload</sup>
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/role/common/cache/text.yaml#L414 65 seconds]<sup>text</sup> / [https://github.com/wikimedia/puppet/blob/1410c8aa6043d002aaf32ca49cdc4bd4c3434927/hieradata/role/common/cache/upload.yaml#L34 35 seconds]<sup>upload</sup>
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/role/common/cache/text.yaml#L415 33 seconds]<sup>text</sup> / [https://github.com/wikimedia/puppet/blob/1410c8aa6043d002aaf32ca49cdc4bd4c3434927/hieradata/role/common/cache/upload.yaml#L35 60 seconds]<sup>upload</sup>
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/role/common/cache/text.yaml#L415 33 seconds]<sup>text</sup> / [https://github.com/wikimedia/puppet/blob/1410c8aa6043d002aaf32ca49cdc4bd4c3434927/hieradata/role/common/cache/upload.yaml#L35 60 seconds]<sup>upload</sup>
|-
|varnish-backend
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/role/common/cache/text.yaml#L29 3 seconds]<sup>text</sup>
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/role/common/cache/text.yaml#L30 63 seconds]<sup>text</sup>
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/role/common/cache/text.yaml#L31 31 seconds]<sup>text</sup>
|-
|-
|ats-backend
|ats-backend
|N/A (fused together with TTFB)
|[https://github.com/wikimedia/puppet/blob/765d39f66320a4def7adccaa8a63fc970e278eb0/hieradata/common/profile/trafficserver/backend.yaml#L388 10 seconds]
|[https://github.com/wikimedia/puppet/blob/production/hieradata/common/profile/trafficserver/backend.yaml#L324 180 seconds]<sup>GET</sup> / [https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/common/profile/trafficserver/backend.yaml#L327 180 seconds]<sup>POST,PUT</sup>
|[https://github.com/wikimedia/puppet/blob/765d39f66320a4def7adccaa8a63fc970e278eb0/hieradata/common/profile/trafficserver/backend.yaml#L393 180 seconds]
|[https://github.com/wikimedia/puppet/blob/a9b571595f0e97fe335e81a0b03d31e284271ec8/hieradata/common/profile/trafficserver/backend.yaml#L329 200 seconds]
|[https://github.com/wikimedia/puppet/blob/765d39f66320a4def7adccaa8a63fc970e278eb0/hieradata/common/profile/trafficserver/backend.yaml#L393 180 seconds]
|}
|}
After leaving the backend caching layer, the request reaches the appserver. Here are described the timeouts that apply to appservers and api:
After leaving the backend caching layer, the request reaches the appserver. Here are described the timeouts that apply to appservers and api:

Revision as of 10:34, 21 January 2020

This page is an attempt of documenting the timeouts involved in a request performed by a user against a service behind WMF caching layer.


The entry point for a user could be nginx or ats-tls depending on the service and the cache node assigned to the user IP:

TLS termination layer SSL handshake timeout connect timeout (origin server) TTFB (origin server) successive reads (origin server) Keepalive timeout (client)
nginx (deprecated) 60 seconds (nginx default value) 10 seconds (nginx default value) 180 seconds 180 seconds (same config parameter as TTFB) 60 seconds
ats-tls 60 seconds 3 seconds 180 seconds 180 seconds 120 seconds

Currently a big difference between nginx and ats-tls can be found on how they handle POST requests. nginx buffers the whole request completely before relying it to the origin (varnish-frontend) while ats-tls doesn't buffer it and relays the connection to varnish-frontend as soon as possible. On nginx, the timeout to fulfil the POST body is 60 seconds between read operations, this is the default value and it isn't explicitly configured.

Our caching system is split in two layers (frontend and backend). There is one implementation of the frontend layer (varnish) and two implementations of the backend layer (varnish-be and ats-be).

caching layer connect timeout TTFB successive reads
varnish-frontend 3 secondstext / 5 secondsupload 65 secondstext / 35 secondsupload 33 secondstext / 60 secondsupload
ats-backend 10 seconds 180 seconds 180 seconds

After leaving the backend caching layer, the request reaches the appserver. Here are described the timeouts that apply to appservers and api:

layer request timeout
Nginx (TLS/ats-be requests) N/A (same timeouts as the nginx used for TLS termination)
Envoy(TLS/ats-be requests) 1 secondconnect timeout / 65 secondsroute timeout
Apache 202 seconds
PHP 201 secondsappservers / 201 secondsapi
Excimer 60 secondsGET / 200 secondsPOST

Note: Those timeouts might be larger than the ones on the caching layer, mainly to properly service internal clients