You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

HHVM/Troubleshooting: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Elukey
No edit summary
imported>Mainframe98
(HHVM isn't used anymore)
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Archive}}
== Problem determination and data collection ==
== Problem determination and data collection ==


* <code>hhvmadm check-health</code>: emits useful statistics like load, queued reqs, etc..
* <code>hhvm-dump-debug</code>: captures a stack trace of HHVM and writes it to stdout and to <code>/tmp/hhvm.$PID.bt</code>. Run <code>hhvm-dump-debug --full</code> to generate a core dump as well.
* <code>hhvm-dump-debug</code>: captures a stack trace of HHVM and writes it to stdout and to <code>/tmp/hhvm.$PID.bt</code>. Run <code>hhvm-dump-debug --full</code> to generate a core dump as well.
* See [[Debugging in production#Locally]].
* <code><nowiki>links -dump http://127.0.0.1/server-status -width 80</nowiki></code>: produce a full Apache status report, showing which requests are currently in flight.
* <code><nowiki>links -dump http://127.0.0.1/server-status -width 80</nowiki></code>: produce a full Apache status report, showing which requests are currently in flight.
* <code>furl</code> is a cURL-like tool for FastCGI. You can use it to bypass Apache and request pages directly from HHVM. For example: <code><nowiki>furl http://en.wikipedia.org/wiki/Main_Page</nowiki></code>. Requests always go to localhost.
* <code>furl</code> is a cURL-like tool for FastCGI. You can use it to bypass Apache and request pages directly from HHVM. For example: <code><nowiki>furl http://en.wikipedia.org/wiki/Main_Page</nowiki></code>. Requests always go to localhost.
Line 9: Line 12:


== Light process dying ==
== Light process dying ==
HHVM runs requests in different threads but in a single process. fork() has to lock the entire memory space of the process so it can copy it, which blocks all requests that are currently in flight. If HHVM forks in advance, it can just have the child process sitting there, and then when a thread needs to shell out it can just call exec*.
If you see light processes deaths in the logs it could be because they always log when they exit (noisy but ignorable). Please check the parent page for an explanation of what a light process is.
 
If you see light processes deaths in the logs it could be because they always log when they exit (noisy but ignorable).
 
There are five pre-forked light processes so that worker threads, serving traffic, don't have to contend for one when they need an additional process forked.
 
[[Category:MediaWiki production]]

Latest revision as of 15:59, 21 December 2021

Problem determination and data collection

  • hhvmadm check-health: emits useful statistics like load, queued reqs, etc..
  • hhvm-dump-debug: captures a stack trace of HHVM and writes it to stdout and to /tmp/hhvm.$PID.bt. Run hhvm-dump-debug --full to generate a core dump as well.
  • See Debugging in production#Locally.
  • links -dump http://127.0.0.1/server-status -width 80: produce a full Apache status report, showing which requests are currently in flight.
  • furl is a cURL-like tool for FastCGI. You can use it to bypass Apache and request pages directly from HHVM. For example: furl http://en.wikipedia.org/wiki/Main_Page. Requests always go to localhost.
  • curl localhost:9002/dump-apc will dump all the current APC keys to /tmp/apc_dump.
  • stacktraces, if available, are in /var/log/hhvm
  • core dumps are in /var/tmp/core or sysctl kernel.core_pattern

Light process dying

If you see light processes deaths in the logs it could be because they always log when they exit (noisy but ignorable). Please check the parent page for an explanation of what a light process is.