HHVM/Troubleshooting

Problem determination and data collection

• hhvm-dump-debug: captures a stack trace of HHVM and writes it to stdout and to /tmp/hhvm.$PID.bt. Run hhvm-dump-debug --full to generate a core dump as well.
• links -dump http://127.0.0.1/server-status -width 80: produce a full Apache status report, showing which requests are currently in flight.
• furl is a cURL-like tool for FastCGI. You can use it to bypass Apache and request pages directly from HHVM. For example: furl http://en.wikipedia.org/wiki/Main_Page. Requests always go to localhost.
• curl localhost:9002/dump-apc will dump all the current APC keys to /tmp/apc_dump.
• stacktraces, if available, are in /var/log/hhvm
• core dumps are in /var/tmp/core or sysctl kernel.core_pattern

Light process dying

HHVM runs requests in different threads but in a single process. fork() has to lock the entire memory space of the process so it can copy it, which blocks all requests that are currently in flight. If HHVM forks in advance, it can just have the child process sitting there, and then when a thread needs to shell out it can just call exec*.

If you see light processes deaths in the logs it could be because they always log when they exit (noisy but ignorable).

There are five pre-forked light processes so that worker threads, serving traffic, don't have to contend for one when they need an additional process forked.