Help:Toolforge/Web/Lighttpd

Toolforge

<inputbox> type=fulltext prefix="Help:Tool" searchbuttonlabel= Search placeholder=Search Toolforge Documentation width=35 break=no </inputbox>

• Toolforge technical documentation
• Toolforge administrator documentation
• Help and communication
• List of tools
• Terms and conditions
• Rules
• Glossary
• Technical documentation resources

Getting started

• Cloud Services, Cloud VPS, and Toolforge overview
• About Toolforge
• Toolforge quickstart guide
• How-tos, tutorials and walkthroughs
• Create a Wikimedia developer account
• Access Toolforge instances with PuTTY and WinSCP
• Troubleshooting Toolforge

Developing tools

• Tool accounts
• Developing successful tools
• Develop tools in Java
• Develop tools in Mono/.NET
• Develop tools in Python
• Develop tools using Pywikibot framework
• Develop tools in Rust
• Web servers for tools
• Grid engine
• Kubernetes

Additional documentation

• Version control in Toolforge
• Elasticsearch for Toolforge
• Redis for Toolforge
• Dumps and Toolforge
• Database

[]

Overview

Lighttpd is the HTTP server used by both the lighttpd and lighttpd-plain types supported by webservice. These types are supported by the Grid Engine backend. On the Kubernetes backend, the php7.4 type (and other PHP types) also uses lighttpd.

• The document root is $HOME/public_html.
  • See Changing the document root for some ideas on how to change that.
• Error logs from the lighttpd process are stored in $HOME/error.log
• PHP scripts are automatically run using a FastCGI helper process.
• The lighttpd web server is configurable (including adding other FastCGI handlers). A $HOME/.lighttpd.conf file can be used to change the default configuration.
• Everything runs as the tool user, regardless of file ownership.

The web server reads any configuration in $HOME/.lighttpd.conf and merges it with the default configuration. Most tools will not need custom configuration.

Sometimes merge fails if an option is already set in the default configuration. When this happens, try using option += value instead of option = value. Some settings cannot be changed, such as fastcgi.server += ( ".php" =>....

Default configuration

This is the default (if you don't specify any other/additional settings in your tool's .lighttpd.conf)

See lighttpd.py in toolsws/wstypes/ for the canonical configuration.

Example configurations

FCGI Flask config

fastcgi.server += ( "/gerrit-patch-uploader" =>
    ((
        "socket" => "/tmp/patchuploader-fcgi.sock",
        "bin-path" => "/data/project/gerrit-patch-uploader/src/gerrit-patch-uploader/app.fcgi",
        "check-local" => "disable",
        "max-procs" => 1,
    ))
)

For Flask, the fcgi handler looks like this: https://github.com/valhallasw/gerrit-patch-uploader/blob/master/app.fcgi

URL rewrite

Documentation: ModRewrite

Note that rewrite rules always execute before redirect rules (regardless of their order in the config file).

A common rewrite scenario is a PHP application that uses a front controller script (often index.php) to decide how to process each request based on the path of the request. The url.rewrite-if-not-file directive can be used to do this while still letting lighttpd serve static files from $HOME/public_html normally:

url.rewrite-if-not-file += ( "(.*)" => "/index.php/$0" )

Alternately you can use url.rewrite-once to pick and choose which requests to rewrite:

url.rewrite-once += (
    ".*\.(js|css)" => "$0",
    "^(/.*)" => "/index.php$1"
)

url.rewrite-once += ( "/id/([0-9]+)" => "/index.php?id=$1",
                      "/link/([a-zA-Z]+)" => "/index.php?link=$1" )

The $0 matches the entire match from the left-hand side regular expression.

Header, mimetype, character encoding, error handler

# Allow Cross-Origin Resource Sharing (CORS)
# Please read up on CORS before setting such headers https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
# If you are handling anything with authentication and/or multiple origins you likely should set the correct header for a specific webresource, instead of setting them for all calls.
setenv.add-response-header += ( "Access-Control-Allow-Origin" => "en.wikipedia.org",
                                 "Access-Control-Allow-Methods" => "POST, GET, OPTIONS" )

# Set cache-control directive for static files and resources
$HTTP["url"] =~ "\.(jpg|gif|png|css|js|txt|ico)$" {
	setenv.add-response-header += ( "Cache-Control" => "max-age=86400, public" )
}

mimetype.assign += (
    # Add custom mimetype
    ".bulk"  => "text/plain",
    # Avoid [[Mojibake]] in JavaScript files
    ".js"   => "application/javascript; charset=utf-8",
    # Default MIME type with UTF-8 character encoding
    ""      => "text/plain; charset=utf-8"
)

# Add custom error-404 handler
server.error-handler-404 += "/error-404.php"

Details: ModSetEnv  Mimetype-Assign   Error-Handler-404   HTTP access control (CORS)

Directory or file index

# Enable basic directory index
$HTTP["url"] =~ "^/?" {
	dir-listing.activate = "enable"
}

Deny access to hidden files

# Deny access to hidden files
$HTTP["url"] =~ "/\." {
	url.access-deny = ("")
}

Details: ModAccess

Custom index

# Enable index for specific directory
$HTTP["url"] =~ "^/download($|/)" {
	dir-listing.activate = "enable"
}

# Custom index file or custom directory generator
index-file.names += ("index.py")

Details: ModDirlisting

Additional debug logging

Documentation: DebugVariables

Add the line:

# Enable request logging
debug.log-request-handling = "enable"

The debug output will be written to the error.log file.

Apache-like cgi-bin directory

Add the following stanza:

$HTTP["url"] =~ "^/cgi-bin" {
	cgi.assign = ( "" => "" )
}

This does require that cgi-bin be under your public_html rather than alongside it.

To run CGI from any directory under your public_html only need this one line (w/out the $HTTP["url"] .. block)

cgi.assign += ( ".cgi" => "" )

The part to the left is the file name or extension ("" = any). The part to the right is the program which will run it ("" = any). Another example

cgi.assign += ( "script.sh" => "/bin/bash" )

Enable status and statistics

# Source: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd#Enable_status_and_statistics
server.modules += ("mod_status")
status.status-url = "/server-status"
status.statistics-url = "/server-statistics"

Details: ModStatus

Web logs

Lighttpd error logs can be found in the tool account's $HOME/error.log; this includes the standard error of invoked scripts.

Beginning in October 2019, Lighttpd based webservices do not write access logs by default. Tool maintainers will be reminded of this by the following warning when starting a webservice:

******************************************************************************
Note that access.log is no longer enabled by default (see https://w.wiki/9go)
******************************************************************************

If a tool has a need for access log data, it can be enabled by adding the following line in the tool account's $HOME/.lighttpd.conf (replace <tool> with your tool account name):

accesslog.filename = "/data/project/<tool>/access.log"

You'll need to do a webservice restart to activate the new settings.

Please note that the web logs are anonymized in accordance with the Wikimedia Foundation’s privacy policy by preventing the tool's webservice from seeing the IP address of the visitor.

Error pages

Tracked in Phabricator
Task T103662

The proxy provides its own error pages when your application returns HTTP/500, HTTP/502 or HTTP/503. This behavior is intended to be friendly to tool users, but it can also make debugging more difficult for tool maintainers. phab:T103662 may change the default behavior if consensus is reached.

Changing the document root

With symlinks

The easiest way to change the document root is with a symlink to $HOME/public_html. However, before this is done, the existing public_html directory needs to be deleted or moved. This is because ln -s $HOME/foo $HOME/public_html would make $HOME/public_html/foo if the $HOME/public_html directory exists. Deleting the directory is done with rm -rf $HOME/public_html to delete the directory and all of its contents, if you do not need anything in there, or with mv $HOME/public_html $HOME/oldpublic_html to move the directory to $HOME/oldpublic_html.

To make the symlink, ln -s $HOME/foo $HOME/public_html/ would make the contents of $HOME/foo available in $HOME/public_html. Replace $HOME/foo in the example with the directory you want lighttpd to serve.

With aliases

Note that you cannot add an alias URL for /toolname because this has already been defined and can't be overridden in the local conf file.

PHP

The lighttpd webservice type includes support for running PHP scripts from files with a .php in $HOME/public_html using a FastCGI helper process.

PHP ini settings can be changed for your tool by creating a $HOME/public_html/.user.ini configuration file. This can be useful for setting the default timezone with date.timezone. See the documentation at php.net for more details.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

• Chat in real time in the IRC channel #wikimedia-cloud ^connect, the bridged Telegram group, or the bridged Mattermost channel
• Discuss via email after you subscribed to the cloud@ mailing list

Receive mail announcements about critical changes

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)

Track work tasks and report bugs

Use the Phabricator workboard #Cloud-Services for bug reports and feature requests about the Cloud VPS infrastructure itself

Learn about major near-term plans

Read the News wiki page

Read news and stories about Wikimedia Cloud Services

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)