User:AndreaWest/WDQS Testing

This page overviews a design and specific suggestions for Wikidata SPARQL query testing. These tests will be useful to evaluate Blazegraph backend alternatives and to (possibly) establish a Wikidata SPARQL benchmark for the industry.

Goals

• Definition of multiple test sets exercising the SPARQL functions and complexities seen in actual Wikidata queries, as well as extensions, federated query, and workloads
  • Definition of specific INSERT, DELETE, CONSTRUCT and SELECT queries for performance and capabilities analysis
  • Definition of read/write workloads for stress testing
  • Tests of system characteristics and SPARQL compliance, and to evaluate system behavior under load

Test Design

Design based on insights gathered (largely) from the following papers:

• An Analytical Study of Large SPARQL Query Logs
• Getting the Most out of Wikidata: Semantic Technology Usage in Wikipedia’s Knowledge Graph
• Navigating the Maze of Wikidata Query Logs

Also, the following analyses (conducted by members of the WDQS team) examined more recent data:

• WDQS Queries Analysis
• Subpages linked from https://wikitech.wikimedia.org/wiki/User:AKhatun

Testing SPARQL 1.1 and GeoSPARQL Compliance

Testing compliance to the SPARQL 1.1 specification (using the W3C test suite) will be accomplished using a modified form of the Tests for Triplestore (TFT) codebase. Details are provided on the Running TFT page.

GeoSPARQL testing will be accomplished similarly, and is also described on that same page.

Testing Wikidata-Specific Updates and Queries

This section expands on the specific SPARQL language constructs (such as FILTER, OPTIONAL, GROUP BY, ...), and query and update patterns that will be tested. Testing includes federated and geospatial queries, and support for the (evolution of the) label, GAS and MediaWiki local SERVICEs.

As regards SPARQL, tests are defined to exercise:

• SELECT, ASK and CONSTRUCT queries, as well as INSERT and DELETE updates
• Language keywords
  • Solution modifiers - Distinct, Limit, Offset, Order By, Offset
  • Assignment operators - Bind, Values
  • Algebraic operators - Filter, Union, Optional, Exists, Not Exists, Minus
  • Aggregation operators - Count, Min/Max, Avg, Sum, Group By, Group_Concat, Sample, Having
• Subqueries
• With both constants and variables in the triples
• With varying numbers of triples (from 1 to 50+)
• With combinations (co-occurrences) of the above language constructs
• Utilizing different property path lengths and structures
  • For example, property paths of the form, a*, ab*, ab*c, abc*, a|b, a*|b*, etc.
• Using different graph patterns
  • From simple chains of nodes (such as a 'connected to' b 'connected to' c, e.g., a - b - c) to
  • "Stars" (consisting of a set of nodes where there is only 1 path between any 2 nodes and at most one node can have more than 2 neighbors - for example, a 'connected to' b + c also 'connected to' b + b - d - e - f)
  • "Trees" (consisting of a set of nodes where there is only 1 path between any 2 nodes, e.g., a collection of stars) to
  • "Petals" (where there may be multiple paths between 2 nodes - for example, a - b - c or a - z - c defines two paths from a to c) to
  • "Flowers" (which have chains + trees + petals) to
  • "Bouquets" (which have component flowers)
  • Using the terminology of Navigating the Maze of Wikidata Query Logs
• Cold-start and warm-start scenarios (to understand the effects of caching)
• Mixes of highly selective, equally selective and non-selective triples (to understand optimization)
• Small and large result sets, some with the potential for large intermediate result sets

These tests for capabilities are defined using static queries. They will be executed using the updated TFT framework to evaluate a triple store's/endpoint's support (or lack of support) for each of the Wikidata requirements, as well as the correctness and completeness of the response. In addition, the tests will be executed using the modified Iguana framework, to get an understanding of execution times.

Since TFT loads the test data for each query, a small data set is used in this environment. It is available from the wikidata-tests repository, in the tft directory (link to be provided), and will be included in the TFT repository as a submodule. The corresponding test definition for use in the Iguana framework is defined in the iguana-compliance directory (link to be provided). Details coming.

Workload Testing

This evaluation utilizes combinations of the above queries/updates (and others) with the proportions of different query complexities defined based on these investigations:

• Wikidata analysis
• WDQS Queries Analysis
• WDQS Triples Analysis

The loading will be based on the:

• Highest (+ some configurable percentage) and lowest number of "queries per second" (for a single server)
  • As captured on the WDQS queries dashboard
• Highest (+ some configurable percentage) and lowest, added and deleted "triples ingestion rate" (for a single server)
  • As captured on the Streaming Updater dashboard

Note that these workloads reflect both user and bot queries.

The tests are defined using query patterns based on the compliance queries from above, with additional updates and queries informed by the analyses. They will be executed using a modified version of the Iguana Framework on a single machine, using multiple "client" threads/workers. The following statistics will be reported:

• Total execution time for each query overall and by worker
• Minimum and maximum execution times for each query (if successful) by worker and across all workers
• Mean and geometric mean of each query (that successfully executed) by worker and across all workers
• Mean and geometric mean of each query using its execution time (if successful) or a penalized amount (= timeout by default) for failed queries
  • By query overall and by worker
  • This adjusts for queries completing quickly due to errors (e.g., they will have a low execution time but not produce results)
• Number of queries overall that executed and completed by worker and dataset
• Number of queries that timed out by worker and dataset
• Average number of queries per second (across all queries) that can be processed by a triple store for the data set

As above, the test details are defined in the wikidata-tests repository, in the iguana-stresstest directory (link to be provided). The stress test/workload environment assumes that the complete Wikidata RDF is loaded, and will be executed using the modified Iguana framework. More details coming.

Testing the Evaluation Infrastructure

The full Wikidata dump will be used for evaluation testing. However, a small subset of Wikidata has been created as a test data set, to evaluate the testing infrastructure. The details of that data set and its evaluation using a local Stardog installation are shown on this page.