You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

User:AKhatun/Wikidata Subgraph Analysis: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>AKhatun
(Add subgraph info table)
imported>AKhatun
(Add triples per item section)
Line 71: Line 71:
'''Number of days to recovery'''
'''Number of days to recovery'''
Given the current rate of growth, how long would it take wikidata to get back to its original size again if some amount of triples were removed from it? This helps us estimate what to temporarily remove from Wikidata in the siatuation Wikidata backend maxes out. The growth rate of triples is not constant, but considering the growth an approximate straight line, in [https://grafana.wikimedia.org/goto/pyO_iMRnk grafana dashboard], Wikidata grows at a rate of '''4.77M''' triples per day. This rate was calculated from the number of triples at the start and end of a 90-day interval (11/3/21 to 6/6/21). It could be faster or a bit slower than this. This will give us a wide approximation of the number of days we can gain by removing some parts of Wikidata.
Given the current rate of growth, how long would it take wikidata to get back to its original size again if some amount of triples were removed from it? This helps us estimate what to temporarily remove from Wikidata in the siatuation Wikidata backend maxes out. The growth rate of triples is not constant, but considering the growth an approximate straight line, in [https://grafana.wikimedia.org/goto/pyO_iMRnk grafana dashboard], Wikidata grows at a rate of '''4.77M''' triples per day. This rate was calculated from the number of triples at the start and end of a 90-day interval (11/3/21 to 6/6/21). It could be faster or a bit slower than this. This will give us a wide approximation of the number of days we can gain by removing some parts of Wikidata.
== Triples ==
The triples within a subgraph can be of various types. They can be:
* truthy triples like <code>wdt</code>
* non-wikidata direct triples like rdfs:label, schema:name etc
* full statements that hold other information
See more about these statement types here: [[mw:wiki/Wikibase/Indexing/RDF_Dump_Format#Statement_types|RDF_Dump_Format#Statement_types]].
== Table of top 50 subgraph information ==


{| class="wikitable sortable"
{| class="wikitable sortable"
|+ Top 50 Subgraphs in Wikidata
|+ Top 50 Subgraphs in Wikidata
|-
|-
! Rank !! Subgraph !! Subgraph Name !! Number of items !! % of WD items !! Number of triples !! % of WD Triples !! Number of days to recover
! Rank !! Subgraph !! Subgraph Name !! Number of items !! % of WD items !! Number of triples !! % of WD Triples !! Number of days to recover!!%of truthy statements!!%of non-wikidata direct statements!!%of full statements
|-
|-
|1||Q13442814||scholarly article||37,362,641||39.75||6,539,020,889||49.73||1370.86
|1||Q13442814||scholarly article||37,362,641||39.75||6,539,020,889||49.73||1370.86||12.62||24.28||63.25
|-
|-
|2||Q6999||astronomical object||8,412,914||8.95||1,136,682,291||8.64||238.3
|2||Q6999||astronomical object||8,412,914||8.95||1,136,682,291||8.64||238.3||10.20||14.24||76.07
|-
|-
|3||Q5||human||9,315,444||9.91||954,536,943||7.26||200.11
|3||Q5||human||9,315,444||9.91||954,536,943||7.26||200.11||13.31||20.06||60.94
|-
|-
|4||Q4167836||Wikimedia category||4,840,195||5.15||753,127,982||5.73||157.89
|4||Q4167836||Wikimedia category||4,840,195||5.15||753,127,982||5.73||157.89||1.19||86.06||5.19
|-
|-
|5||Q16521||taxon||3,180,248||3.38||367,926,462||2.8||77.13
|5||Q16521||taxon||3,180,248||3.38||367,926,462||2.8||77.13||10.11||37.22||42.97
|-
|-
|6||Q101352||family name||481,445||0.51||187,299,892||1.42||39.27
|6||Q101352||family name||481,445||0.51||187,299,892||1.42||39.27||1.59||93.49||6.62
|-
|-
|7||Q4167410||Wikimedia disambiguation page||1,359,804||1.45||180,124,174||1.37||37.76
|7||Q4167410||Wikimedia disambiguation page||1,359,804||1.45||180,124,174||1.37||37.76||0.89||88.32||3.83
|-
|-
|8||Q7187||gene||1,196,361||1.27||122,421,508||0.93||25.66
|8||Q7187||gene||1,196,361||1.27||122,421,508||0.93||25.66||14.13||12.51||73.11
|-
|-
|9||Q11266439||Wikimedia template||845,852||0.9||114,308,711||0.87||23.96
|9||Q11266439||Wikimedia template||845,852||0.9||114,308,711||0.87||23.96||0.78||87.06||3.20
|-
|-
|10||Q11173||chemical compound||1,223,387||1.3||91,228,463||0.69||19.13
|10||Q11173||chemical compound||1,223,387||1.3||91,228,463||0.69||19.13||12.69||35.72||50.86
|-
|-
|11||Q8054||protein||986,599||1.05||88,483,828||0.67||18.55
|11||Q8054||protein||986,599||1.05||88,483,828||0.67||18.55||14.26||13.51||72.21
|-
|-
|12||Q3305213||painting||539,468||0.57||56,769,083||0.43||11.9
|12||Q3305213||painting||539,468||0.57||56,769,083||0.43||11.9||12.10||24.46||63.35
|-
|-
|13||Q13100073||village-level division in China||588,477||0.63||51,615,572||0.39||10.82
|13||Q13100073||village-level division in China||588,477||0.63||51,615,572||0.39||10.82||6.84||62.73||30.41
|-
|-
|14||Q11424||film||263,070||0.28||47,176,067||0.36||9.89
|14||Q11424||film||263,070||0.28||47,176,067||0.36||9.89||14.37||12.74||66.01
|-
|-
|15||Q486972||human settlement||563,958||0.6||39,590,792||0.3||8.3
|15||Q486972||human settlement||563,958||0.6||39,590,792||0.3||8.3||10.84||22.94||49.32
|-
|-
|16||Q13406463||Wikimedia list article||334,939||0.36||33,742,245||0.26||7.07
|16||Q13406463||Wikimedia list article||334,939||0.36||33,742,245||0.26||7.07||2.45||78.06||10.48
|-
|-
|17||Q13433827||encyclopedia article||512,141||0.55||33,373,227||0.25||7.0
|17||Q13433827||encyclopedia article||512,141||0.55||33,373,227||0.25||7.0||9.03||45.52||39.76
|-
|-
|18||Q8502||mountain||525,553||0.56||33,340,188||0.25||6.99
|18||Q8502||mountain||525,553||0.56||33,340,188||0.25||6.99||11.37||27.79||50.39
|-
|-
|19||Q2668072||collection||500,968||0.53||32,670,637||0.25||6.85
|19||Q2668072||collection||500,968||0.53||32,670,637||0.25||6.85||15.24||12.30||72.71
|-
|-
|20||Q79007||street||578,926||0.62||30,252,119||0.23||6.34
|20||Q79007||street||578,926||0.62||30,252,119||0.23||6.34||13.86||24.24||59.88
|-
|-
|21||Q4022||river||399,552||0.42||28,833,476||0.22||6.04
|21||Q4022||river||399,552||0.42||28,833,476||0.22||6.04||11.34||24.99||52.15
|-
|-
|22||Q30612||clinical trial||356,838||0.38||27,731,502||0.21||5.81
|22||Q30612||clinical trial||356,838||0.38||27,731,502||0.21||5.81||16.05||12.33||71.78
|-
|-
|23||Q532||village||274,840||0.29||26,483,275||0.2||5.55
|23||Q532||village||274,840||0.29||26,483,275||0.2||5.55||9.69||26.25||45.22
|-
|-
|24||Q17633526||Wikinews article||286,950||0.3||21,830,150||0.17||4.58
|24||Q17633526||Wikinews article||286,950||0.3||21,830,150||0.17||4.58||3.68||72.76||16.46
|-
|-
|25||Q482994||album||269,095||0.29||21,181,015||0.16||4.44
|25||Q482994||album||269,095||0.29||21,181,015||0.16||4.44||12.20||22.64||53.05
|-
|-
|26||Q23397||lake||260,135||0.28||18,053,096||0.14||3.78
|26||Q23397||lake||260,135||0.28||18,053,096||0.14||3.78||11.32||25.43||53.19
|-
|-
|27||Q54050||hill||327,277||0.35||17,228,390||0.13||3.61
|27||Q54050||hill||327,277||0.35||17,228,390||0.13||3.61||12.67||23.07||56.77
|-
|-
|28||Q16970||church building||211,291||0.22||16,821,530||0.13||3.53
|28||Q16970||church building||211,291||0.22||16,821,530||0.13||3.53||14.03||15.06||63.56
|-
|-
|29||Q41176||building||265,925||0.28||16,293,008||0.12||3.42
|29||Q41176||building||265,925||0.28||16,293,008||0.12||3.42||14.21||14.99||68.06
|-
|-
|30||Q56436498||village in India||145,824||0.16||15,383,416||0.12||3.23
|30||Q56436498||village in India||145,824||0.16||15,383,416||0.12||3.23||8.34||22.23||64.50
|-
|-
|31||Q4830453||business||193,858||0.21||14,101,220||0.11||2.96
|31||Q4830453||business||193,858||0.21||14,101,220||0.11||2.96||13.06||16.23||60.48
|-
|-
|32||Q47150325||calendar day of a given year||189,366||0.2||14,078,486||0.11||2.95
|32||Q47150325||calendar day of a given year||189,366||0.2||14,078,486||0.11||2.95||6.85||56.25||34.18
|-
|-
|33||Q3947||house||197,736||0.21||12,468,434||0.1||2.61
|33||Q3947||house||197,736||0.21||12,468,434||0.1||2.61||15.26||15.18||70.67
|-
|-
|34||Q3331189||version, edition, or translation||157,486||0.17||10,997,589||0.08||2.31
|34||Q3331189||version, edition, or translation||157,486||0.17||10,997,589||0.08||2.31||15.41||14.70||69.39
|-
|-
|35||Q18593264||item of collection or exhibition||147,402||0.16||10,732,969||0.08||2.25
|35||Q18593264||item of collection or exhibition||147,402||0.16||10,732,969||0.08||2.25||16.96||9.69||73.38
|-
|-
|36||Q27020041||sports season||158,877||0.17||10,693,504||0.08||2.24
|36||Q27020041||sports season||158,877||0.17||10,693,504||0.08||2.24||12.11||15.12||53.43
|-
|-
|37||Q355304||watercourse||174,620||0.19||10,080,421||0.08||2.11
|37||Q355304||watercourse||174,620||0.19||10,080,421||0.08||2.11||12.17||25.07||54.79
|-
|-
|38||Q7725634||literary work||164,860||0.18||10,049,521||0.08||2.11
|38||Q7725634||literary work||164,860||0.18||10,049,521||0.08||2.11||13.85||16.69||58.88
|-
|-
|39||Q23442||island||148,587||0.16||9,885,277||0.08||2.07
|39||Q23442||island||148,587||0.16||9,885,277||0.08||2.07||11.40||22.52||50.06
|-
|-
|40||Q11060274||print||119,806||0.13||9,700,063||0.07||2.03
|40||Q11060274||print||119,806||0.13||9,700,063||0.07||2.03||14.85||9.74||76.96
|-
|-
|41||Q811979||architectural structure||145,957||0.16||9,666,936||0.07||2.03
|41||Q811979||architectural structure||145,957||0.16||9,666,936||0.07||2.03||10.06||9.91||52.63
|-
|-
|42||Q5084||hamlet||118,188||0.13||9,013,534||0.07||1.89
|42||Q5084||hamlet||118,188||0.13||9,013,534||0.07||1.89||10.94||18.63||55.55
|-
|-
|43||Q9842||primary school||157,451||0.17||8,916,373||0.07||1.87
|43||Q9842||primary school||157,451||0.17||8,916,373||0.07||1.87||13.99||16.22||68.98
|-
|-
|44||Q19389637||biographical article||151,026||0.16||8,238,397||0.06||1.73
|44||Q19389637||biographical article||151,026||0.16||8,238,397||0.06||1.73||12.76||21.07||57.51
|-
|-
|45||Q21014462||cell line||128,805||0.14||7,955,975||0.06||1.67
|45||Q21014462||cell line||128,805||0.14||7,955,975||0.06||1.67||10.62||36.18||54.60
|-
|-
|46||Q47521||stream||124,853||0.13||6,654,366||0.05||1.4
|46||Q47521||stream||124,853||0.13||6,654,366||0.05||1.4||12.95||19.67||58.11
|-
|-
|47||Q59199015||group of stereoisomers||111,599||0.12||5,843,270||0.04||1.23
|47||Q59199015||group of stereoisomers||111,599||0.12||5,843,270||0.04||1.23||15.43||18.82||67.23
|-
|-
|48||Q61443690||branch post office||129,183||0.14||5,313,033||0.04||1.11
|48||Q61443690||branch post office||129,183||0.14||5,313,033||0.04||1.11||14.59||14.86||70.54
|-
|-
|49||Q49008||prime number||127,545||0.14||5,188,768||0.04||1.09
|49||Q49008||prime number||127,545||0.14||5,188,768||0.04||1.09||10.01||36.78||52.40
|-
|-
|50||Q4164871||position||120,117||0.13||4,720,668||0.04||0.99
|50||Q4164871||position||120,117||0.13||4,720,668||0.04||0.99||12.72||32.76||52.28
|}
|}


== Triples ==
== Triples per item ==
=== Direct triples vs statements ===
While it is interesting to see how big a subgraph is and how many items it has, it is helpful to know how many triples each item has typically in a given subgraph. A very basic idea can be gained from density of a subgraph, where <code>density = #of triples / #of items</code>, in [https://tanny411.github.io/Wikidata-WDQS-Analysis/subgraph_stats.html subgraph stats]. Below is a diagram of box plot showing triples per item distribution for the top 50 subgraphs. The box plot omits min and max values, shows only mean, median, Q1, and Q3.
=== Triples per item ===
 
[[File:subgraph_boxplot.png|1100px]]
 
== Properties count ==
== Properties count ==
=== Top predicates ===
=== Top predicates ===
= Rate of growth of subgraphs =
= Rate of growth of subgraphs =
Here is an interactive chart showing the growth of the top 50 subgraphs over a period of one month: [https://tanny411.github.io/Wikidata-WDQS-Analysis/subgraph_growth_rate.html subgraph_growth_rate]
= Subgraph Connectivity =
= Subgraph Connectivity =
== Through properties ==
== Through properties ==

Revision as of 11:47, 28 October 2021

TL;DR

What are subgraphs?

Wikidata contains all kinds of data from various aspects of knowledge. All of these data are highly inter-connected, but we can find some patterns. We find subgraphs within Wikidata and find out how large these subgraphs are, how connected they are, and finally how much these subgraphs are used (queried).

In order to find subgraphs, the following steps were taken:

  • Consider all items that are instance of (P31) the same item to be under a subgraph. For example: all items that are instance of Q13442814 are part of one subgraph.
  • Some subgraphs were merged where it was obvious. For example: all subclasses of astronomical object were considered part of astronomical object as they were all indeed some sort of astronomical object. This method of sublcass merging is not applicable everywhere without manual inspection.
  • Some large subgraphs were almost completely part of another subgraph. For example: all items under Review Articles are also instance of scholarly article. In such case, review articles was not considered a separate subgraph.

Subgraph sizes

Using only instance of, Wikidata has 82,919 subgraphs. The distribution of the sizes of these subgraphs has a clear long tail, with very few subgraphs incorporating most items in Wikidata. Subgraph size can be calculated in two ways:

  • The number of items it contains
  • The number of triples related to the items in a subgraph. This is what we refer as subgraph size from here on.

Takeaways:

  • Most calculations from here on will take the top 50 subgraphs, which form 85% of Wikidata
  • 340 top subgraphs (0.5% of all subgraphs, after merging some) form 90% of Wikidata (91% of all items and 90% of all triples). These subgraphs have >=10,000 items each.
  • Rest 99.5% of the subgraphs have <10,000 items each, and together form 10% of Wikidata.

Below is the distribution of the number of items in a subgraphs.

File:Number of groups vs number of items.png File:Number of groups vs number of items log.png

To be more specific,

Subgraph item distribution
Number of subgraphs Number of items
There are 54,602 subgraph(s) with more than 1 item(s)
23,724 10
6,625 100
1,712 1,000
392 10,000
63 100,000
10 1,000,000
1 10,000,000

Below is the subgraph size comparison of top 340 subgraphs in Wikidata (90%).

File:Subgraph distribution triples.png File:Subgraph distribution percents.png

Below is the subgraph size comparison of top 50 subgraphs in Wikidata (85%).

File:Top 50 subgraph distribution triples.png File:Top 50 subgraph distribution percents.png

Here is an interactive graph showing the comparison of subgraph sizes in terms of item count and triple count: subgraph stats.

Here are some subgraph size visualizations in WDQS:

  • Size as percentage of Wikidata each subgraph occupies: query link
  • Size as percentage of Wikidata items each subgraph contains: query link

Number of days to recovery Given the current rate of growth, how long would it take wikidata to get back to its original size again if some amount of triples were removed from it? This helps us estimate what to temporarily remove from Wikidata in the siatuation Wikidata backend maxes out. The growth rate of triples is not constant, but considering the growth an approximate straight line, in grafana dashboard, Wikidata grows at a rate of 4.77M triples per day. This rate was calculated from the number of triples at the start and end of a 90-day interval (11/3/21 to 6/6/21). It could be faster or a bit slower than this. This will give us a wide approximation of the number of days we can gain by removing some parts of Wikidata.

Triples

The triples within a subgraph can be of various types. They can be:

  • truthy triples like wdt
  • non-wikidata direct triples like rdfs:label, schema:name etc
  • full statements that hold other information

See more about these statement types here: RDF_Dump_Format#Statement_types.

Table of top 50 subgraph information

Top 50 Subgraphs in Wikidata
Rank Subgraph Subgraph Name Number of items % of WD items Number of triples % of WD Triples Number of days to recover %of truthy statements %of non-wikidata direct statements %of full statements
1 Q13442814 scholarly article 37,362,641 39.75 6,539,020,889 49.73 1370.86 12.62 24.28 63.25
2 Q6999 astronomical object 8,412,914 8.95 1,136,682,291 8.64 238.3 10.20 14.24 76.07
3 Q5 human 9,315,444 9.91 954,536,943 7.26 200.11 13.31 20.06 60.94
4 Q4167836 Wikimedia category 4,840,195 5.15 753,127,982 5.73 157.89 1.19 86.06 5.19
5 Q16521 taxon 3,180,248 3.38 367,926,462 2.8 77.13 10.11 37.22 42.97
6 Q101352 family name 481,445 0.51 187,299,892 1.42 39.27 1.59 93.49 6.62
7 Q4167410 Wikimedia disambiguation page 1,359,804 1.45 180,124,174 1.37 37.76 0.89 88.32 3.83
8 Q7187 gene 1,196,361 1.27 122,421,508 0.93 25.66 14.13 12.51 73.11
9 Q11266439 Wikimedia template 845,852 0.9 114,308,711 0.87 23.96 0.78 87.06 3.20
10 Q11173 chemical compound 1,223,387 1.3 91,228,463 0.69 19.13 12.69 35.72 50.86
11 Q8054 protein 986,599 1.05 88,483,828 0.67 18.55 14.26 13.51 72.21
12 Q3305213 painting 539,468 0.57 56,769,083 0.43 11.9 12.10 24.46 63.35
13 Q13100073 village-level division in China 588,477 0.63 51,615,572 0.39 10.82 6.84 62.73 30.41
14 Q11424 film 263,070 0.28 47,176,067 0.36 9.89 14.37 12.74 66.01
15 Q486972 human settlement 563,958 0.6 39,590,792 0.3 8.3 10.84 22.94 49.32
16 Q13406463 Wikimedia list article 334,939 0.36 33,742,245 0.26 7.07 2.45 78.06 10.48
17 Q13433827 encyclopedia article 512,141 0.55 33,373,227 0.25 7.0 9.03 45.52 39.76
18 Q8502 mountain 525,553 0.56 33,340,188 0.25 6.99 11.37 27.79 50.39
19 Q2668072 collection 500,968 0.53 32,670,637 0.25 6.85 15.24 12.30 72.71
20 Q79007 street 578,926 0.62 30,252,119 0.23 6.34 13.86 24.24 59.88
21 Q4022 river 399,552 0.42 28,833,476 0.22 6.04 11.34 24.99 52.15
22 Q30612 clinical trial 356,838 0.38 27,731,502 0.21 5.81 16.05 12.33 71.78
23 Q532 village 274,840 0.29 26,483,275 0.2 5.55 9.69 26.25 45.22
24 Q17633526 Wikinews article 286,950 0.3 21,830,150 0.17 4.58 3.68 72.76 16.46
25 Q482994 album 269,095 0.29 21,181,015 0.16 4.44 12.20 22.64 53.05
26 Q23397 lake 260,135 0.28 18,053,096 0.14 3.78 11.32 25.43 53.19
27 Q54050 hill 327,277 0.35 17,228,390 0.13 3.61 12.67 23.07 56.77
28 Q16970 church building 211,291 0.22 16,821,530 0.13 3.53 14.03 15.06 63.56
29 Q41176 building 265,925 0.28 16,293,008 0.12 3.42 14.21 14.99 68.06
30 Q56436498 village in India 145,824 0.16 15,383,416 0.12 3.23 8.34 22.23 64.50
31 Q4830453 business 193,858 0.21 14,101,220 0.11 2.96 13.06 16.23 60.48
32 Q47150325 calendar day of a given year 189,366 0.2 14,078,486 0.11 2.95 6.85 56.25 34.18
33 Q3947 house 197,736 0.21 12,468,434 0.1 2.61 15.26 15.18 70.67
34 Q3331189 version, edition, or translation 157,486 0.17 10,997,589 0.08 2.31 15.41 14.70 69.39
35 Q18593264 item of collection or exhibition 147,402 0.16 10,732,969 0.08 2.25 16.96 9.69 73.38
36 Q27020041 sports season 158,877 0.17 10,693,504 0.08 2.24 12.11 15.12 53.43
37 Q355304 watercourse 174,620 0.19 10,080,421 0.08 2.11 12.17 25.07 54.79
38 Q7725634 literary work 164,860 0.18 10,049,521 0.08 2.11 13.85 16.69 58.88
39 Q23442 island 148,587 0.16 9,885,277 0.08 2.07 11.40 22.52 50.06
40 Q11060274 print 119,806 0.13 9,700,063 0.07 2.03 14.85 9.74 76.96
41 Q811979 architectural structure 145,957 0.16 9,666,936 0.07 2.03 10.06 9.91 52.63
42 Q5084 hamlet 118,188 0.13 9,013,534 0.07 1.89 10.94 18.63 55.55
43 Q9842 primary school 157,451 0.17 8,916,373 0.07 1.87 13.99 16.22 68.98
44 Q19389637 biographical article 151,026 0.16 8,238,397 0.06 1.73 12.76 21.07 57.51
45 Q21014462 cell line 128,805 0.14 7,955,975 0.06 1.67 10.62 36.18 54.60
46 Q47521 stream 124,853 0.13 6,654,366 0.05 1.4 12.95 19.67 58.11
47 Q59199015 group of stereoisomers 111,599 0.12 5,843,270 0.04 1.23 15.43 18.82 67.23
48 Q61443690 branch post office 129,183 0.14 5,313,033 0.04 1.11 14.59 14.86 70.54
49 Q49008 prime number 127,545 0.14 5,188,768 0.04 1.09 10.01 36.78 52.40
50 Q4164871 position 120,117 0.13 4,720,668 0.04 0.99 12.72 32.76 52.28

Triples per item

While it is interesting to see how big a subgraph is and how many items it has, it is helpful to know how many triples each item has typically in a given subgraph. A very basic idea can be gained from density of a subgraph, where density = #of triples / #of items, in subgraph stats. Below is a diagram of box plot showing triples per item distribution for the top 50 subgraphs. The box plot omits min and max values, shows only mean, median, Q1, and Q3.

File:Subgraph boxplot.png

Properties count

Top predicates

Rate of growth of subgraphs

Here is an interactive chart showing the growth of the top 50 subgraphs over a period of one month: subgraph_growth_rate

Subgraph Connectivity

Through properties

Through items

Through triples

Triples connecting 'to' each subgraph

Triples connecting 'from' each subgraph

Venn

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=User:AKhatun/Wikidata_Subgraph_Analysis&oldid=584506"