You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Stream.wikimedia.org: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Krinkle
No edit summary
imported>BryanDavis
(RCStream is gone)
 
(6 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[File:RCStream_example.png|thumb|374px|Example client at [http://codepen.io/Krinkle/pen/laucI/?editors=101 codepen.io/Krinkle/pen/laucI]]]
'''stream.wikimedia.org''' hosts services that allow one to subscribe to real-time events from Wikimedia wikis.
'''stream.wikimedia.org''' is a service to provide a live data stream of edits on Wikimedia wikis that anyone can tap and use to power editor tools and web apps, create beautiful visualisations, inform research, and extend MediaWiki.
 
It uses [[RCStream]] to subscribe to the [[mw:Manual:RCFeed|RCFeed]] of Wikimedia's production cluster, and publishes this on the endpoint <code>stream.wikimedia.org/rc</code>. As a web developer, one can tap the stream using JavaScript. As an app developer, one can use a suitable client library for your platform.
 
Consuming RCStream is a cleaner approach than parsing the change messages from [[irc.wikimedia.org]]. Wikimedia Foundation continues to operate both (configured via <code>$wgRCFeeds</code> in [[wmf-config]]).
 
The RCStream server also responds at http://stream.wikimedia.org/rcstream_status with a simple text message; check this if you do not receive any events.
 
{{Alert|content=As of January 2015, RCStream implements version 0.9 of the Socket.IO protocol, not 1.0 ([[phab:T68232]]). See also [https://github.com/Automattic/socket.io/tree/0.9.17 socket.io 0.9] and [https://github.com/Automattic/socket.io-client/tree/0.9.17 socket.io-client 0.9] on GitHub for more information.}}
 
== Usage ==
 
* Several researchers are pulling the whole recent changes stream with all associated content (see wiki-research-l, December 2014)
* [[m:CVN|CVN]] (Not yet, as of February 2015)
 
== Clients and alternative access points ==
 
* [http://codepen.io/Krinkle/pen/laucI Demo client (CodePen)], Example listener for stream.wikimedia.org using JavaScript
* [http://datasift.com/source/44/wikipedia Datasift]: Stream of edits to the English Wikipedia (includes page content)
* http://wikimedia.meteor.com/
* [[mw:Manual:Pywikibot|pywikibot]]
 
== Server setup ==
[[RCStream]] runs on a set of backend servers (currently, <code>rcs100x</code>; [https://github.com/wikimedia/operations-puppet/blob/1dffa7ab/manifests/site.pp#L2087-L2095 puppet node]; [https://github.com/wikimedia/operations-puppet/blob/1dffa7ab/manifests/role/rcstream.pp puppet role]). Backend nodes: rcs1001, rcs1002.
 
The backend servers run instances of [[RCStream]], with an nginx reverse proxy on each server.
 
An [[LVS]] load balancer (<code>stream-lb</code>) is situated in front of the backend servers.
 
The [[Beta cluster]] has a simplified setup on a single VM instance running the rcstream role, exposed as http://stream.wmflabs.org.
 
== See also ==
* [[RCStream]]: The application running on this domain.
* [[irc.wikimedia.org]]: The service that will be superseded by stream.wikimedia.org.
* [https://github.com/wikimedia/operations-puppet/tree/1dffa7ab/modules/rcstream puppet-rcstream module]: Wikimedia's puppet module for RCStream.
* [https://github.com/wikimedia/operations-puppet/blob/1dffa7ab/manifests/role/rcstream.pp puppet-rcstream role]: Wikimedia's puppet role for stream.wikimedia.org.


* <code>stream.wikimedia.org/v2</code>: '''[[EventStreams]]''' - REST API using the [[w:Server-sent events|Server-Sent Events]] protocol (SSE). See https://stream.wikimedia.org/?doc.


{{lowercase}}
{{lowercase}}
[[Category:Services]]
[[Category:Services]]
[[Category:Recent changes]]

Latest revision as of 17:47, 6 July 2018

stream.wikimedia.org hosts services that allow one to subscribe to real-time events from Wikimedia wikis.