You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Developer Portal: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
imported>BryanDavis
(BryanDavis moved page Developer Portal to Developer.wikimedia.org: Make service name the primary page name)
 
Line 1: Line 1:
{{ptag|Wikimedia-Developer-Portal}}
#REDIRECT [[Developer.wikimedia.org]]
 
The '''Developer Portal''' is a central entry point for Wikimedia technical documentation. The portal is a static site built using [https://www.mkdocs.org/ MkDocs] and [https://squidfunk.github.io/mkdocs-material/ Material for MkDocs]. It uses a custom plugin to generate [https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html PO files] and integrate with translatewiki.net. For more information about the project, see [[mw:Developer Advocacy/Developer Portal]].
 
{{Clickable button 2|View the demo|class=mw-ui-progressive|url=https://developer-portal.wmcloud.org/}} {{Clickable button 2|Browse the repository|url=https://gerrit.wikimedia.org/g/wikimedia/developer-portal}}
 
== How it works ==
 
=== Rendering page content ===
 
The purpose of the Developer Portal is to help users find key documentation pages. The site is designed to make it easy to remix documentation links into new pages tailored for different purposes. Data in the Dev Portal is organized into three main types of data:
 
; documents
: YAML files in ''data/documents'' describe key documents. Each document can be assigned to one or more categories.
; categories
: YAML files in ''data/categories'' describe categories. Behind the scenes, our ''macros/__init__.py'' local plugin collects categories and their corresponding documents into data that can be used in jinja templates.
; markdown files
: Markdown files reside in ''src/'' subdirectories which correspond to the top-level sections of the dev portal site. Jinja template syntax inside those markdown files pulls in content from categories and documents.
 
=== Rendering the navigation ===
 
The Material theme for Mkdocs generates [https://squidfunk.github.io/mkdocs-material/setup/setting-up-navigation/#navigation-sections navigation sections] based on the documents in a given subdirectory. It renders tables of contents based on the structure of an individual markdown file (its headings).
 
The <code>navigation.indexes</code> setting in mkdocs.yml implements [https://squidfunk.github.io/mkdocs-material/setup/setting-up-navigation/#section-index-pages section index pages]. This causes the index.md file for a directory to be used as the landing page for a section. We use this feature because our index.md files (in /src) only include links to the other pages in the directory, so we don't need or want them to be displayed as an additional item to click in the left nav.
 
While it could be simpler to use a single markdown file for each site section, and rely on the TOC for navigation, this has drawbacks:
*  The TOC disappears on small screens, effectively removing navigation to key subsections.
*  Using multiple markdown files enables navigation through and between site sections via buttons at the bottom of the screen.  This is a nice additional navigation option to supplement the left and/or horizontal menus.
*  Using a single index.md file populated via YAML for each site section means that nearly all the site structure will be rendered via YAML and templates. This means a larger number of YAML files, and a more difficult structure to understand if you're encountering the site for the first time. It's easier to understand how the site is rendered -- and thus better for future maintainers -- when more of the structure is coming from markdown and less from YAML and Jinja magic.
 
For more details about why we made these implementation decisions, see [https://miro.com/app/board/o9J_lkldIfg=/?moveToWidget=3074457367215289187&cot=14 this Miro board] (restricted access).
 
== How to use it ==
 
The Material theme for Mkdocs renders content in ways that can be confusing due to our usage of templates to generate page content based on yaml docs and categories (see the /data subdirectory).
 
This section describes the content requirements that doc editors and creators must follow in order for the site to render correctly and consistently.
 
=== Hide TOC in subdirectory index files ===
 
index.md files in subdirectories of the /src directory must hide the table of contents. This is necessary because the content of most of these pages is navigational: the section headers in the page mirror the list of other pages in the directory/navigation section, so the navigation and the table of contents render the same content. This is only true for subdirectories of the /src directory–not for the top-level index.md nor for any other markdown files.
 
To hide the table of contents, include the following at the top of the index.md file:
 
<syntaxhighlight lang="yaml">
---
hide:
  - toc
---
</syntaxhighlight>
 
=== Mirror text copy in index files and category files ===
 
In general (but not always), each section (<code>##</code>) in a subdirectory index.md file should contain:
 
* A text snippet that is the same as the category description used in the Jinja template on the subpage to which that section links
* A link to another md file (subpage) in that directory
<!--TODO: add examples to make this clearer -->
 
=== Render a document ===
To render a single document, add this syntax to a markdown file:
 
<syntaxhighlight lang="jinja">
{{ document.render( "document-filename-here" ) }}
</syntaxhighlight>
 
This renders the document with an h2 heading. To customize the heading level, add the level as an argument.
 
For example, this syntax renders a document with an h3 heading:
<syntaxhighlight lang="jinja">
{{ document.render( "document-filename-here", 3 ) }}
</syntaxhighlight>
 
=== Render a category ===
To render a category and its assigned documents, add this syntax to a markdown file.
 
Render one category per page:
<syntaxhighlight lang="jinja">
{{ category.render( "category-name-here" ) }}
</syntaxhighlight>
 
Render multiple categories per page:
<syntaxhighlight lang="jinja">
{{ category.render( "category-name-here", 2 ) }}
</syntaxhighlight>
 
=== Order documents in a category ===
 
To control the order that documents appear in a category, use this syntax in the document YAML definition:
 
<syntaxhighlight lang="yaml">
categories:
    category-a: 1
    category-b: 2
    category-c: 0
</syntaxhighlight>
 
In this example, the document will be ordered first in category-a, second in category-b, and unordered in category-c. Note that if a document needs to be ordered in ''any'' of its category, all listed categories must use the dictionary syntax instead of the list syntax (<code>category-a: 1</code> instead of <code>- category-a</code>).
 
If a category contains both ordered and unordered documents, the unordered documents are listed first since their index is 0.
 
== Content conventions ==
 
* In most cases, documentation links should link to the top of the page, not to a specific section. Exceptions include linking to specific endpoints or endpoint groups in an OpenAPI specification or in cases where it is not possible to break up a long page.
* All documentation links should be stored in data/documents, not in markdown files, with the exception of the intro on the site homepage.
 
== Mkdocs plugins ==
=== Macros module ===
The upstream [https://github.com/fralau/mkdocs_macros_plugin macros plugin] adds support for [https://jinja.palletsprojects.com/en/3.0.x/ jinja templates] to the markdown pages of a MkDocs website. We [https://mkdocs-macros-plugin.readthedocs.io/en/latest/macros/ extend] this plugin using a [[gerrit:plugins/gitiles/wikimedia/developer-portal/+/refs/heads/main/macros/__init__.py|custom "macros" Python module]]. This exposes various variables and macros to the jinja layer.
 
==== Variables ====
; {{Anchor|macro-var-categories}}categories
: Dict of keyed by category name of information loaded from YAML files found in ''data/categories''. In addition to the data found by parsing the YAML file, each value contains a "documents" list which includes every document from ''data/documents'' which was tagged with the category.
; {{Anchor|macro-var-documents}}documents
: Dict keyed by file name (minus extension) of document information loaded from YAML files found in ''data/documents''.
 
==== Macros ====
; <syntaxhighlight lang=python inline>category_data( name )</syntaxhighlight>
: Get the data for a [[#macro-var-categories|category]] or an empty dict if the category does not exist.
; <syntaxhighlight lang=python inline>document_data( name )</syntaxhighlight>
: Get the data for a [[#macro-var-documents|document]] or an empty dict if the document does not exist.
 
=== JinjaWrapperPlugin ===
Wrap markdown files in a jinja template.
 
This plugin works in concert with the [[#Macros_module|macros plugin]] to setup each markdown file as an extension of the [[gerrit:plugins/gitiles/wikimedia/developer-portal/+/refs/heads/main/data/includes/markdown_base.jinja|''data/includes/markdown_base.jinja'']] template prior to it being rendered by the macros plugin. This in turn allows us to setup common imports and macros for use by all markdown files while keeping boilerplate in the markdown files themselves to a minimum.
 
=== TranslatePlugin ===
{{Tracked|T297168|Resolved}}
Translate content via Gnutext PO files.
 
This local MkDocs plugin maintains a translation catalog based on English language content extracted from the [[gerrit:plugins/gitiles/wikimedia/developer-portal/+/refs/heads/main/src/|''src/**/*.md'']] Markdown documents and other configuration files as a [[w:Gettext|GNU Portable Object]] (PO) file suitable for import by [[translatewiki:|translatewiki.net]]. It also generates internationalized static pages based on non-English PO files found in subdirectories of [[gerrit:plugins/gitiles/wikimedia/developer-portal/+/refs/heads/main/data/locale/|''data/locale/'']]. Translation units are extracted from Markdown documents after the Jinja templates are expanded.
 
This plugin was heavily inspired by both [https://github.com/mondeja/mkdocs-mdpo-plugin/ mkdocs-mdpo-plugin] and [https://github.com/ultrabug/mkdocs-static-i18n/ mkdocs-static-i18n] which attempt to provide similar functionality.
 
The plugin hooks into a number of [https://www.mkdocs.org/dev-guide/plugins/#events lifecycle events] exposed by MkDocs:
 
; on_config
:* Create tmp dir for generated non-English Markdown files
:* Scan disk for translation dictionaries (PO files)
:* Add discovered languages to [https://squidfunk.github.io/mkdocs-material/setup/changing-the-language/#site-language-selector theme's alternate language configuration]
:* Add supported locales to [https://www.mkdocs.org/user-guide/configuration/#lang search plugin config]
 
; on_files
:* Extract translation units from English mkdocs.yml config values
:* Create a parallel content tree in the tmp dir for each target language for translation
 
; on_nav
:* If target language is English, extract translation units from nav config
:* If target language is non-English, translate non-English nav config
 
; on_page_markdown
:* If target language is non-English, translate Markdown using PO data
:* If target language is English, extract translation units from Markdown
 
; on_page_context
:* Update alternate links to point to current page in other locales
 
; on_post_build
:* For each supported locale, build translated pages using a truncated version of the MkDocs internal build process
:* Remove obsolete translation units from English PO file
:* Remove generated tmp directory and contents
 
; on_build_error
:* Remove generated tmp directory and contents
 
== Local development environment ==
 
=== Quick start ===
{{Codesample|lang=shell-session|scheme=light|code=
$ git clone https://gerrit.wikimedia.org/r/wikimedia/developer-portal
$ cd developer-portal
$ make start
$ open http://127.0.0.1:9000/
 
$ vim src/index.md
$ make build
$ open http://127.0.0.1:9000/
}}
{{Note|<code>open</code> is a MacOS command to open a file, directory, or url in the default application for that media type. On a Linux host you can either manually open the URL or use a similar URL opening script like <code>xdg-open</code>, <code>sensible-browser</code>, <code>x-www-browser</code>, or <code>gnome-open</code>.}}
 
=== How it works ===
The local development environment uses [https://docs.docker.com/compose/ docker-compose] and [[w:GNU Make|GNU Make]] to manage a container hosting the project. This container is used to build and serve the compiled mkdocs project.
 
== Demo server ==
{{Tracked|T297800|Resolved}}
 
A demo server is available at https://developer-portal.wmcloud.org/. This proxy points to an instance in the [[Nova Resource:Devportal|"devportal" Cloud VPS project]] which uses [https://docs.podman.io/en/latest/index.html Podman] to run the latest developer portal container as built by [[PipelineLib]]. A systemd timer runs [https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html podman auto-update] every 5 minutes to poll for newer container versions.
 
The demo server was manually built and does not currently have any Puppet automation to build a new instance. The manual work needed however is a relatively small amount of effort thanks to the magic of podman:
{{Codesample|lang=shell-session|scheme=light|code=
$ sudo apt install podman
$ sudo podman pull docker-registry.wikimedia.org/wikimedia/wikimedia-developer-portal:latest
$ sudo podman run --name developer-portal --label 'io.containers.autoupdate=image' --rm --detach --publish 0.0.0.0:8000:8000 docker-registry.wikimedia.org/wikimedia/wikimedia-developer-portal:latest
$ sudo podman generate systemd --new --name developer-portal {{!}} sudo tee /etc/systemd/system/developer-portal.service
$ sudo systemctl enable developer-portal
$ sudo systemctl start developer-portal
$ sudo systemctl edit podman-auto-update.timer # Set to run every 5m
$ sudo systemctl enable podman-auto-update.service
$ sudo systemctl start podman-auto-update.service
}}
 
== Production deployment ==
{{See also|Kubernetes/Deployments}}
 
# Create a patch to update main_app.version in {{Gitweb|project=operations/deployment-charts|file=helmfile.d/services/developer-portal/values.yaml|text=helmfile.d/services/developer-portal/values.yaml}} to the desired tag of the [https://docker-registry.wikimedia.org/wikimedia/wikimedia-developer-portal/tags/ wikimedia/wikimedia-developer-portal container]. ([[gerrit:799416|example]])
# CR+2 the helmfile.d patch in gerrit to start the merge process.
# Ssh to the currently active [[Deployment server|deployment server]] (''deployment.eqiad.wmnet'').
# ''cd /srv/deployment-charts/helmfile.d/services/developer-portal''
# Verify that your changes from step 1 are checked out. The git clone is updated once a minute by a timer, so if your changes are not present yet double check that zuul has merged the change and then wait patiently for the update to be fetched.
# Deploy the chart to the staging cluster: ''helmfile -e staging -i apply''
# Deploy the chart to the codfw cluster: ''helmfile -e codfw -i apply''
# Deploy the chart to the eqiad cluster: ''helmfile -e eqiad -i apply''
 
== Troubleshooting ==
 
=== AttributeError: 'NoneType' object has no attribute 'items' ===
 
* Check for a Markdown file with an empty link. (You can search the /src directory for <code>]()</code>.) In the Dev Portal, links in Markdown must have a value for the URL.
 
{{ContentGrid
|content=
{{Colored box|title=Yes|icon=OOjs UI icon check-constructive.svg|content=<pre>[link text](placeholder)</pre>}}
 
{{Colored box|title=No|icon=OOjs UI icon close-ltr-destructive.svg|content=<pre>[link text]()</pre>}}
}}
 
* Check for a document file with an empty categories list. If a document has no categories, the <code>categories</code> line should be removed.
 
{{ContentGrid
|content=
{{Colored box|title=Yes|icon=OOjs UI icon check-constructive.svg|content=<syntaxhighlight lang="yaml">---
title: My title
description: My description
links:
  - url: https://...
    label: Learn more
categories:
  - example-category</syntaxhighlight>
<syntaxhighlight lang="yaml">---
title: My title
description: My description
links:
  - url: https://...
    label: Learn more</syntaxhighlight>}}
 
{{Colored box|title=No|icon=OOjs UI icon close-ltr-destructive.svg|content=<syntaxhighlight lang="yaml">--
title: My title
description: My description
links:
  - url: https://...
    label: Learn more
categories:
# no categories
</syntaxhighlight>}}
}}
 
=== TypeError: unhashable type: 'dict'===
* Check YAML syntax for files in which you're attempting to [[#Order_documents_in_a_category | Order documents in a category]]. If you're adding an order argument to the category field, you must not prefix the category name with a hyphen:
 
 
{{ContentGrid
|content=
{{Colored box|title=Yes|icon=OOjs UI icon check-constructive.svg|content=<syntaxhighlight lang="yaml">
---
title: Hackathons
description: A hackathon brings together people with different skills to work together on a common technical goal.
links:
  - url: https://www.mediawiki.org/wiki/Special:MyLanguage/Hackathons
    label: Read more on mediawiki.org
categories:
  events: 1</syntaxhighlight>}}
 
{{Colored box|title=No|icon=OOjs UI icon close-ltr-destructive.svg|content=<syntaxhighlight lang="yaml">
---
title: Hackathons
description: A hackathon brings together people with different skills to work together on a common technical goal.
links:
  - url: https://www.mediawiki.org/wiki/Special:MyLanguage/Hackathons
    label: Read more on mediawiki.org
categories:
  - events: 1 # hyphen in this line must be removed
</syntaxhighlight>}}
}}
 
== Contribute ==
 
=== Propose content changes ===
 
Please [https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?description=%2F%2FPlease%20fill%20out%20this%20form%20to%20request%20content%20changes%20on%20https%3A%2F%2Fdeveloper.wikimedia.org%20and%20use%20the%20preview%20below!%2F%2F%0D%0A%0D%0A**Only%20if%20you%20propose%20to%20change%20existing%20content%3A**%0D%0A%0D%0A*%20Paste%20the%20current%20content%20(text%20and%20URL)%3A%20%0D%0A%0D%0A*%20Provide%20the%20proposed%20new%20content%20(text%20and%20URL)%3A%0D%0A%0D%0A*%20Provide%20a%20short%20explanation%20why%20you%20are%20proposing%20this%20change%3A%0D%0A%0D%0A**Only%20if%20you%20propose%20to%20add%20new%20content%3A**%0D%0A%0D%0A**%20Provide%20a%20full%20URL%20address%20of%20the%20web%20page%20or%20web%20site%20to%20link%20from%20the%20Developer%20Portal%3A%0D%0A%0D%0A*%20Provide%20a%20short%20text%20that%20describes%20the%20topic%20and%20URL%3A%0D%0A%0D%0A*%20%5BOptional%5D%20Provide%20a%20link%20trail%20where%20in%20the%20Developer%20Portal%20you'd%20expect%20this%20information%20to%20be%20found%3A%0D%0A%0D%0A**%20Provide%20a%20short%20explanation%20why%20you%20are%20proposing%20this%20change%20and%20why%20this%20topic%20and%20URL%20should%20be%20added%20to%20the%20developer%20portal.%20Note%20that%20we%20prefer%20to%20link%20to%20overview%20pages%20for%20a%20specific%20topic%20area%3A%0D%0A%0D%0A%0D%0A---%0D%0A%0D%0AIf%20you%20would%20like%20to%20prepare%20this%20content%20change%20yourself%3A%20Please%20%5Bfollow%20the%20%22Quick%20start%22%20section%5D(https%3A%2F%2Fgerrit.wikimedia.org%2Fg%2Fwikimedia%2Fdeveloper-portal%2F)%20and%20%5Bprovide%20a%20changeset%20in%20a%20Gerrit%20branch%5D(https%3A%2F%2Fwww.mediawiki.org%2Fwiki%2FGerrit%2FTutorial)%20following%20the%20%5Bcommit%20message%20guidelines%5D(https%3A%2F%2Fwww.mediawiki.org%2Fwiki%2FGerrit%2FCommit_message_guidelines).%20Thanks!&projects=wikimedia-developer-portal propose content additions or content changes in a Phabricator task]. ([[mw:Phabricator/Help|Learn more about Phabricator.]])
 
We would like to keep content on the portal limited. Pages should not duplicate other pages. Pages must be high quality and helpful. Please be aware of the [[mw:Developer_Advocacy/Developer_Portal#User_journeys|user journeys]]. Pages which are proposed to be linked must follow the [[mw:Documentation/Review_template|documentation guide]] for accuracy and consistency.
 
The proposed changes will be reviewed by the [[mw:Developer Advocacy|Developer Advocacy]] team.
 
Textual content is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ CC-BY-SA 3.0].
 
=== Propose code changes ===
 
Please [https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wikimedia-developer-portal propose code changes to the developer portal code base itself in a Phabricator task]. ([[mw:Phabricator/Help|Learn more about Phabricator.]])
 
The codebase is located at https://gerrit.wikimedia.org/g/wikimedia/developer-portal/. To prepare a code change yourself, set up your [[#Local development environment]] and [[mw:Gerrit/Tutorial|provide a changeset in a Gerrit branch]] following the [[mw:Gerrit/Commit message guidelines|commit message guidelines]].
 
The proposed changes will be reviewed by the [[mw:Developer Advocacy|Developer Advocacy]] team.
 
Code is licensed under [https://www.gnu.org/licenses/gpl-3.0.html GNU GPL version 3 or later].
 
=== Translate content to your language ===
 
Please translate content at https://translatewiki.net/wiki/Translating:WikimediaDeveloperPortal
 
Your translations will be regularly merged into the codebase at https://gerrit.wikimedia.org/g/wikimedia/developer-portal/. After a while, the latest codebase will be deployed on https://developer.wikimedia.org/ and translations become available to anyone visiting the website.

Latest revision as of 21:32, 27 May 2022

Redirect to:

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Developer_Portal&oldid=590829"