Homer

Homer (previously jnt) is our homemade network configuration manager.

It takes variables from Netbox and yaml files, run them through jinja templates to generate Juniper compatible configuration.

Homer can then send those configurations to selected network devices, for a diff or a safe commit.

The tool is written to not be Wikimedia specific. It only supports Junos but can easily be extended to other platforms.

Its doc is available on https://doc.wikimedia.org/homer/master/

Its code on Gerrit https://gerrit.wikimedia.org/g/operations/software/homer

Its bug and feature requests on Phabricator: https://phabricator.wikimedia.org/tag/homer/

This page focuses on Wikimedia's deployment.

Deployment

Homer is deployed via Puppet and Scap to the cumin (fleet management) hosts.

You can find its deploy repository here https://gerrit.wikimedia.org/g/operations/software/homer/deploy

And its Puppet module there https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/homer

In addition it's available on Pypi: https://pypi.org/project/homer/

Releasing a new version

• Make a release patch updating the changelog (see this example patch).
• Once it's merged, update the local checkout and make a git tag. Ideally an annotated one (requires a GPG key and have git configured to use it, see signingkey):

$ RELEASE=v0.1.0
$ git tag -s -a "${RELEASE}" -m "${RELEASE}" -m "[Release Notes](CHANGELOG.rst)"

• Push the generated tag: git push origin "${RELEASE}"
• Move to the homer-deploy checkout:

$ cd src/
$ git pull
$ git log -1  # to check to be at the right commit
$ cd ..
# At this point git status would show that there is a diff for the 'src' path, indicating the different SHA1 of the git submodule
# Ensure that docker is running
$ make -f Makefile.build all
# Verify that the generated wheels are correct
# At this point the frozen-requirements.txt file will most likely have some changes and the artifacts/artifacts.stretch.tar.gz will be different
git add .
git commit -m "Release ${RELEASE}"
git review

• Once the above patch has been merged (C+2, V+2 + submit), move to the deployment server in /srv/deployment/homer/deploy
• Pull the latest changes: git pull
• Verify that in the src/ directory we're at the correct commit (check also with git status)
• Deploy the new release: scap deploy --verbose "Homer release v... - T..."

Daily diffs

A cron job runs Homer every 12h (24h per cumin hosts) to compare the live network configuration with our intended state. Any discrepancies is emailed to the rancid-core alias.

Usage 🚀

Making changes

Note that Homer explicitly asks you when its about to modify the live network configuration (Type "yes" to commit, "no" to abort.) and will prompt you with a diff of the changes beforehand.

Editing the private repository

Manually edit then commit the files on ssh://cumin1001.eqiad.wmnet:/srv/homer/private .

git will sync them with the other cumin host. And will email a summary of the changes to SREs.

Make sure to mirror all your changes on the mock-private repo: https://gerrit.wikimedia.org/g/operations/homer/mock-private

This repository doesn't have CI, please be extra careful.

Editing the public repository

Similar to our other public repositories, send CRs to https://gerrit.wikimedia.org/g/operations/homer/public , try not to self-+2 your changes without other review.

Editing Netbox

Data is also pulled from Netbox, always make sure that Netbox is accurate before using Homer.

Running Homer from cumin hosts (recommended)

Get familiar with the command line: https://doc.wikimedia.org/homer/master/homer.html everything else is taken care of.

The public repository is regularly updated by Puppet.

When pushing configurations, homer will ssh to the network devices using the Homer user. You need to be in the ops group to be able to use its private key.

Some examples:

• homer "*" diff All devices
• homer "cr*ams*" diff esams and knams core routers
• homer "mr*" commit "My commit message" All management routers

Running Homer from your local machine (less recommended)

• Clone the public repo: https://gerrit.wikimedia.org/g/operations/homer/public
• Clone private repo: ssh://cumin1001.eqiad.wmnet:/srv/homer/private
• Clone deploy repo: https://gerrit.wikimedia.org/g/operations/software/homer/deploy
• Install Homer with either:
  • pip install homer
  • https://gerrit.wikimedia.org/g/operations/software/homer + python3 setup.py install (if you live on the edge)
• Make the plugins included in the deploy repo available in the Python path:
  • If homer's code is checked out, just create a symlink in the root's of homer's checkout to the homer_plugins/ directory in the deploy repo. If they are all checkout in the same root directory, from within the homer's checkout run: ln -s ../homer-deploy/plugins/ homer_plugins
  • If homer is installed via pip, find the site_packages directory where homer is installed, usually something like venv/lib/python3.X/site-packages/ and add there a symlink to the plugins like ln -s /PATH_TO_DEPLOY_REPO/plugins/ homer_plugins.
• Create your configuration file to match https://doc.wikimedia.org/homer/master/configuration.html
  • Including the plugin setup: homer_plugins.wmf-netbox
• Get familiar with the command line: https://doc.wikimedia.org/homer/master/homer.html

When pushing configurations, your machine will ssh directly to the network devices, which mean that you have to have an account there, with the proper permissions.

It's common to test a change locally with the "diff" option. Once satisfied with the result, please merge your change on Gerrit before pushing them with the "commit" action.

Style guides

YAML files

Set configuration options doc as comment in the most common denominator file, with a note if it can be applied to more specific sites/roles/devices. See existing options.

TODO

https://json-schema.org

Templates

It's ok to give up on indentation.

Capirca (ACL generation)

This is still a POC.

Task: https://phabricator.wikimedia.org/T273865

Capirca is an actively maintained open source tool made by Google to generate multi-platform ACLs based on generic policy and definitions files.

Network configuration coverage

CR

Done

groups {}
apply-groups [ re0 re1 ];
system {}
interfaces {}
logical-systems {}
services {}
snmp {}
forwarding-options {}
routing-options {}
protocols {
    ospf {}
    ospf3 {}
    lldp {}
    bgp {} # Transit, LVS, Netflow ,k8s, Anycast
}
policy-options {}
firewall {}
routing-instances {}
chassis {} (partial)

TODO

chassis {} (partial)
routing-options {}  # TODO: statics
protocols {
    router-advertisement {}
    bgp {}  # TODO: confed. IXPs are out of scope (dedicated tool like peering-manager)
}

ASW

Done

system {}
interfaces {}
snmp {}
protocols {}
routing-options {}
virtual-chassis {}
vlans {}
chassis {}

MR

Done

groups {}
system {}
snmp {}
interfaces {}
protocols {}
routing-options {}
policy-options {}
security {
    zones {}
    alg {}
    forwarding-options {}
    screen {}
    nat {}
}

TODO

security {
    address-book {}  # Capirca?
    policies {}  # Capirca?
}
routing-instances {}
applications {}  # Capirca?

MSW (all done)

system {}
interfaces {}
snmp {}
protocols {}
routing-options {}
vlans {}

CLOUDSW

TODO

Common/known issues

(Almost) None.

• The "commit" action will not work on the first try with the mr1* devices, but homer will retry.