You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Juniper router upgrade
Jump to navigation
Jump to search
Preparation
- Download the proper image to apt1001:/srv/junos/
- We now only use 64bits vmhost
- Based on upgrade task and Juniper recommended
All the steps bellow should be done with:
cumin1001:~$ sudo cookbook sre.network.prepare-upgrade <image-filename>.tgz <router-fqdn>
- Make room for the image
request system storage cleanup
- Save rescue config (just in case)
request system configuration rescue save
- Copy image
file copy "https://apt.wikimedia.org/junos/$filename.tgz" /var/tmp/
routing-instance mgmt_junos
- Check checksum
file checksum md5 /var/tmp/$filename.tgz
- Compare with checksum on Juniper's website
Upgrade
- Check if console port(s) is(/are) working
- Depool site (optional)
- Drain traffic away from router
- Apply
GRACEFUL_SHUTDOWN
- https://phabricator.wikimedia.org/T211728 - Disable the peers
deactivate protocols bgp group Transit4
deactivate protocols bgp group Transit6
deactivate protocols bgp group IX4
deactivate protocols bgp group IX6
- Adjust OSPF metrics
- If eqiad/codfw drain the pfw3 link by increasing the MED value on both sides
- Apply
- Ensure router is not VRRP master
show vrrp summary
set groups vrrp interfaces <*> unit <*> family inet address <*> vrrp-group <*> priority 70
set groups vrrp interfaces <*> unit <*> family inet6 address <*> vrrp-inet6-group <*> priority 70
- Downtime host in Icinga and LibreNMS
If Multi RE:
- Remove
graceful-switchover
deactivate chassis redundancy graceful-switchover
request system configuration rescue save
(to not have the above statement in the rescue config)
- Install image on backup RE
request vmhost software add /var/tmp/$filename.tgz re1
- Reboot RE1
request vmhost reboot re1
- Once back up (
show chassis routing-engine
), perform RE switchover (impactful)request chassis routing-engine master switch
- Once done, repeat previous 3 steps for re0
- Rollback "Remove
graceful-switchover
"
If single RE:
- Install image on RE
request vmhost software add /var/tmp/$filename.tgz
- Reboot router
request vmhost reboot
Both single and dual RE:
- Check if router is healthy
show log messages | last
show system alarms
show ospf(3) interface
show bgp summary
- All green in Icinga and LibreNMS
Cleanup
request system storage cleanup
- Remove Icinga and LibreNMS downtimes
- Rollback "Drain traffic away from router"
- Rollback VRRP change if any
- Save rescue config (just in case)
request system configuration rescue save
- On vmhost devices, save the disk snapshot to the backup partition
request vmhost snapshot
for single RE devicesrequest vmhost snapshot routing-engine both
for dual RE devices