Stuck global renames

From Wikitech-static
Jump to navigation Jump to search

Recovering a "locked" account

Sometimes you'll find that a rename becomes stuck at a project (see for example phab:T137973). In that case, please follow these instructions:

  • Wait at least 3 hours since the rename started. The system will automatically retry failed renames a few times.
  • Please open a task in Phabricator (direct link with title and projects already added). In there please post a link to the Special:GlobalRenameProgress link to the account whose rename has become stuck and mention the old and new name.
  • A developer with production shell access should:
    • Check for errors in Logstash before continuing.
      • Searching the old or new username in the log channels CentralAuthRename and Renameuser (dashboard) should catch the more obvious things and narrow down the time range (based on when the job ran on the preceding wikis).
      • Searching for the job runner URL might give more elusive errors like exceptions and DB timeouts (dashboard; try to filter by wiki and/or time range).
      • If the error suggests global renaming itself is broken, that should be fixed first. (Usually it's some kind of timeout; it's OK to just re-run in that case.)
    • Verify that no rename jobs are running on the affected wiki (the one showing up as failed in the GlobalRenameProgress page):
      mwscript showJobs.php --wiki=<...> --type LocalRenameUserJob
      mwscript showJobs.php --wiki=<...> --type RenameUserJob
    • Run fixStuckGlobalRename.php maintenance script to requeue the job:
      mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=<...> --logwiki=metawiki '<old username>' '<new username>'
      • wiki: the database name of the wiki where the rename job has become stuck, e.g. ptwiki. If it is stuck on multiple wikis, use the one that comes first alphabetically.
      • logwiki: the database name of the wiki where the global rename was started. This will always be metawiki.
      • --ignorestatus is a parameter that can be added to fix a global rename that's stuck in running state even though the job is not actually running (so trying to run the job without this switch gives skipping duplicate rename from...). This used to happen a lot in the past due to fragile error handling; shouldn't be the case anymore. Use with care; could make a mess if the job really is running.
    • Log in the production SAL that you've run the script, by issuing an IRC command in #wikimedia-operations such as:
      !log T12345 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ptwiki --logwiki=metawiki 'OldName' 'NewName'
  • In case of doubt, ask the maintainers of Renameuser or CentralAuth.