MediaWiki deployment
Translatewiki.net uses oregano to deploy MediaWiki. Oregano is simple deployment tool. Oregano is documented in a blog post.
There is a script called twn-update-all
that updates MediaWiki core, extensions and composer dependencies and updates the localisation cache. It does not run schema changes (update.php). We run the master branch of MediaWiki and extensions. For this reason the deployer should be aware of the latest MediaWiki changes and whether they require configuration or schema changes. The script automatically uses sudo -u betawiki
and will fail if you don't have access.
The oregano commands must be run with sudo -u betawiki
. oregano
itself is in path and it must be run in the /srv/mediawiki
directory. It is recommended to add alias b for the sudo command. After deploying a new version, please monitor messages from Rakkaus in #translatewiki on Libera.Chat IRC and be ready to rollback in case of issues. If you notice new PHP notices or warnings, those should be investigated and reported to Phabricator where applicable. You can also inspect the log file in /www/translatewiki.net/logs/error_php
directly.
Deployment window
Regular deployment window checklist:
- Check error logs before production update.
- Prepare deployment plan.
- (If root) update packages: apt update, apt upgrade.
- Update production environment but do not deploy (do not create and release a tag). See deployment example below.
- Test on canary while checking error logs.
- If everything is OK, deploy and monitor error logs.
- Check #affects-translatewiki.net to see if issues are resolved (e.g. no more seen in logs), or need updating
Report errors as issues in Phabricator with #affects-translatewiki.net tag.
Full deployment example
cd /srv/mediawiki
# Update
b twn-update-all
b twn-update-database
# Test on canary
# Deploy and release
b oregano tag
b oregano deploy
How to see the difference against deployed code
If you want to see how the working directory differs from the currently deployed code, you can run
cd /srv/mediawiki
diff -urx .git targets/production workdir
You can use this before deployment to confirm that the working directory is clean, on manual deployment (e.g. single extension) to confirm only expected changes are deployed, or on full deployment to inspect what has changed.
How to test code changes before deploying
It is possible to test code changes before deploying them to all users with oregano
.
You can enable the gadget "Add button to toggle canary mode (see MediaWiki deployment)." to get a link to toggle canary mode added to your user tool menu.
You know that canary mode is enabled if you see orange background stripes and a "You are using a canary" message near the page title. This varies a bit depending on the skin and which page you are on, so for example on Timeless the colored stripes are only visible at the very bottom, and the message is not visible on our main page.
You can now test your changes. Once you are finished, do remember to disable canary mode.
How to handle resolve conflicts during update
We occasionally cherry-pick patches or do local patches for various reasons. Although we try to keep those at minimum, updating the code may fail due to a merge conflict caused by some of these patches. You will notice this in the output of the command and often also L10N cache rebuild will fail because of fatal errors due to those conflicts.
Here is how to get rid of a conflicting patch (i.e. because an updated version was merged to master):
cd /srv/mediawiki/workdir/extensions/SomeExtension
b git rebase --abort
b git rebase -i # Drop the affected patches
Other times the patches might still be needed, in which case you need to solve the merge conflicts :(
How to rollback a deployment
If the deployment causes issues, such as bringing the site down, or some feature not working, or lot of log spam, it is recommended that you revert the deployment. This command reverts the site to the previously deployed version:
b oregano rollback
After the immediate issues are averted, it is time to investigate what went wrong, fix it, and think if this can be prevented in the future.
How to do a configuration change
Submit a patch to the translatewiki repository in Gerrit. Get the patch merged. Self-merges are not forbidden, but discouraged.
How to check MediaWiki version
Some configuration changes (such as removing language configuration for languages that have been exported to core for the first time) need a specific commit from MediaWiki core to have be deployed on Translatewiki.net before it's safe to deploy those configuration changes. To check which MediaWiki version is deployed, you can use:
version
Changes to MediaWiki configuration
This section applies to all files in mw-config directory.
Follow the instructions for a full deployment, but use twn-update-config
instead as follows:
cd /srv/mediawiki
# Update
twn-update-config
# Test on canary
# Deploy and release
b oregano tag
b oregano deploy
Configuration files
- PrivateSettings.php and DatabaseSettings.php: Keys, passwords and other things which should not public. This file is only editable locally on the server.
- LanguageSettings.php and FallbackSettings.php: Settings related to configuration of available translation languages
- PermissionSettings.php: Most $wgGroupPermission things go here
- TranslateSettings.php: Settings related to the Translate extensions
- TranslatewikiSettings.php: Everything else
Changes to group configuration
The updated group configuration will be applied immediately! If the configuration depends on code change, be extremely sure that that code is first deployed to production with oregano
or you will bring the site down.
- Go to
cd /home/betawiki/config
and edit the broken configuration with text editor (b nano ...php
, orb git revert HEAD
. - Commit the working code locally with
b git commit -av
- Submit a fix to Gerrit and get it merged
- Run
cd /home/betawiki/config
,b git fetch
andb git rebase -i origin/master
– remove your fixup commit to avoid merge conflicts. If you still get conflicts,b git rebase --abort
should help to get back to the previous state.
Updating only specific extensions
If you need to update one extension, for example UniversalLanguageSelector:
cd /srv/mediawiki/workdir/extensions/UniversalLanguageSelector
b git pull --rebase
cd /srv/mediawiki/workdir
b composer -n update
php maintenance/rebuildLocalisationCache.php --threads=4
cd /srv/mediawiki
b oregano tag
b oregano deploy
For the common case of updating Translate, UniversalLanguageSelector, TwnMainPage and cldr, there is a shortcut twn-update-maintained
that does all of the above for these extensions and prints a list of changes in each extension.
Remember to run twn-update-database
manually if needed.
Handling logspam and other issues
During log triage or canary testing you may find things that are broken. Those issues should be reported to Phabricator. You can check the #affects-translatewiki.net tag if the issue is already known. If the issue is there, perform a general search in Phabricator to see if the issue has been reported. If it is, add the #affects-translatewiki.net to it. If it isn't, file a new issue with as many details as possible, but not exposing private information like IP addresses. Add the #affects-translatewiki.net tag and other relevant tags to it.
Discuss with the co-deployers whether the issue is severe enough that it needs a workaround or even halting the deployment.