Ultimate openSUSE Leap Upgrade Guide
Quick Summary: In this playbook we demonstrate how to perform an enterprise-grade minor version upgrade of openSUSE.
Last updated: | Leave a comment
Introduction
We’re big fans of SUSE Linux (SLES and openSUSE), it’s one of several operating systems we use here at XOMedia for development, testing and production purposes.
SUSE Linux is a popular choice for both home and business users. It is a powerful and versatile operating system that is easy to use and customize. It shares a code base with SUSE Linux Enterprise (SLES), which gives it a level of stability unmatched by other Linux distributions.
Trivia: The first version of SUSE was rolled from Slackware, the oldest distribution that is still maintained today.
NOTE: As of openSUSE 15, SUSE made its “Leap” variant directly upgradable to SUSE Linux Enterprise.
In this Ultimate guide we are going to detail how to perform an enterprise-grade minor version upgrade of openSUSE. The upgrade will be performed [live / online / in-place, e.g., applications running with active logged in users]. This approach will minimize system, user and application downtime.
Ultimate guide entails that if you purchase SUSE (or any vendor) professional services to perform your installations and upgrades, you end up with a certified implementation that you can guarantee. This is especially important if you’re building out systems that will support 25+ million customers. Many well-meaning enthusiast read and fumble through a home installation and then publish these results on a Blog. Don’t bet your business or risk your career on anything less.
This procedure has been tested with the following openSUSE versions:
15.3 > 15.4 > 15.5NOTE: Transcripts in this article apply to 15.3 > 15.4 (but also apply to any 15.x point release).
TIP: Although this article covers openSUSE, the procedure outlined in this article can serve as a generic template for upgrading any operating system. Many of the concepts introduced are agnostic, timeless principles that can be applied across all systems (including Windows).
The supported upgrade starting point is an openSUSE release with all current updates applied.
We have seen some people skip minor number upgrades (e.g., 15.2 to 15.5). We do not recommend this. If you do, know what you’re skipping by reviewing the release notes. In any event, this is why we have viable backups and conduct sanities.
For example, the following remarks appear conflicting:
https://en.opensuse.org/SDB:System_upgrade_to_Leap_15.5
‘Warning: Do not skip a release when upgrading! Example: do not upgrade from 15.3 to 15.5. Instead, from 15.3 upgrade to 15.4, and only then from 15.4 to upgrade to 15.5.’
https://en.opensuse.org/SDB:Offline_upgrade
‘Official Support: As far as I know, you can upgrade between two versions that are or were currently supported, like from 13.1 to 42.1. But this assertion needs an official statement I can not make.’
Major version upgrades: Serious issues can occur executing major version upgrades (e.g., 12.x to 13.x) across all *NIX operating platforms. Some vendors do not support this (e.g., Red Hat at the time of this writing). Despite this warning, people still attempt this, getting themselves into serious trouble years (and thousands of dollars) later. If you do go this route – upgrade from 12.3 to 13.1 and then from 13.2, and so on and so forth. Performing a fresh install for major version upgrades is safer. It also allows you to retest / re-evaluate your procedures resulting in a fresh install without all of the cruft.
If this is a production system, it’s assumed you have already tested the upgrade in Dev > Test > Staging. Even if you do not have non-prod environments, you can still test your procedure using a virtual environment (e.g., VMWare / VirtualBox).
Log out active users, quiesce applications and the system as much as possible to minimize upgrade errors. Use good technical judgment to determine this in your case.
If uptime is paramount: You can minimize the outage window by executing a live upgrade ahead of the outage and simply reboot the system during a small outage to commit changes.
If uptime is not critical: You can execute an (offline) upgrade booted from an ISO image (DVD/USB). The advantages of this approach are that all applications, files (databases) and libraries are closed (not in use) during the upgrade process. This may result in a longer maintenance window, but may be much safer in your situation. Again, use good technical judgment.
Warning: Avoid over-the-wire upgrades (an approach that sources only required files, minimizing bandwidth use) – explained further later in this article.
Regardless of which approach is used, you should always have ‘validated’ backups. This backup should be retained indefinitely.
BONUS: An easy, safe database backup procedure has been provided at the end of this article > ‘Database Backup’.
If this is a VM, take a snapshot at each major stage of the upgrade process:
1) Snapshot #1: Adding an addition disk to increase the root volume size. 2) Snapshot #2: The upgrade. 3) Snapshot #3: Post upgrade patches.
If there was an error in judgment with the selected upgrade approach, you can always restore / rollback and reassess your upgrade approach.
Warning: Do not forget to delete (commit your snapshots) once all goes well. Leaving snapshots lingering can lead to bad things later down the line (including loss of system(s)). We were once involved in a P1 where a customer left a snapshot lingering on a very active system for over a year. They lost the vPostgres DB (a critical component of vCenter). Having a large snapshot or too many snapshots can lead to performance degradation (or worse) losing an entire cluster (and multiple VMs). The bigger the snapshot (e.g., the longer it lingers) – the more performance issues and the longer a system may be offline committing the snapshot.
Pre-checks
Review the Release_Notes for your target version, and intermediate versions if you’re jumping versions (e.g., 15.2 to 15.5).
Review the openSUSE:Most_annoying_bugs page for the target version and intermediate versions.
Also review the openSUSE upgrade documentation for any new requirements that may have been introduced in the upgrade process between minor versions.
For example, the new 4096 bit RSA signing key was introduced in releases 15.4 / 15.5:
https://en.opensuse.org/SDB:System_upgrade#0._New_4096_bit_RSA_signing_key
Performing the upgrade 0. New 4096 bit RSA signing key"
You can verify if you need to install keys with:
[root@devlinux:/root]# rpm -qa | grep gpg-pubkey gpg-pubkey-3adca79d-550f2d7f gpg-pubkey-45f2c3d5-5e81efb9 gpg-pubkey-3dbdc284-53674dd4 gpg-pubkey-ef5975ca-6421ce2b gpg-pubkey-29b700a4-62b07e22 gpg-pubkey-65176565-61a0ee8f gpg-pubkey-d38b4796-570c8cd3 gpg-pubkey-3fa1d6ce-63c9481c gpg-pubkey-39db7c82-5f68629b gpg-pubkey-7fac5991-4615767f gpg-pubkey-25db7ae0-645bae34 gpg-pubkey-307e3d54-5aaa90a5 [root@devlinux:/root]#
Storage
Ensure you have adequate storage across all system partitions / file systems to accommodate the upgrade (in our situation, the root volume had 3.8GB free and still ran out of free space). Each scenario is different, use good judgment. If you’re not careful, you can lose your system.
You can remove unused packages and dependencies to free up space:
NOTE: Execute an ad hoc before and after report to determine what and how much was freed:
mkdir /root/UPGRADE/15.5; cd /root/UPGRADE/15.5; df -h > df-b4.dat ; rpm -qa > rpm-b4.dat
[root@devlinux:/root]# zypper clean -a
All repositories have been cleaned up.
[root@devlinux:/root]#
3rd Party REPOs
Disable any 3rd party REPOs before starting the upgrade, or lower the priority of these REPOs to make sure packages from the default system repositories take precedence.
You can re-enable them after the upgrade and edit their version string to match the version number of the upgraded system.
List all REPOs with zypper lr, note the number on the left of the REPO you want to disable. You can then disable it with (zypper mr -d 3):
Baseline Checks
NOTE: Be sure to save all command output / files somewhere safe so they can be used for the post-install sanities to determine what has been upgraded.
Take before and after baselines (using cfg2html and XOMedia’s sysinfo script).
Execute a baseline sanity reboot so you know you’re upgrading a viable system (example, a faulty /etc/fstab).
Once the upgrade completes, we will conduct system and application sanities before turning the environment back over to the customer(s).
OS:
[root@devlinux:/root]# cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.3"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.3"
PRETTY_NAME="openSUSE Leap 15.3"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.3"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
Kernel:
[root@devlinux:/root]# uname -a
Linux devlinux 5.3.18-150300.59.106-default #1 SMP Mon Dec 12 13:16:24 UTC 2022 (774239c) x86_64 x86_64 x86_64 GNU/Linux
Refresh REPO, metadata update:
[root@devlinux:/root]# zypper refresh
Retrieving repository 'google-chrome' metadata ....................................................................................................................................................................................................................[done]
Building repository 'google-chrome' cache .........................................................................................................................................................................................................................[done]
Repository 'jenkins' is up to date.
Repository 'Online updates for openSUSE Leap:15.0 (standard)' is up to date.
Repository 'oss' is up to date.
Retrieving repository 'oss_update' metadata .......................................................................................................................................................................................................................[done]
Building repository 'oss_update' cache ............................................................................................................................................................................................................................[done]
Repository 'Update repository of openSUSE Backports' is up to date.
Repository 'Non-OSS Repository' is up to date.
Repository 'Main Repository' is up to date.
Repository 'Update repository with updates from SUSE Linux Enterprise 15' is up to date.
Repository 'Update Repository (Non-Oss)' is up to date.
All repositories have been refreshed.
Enter the following command to apply pending updates to the current release. Our starting point should always be a fully upgraded current release:
[root@devlinux:/root]# zypper update Loading repository data... Reading installed packages... The following package update will NOT be installed: yum The following 14 packages are going to be upgraded: google-chrome-stable jenkins libavcodec57 libavformat57 libavutil55 libsass-3_6_1-1 libswresample2 libswscale4 transmission-common transmission-daemon transmission-gtk transmission-gtk-lang xdg-utils yum-lang The following 3 NEW packages are going to be installed: libwebp6 libwebpmux2 yast2-trans The following 2 packages are going to be REMOVED: yast2-trans-en yast2-trans-en_US 14 packages to upgrade, 3 new, 2 to remove. Overall download size: 191.6 MiB. Already cached: 0 B. After the operation, 5.0 MiB will be freed. Continue? [y/n/v/...? shows all options] (y): Retrieving package jenkins-2.401.3-1.2.noarch (1/17), 93.6 MiB ( 93.9 MiB unpacked) Retrieving: jenkins-2.401.3-1.2.noarch.rpm ............................................................................................................................................................................................................[done (1.9 MiB/s)] Retrieving package libavutil55-3.4.4-lp150.6.1.x86_64 (2/17), 213.2 KiB (432.9 KiB unpacked) Retrieving: libavutil55-3.4.4-lp150.6.1.x86_64.rpm ....................................................................................................................................................................................................[done (5.2 KiB/s)] Retrieving package libsass-3_6_1-1-3.6.1-lp150.2.3.1.x86_64 (3/17), 735.7 KiB ( 2.7 MiB unpacked) Retrieving: libsass-3_6_1-1-3.6.1-lp150.2.3.1.x86_64.rpm ................................................................................................................................................................................................[done (465 B/s)] Retrieving package transmission-common-2.94-lp150.2.3.1.noarch (4/17), 318.0 KiB (749.2 KiB unpacked) Retrieving: transmission-common-2.94-lp150.2.3.1.noarch.rpm .............................................................................................................................................................................................[done (465 B/s)] Retrieving package transmission-gtk-2.94-lp150.2.3.1.x86_64 (5/17), 408.9 KiB (918.1 KiB unpacked) Retrieving: transmission-gtk-2.94-lp150.2.3.1.x86_64.rpm .............................................................................................................................................................................................[done (28.7 KiB/s)] Retrieving package transmission-gtk-lang-2.94-lp150.2.3.1.noarch (6/17), 399.3 KiB ( 2.0 MiB unpacked) Retrieving: transmission-gtk-lang-2.94-lp150.2.3.1.noarch.rpm ...........................................................................................................................................................................................[done (931 B/s)] Retrieving package transmission-daemon-2.94-lp150.2.3.1.x86_64 (7/17), 321.5 KiB (653.3 KiB unpacked) Retrieving: transmission-daemon-2.94-lp150.2.3.1.x86_64.rpm ...........................................................................................................................................................................................[done (1.2 KiB/s)] Retrieving package xdg-utils-20170508-lp150.3.3.2.noarch (8/17), 89.4 KiB (344.9 KiB unpacked) Retrieving: xdg-utils-20170508-lp150.3.3.2.noarch.rpm ................................................................................................................................................................................................[done (86.7 KiB/s)] Retrieving package yum-lang-3.4.3-lp150.4.4.1.noarch (9/17), 201.1 KiB ( 1.2 MiB unpacked) Retrieving: yum-lang-3.4.3-lp150.4.4.1.noarch.rpm .......................................................................................................................................................................................................[done (931 B/s)] Retrieving package libswscale4-3.4.4-lp150.6.1.x86_64 (10/17), 186.2 KiB (524.3 KiB unpacked) Retrieving: libswscale4-3.4.4-lp150.6.1.x86_64.rpm ......................................................................................................................................................................................................[done (931 B/s)] Retrieving package libswresample2-3.4.4-lp150.6.1.x86_64 (11/17), 98.4 KiB (136.3 KiB unpacked) Retrieving: libswresample2-3.4.4-lp150.6.1.x86_64.rpm .................................................................................................................................................................................................[done (2.3 KiB/s)] Retrieving package yast2-trans-84.87.20210502.7b34dbceae-1.1.noarch (12/17), 311.6 KiB ( 0 B unpacked) Retrieving: yast2-trans-84.87.20210502.7b34dbceae-1.1.noarch.rpm ........................................................................................................................................................................................[done (328 B/s)] Retrieving package libwebp6-0.5.0-150000.3.7.1.x86_64 (13/17), 173.5 KiB (370.7 KiB unpacked) Retrieving: libwebp6-0.5.0-150000.3.7.1.x86_64.rpm ................................................................................................................................................................................................................[done] Retrieving package libwebpmux2-0.5.0-150000.3.7.1.x86_64 (14/17), 27.2 KiB ( 38.3 KiB unpacked) Retrieving: libwebpmux2-0.5.0-150000.3.7.1.x86_64.rpm .................................................................................................................................................................................................[done (8.3 KiB/s)] Retrieving package google-chrome-stable-115.0.5790.170-1.x86_64 (15/17), 92.3 MiB (301.9 MiB unpacked) Retrieving: google-chrome-stable-115.0.5790.170-1.x86_64.rpm ..........................................................................................................................................................................................[done (6.2 MiB/s)] Retrieving package libavcodec57-3.4.4-lp150.6.1.x86_64 (16/17), 1.4 MiB ( 4.4 MiB unpacked) Retrieving: libavcodec57-3.4.4-lp150.6.1.x86_64.rpm .................................................................................................................................................................................................[done (114.1 KiB/s)] Retrieving package libavformat57-3.4.4-lp150.6.1.x86_64 (17/17), 876.3 KiB ( 2.1 MiB unpacked) Retrieving: libavformat57-3.4.4-lp150.6.1.x86_64.rpm .................................................................................................................................................................................................[done (90.0 KiB/s)] Checking for file conflicts: ......................................................................................................................................................................................................................................[done] ( 1/17) Installing: jenkins-2.401.3-1.2.noarch ....................................................................................................................................................................................................................[done] ( 2/17) Installing: libavutil55-3.4.4-lp150.6.1.x86_64 ............................................................................................................................................................................................................[done] ( 3/17) Installing: libsass-3_6_1-1-3.6.1-lp150.2.3.1.x86_64 ......................................................................................................................................................................................................[done] ( 4/17) Installing: transmission-common-2.94-lp150.2.3.1.noarch ...................................................................................................................................................................................................[done] ( 5/17) Installing: transmission-gtk-2.94-lp150.2.3.1.x86_64 ......................................................................................................................................................................................................[done] ( 6/17) Installing: transmission-gtk-lang-2.94-lp150.2.3.1.noarch .................................................................................................................................................................................................[done] ( 7/17) Installing: transmission-daemon-2.94-lp150.2.3.1.x86_64 ...................................................................................................................................................................................................[done] ( 8/17) Installing: xdg-utils-20170508-lp150.3.3.2.noarch .........................................................................................................................................................................................................[done] ( 9/17) Installing: yum-lang-3.4.3-lp150.4.4.1.noarch .............................................................................................................................................................................................................[done] (10/17) Installing: libswscale4-3.4.4-lp150.6.1.x86_64 ............................................................................................................................................................................................................[done] (11/17) Installing: libswresample2-3.4.4-lp150.6.1.x86_64 .........................................................................................................................................................................................................[done] (12/17) Installing: yast2-trans-84.87.20210502.7b34dbceae-1.1.noarch ..............................................................................................................................................................................................[done] (13/17) Installing: libwebp6-0.5.0-150000.3.7.1.x86_64 ............................................................................................................................................................................................................[done] (14/17) Installing: libwebpmux2-0.5.0-150000.3.7.1.x86_64 .........................................................................................................................................................................................................[done] (15/17) Installing: google-chrome-stable-115.0.5790.170-1.x86_64 ..................................................................................................................................................................................................[done] (16/17) Installing: libavcodec57-3.4.4-lp150.6.1.x86_64 ...........................................................................................................................................................................................................[done] (17/17) Installing: libavformat57-3.4.4-lp150.6.1.x86_64 ..........................................................................................................................................................................................................[done] There are running programs which still use files and libraries deleted or updated by recent upgrades. They should be restarted to benefit from the latest updates. Run 'zypper ps -s' to list these programs.
Reboot:
[root@devlinux:/root]# systemctl reboot
Connection to 192.168.1.30 closed by remote host.
Connection to 192.168.1.30 closed.
Apply post-install patches (if any are pending) and reboot once again.
After all updates and patching are applied, the tray update indicator should reflect that everything is indeed up to date:
Once the system is back online, verify that the reboot actually took place:
[root@devlinux:/root]# uptime 17:26:17 up 0:00, 1 user, load average: 1.46, 0.35, 0.12
Did the kernel version change in this update?
[root@devlinux:/root]# uname -a
Linux devlinux 5.3.18-150300.59.106-default #1 SMP Mon Dec 12 13:16:24 UTC 2022 (774239c) x86_64 x86_64 x86_64 GNU/Linux
Once again, verify that there are no 3rd party REPOs enabled:
[root@devlinux:/root]# zypper repos --uri
Repository priorities are without effect. All enabled repositories share the same priority.
# | Alias | Name | Enabled | GPG Check | Refresh | URI
---+-----------------------------+---------------------------------------------------------------------------------------------+---------+-----------+---------+------------------------------------------------------------------------
1 | PlexRepo | PlexRepo | No | ---- | ---- | https://downloads.plex.tv/repo/rpm/x86_64/
2 | google-chrome | google-chrome | Yes | (r ) Yes | Yes | https://dl.google.com/linux/chrome/rpm/stable/x86_64
3 | jenkins | jenkins | Yes | (r ) Yes | Yes | https://pkg.jenkins.io/opensuse-stable/
4 | openSUSE-Leap-15.3-2 | openSUSE-Leap-15.3-2 | No | ---- | ---- | cd:/?devices=/dev/disk/by-id/ata-VBOX_CD-ROM_VB2-01700376
5 | openSUSE_Leap_15.0_Update | Online updates for openSUSE Leap:15.0 (standard) | Yes | (r ) Yes | No | http://download.opensuse.org/update/leap/15.0/oss/
6 | oss | oss | Yes | (r ) Yes | No | https://download.opensuse.org/distribution/leap/15.3/repo/oss/
7 | oss_update | oss_update | Yes | (r ) Yes | No | https://download.opensuse.org/update/leap/15.3/oss/
8 | repo-backports-debug-update | Update repository with updates for openSUSE Leap debuginfo packages from openSUSE Backports | No | ---- | ---- | http://download.opensuse.org/update/leap/15.3/backports_debug/
9 | repo-backports-update | Update repository of openSUSE Backports | Yes | (r ) Yes | Yes | http://download.opensuse.org/update/leap/15.3/backports/
10 | repo-debug | Debug Repository | No | ---- | ---- | http://download.opensuse.org/debug/distribution/leap/15.3/repo/oss/
11 | repo-debug-non-oss | Debug Repository (Non-OSS) | No | ---- | ---- | http://download.opensuse.org/debug/distribution/leap/15.3/repo/non-oss/
12 | repo-debug-update | Update Repository (Debug) | No | ---- | ---- | http://download.opensuse.org/debug/update/leap/15.3/oss/
13 | repo-debug-update-non-oss | Update Repository (Debug, Non-OSS) | No | ---- | ---- | http://download.opensuse.org/debug/update/leap/15.3/non-oss/
14 | repo-non-oss | Non-OSS Repository | Yes | (r ) Yes | Yes | http://download.opensuse.org/distribution/leap/15.3/repo/non-oss/
15 | repo-oss | Main Repository | Yes | (r ) Yes | Yes | http://download.opensuse.org/distribution/leap/15.3/repo/oss/
16 | repo-sle-debug-update | Update repository with debuginfo for updates from SUSE Linux Enterprise 15 | No | ---- | ---- | http://download.opensuse.org/debug/update/leap/15.3/sle/
17 | repo-sle-update | Update repository with updates from SUSE Linux Enterprise 15 | Yes | (r ) Yes | Yes | http://download.opensuse.org/update/leap/15.3/sle/
18 | repo-source | Source Repository | No | ---- | ---- | http://download.opensuse.org/source/distribution/leap/15.3/repo/oss/
19 | repo-update-non-oss | Update Repository (Non-Oss) | Yes | (r ) Yes | Yes | http://download.opensuse.org/update/leap/15.3/non-oss/
[root@devlinux:/root]#
Create a backup of your existing REPOS:
[root@devlinux:/root]# cd /etc/zypp[root@devlinux:/etc/zypp]# cp -rp repos.d repos.d.bkpVerify if any of the ‘baseurl’ parms are hard coded (2 were in our setup):
[root@devlinux:/root]# cd /etc/zypp/repos.d[root@devlinux:/etc/zypp/repos.d]# grep baseurl /etc/zypp/repos.d/*.repo /etc/zypp/repos.d/google-chrome.repo:baseurl=https://dl.google.com/linux/chrome/rpm/stable/x86_64 /etc/zypp/repos.d/jenkins.repo:baseurl=https://pkg.jenkins.io/opensuse-stable/ /etc/zypp/repos.d/openSUSE-Leap-15.3-2.repo:baseurl=cd:/?devices=/dev/disk/by-id/ata-VBOX_CD-ROM_VB2-01700376 /etc/zypp/repos.d/openSUSE_Leap_15.0_Update.repo:baseurl=http://download.opensuse.org/update/leap/15.0/oss/ /etc/zypp/repos.d/oss.repo:baseurl=https://download.opensuse.org/distribution/leap/15.4/repo/oss/ /etc/zypp/repos.d/oss_update.repo:baseurl=https://download.opensuse.org/update/leap/15.4/oss/ /etc/zypp/repos.d/plex.repo:baseurl=https://downloads.plex.tv/repo/rpm/$basearch/ /etc/zypp/repos.d/repo-backports-debug-update.repo:baseurl=http://download.opensuse.org/update/leap/$releasever/backports_debug/15.4 /etc/zypp/repos.d/repo-backports-update.repo:baseurl=http://download.opensuse.org/update/leap/$releasever/backports/ /etc/zypp/repos.d/repo-debug-non-oss.repo:baseurl=http://download.opensuse.org/debug/distribution/leap/$releasever/repo/non-oss/ /etc/zypp/repos.d/repo-debug-update-non-oss.repo:baseurl=http://download.opensuse.org/debug/update/leap/$releasever/non-oss/ /etc/zypp/repos.d/repo-debug-update.repo:baseurl=http://download.opensuse.org/debug/update/leap/$releasever/oss/ /etc/zypp/repos.d/repo-debug.repo:baseurl=http://download.opensuse.org/debug/distribution/leap/$releasever/repo/oss/ /etc/zypp/repos.d/repo-non-oss.repo:baseurl=http://download.opensuse.org/distribution/leap/$releasever/repo/non-oss/ /etc/zypp/repos.d/repo-oss.repo:baseurl=http://download.opensuse.org/distribution/leap/$releasever/repo/oss/ /etc/zypp/repos.d/repo-sle-debug-update.repo:baseurl=http://download.opensuse.org/debug/update/leap/$releasever/sle/ /etc/zypp/repos.d/repo-sle-update.repo:baseurl=http://download.opensuse.org/update/leap/$releasever/sle/ /etc/zypp/repos.d/repo-source.repo:baseurl=http://download.opensuse.org/source/distribution/leap/$releasever/repo/oss/ /etc/zypp/repos.d/repo-update-non-oss.repo:baseurl=http://download.opensuse.org/update/leap/$releasever/non-oss/ [root@devlinux:/etc/zypp/repos.d]#
If so, fix with the following:
sed -i 's/15.3/${releasever}/g' /etc/zypp/repos.d/*.repoVerify REPOs again with ‘grep baseurl /etc/zypp/repos.d/*.repo’
List REPOs with ‘zypper –releasever=15.4 lr -u’, all ‘baseurl’ values should now reflect 15.4.
NOTE:
--releasever should be set to the minor version we are upgrading to.
Verify the URLs in the URI column in a web browser to see if they exist. If they do not, remove (or disable) the REPO(s) (the column ‘Enable’ needs to have a value of ‘No’).
Also, verify that the minor version you want to upgrade to has actually been released:
https://en.wikipedia.org/wiki/OpenSUSE
And:
https://www.opensuse.org/#Leap
https://get.opensuse.org/leap/15.5/
Begin Upgrade
Update the REPO metadata for 15.4:
[root@devlinux:/root]# zypper --releasever=15.4 refresh Warning: Enforced setting: $releasever=15.4 Repository 'google-chrome' is up to date. Repository 'jenkins' is up to date. Repository 'Online updates for openSUSE Leap:15.0 (standard)' is up to date. Repository 'oss' is up to date. Repository 'oss_update' is up to date. Retrieving repository 'Update repository of openSUSE Backports' metadata ..........................................................................................................................................................................................[done] Building repository 'Update repository of openSUSE Backports' cache ...............................................................................................................................................................................................[done] Retrieving repository 'Non-OSS Repository' metadata ...............................................................................................................................................................................................................[done] Building repository 'Non-OSS Repository' cache ....................................................................................................................................................................................................................[done] Retrieving repository 'Main Repository' metadata ..................................................................................................................................................................................................................[done] Building repository 'Main Repository' cache .......................................................................................................................................................................................................................[done] Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata .....................................................................................................................................................................[done] Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ..........................................................................................................................................................................[done] Retrieving repository 'Update Repository (Non-Oss)' metadata ......................................................................................................................................................................................................[done] Building repository 'Update Repository (Non-Oss)' cache ...........................................................................................................................................................................................................[done] All repositories have been refreshed. [root@devlinux:/root]#
We’ll be using the --download-in-advance option here. Ideally, in a large corporate setting you would have a shared REPO or use something like ‘Red Hat Satellite Servers’ or a ‘Capsule Server’ that caches content on the other side of the corporate web proxy in your organization.
We were once engaged in a situation where a team was upgrading 30+ servers via a saturated web proxy. This led to a hideous error that did not appear to be related to a network connection issue. Tickets were opened to the vendor with everyone going in circles for half a year.
Over-the-wire upgrades should be avoided, especially in corporate environments. This update approach can be vulnerable to disruptions caused by unreliable network connections, leading to unexpected and subtle issues during the update process.
Other caveats: If the upgrade process is interrupted due to (e.g., power outage, network disruption etc), you ‘could’ be left with a broken system. It is recommended to upgrade via the ISO image or download all required files prior to executing a disconnected (offline) upgrade.
Start the distribution update (abbreviated transcript):
[root@devlinux:/root]# zypper --releasever=15.4 dup --download-in-advance
Warning: Enforced setting: $releasever=15.4
Loading repository data...
Warning: Repository 'Online updates for openSUSE Leap:15.0 (standard)' appears to be outdated. Consider using a different mirror or server.
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...
The following 1190 packages are going to be upgraded:
ImageMagick ImageMagick-config-7-SUSE Mesa Mesa-dri Mesa-gallium Mesa-libEGL1 Mesa-libGL1 Mesa-libglapi0 Mesa-libva ModemManager ModemManager-bash-completion ModemManager-lang MozillaFirefox MozillaFirefox-branding-openSUSE MozillaThunderbird NetworkManager
NetworkManager-applet NetworkManager-applet-lang NetworkManager-branding-openSUSE NetworkManager-connection-editor NetworkManager-lang PackageKit PackageKit-backend-zypp PackageKit-branding-openSUSE PackageKit-gstreamer-plugin PackageKit-gtk3-module
[...]
The following 19 patterns are going to be upgraded:
apparmor base basesystem documentation enhanced_base fonts fonts_opt imaging minimal_base multimedia sw_management x11 x11_enhanced x11_yast xfce xfce_basis xfce_office yast2_basis yast2_desktop
The following product is going to be upgraded:
openSUSE Leap 15.3 15.3-2 -> 15.4-1
The following 42 packages are going to be downgraded:
cogl-lang gimp gimp-lang gimp-plugins-python gstreamer-plugins-cogl icewm-theme-branding libcogl20 libebl-plugins libgimp-2_0-0 libgimpui-2_0-0 libhogweed4 libnettle6 libply-boot-client5 libply-splash-core5 libply-splash-graphics5 libply5 librsvg-2-2 librsvg-lang
libsgutils2-1_43-2 libvncclient0 libyui-ncurses15 libyui-qt-graph15 libyui-qt15 libyui15 plymouth plymouth-dracut plymouth-plugin-label plymouth-plugin-label-ft plymouth-plugin-two-step plymouth-scripts plymouth-theme-bgrt plymouth-theme-spinner python2-cairo
python2-cffi python2-dbus-python python2-pycparser python2-rpm python2-six python3-rpm rpm util-linux util-linux-lang
The following 2 packages are going to change architecture:
ModemManager-bash-completion x86_64 -> noarch
xorg-x11-libX11-ccache noarch -> x86_64
The following 144 NEW packages are going to be installed:
abseil-cpp bash-sh crypto-policies crypto-policies-scripts ctags distribution-logos-openSUSE-icons dracut-mkinitrd-deprecated fontconfig-lang gcr-ssh-agent geoclue2 google-opensans-fonts google-poppins-fonts grub2-i386-pc-extras ibus ibus-gtk ibus-gtk3 ibus-lang
irqbalance-ui kernel-default-5.14.21-150400.24.74.1 kernel-default-devel-5.14.21-150400.24.74.1 kernel-default-extra-5.14.21-150400.24.74.1 kernel-default-optional-5.14.21-150400.24.74.1 kernel-devel-5.3.18-57.3 kernel-devel-5.14.21-150400.24.74.1
[...]
The following 10 packages are going to be REMOVED:
libfreebl3-hmac libgupnp-1_2-0 libsoftokn3-hmac libyui-ncurses-pkg15 libyui-qt-pkg15 nma-data perl-TermReadLine-Gnu python3-cairo python3-smbc systemd-icon-branding-openSUSE
The following 2 packages require a system reboot:
kernel-default-5.14.21-150400.24.74.1 kernel-preempt
1190 packages to upgrade, 42 to downgrade, 144 new, 10 to remove, 2 to change arch.
Overall download size: 0 B. Already cached: 1.69 GiB. After the operation, additional 1.3 GiB will be used.
Note: System reboot required.
Continue? [y/n/v/...? shows all options] (y): y
In cache MozillaFirefox-branding-openSUSE-68-lp154.5.1.x86_64.rpm (1/1376), 14.6 KiB ( 8.4 KiB unpacked)
In cache branding-openSUSE-15.4.20220322-lp154.2.3.noarch.rpm (2/1376), 11.6 KiB ( 1.6 KiB unpacked)
In cache btrfsprogs-udev-rules-5.14-150400.3.6.noarch.rpm (3/1376), 65.0 KiB ( 387 B unpacked)
[...]
In cache patterns-xfce-xfce_office-20210209-lp154.1.2.x86_64.rpm (1376/1376), 11.1 KiB ( 57 B unpacked)
Checking for file conflicts: .....................................................................................................................................................................................................................................[error]
Detected 1 file conflict:
File /usr/share/bash-completion/completions/lsblk
from install of
util-linux-2.36.2-2.29.x86_64 (oss)
conflicts with file from install of
util-linux-systemd-2.37.2-150400.8.17.1.x86_64 (Update repository with updates from SUSE Linux Enterprise 15)
File conflicts happen when two packages attempt to install files with the same name but different contents. If you continue, conflicting files will be replaced losing the previous content.
Continue? [yes/no] (no): yes
( 1/1376) Installing: MozillaFirefox-branding-openSUSE-68-lp154.5.1.x86_64 ......................................................................................................................................................................................[done]
( 2/1376) Installing: branding-openSUSE-15.4.20220322-lp154.2.3.noarch ..........................................................................................................................................................................................[done]
( 3/1376) Installing: btrfsprogs-udev-rules-5.14-150400.3.6.noarch ..............................................................................................................................................................................................[done]
[...]
Restarting apache (all instances)
Executing %posttrans scripts ......................................................................................................................................................................................................................................[done]
Update notifications were received from the following packages:
mariadb-10.6.14-150400.3.26.1.x86_64 (/var/adm/update-messages/mariadb-10.6.14-150400.3.26.1-something)
View the notifications now? [y/n] (n): y
There are running programs which still use files and libraries deleted or updated by recent upgrades. They should be restarted to benefit from the latest updates. Run 'zypper ps -s' to list these programs.
Since the last system boot core libraries or services have been updated.
Reboot is suggested to ensure that your system benefits from these updates.
NOTE: The above process interrupted due to a missed critical prompt that required acknowledgment. Once acknowledged, you can safely resume the upgrade process.
Unless there’s an issue that needs to be addressed, acknowledge any conflicts that may arise (using good judgment) and resume the upgrade.
For example, in our case, the upgrade was interrupted 3 times:
Retrieving package patterns-xfce-xfce_office-20210209-lp154.1.2.x86_64 (1376/1376), 11.1 KiB ( 57 B unpack
Retrieving: patterns-xfce-xfce_office-20210209-lp154.1.2.x86_64.rpm ..................................................................................................................................................................................................[do
Checking for file conflicts: ........................................................................................................................................................................................................................................[err
Detected 1 file conflict:
File /usr/share/bash-completion/completions/lsblk
from install of
util-linux-2.36.2-2.29.x86_64 (oss)
conflicts with file from install of
util-linux-systemd-2.37.2-150400.8.17.1.x86_64 (Update repository with updates from SUSE Linux Enterprise 15)
File conflicts happen when two packages attempt to install files with the same name but different contents. If you continue, conflicting files will be replaced losing the previous content.
Continue? [yes/no] (no):
Problem occurred during or after installation or removal of packages:
Installation has been aborted as directed.
History:
- ABORT request:
Please see the above error message for a hint.
[root@devlinux:/root]#
Handling the above interruption:
If MySQL/MariaDB is installed, you may be prompted by the following notice during the upgrade. Make sure you have DB Exports and a Cold DB backup. A safe procedure for backing up your database has been provided in: Appendix > ‘Database Backup’. Post upgrade, be sure to conduct Application and DB sanities.
Enter ‘q’ to resume upgrade process.
Use the Enter or Space key to scroll the text by lines or pages.)
Message from package mariadb:
WARNING: You are upgrading from different stable version of MySQL!
Your database will be migrated automatically during next restart of MySQL.
Before you do that make sure you have up to date backup of your data. It
should be mainly in /var/lib/mysql directory.
-----------------------------------------------------------------------------
(Press 'q' to exit the pager.)
/var/tmp/TmpFile.qYfUaQ lines 1-16/16 (END)
You can resume fully interrupted upgrades simply by re-executing the upgrade command:
[root@devlinux:/root]# zypper --releasever=15.4 dup --download-in-advance
Warning: Enforced setting: $releasever=15.4
Building repository 'oss' cache ...................................................................................................................................................................................................................................[done]
Building repository 'Non-OSS Repository' cache ....................................................................................................................................................................................................................[done]
Building repository 'Main Repository' cache .......................................................................................................................................................................................................................[done]
Loading repository data...
Warning: Repository 'Online updates for openSUSE Leap:15.0 (standard)' metadata expired since 2020-02-15 11:04:07 -05.
Warning: Repository metadata expired: Check if 'autorefresh' is turned on (zypper lr), otherwise
manualy refresh the repository (zypper ref). If this does not solve the issue, it could be that
you are using a broken mirror or the server has actually discontinued to support the repository.
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...
The following 7 packages are going to be upgraded:
ibus ibus-gtk ibus-gtk3 ibus-lang libibus-1_0-5 yum zypper-aptitude
The following package is going to be downgraded:
python2-yum
The following NEW package is going to be installed:
ibus-dict-emoji
7 packages to upgrade, 1 to downgrade, 1 new.
Overall download size: 8.7 MiB. Already cached: 0 B. After the operation, additional 35.5 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving: python2-yum-3.4.3-lp150.4.4.1.x86_64 (Online updates for openSUSE Leap:15.0 (standard)) (1/9), 429.5 KiB
Retrieving: python2-yum-3.4.3-lp150.4.4.1.x86_64.rpm .....................................................................................................................................................................................................[done (77 B/s)]
Retrieving: yum-3.4.3-lp150.4.4.1.x86_64 (Online updates for openSUSE Leap:15.0 (standard)) (2/9), 174.1 KiB
Retrieving: yum-3.4.3-lp150.4.4.1.x86_64.rpm ............................................................................................................................................................................................................[done (465 B/s)]
Retrieving: ibus-dict-emoji-1.5.25-150400.1.13.noarch (Main Repository) (3/9), 7.0 MiB
[...]
(8/9) Installing: ibus-lang-1.5.25-150400.1.13.noarch .............................................................................................................................................................................................................[done]
(9/9) Installing: zypper-aptitude-1.14.61-150400.3.24.1.noarch ....................................................................................................................................................................................................[done]
Executing %posttrans scripts ......................................................................................................................................................................................................................................[done]
There are running programs which still use files and libraries deleted or updated by recent upgrades. They should be restarted to benefit from the latest updates. Run ‘zypper ps -s’ to list these programs.
Since the last system boot core libraries or services have been updated.
Reboot is suggested to ensure that your system benefits from these updates.
[root@devlinux:/root]#
What zypper ps -s looks like, but we’re rebooting:
NOTE: Save this output for later reference, it will be part of your post-install check. Once the system is back online we want to ensure server based processes have restarted.
[root@devlinux:/var/lib/mysql]# zypper ps -s The following running processes use deleted files: PID | PPID | UID | User | Command | Service ------+------+------+------------+-------------------------------+-------------------- 931 | 1 | 499 | messagebus | dbus-daemon (deleted) | dbus 938 | 1 | 493 | polkitd | polkitd | polkit 941 | 1 | 0 | root | VBoxDRMClient | vboxclient 1009 | 1 | 0 | root | systemd-logind (deleted) | systemd-logind 1033 | 1 | 461 | jenkins | java (deleted) | jenkins 1451 | 1 | 0 | root | VBoxService | vboxadd-service 1477 | 1346 | 459 | plex | Plex Script Host | plexmediaserver 1497 | 1 | 1001 | syncthing | syncthing (deleted) | syncthing@syncthing 1504 | 1497 | 1001 | syncthing | syncthing (deleted) | syncthing@syncthing 1720 | 1 | 0 | root | agetty (deleted) | getty@tty1 1763 | 1 | 0 | root | lightdm (deleted) | display-manager 1769 | 1763 | 0 | root | Xorg (deleted) | display-manager 1780 | 1763 | 0 | root | lightdm (deleted) | 1794 | 1 | 466 | lightdm | systemd (deleted) | 1795 | 1794 | 466 | lightdm | systemd (deleted) | 1802 | 1780 | 466 | lightdm | lightdm-gtk-greeter (deleted) | 1807 | 1 | 0 | root | systemd (deleted) | 1808 | 1807 | 0 | root | systemd (deleted) | 1835 | 1794 | 466 | lightdm | dbus-daemon (deleted) | 1839 | 1794 | 466 | lightdm | at-spi-bus-launcher (deleted) | 1851 | 1839 | 466 | lightdm | dbus-daemon (deleted) | 1854 | 1794 | 466 | lightdm | gvfsd (deleted) | 1862 | 1794 | 466 | lightdm | gvfsd-fuse (deleted) | 1874 | 1763 | 0 | root | lightdm (deleted) | display-manager 1877 | 1794 | 466 | lightdm | at-spi2-registryd (deleted) | 3618 | 1 | 0 | root | sshd (deleted) | 3620 | 3618 | 0 | root | mksh | 3859 | 1 | 0 | root | sshd (deleted) | 3861 | 3859 | 0 | root | mksh | 3931 | 1 | 0 | root | sshd (deleted) | 3933 | 3931 | 0 | root | mksh | 7331 | 1 | 0 | root | sshd (deleted) | 7333 | 7331 | 0 | root | mksh | 11359 | 1 | 468 | nscd | nscd | nscd 11909 | 1807 | 0 | root | gvfsd (deleted) | 11914 | 1807 | 0 | root | gvfsd-fuse (deleted) | 13387 | 1 | 474 | chrony | chronyd | chronyd You may wish to restart these processes. See 'man zypper' for information about the meaning of values in the above table. Since the last system boot core libraries or services have been updated. Reboot is suggested to ensure that your system benefits from these updates. [root@devlinux:/var/lib/mysql]#
Reboot:
[root@devlinux:/var/lib/mysql]# shutdown -r now
Connection to 192.168.1.30 closed by remote host.
Connection to 192.168.1.30 closed.
Validations
We’re upgraded from 15.3 to 15.4:
[root@devlinux:/var/lib/mysql]# cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.4"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.4"
PRETTY_NAME="openSUSE Leap 15.4"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.4"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"
[root@devlinux:/var/lib/mysql]#
[root@devlinux:/var/lib/mysql]# hostnamectl
Static hostname: devlinux
Icon name: computer-vm
Chassis: vm
Machine ID: 590ae5aab6fe4d05b2b21d2f99bc8ec1
Boot ID: f894d53583674698a411152d7607b6b4
Virtualization: oracle
Operating System: openSUSE Leap 15.4
CPE OS Name: cpe:/o:opensuse:leap:15.4
Kernel: Linux 5.3.18-150300.59.106-default
Architecture: x86-64
[root@devlinux:/var/lib/mysql]#
The kernel was upgraded from…
[root@devlinux:/var/lib/mysql]# uname -a
Linux devlinux 5.3.18-150300.59.106-default #1 SMP Mon Dec 12 13:16:24 UTC 2022 (774239c) x86_64 x86_64 x86_64 GNU/Linux
…to:
[root@devlinux:/root]# uname -a
Linux devlinux 5.14.21-150400.24.74-default #1 SMP PREEMPT_DYNAMIC Thu Jul 27 15:04:26 UTC 2023 (28a2488) x86_64 x86_64 x86_64 GNU/Linux
Use the following commands to further validate:
Manually look over the system log:
less /var/log/messagesExecute egrep -i "error|warning" /var/log/messages, looking for (googling) anything of concern. Depending on your terminal program, output should be color coded. Some errors will be benign.
dmesg | more
journalctl -xe
NOTE: Save the the above outputs as a post-install baseline, which can be used to compare with the pre-install baseline.
Our DB process looks good:
You should conduct a complete Application and Database sanity.
[root@devlinux:/root]# ps -ef | grep mysql
mysql 991 1 0 01:57 ? 00:00:00 /usr/sbin/mysqld --defaults-file=/etc/my.cnf --user=mysql
root 13197 13156 0 02:00 pts/0 00:00:00 grep --color=auto mysql
[root@devlinux:/root]# systemctl status mysql
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-08-10 01:57:19 -05; 2min 51s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 951 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, status=0/SUCCESS)
Process: 981 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, status=0/SUCCESS)
Main PID: 991 (mysqld)
Status: "Taking your SQL requests now..."
Tasks: 8 (limit: 4915)
CGroup: /system.slice/mariadb.service
└─ 991 /usr/sbin/mysqld --defaults-file=/etc/my.cnf --user=mysql
Aug 10 01:57:19 devlinux systemd[1]: Starting MariaDB database server...
Aug 10 01:57:19 devlinux systemd[1]: Started MariaDB database server.
[root@devlinux:/root]#
[root@devlinux:/root]# ps -ef | egrep -i "jenkins|syncthing|VBoxService"
jenkins 872 1 1 01:57 ? 00:00:10 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
syncthi+ 874 1 0 01:57 ? 00:00:00 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
syncthi+ 959 874 0 01:57 ? 00:00:01 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
root 1286 1 0 01:57 ? 00:00:00 /usr/sbin/VBoxService --pidfile /var/run/vboxadd-service.sh
root 14833 13156 0 02:14 pts/0 00:00:00 grep -E --color=auto -i jenkins|syncthing|VBoxService
[root@devlinux:/root]#
All is clean here:
[root@devlinux:/root]# zypper ps -s
No processes using deleted files found.
No core libraries or services have been updated since the last system boot.
Reboot is probably not necessary.
[root@devlinux:/root]#
If all is good, clear the cache:
[root@devlinux:/root]# zypper clean -a
All repositories have been cleaned up.
[root@devlinux:/root]#
Appendix
Database Backup
Here’s an example of a safe, Cold DB backup.
Stop the database:
[root@devlinux:/root]# systemctl list-units | grep mari mariadb.service loaded active running MariaDB database server
[root@devlinux:/root]# systemctl stop mariadb
Verify:
[root@devlinux:/root]# systemctl status mariadb
○ mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Thu 2023-08-10 01:31:53 -05; 35s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 12590 ExecStart=/usr/lib/mysql/mysql-systemd-helper start (code=exited, status=0/SUCCESS)
Main PID: 12590 (code=exited, status=0/SUCCESS)
Status: "MariaDB server is down"
Aug 10 00:51:14 devlinux mysql-systemd-helper[12547]: wikidatabase.watchlist_expiry OK
Aug 10 00:51:14 devlinux mysql-systemd-helper[12547]: Phase 7/7: Running 'FLUSH PRIVILEGES'
Aug 10 00:51:14 devlinux mysql-systemd-helper[12547]: OK
Aug 10 00:51:14 devlinux mysql-systemd-helper[12514]: Everything upgraded successfully
Aug 10 00:51:14 devlinux mysql-systemd-helper[12514]: Shutting down protected MySQL
Aug 10 00:51:15 devlinux mysql-systemd-helper[12514]: Final cleanup
Aug 10 00:51:15 devlinux systemd[1]: Started MariaDB database server.
Aug 10 01:31:53 devlinux systemd[1]: Stopping MariaDB database server...
Aug 10 01:31:53 devlinux systemd[1]: mariadb.service: Deactivated successfully.
Aug 10 01:31:53 devlinux systemd[1]: Stopped MariaDB database server.
[root@devlinux:/var/lib/mysql]#
Execute a Cold backup:
[root@devlinux:/root]# cd /var/lib
[root@devlinux:/var/lib]# cp -rp mysql mysql.cold [root@devlinux:/var/lib]#
Verify:
[root@devlinux:/var/lib]# du -hs mysql* 221M mysql 0 mysql-files 221M mysql.cold [root@devlinux:/var/lib]#
NOTE: You can further verify the originals and copies with md5sum.
Make sure you test this Cold backup either on a different system (aka, standby server) that has the same database application version installed (Ex. MySQL or MariaDB) or moving the current DB directory / files aside and moving the Cold backup you executed in place and starting up the database (conducting a quick Application and Database sanity).
Conclusion
Congratulations, you have a fully upgraded system! We hope this guide was helpful. You can share, upvote, discuss or let us know if you have any further questions in the comment section!
Thanks for reading!
If you enjoyed this post, please share:
XOMedia is a full-service IT solutions provider. Learn how your business can benefit from XOMedia's 30+ years of experience with our Consulting and Partnership services - all work is backed by our 100% guaranteed.
Back to Blog Home