CentOS Installation

Checking the packages in GSI_CORE, GSI_FILESERVER, IB shows a number of packages with identical names. A number of packages has obvious pendants in CentOS, some others cannot be easily identified. Certainly not all packages are relevant for this CentOS-Lustre case.

knife bootstrap

does not work from lxdv84 due to old chef versions etc.

The validation.pem distributed by FAI fits, but not the FAI-client.rb. Instead use a simpler version (also in chef/roles/development/cluster/virgo/execution_node.rb)

chef_server_url  "https://chef.hpc.gsi.de"
log_location     STDOUT
log_level        :fatal
ssl_verify_mode  :verify_none

Thus, create /etc/chef/ with these two files, then run chef-client.

The following packages are added to the default installation (lxmgmt5:/srv/http/centos/kickstart/default.cfg):

• ibutils.x86_64
• infiniband-diags.x86_64
• emacs.x86_64

Perhaps the option --excludedocs should be dropped from default.cfg? (No man pages in installed system)

https://wiki.centos.org/HowTos/Grub2 seems to give some good hints concerning grub.

Install device-mapper-multipath.x86_64.

To run multipath, load the module, modprobe -v dm_multipath, create an /etc/multipath.conf by copying the file from lxfs410, and start it systemctl start multipathd.

• The blacklist_exceptions segment of multipath.conf contains the key word property which is not understood.
• Remove it to get

defaults {
     user_friendly_names yes
     bindings_file /etc/multipath/bindings
     path_grouping_policy failover
}
blacklist {
     devnode "^sda[0-9]*$"
}

To create the Nyx OSS disk mapping, install

• ruby.x86₆₄
• sg3_utils.x8664

Copy Benny's files to the server:

troth@lxdv84:~/git/nyx-oss-udev/sbin$ scp update-nyx-oss-disk-map root@lxfs419:/usr/sbin/
troth@lxdv84:~/git/nyx-oss-udev/udev$ scp nyx_oss_diskid root@lxfs419:/lib/udev/nyx_oss_diskid
troth@lxdv84:~/git/nyx-oss-udev/udev/rules.d$ scp 69-nyx-oss-disk.rules root@lxfs419:/etc/udev/rules.d/

Run the usual:

[root@lxfs419 ]# /usr/sbin/update-nyx-oss-disk-map
[root@lxfs419 ]# udevadm trigger --action=add
[root@lxfs419 ]# udevadm settle

Looks good:

[root@lxfs419 ~]# ls /dev/disk/by-location/
0-0-10  0-0-19  0-0-7  0-1-6   1-0-13  1-0-3  1-1-10  1-1-19  encb0   encb17  encb8   encf15  encf5  srvb4   srvf13  srvf21
0-0-11  0-0-2   0-0-8  0-1-7   1-0-14  1-0-4  1-1-11  1-1-2   encb1   encb18  encb9   encf16  encf6  srvb5   srvf14  srvf22
0-0-12  0-0-20  0-0-9  0-1-8   1-0-15  1-0-5  1-1-12  1-1-3   encb10  encb19  encf0   encf17  encf7  srvb6   srvf15  srvf3
0-0-13  0-0-21  0-1-0  0-1-9   1-0-16  1-0-6  1-1-13  1-1-4   encb11  encb2   encf1   encf18  encf8  srvb7   srvf16  srvf4
0-0-14  0-0-22  0-1-1  1-0-0   1-0-17  1-0-7  1-1-14  1-1-5   encb12  encb3   encf10  encf19  encf9  srvb8   srvf17  srvf5
0-0-15  0-0-3   0-1-2  1-0-1   1-0-18  1-0-8  1-1-15  1-1-6   encb13  encb4   encf11  encf2   srvb0  srvb9   srvf18  srvf6
0-0-16  0-0-4   0-1-3  1-0-10  1-0-19  1-0-9  1-1-16  1-1-7   encb14  encb5   encf12  encf20  srvb1  srvf10  srvf19  srvf7
0-0-17  0-0-5   0-1-4  1-0-11  1-0-2   1-1-0  1-1-17  1-1-8   encb15  encb6   encf13  encf3   srvb2  srvf11  srvf2   srvf8
0-0-18  0-0-6   0-1-5  1-0-12  1-0-20  1-1-1  1-1-18  1-1-9   encb16  encb7   encf14  encf4   srvb3  srvf12  srvf20  srvf9

To put Benny's nyx-oss-udev into a rpm, just use fpm, directory-to-rpm:

• Create /tmp/nyx-oss-udev with the target directories as subdirectories, and copy the three files to their targets:

troth@lxdv84:~$ ls -lR /tmp/nyx-oss-udev
/tmp/nyx-oss-udev:
total 0
drwxr-xr-x 3 troth hpc 17 Jul 14 18:29 etc
drwxr-xr-x 3 troth hpc 17 Jul 14 18:29 lib
drwxr-xr-x 3 troth hpc 17 Jul 14 18:28 usr

/tmp/nyx-oss-udev/etc:
total 0
drwxr-xr-x 3 troth hpc 20 Jul 14 18:29 udev

/tmp/nyx-oss-udev/etc/udev:
total 0
drwxr-xr-x 2 troth hpc 34 Jul 14 18:29 rules.d

/tmp/nyx-oss-udev/etc/udev/rules.d:
total 4
-rw-r--r-- 1 troth hpc 382 Jul 14 18:29 69-nyx-oss-disk.rules

/tmp/nyx-oss-udev/lib:
total 0
drwxr-xr-x 2 troth hpc 27 Jul 14 18:29 udev

/tmp/nyx-oss-udev/lib/udev:
total 4
-rwxr-xr-x 1 troth hpc 1486 Jul 14 18:29 nyx_oss_diskid

/tmp/nyx-oss-udev/usr:
total 0
drwxr-xr-x 2 troth hpc 36 Jul 14 18:28 sbin

/tmp/nyx-oss-udev/usr/sbin:
total 8
-rwxr-xr-x 1 troth hpc 5347 Jul 14 18:28 update-nyx-oss-disk-map

The run fpm

fpm -s dir -t rpm -C /tmp/nyx-oss-udev --name nyx-oss-udev --version 1.0.0 --iteration 1  --description "nys-oss-udev"

The resulting nyx-oss-udev-1.0.0-1.x86_64.rpm is rpm-installable on lxfs419.

• To install Lustre packages, the newest release is downloaded from https://downloads.hpdd.intel.com .
• The desired parts of that release are copied to the mirror, creating lxrpt2:/srv/centos/lustre-repository/lustre-2.10.0/el7.3.1611/server and lxrpt2:/srv/centos/lustre-repository/lustre-2.10.0/el7.3.1611/client.
• The subdir repodata already exists upstream, but createrepo --update $PWD in the server or client directory overwrites these.

In the Chef role for Hebe OSS, cent_oss.rb, the following yum.repos.d/ config is sufficient

'/etc/yum.repos.d/lustre-server.repo' => {
  content: '
    [lustre-server]
    name = lustre-server
    baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.0/el7.3.1611/server
    enabled = 1
    gpgcheck = 0
  '
},

The name lustre-server appears only here, not on the mirror. Nevertheless, yum repolist shows gsi-mirror, gsi-repo, and lustre-server.

• After addition of the lustre-server repo, yum update installs

Installing:
 kernel                         x86_64              3.10.0-514.21.1.el7_lustre               lustre-server               37 M
Updating:
 kernel-tools                   x86_64              3.10.0-514.21.1.el7_lustre               lustre-server              4.0 M
 kernel-tools-libs              x86_64              3.10.0-514.21.1.el7_lustre               lustre-server              3.9 M
 python-perf                    x86_64              3.10.0-514.21.1.el7_lustre               lustre-server              4.0 M

• Installation of lustre.x86_64 (contains the utilities) fails because amongst others, kmod-lustre-osd-ldiskfs-2.10.0-1.el7.x86_64 is a dependency, and this requires ldiskfsprogs.

Of course, on an OSS we don't want ldiskfs.

Installation of ZFS follows https://github.com/zfsonlinux/zfs/wiki/RHEL-&-CentOS.

• The package to start with is http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm, but downloading and copying to lxrpt2 + createrepo --update $PWD does not make this package available on lxfs419.
• Add a proxy to /etc/yum.conf: proxy=http://proxy-ib2.gsi.de:3128 and install directly

yum install http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm

• Add the gpg key for zfsonlinux

gpg --quiet --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

• The next step of the zfsonlinux-instruction, yum install kernel-devel zfs, fails because dkms is missing.
  • Seems that dkms is not a standard package in CentOS, although it is used everywhere…
• The official workaround is to make use of epel repositories (https://www.tecmint.com/how-to-enable-epel-repository-for-rhel-centos-6-5/)
• Get the epel into the system:

yum install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-10.noarch.rpm

• yum repolist now reads

Loaded plugins: fastestmirror
epel/x86_64/metalink                                                                                   |  23 kB  00:00:00     
epel                                                                                                   | 4.3 kB  00:00:00     
(1/3): epel/x86_64/updateinfo                                                                          | 791 kB  00:00:00     
(2/3): epel/x86_64/group_gz                                                                            | 170 kB  00:00:00     
(3/3): epel/x86_64/primary_db                                                                          | 4.8 MB  00:00:00     
Loading mirror speeds from cached hostfile
 * epel: epel.besthosting.ua
repo id                                    repo name                                                                    status
epel/x86_64                                Extra Packages for Enterprise Linux 7 - x86_64                               11,888
gsi-mirror                                 gsi-mirror                                                                    9,363
gsi-repo                                   gsi-repo                                                                        0+2
lustre-server                              lustre-server                                                                  25+3
zfs/x86_64                                 ZFS on Linux for EL7 - dkms                                                      54
repolist: 21,330

• The option --enablerepo=epel mentioned on the tecmint-webpage seems to be enabled by default.
• Install dkms now works

yum --enablerepo=epel install dkms

==============================================================================================================================
 Package                     Arch                Version                                     Repository                  Size
==============================================================================================================================
Installing:
 dkms                        noarch              2.3-5.20170523git8c3065c.el7                epel                        78 k
Installing for dependencies:
 cpp                         x86_64              4.8.5-11.el7                                gsi-mirror                 5.9 M
 gcc                         x86_64              4.8.5-11.el7                                gsi-mirror                  16 M
 glibc-devel                 x86_64              2.17-157.el7                                gsi-mirror                 1.1 M
 glibc-headers               x86_64              2.17-157.el7                                gsi-mirror                 668 k
 kernel-devel                x86_64              3.10.0-514.21.1.el7_lustre                  lustre-server               19 M
 kernel-headers              x86_64              3.10.0-514.21.1.el7_lustre                  lustre-server              4.8 M
 libmpc                      x86_64              1.0.1-3.el7                                 gsi-mirror                  51 k
 mpfr                        x86_64              3.1.1-4.el7                                 gsi-mirror                 203 k

• The gpg key file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 has a matching fingerprint and is imported.

yum install kernel-devel zfs

==============================================================================================================================
 Package                      Arch                     Version                                    Repository             Size
==============================================================================================================================
Installing:
 zfs                          x86_64                   0.6.5.11-1.el7.centos                      zfs                   337 k
Installing for dependencies:
 libnvpair1                   x86_64                   0.6.5.11-1.el7.centos                      zfs                    40 k
 libuutil1                    x86_64                   0.6.5.11-1.el7.centos                      zfs                    46 k
 libzfs2                      x86_64                   0.6.5.11-1.el7.centos                      zfs                   128 k
 libzpool2                    x86_64                   0.6.5.11-1.el7.centos                      zfs                   423 k
 spl                          x86_64                   0.6.5.11-1.el7.centos                      zfs                    30 k
 spl-dkms                     noarch                   0.6.5.11-1.el7.centos                      zfs                   444 k
 zfs-dkms                     noarch                   0.6.5.11-1.el7.centos                      zfs                   1.9 M

yum install lustre-osd-zfs-mount.x86_64

==============================================================================================================================
 Package                              Arch                   Version                      Repository                     Size
==============================================================================================================================
Installing:
 lustre-osd-zfs-mount                 x86_64                 2.10.0-1.el7                 lustre-server                  10 k

yum install lustre-dkms.noarch

==============================================================================================================================
 Package                       Arch                     Version                         Repository                       Size
==============================================================================================================================
Installing:
 lustre-dkms                   noarch                   2.10.0-1.el7                    lustre-server                    13 M
Installing for dependencies:
 expect                        x86_64                   5.45-14.el7_1                   gsi-mirror                      262 k

yum install kmod-lustre.x86_64

==============================================================================================================================
 Package                       Arch                     Version                         Repository                       Size
==============================================================================================================================
Installing:
 kmod-lustre                   x86_64                   2.10.0-1.el7                    lustre-server                   3.7 M

yum install lustre.x86_64

==============================================================================================================================
 Package                        Arch              Version                                      Repository                Size
==============================================================================================================================
Installing:
 lustre                         x86_64            2.10.0-1.el7                                 lustre-server            738 k
Installing for dependencies:
 libyaml                        x86_64            0.1.4-11.el7_0                               gsi-mirror                55 k
 lm_sensors-libs                x86_64            3.4.0-4.20160601gitf9185e5.el7               gsi-mirror                41 k
 net-snmp-agent-libs            x86_64            1:5.7.2-24.el7_2.1                           gsi-mirror               702 k
 net-snmp-libs                  x86_64            1:5.7.2-24.el7_2.1                           gsi-mirror               747 k

Although the zfs packages seem to have installed without problems, zpool cannot execute because the module zfs is not loaded, and modprobe zfs does not find the module.

• First of all, reboot, to get the kernel 3.10.0-514.21.1.el7_lustre.x86_64 running.
• Still Module zfs not found.
• Check the modules directory:

[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# pwd
/lib/modules/3.10.0-514.21.1.el7_lustre.x86_64
[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# ls
build          modules.alias.bin    modules.dep      modules.modesetting  modules.symbols      vdso
extra          modules.block        modules.dep.bin  modules.networking   modules.symbols.bin  weak-updates
kernel         modules.builtin      modules.devname  modules.order        source
modules.alias  modules.builtin.bin  modules.drm      modules.softdep      updates

[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# find . -name "*zfs*"

[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# find . -name "*lustre*"
./extra/lustre
./extra/lustre/fs/lustre.ko

[root@lxfs419 weak-updates]# cd ../extra/
[root@lxfs419 extra]# ls
lustre  splat.ko  spl.ko  zavl.ko  zcommon.ko  zfs.ko  znvpair.ko  zpios.ko  zunicode.ko
[root@lxfs419 extra]# modinfo ./zfs.ko 
filename:       /lib/modules/3.10.0-514.21.1.el7_lustre.x86_64/extra/./zfs.ko
...

• So the module file is there, actually. Perhaps run depmod?

[root@lxfs419 extra]# yum search depmod
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * epel: epel.besthosting.ua
Warning: No matches found for: depmod
No matches found

• So, CentOS does not need depmod?
• Check modprobe:

[root@lxfs419 extra]# man modprobe
No manual entry for modprobe

• Right, CentOS also does not need man pages by default.
  • Reinstallation of a package usually installs its manpages, too.

yum whatprovides /usr/sbin/modprobe
kmod-20-9.el7.x86_64

yum reinstall kmod-20-9.el7.x86_64

• Man page of modprobe is now installed but says nothing.
• According to the web, depmod comes with the package module-init-tools

[root@lxfs419 extra]# yum install module-init-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * epel: epel.besthosting.ua
Package kmod-20-9.el7.x86_64 already installed and latest version
Nothing to do

• So what? Already installed?

[root@lxfs419 extra]# depmod --help
Usage:
	depmod -[aA] [options] [forced_version]
...

• WFT?

[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# pwd
/lib/modules/3.10.0-514.21.1.el7_lustre.x86_64
[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# ls
build          modules.alias.bin    modules.dep      modules.modesetting  modules.symbols      vdso
extra          modules.block        modules.dep.bin  modules.networking   modules.symbols.bin  weak-updates
kernel         modules.builtin      modules.devname  modules.order        source
modules.alias  modules.builtin.bin  modules.drm      modules.softdep      updates
[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# find . -name *zfs*
./extra/zfs.ko
[root@lxfs419 3.10.0-514.21.1.el7_lustre.x86_64]# modprobe -v zfs
insmod /lib/modules/3.10.0-514.21.1.el7_lustre.x86_64/kernel/lib/zlib_deflate/zlib_deflate.ko 
...

• Basically, reinstallation of kmod-20-9.el7.x86_64 has triggered (creation?,) existence and loadability of zfs.
• Creation of a zpool works as in Debian.

To avoid the conflict that showed up later with building the osd-zfs module, where in the end the DKMS approach was dismissed and the kmod approach accepted, stick to the section kABI-tracking kmod and just put the following repo on the server:

# cat /etc/yum.repos.d/zfs.repo 
#
# DO NOT CHANGE THIS FILE MANUALLY!
#
# This file is managed by the Chef configuration management system
#

[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7.3/kmod/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

• Also install the key RPM-GPG-KEY-zfsonlinux

yum install zfs

will install all necessary packages, also creating the zfs module.

==============================================================================================================================================================================================================================================================
 Package                                                      Arch                                                Version                                                                       Repository                                               Size
==============================================================================================================================================================================================================================================================
Installing:
 zfs                                                          x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                382 k
Installing for dependencies:
 kmod-spl                                                     x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                105 k
 kmod-zfs                                                     x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                866 k
 libnvpair1                                                   x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                 29 k
 libuutil1                                                    x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                 35 k
 libzfs2                                                      x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                129 k
 libzpool2                                                    x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                586 k
 lm_sensors-libs                                              x86_64                                              3.4.0-4.20160601gitf9185e5.el7                                                gsi-mirror                                               41 k
 spl                                                          x86_64                                              0.7.0-1.el7_3                                                                 zfs-kmod                                                 29 k
 sysstat                                                      x86_64                                              10.1.5-11.el7                                                                 gsi-mirror                                              310 k

The package kmod-lustre-osd-zfs cannot be installed: mountains of ksym(...) errors. Without it, no module osd-zfs exists. Lustre and ZFS can be installed, a zpool can be created and formatted with Lustre, but it cannot be mounted.

• Seems like a compilation of Lustre locally is necessary. Several discussions on the web point to that (cf. http://lustre-discuss.lustre.narkive.com/VmraUPnP/can-t-load-module-osd-zfs)
• Follow https://lists.01.org/pipermail/hpdd-discuss/2016-December/003047.html [HPDD-discuss] Error when attempting to mount ZFS with Lustre

During several attempts it became clear that the lustre-repo is constructed above gets in the way.

• Running yum update unconditionally with lustre-server.repo enabled will install the Lustre-Kernel kernel-3.10.0-514.21.1.el7_lustre.x86_64.rpm which should not be necessary on a ZFS system.
• The repo also installs kernel-devel and kernel-headers in the Lustre variant which also collide with the build.

Instead, start a fresh installation, follow https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS with installation of zfs-release.el7_3.noarch.rpm and subsequently kernel-devel, zfs. Then follow https://lists.01.org/pipermail/hpdd-discuss/2016-December/003047.html religiously:

[root@lxfs419 ~]# yum install yum-utils-1.1.31-40.el7.noarch
[root@lxfs419 ~]# yum-config-manager --disable zfs
[root@lxfs419 ~]# yum-config-manager --enable zfs-kmod
[root@lxfs419 ~]# yum -y install kmod-zfs-devel libzfs2-devel

troth@lxdv84:~/Downloads/Lustre/CentOS/lustre-2.10.0/el7.3.1611/public/lustre/lustre-2.10.0/el7.3.1611/server/SRPMS$ scp lustre-2.10.0-1.src.rpm root@lxfs419:

[root@lxfs419 ~]# yum -y install rpm-build
[root@lxfs419 ~]# yum -y install libselinux-devel libtool 
[root@lxfs324 ~]# yum -y install kernel-devel-3.10.0-862.14.4.el7.x86_64
[root@lxfs419 ~]# uname -r
3.10.0-514.el7.x86_64
[root@lxfs419 ~]# rpm -qa | grep kernel | grep `uname -r`
kernel-devel-3.10.0-514.el7_lustre.x86_64
kernel-headers-3.10.0-514.el7_lustre.x86_64
kernel-3.10.0-514.el7_lustre.x86_64

• An important deviation from these Web-instructions was to add --without ldiskfs to the rpmbuild command: otherwise, after compilation the rpmbuild fails when trying to copy the (non-existing) file osd-ldiskfs.ko between directories.

[root@lxfs419 ~]# rpmbuild --rebuild --with zfs --without ldiskfs lustre-2.10.0-1.src.rpm
[root@lxfs419 ~]# yum -y install createrepo
[root@lxfs419 ~]# rpmbuild=`readlink -f ~/rpmbuild/RPMS`
[root@lxfs419 ~]# createrepo ${rpmbuild}
[root@lxfs419 ~]# cat <<END > /etc/yum.repos.d/lustre.repo
[lustre]
name=CentOS-$releasever - Lustre locally built RPMS
baseurl=file://${rpmbuild}
enabled=1
gpgcheck=0
END

• Installation of the newly built packages still fails at first, because zfs-dkms and spl-dkms had been installed from the zfsonlinus-Epel and now conflict the new ones.
  • Erase and install:

[root@lxfs419 ~]# yum erase zfs-dkms-0.6.5.11-1.el7.centos.noarch spl-dkms-0.6.5.11-1.el7.centos.noarch 
[root@lxfs419 ~]# yum -y install kmod-lustre-osd-zfs* lustre

After finding and importing a previously created osspool0, it was mounted with lxmds20 as the MDS (mkfs.lustre --reformat --mgs --mdt --fsname=hebcent --index=0 --servicenode=10.20.3.1@o2ib5 --mgsnode=10.20.3.1@o2ib5 --mtail ^Csoptions"-E stride=4,stripe-width=20 -O flex_{bg,mmp,uninitbg}" /dev/sda=)

The Lustre packages created on lxfs418 with an installed zfs version 0.7.0-1 are copied to the mirror, lxrpt2:/srv/centos/lustre-repository/lustre-2.10.0/el7.3.1611/zfs-server

• Set up an entry for that repo:

# cat /etc/yum.repos.d/zfs-server.repo 
#
# DO NOT CHANGE THIS FILE MANUALLY!
#
# This file is managed by the Chef configuration management system
#

[zfs-server]
name = zfs-server
baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.0/el7.3.1611/zfs-server
enabled = 1
gpgcheck = 0

yum install kmod-lustre.x86_64 kmod-lustre-osd-zfs.x86_64 lustre.x86_64 lustre-osd-zfs-mount.x86_64

wget and upload https://downloads.hpdd.intel.com/public/e2fsprogs/1.42.13.wc5/el7/ to lxrpt2:/srv/centos/lustre-repository/e2fsprogs

On lxmsd20, yum install e2fsprogs: one version is already installed, but the install command gets the Intel package, probably due to the higher version number.

With the correct e2fsprogs, the Lustre packages are installable on lxmds20, here including

kmod-lustre.x86_64 kmod-lustre-osd-ldiskfs lustre.x86_64 lustre-osd-ldiskfs-mount.x86_64

but not lustre-dkms.noarch (while dkms fomr the epel repo is installed), because of zfs-dependencies. Obviously, this package is for ZFS servers only.

The EPEL repo holding drbd packages is installable:

wget http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm + rpm -i ...

This gives access to drbd84-utils.x86_64 and kmod-drbd84.x86_64, but the modules from kmod-drbd84 are not compatible with the Linux kernel.

Follow instead https://www.howtoforge.com/tutorial/how-to-install-and-setup-drbd-on-centos-6/

mkdir -p /root/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
cd drbd-8.4.10-1
drbd-8.4.10-1]# make km-rpm

• This works for the module package, gives drbd-km-3.10.0_514.21.1.el7_lustre.x86_64-8.4.10-1.x86_64.rpm, which is installable and gives loadable module.
• The utils however cannot be compiled: ./configure works, but make rpm fails because failed to load external entity http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl
• Bump /etc/environment to the kronos.hpc.gsi.de version

ftp_proxy="http://proxy-ib2.gsi.de:3128/"
FTP_PROXY="http://proxy-ib2.gsi.de:3128/"
HTTP_PROXY="http://proxy-ib2.gsi.de:3128/"
http_proxy="http://proxy-ib2.gsi.de:3128/"
HTTPS_PROXY="http://proxy-ib2.gsi.de:3128/"
https_proxy="http://proxy-ib2.gsi.de:3128/"
no_proxy="localhost,127.0.0.1,10.10.0.0/16,chef.hpc.gsi.de"
NO_PROXY="localhost,127.0.0.1,10.10.0.0/16,chef.hpc.gsi.de"
RSYNC_PROXY="proxy-ib2.gsi.de:3128"
rsync_proxy="proxy-ib2.gsi.de:3128"
TZ=":/etc/localtime"

• Log out, log in again !!
• Install gcc++: yum install gcc-c++.x86_64

yum -y install gcc make automake autoconf libxslt libxslt-devel flex rpm-build

• Now make km-rpm works
• The compiled rpms in /root/rpmbuild/RPMS/x86_64 refuse to install because of conflicts of the type

file /usr/lib/debug/usr/sbin/drbdadm.debug conflicts between attempted installs of drbd-debuginfo-8.9.11-1.el7.centos.x86_64 and drbd-debuginfo-8.9.11-1.el7.centos.x86_64

• Perhaps this is due to the prior installation of the drbd84-utils package from the repo. This package had been yum remove'd, but perhaps rpm does not really know about it?
• rpm -iv --replacefiles is the command that works.

The tool of choice seems to be sssd here - nslcd is not needed/installed.

• The nslcd.keytab mentioned below has this name, nslcd, just for convenience: otherwise, a corresponding change on the keytab server would be necessary.
• Some kerberos-stuff seems to be installed by default: there is an /etc/krb5.conf and /etc/krb5.conf.d

yum install sssd

• This also creates /etc/sssd/, so the following config files can be written by Chef:

Create the following config files:

• sssd

# cat /etc/sssd/sssd.conf
[sssd]
config_file_version = 2
services = nss
domains = LDAP

[nss]
filter_groups = root
filter_users = root

[domain/LDAP]
id_provider = ldap
ldap_uri = ldap://lxldap04.gsi.de
ldap_search_base = dc=gsi,dc=de
krb5_server = lxkdc1.gsi.de
krb5_realm = GSI.DE
ldap_krb5_keytab = /etc/nslcd.keytab
ldap_sasl_authid = nslcd/lxmds19.gsi.de
ldap_sasl_mech = GSSAPI
ldap_schema = rfc2307
ldap_user_search_base = ou=people,dc=gsi,dc=de
ldap_group_search_base = ou=groups,dc=gsi,dc=de
enumerate = true

• Putting this sssd.conf as pure text into a role is fail: of course, the line with nslcd/lxmds19.gsi.de has to be adapted!
• krb5

# cat krb5.conf
# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]

[domain_realm]

• ldap

# cat /etc/openldap/ldap.conf 
#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

#BASE	dc=example,dc=com
#URI	ldap://ldap.example.com ldap://ldap-master.example.com:666

#SIZELIMIT	12
#TIMELIMIT	15
#DEREF		never

TLS_CACERTDIR /etc/openldap/cacerts

# Turning this off breaks GSSAPI used with krb5 when rdns = false
SASL_NOCANON	on
URI ldap://lxldap03.gsi.de/
BASE dc=gsi,dc=de

• The directory mentioned here, /etc/openldap/cacerts, is copied from server to server - is that correct?
  • On lxmds17, this directory was not created, but id troth still works.

keytabs can be installed with knife and wallet

troth@lxdv84:~$ knife krb5 info lxmds19.gsi.de
Information for lxmds19.gsi.de
...
Also found nslcd/lxmds19.gsi.de@GSI.DE
Also found verify/lxmds19.gsi.de@GSI.DE

• knife krb5 -a get lxmds19.gsi.de creates lxmds19:/etc/krb5.keytab
  • Since this installation is not running the usual kerb-cookbooks, the nslcd-keytab has to be place by hand as well:

troth@lxdv84:~$ wallet get keytab nslcd/lxmds19.gsi.de@GSI.DE -f /tmp/lxmds19nslcd

• This may not work due to

wallet: troth@GSI.DE not authorized to create keytab:nslcd/lxmds19.gsi.de@GSI.DE

• Destroy the keytabs that already exist

knife krb5 -a destroy lxmds19.gsi.de

• Get a new krb5 keytab locally:

knife krb5 get lxmds19.gsi.de

• Make yourself master of the universe

kinit -t /tmp/lxmds19.gsi.de20180226-988-po3mxt.keytab host/lxmds19.gsi.de

• Now get the nslcd.keytab with wallet

wallet get keytab nslcd/lxmds19.gsi.de@GSI.DE -f /tmp/lxmds19nslcd

• Move /tmp/lxmds19nslcd to lxmds19:/etc/nslcd.keytab
• Move /tmp/lxmds19.gsi.de20180226-988-po3mxt.keytab to lxmds19:/etc/krb5.keytab
• Enable and start sssd

systemctl enable sssd.service
systemctl start sssd.service

• According to Andreas Dilger, ZFS versions 0.7.10 and 0.7.11 have serious bugs and should not be used.
• Upgrade on lxfstest04 and create the necessary Lustre packages there (procedure as above), creating

kmod-lustre-2.10.5-1.el7.cern.x86_64.rpm          kmod-lustre-tests-2.10.5-1.el7.cern.x86_64.rpm  lustre-debuginfo-2.10.5-1.el7.cern.x86_64.rpm  lustre-osd-zfs-mount-2.10.5-1.el7.cern.x86_64.rpm    lustre-tests-2.10.5-1.el7.cern.x86_64.rpm kmod-lustre-osd-zfs-2.10.5-1.el7.cern.x86_64.rpm  lustre-2.10.5-1.el7.cern.x86_64.rpm lustre-iokit-2.10.5-1.el7.cern.x86_64.rpm      lustre-resource-agents-2.10.5-1.el7.cern.x86_64.rpm

These packages are put on the mirror, again creating a subdirectory named by the Lustre and the ZFS version.

The updated yum repo files read

# cat zfs.repo

[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7.5/kmod/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

# cat zfs.repo

[zfs-server]
name = zfs-server
# baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.2/el7.4/zfs-server
baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.5/zfs-0.7.9/zfs-server
enabled = 1
gpgcheck = 0

• Start by getting all security updates:

root@lxfs421:~# yum --security update

• Running update will install all new packages, effectivly upgrading to CentOS 7.5

root@lxfs434:~# yum update --skip-broken

• –skip-broken skips over the ZFS packages and the depending Lustre packages - without it, yum will try to satisfy dependencies with zfs 0.7.11, and more importantly it will fail to fullfill the dependencies of the existing Lustre packages - and the new Lustre packages are not in the scope of yum update.

• To get the new ZFS packages, first clean yum

root@lxfs434:~# yum clean all && rm -rf /var/cache/yum && ainsl /etc/yum.conf http_caching=none

• Install the packages, explicitly specifying the desired version:

root@lxfs434:~# yum install kmod-zfs-0.7.9-1.el7_5.x86_64 kmod-spl-0.7.9-1.el7_5.x86_64 zfs-0.7.9-1.el7_5.x86_64 spl-0.7.9-1.el7_5.x86_64 libzfs2-0.7.9-1.el7_5.x86_64 libzpool2-0.7.9-1.el7_5.x86_64 libnvpair1-0.7.9-1.el7_5.x86_64

• This will pull in the Lustre packages from lustre-repository/lustre-2.10.5/zfs-0.7.9/zfs-server as dependencies.
• However, the kernel modules from the ZFS packages go to /lib/modules/3.10.0-862.2.3.el7.x86_64/extra, while the newest kernel is 3.10.0-862.14.4.el7.x86_64.
  • An option to make yum install the package as-if for the desired kernel is not known.
• Furthermore: if the machine is booted first into 3.10.0-862.14.4 and the zfs packages are installed afterwards, the modules still go to the 2.3.../extra directory.
  • However, yum also creates symlinks from that /extra/ directory to 3.10.0-862.14.4.el7.x86_64/weak-updates
• Hence write down all necessary commands in a script:

# cat link_modules.sh

#!/bin/bash

mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/spl/spl
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/spl/splat
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/avl
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/icp
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/nvpair
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/unicode
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/zcommon
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/zfs
mkdir -p /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/zpios

ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/spl/spl/spl.ko          /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/spl/spl/spl.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/spl/splat/splat.ko      /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/spl/splat/splat.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/zfs/avl/zavl.ko         /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/avl/zavl.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/zfs/icp/icp.ko          /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/icp/icp.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/zfs/nvpair/znvpair.ko   /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/nvpair/znvpair.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/zfs/unicode/zunicode.ko /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/unicode/zunicode.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/zfs/zcommon/zcommon.ko  /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/zcommon/zcommon.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/zfs/zfs/zfs.ko          /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/zfs/zfs.ko
ln -s /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/zfs/zpios/zpios.ko      /lib/modules/3.10.0-862.14.4.el7.x86_64/weak-updates/zfs/zpios/zpios.ko


depmod -w 3.10.0-862.14.4.el7.x86_64

• Remember grub2-set-default 0 to boot the wanted kernel, for i in $(seq 0 6); do umount /srv/OST$i; done, lustre_rmmod and reboot.

• Run first on the DRBD-secondaries:

root@lxmds20:~# yum --security update
root@lxmds20:~# yum  update --skip-broken
root@lxmds20:~# drbdadm down mds
root@lxmds20:~# grub2-set-default 0
root@lxmds20:~# shutdown -r now

• On lxmds18, the DRBD packages are then created as described above/in the Wiki, from drbd-8.4.11-1.tar.gz. The packages are placed on the mirror, same location (http://mirror.gsi.de/centos/drbd-repository/8.9/el7)
• Download new e2fsprogs and patchless-ldiskfs-server from lustre.org and put these on the mirror, too. The URLs read

baseurl = http://mirror.gsi.de/centos/lustre-repository/e2fsprogs/1.44.3.wc1/el7
baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.5/el7.5.1804/patchless-ldiskfs-server

• Get DRBD modules

root@lxmds20:~# yum install drbd-km-3.10.0_862.14.4.el7.x86_64.x86_64

• Install e2fsprogs and Lustre

root@lxmds20:~# yum install e2fsprogs-1.44.3.wc1-0.el7.x86_64
root@lxmds20:~# yum install kmod-lustre-osd-ldiskfs-2.10.5-1.el7.x86_64 lustre-osd-ldiskfs-mount-2.10.5-1.el7.x86_64 lustre-2.10.5-1.el7.x86_64 kmod-lustre-2.10.5-1.el7.x86_64

On some machines, yum stubbornly refuses to see the e2fsprogs package (the -devel packages of the same version are visible). In that case use scp and yum localinstall

The switch from lxmds19 to lxmds20 means umounting the MDS on lxmds19. This takes a long time (feeling), actually 5 min:

Nov 08 09:42:24 lxmds19.gsi.de kernel: Lustre: Failing over hebe-MDT0000

…

Nov 08 09:47:13 lxmds19.gsi.de kernel: Lustre: server umount hebe-MDT0000 complete

Mounting the MDS on lxmds20 and then lxmds22 works, reconnects to the new IPs likewise.

lxfstest03 has already most of the development tools necessary for DKMS packages.

• The manual asks for installation of epel-release, kernel-devel and zfs.
• The latter does not work. Instead, first remove the kmod-packages:

yum remove kmod-zfs kmod-spl

• Then try to do the dkms packages: yum install zfs-dkms.
  • The upstream code example for the yum repository for zfs-dkms contains the line baseurl=http://download.zfsonlinux.org/epel/7/$basearch/ which puts zfs 0.6.5.9 on the system - slightly too old …
  • Update the repo line to read baseurl=http://download.zfsonlinux.org/epel/7.6/$basearch/
  • Run yum remove zfs-dkms-0.6.5.9-1.el7.centos.noarch spl-dkms-0.6.5.9-1.el7.centos.noarch
• Now, yum install zfs does the trick. The module is created, the old zpool can be imported.

• Cern-Repo aktivieren: in /etc/yum.repos.d/CentOS-Base.repo einfuegen

baseurl=http://linuxsoft.cern.ch/cern/centos/$releasever/os/$basearch/

• Im /etc/yum.repos.d/zfs-server.repo auf Verzeichnis 2.10.6 wechseln

baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.6/zfs-0.7.9/zfs-server

Das Cern-Repo wird gebraucht, weil die 2.10.6-Pakete nach kernel >= 3.10.0-957 fragen - das gibt es nur dort.

• Installiert wird dann eher ein noch neuerer Kernel, aber die Module aus den 2.10.6-er Paketen laufen dann auch mit kernel-3.10.0-862.14.4

On lxfs387-400, the Mellanox firmware does not speak iPXE. Installation of CentOS via PXE does not work either (network timeout). Hence installation with the DVD.

• The CentOS mirror holds installation media: 7.5.1804/isos/x86_64
  • The smaller image CentOS-7-x86_64-Minimal-1804.iso does not offer IB support
  • CentOS-7-x86_64-DVD-1804.iso works
• The image has to be plugged into the KVM window of IPMIView
• In the software selection part, in the second column, check IB support and infrastructure server
• The installation does not allow to pick specific packages and it does not configure the IB interface

After booting into the installed system, at least the IB interface should be up (ibstat should exist).

• Configure /etc/sysconfig/network-scripts/ifcfg-ib0 according to examples from other servers.
  • In particular set BOOTPROTO=dhcp -> BOOTPROTO=none, ONBOOT=no -> ONBOOT=yes and add IPADDR, NETMASK, GATEWAY and DNS1
• Copy /etc/environment from some other server
• Install emacs and joe

Even though the appropriate chef-package is in gsi-reposotory, it is not installable directly.

• Copy chef-13.7.16-1.el7.x86_64.rpm to the server, install
• mkdir /etc/chef
• Copy client.rb and validation.pem from lxmgmt5:/srv/http/centos/chefstart
• Run chef-client (stops with errors on the first run)
• Install zfs: yum install zfs-0.7.9-1.el7_5.x86_64 spl-0.7.9-1.el7_5.x86_64
• Run Chef-client (should succeed now)

The OSS role is role[lustre_hebe_oss]

• modprobe -v dm_multipath
• systemctl restart multipathd
• update-sas-oss-disk-map
• udevadm trigger --action=add
• udevadm settle
• Check /dev/disk/by-location/

8388608*8 = 67108864 540757909504*0.75 = 405568432128 131072*8 1048576

cd /sys/module/zfs/parameters/
cat zfs_vdev_scheduler
echo  -n deadline > zfs_vdev_scheduler
echo 1 > metaslab_debug_unload
echo 16 > zfs_vdev_async_read_max_active
echo 16 > zfs_vdev_sync_read_max_active
echo 16 > zfs_vdev_sync_read_min_active
cat zfetch_max_distance
free  -b
arc_summary.py 
echo 405568432128 > zfs_arc_max 
echo 54075790950 > zfs_dirty_data_max
echo 67108864 > zfetch_max_distance
cat zfs_vdev_async_read_max_active
cat zfs_vdev_scheduler
cat /sys/module/zfs/parameters/zfs_vdev_scrub_max_active
cat /sys/module/zfs/parameters/zfs_vdev_sync_read_max_active 
cat /sys/module/zfs/parameters/zfs_vdev_sync_read_max_active 
cat zfs_vdev_sync_read_max_active
cat zfs_vdev_sync_read_min_active
cat zfs_vdev_async_read_min_active 
cat zfs_vdev_async_read_max_active 
echo 20 > zfs_vdev_async_write_active_min_dirty_percent
echo 5 > zfs_vdev_async_write_min_active
cat zfs_prefetch_disable
cat zfs_arc_max
free -g
cat /proc/meminfo 
grep MemTotal /proc/meminfo 
cat zfs_dirty_data_max
cat zfs_arc_max
free -g
free -b
zpool status
zfs get recordsize
zfs set recordsize=1M osspool6
zfs get recordsize
zfs set recordsize=1M osspool5
zfs get recordsize
zfs set recordsize=1M osspool4
zfs set recordsize=1M osspool3
zfs set recordsize=1M osspool2
zfs set recordsize=1M osspool1
zfs set recordsize=1M osspool0
zfs get recordsize
echo 1048576 > zfs_vdev_aggregation_limit
cd /sys/module/zfs/parameters/
cat zfs.txg.timeout
ls *write*
man 5 spl-module-parameters
#
cd /proc/fs/lustre/
cd ost
ls
cd OSS
ls
cd ost
ls
cat threads_max 
cat threads_started 
cd ..
cd ost_create/
ls
cat threads_max 
cat /sys/fs/lustre/version 
cat /proc/spl/kstat/zfs/arcstats 
cat /proc/spl/kstat/zfs/arcstats | grep size
cat /proc/spl/kstat/zfs/arcstats | grep c_max
zfs get compressratio

Upgrade of lxfstest03 very tedious because the wrong ZFS version (0.7.12) was installed.

• Closer inspection of the Whamcloud repos show that Lustre 2.10.8 still goes with zfs 0.7.9.
• yum update did update many packages but not change to CentOS 7.6 or similar -?
• mirror.gsi.de gets a copy of Whamclouds rpm repository: use of upstream directly does not work because of missing filelist.xml in the repodata directory -WTF?
  • This copy contains only the 2.10.8/el7/server and 2.10.8/el7/patchless-ldiskfs-server parts.
• Putting these repos and the repo for zfs 0.7.12 into /etc/yum.repos.d/ makes yum update try to upgrade ZFS and Lustre as well, but it fails with the ksyms
• The attempt to install or build the Lustre packages fails completely because at first, the zfs 0.7.12 packages for CentOS 7.5 were installed: zfs-0.7.12-1.el7_5.x86_64 - no good
• At the end, the ZFS and Lustre packages are removed at first.
• Then the zfs 0.7.12 packages are installed:

yum install zfs-0.7.12-1.el7_6.x86_64 spl-0.7.12-1.el7_6.x86_64 kmod-zfs-0.7.12-1.el7_6.x86_64 kmod-spl-0.7.12-1.el7_6.x86_64 libzpool2-0.7.12-1.el7_6.x86_64 libzfs2-0.7.12-1.el7_6.x86_64  libnvpair1-0.7.12-1.el7_6.x86_64

• The Lustre source is put into /root, lustre-2.10.8-1.src.rpm, and the devel tools are installed: yum install kmod-zfs-devel-0.7.12.1.el7_6 libzfs2-devel-0.7.12.1.el7_6 rpm-build libselinux-devel libtool

rpmbuild --rebuild --with zfs --without ldiskfs lustre-2.10.8-1.src.rpm

fails because of LU-11507, lustre/osd-zfs/osd_internal.h:929:2: error: implicit declaration of function 'refcount_add' This is due to a change in ZFS, https://github.com/openzfs/zfs/pull/7932 The previous attempt at patching was to introduce the following in osd_internal.h:

#ifdef HAVE_ZFS_REFCOUNT_ADD
#define ZFS_REFCOUNT_ADD        zfs_refcount_add
#else
#define ZFS_REFCOUNT_ADD        refcount_add
#endif

/*                                                                                                                                                                                                                                                                                       
 * Helpers.                                                                                                                                                                                                                                                                              
 */

and replace the occurences of refcound_add by ZFS_REFCOUNT_ADD. This time, this leads to more errors about implicit definitions of the function. -> Instead replace refcound_add by zfs_refcount_add wherever applicable.

• Staring afresh with rpmbuild ... does not work, because it cleans the working directory first. -> Either patch the source properly or continue the build after the error message.
• Run make -j40 -s and then make rpms, puts the rpms directly into /root/rpmbuild/BUILD/lustre-2.10.8. Proper make-command to continue with rpmbuild is unknown.
• Installation of a mixture of upstream lustre 2.10.8 and the self compiled ones fails. -> Install only the local packages.
• Mounting the OSTs results in error messages on the MDS: = client hebetest-MDT0000-lwp-OST000e_UUID does not support ibits lock, either very old or an invalid client=
  • However, recovery completes and the OST seems to be writable.

mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/spl/spl mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/spl/splat mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/avl mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/icp mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/nvpair mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/unicode mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/zcommon mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/zfs mkdir -p /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/zpios

ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/spl/spl/spl.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/spl/spl/spl.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/spl/splat/splat.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/spl/splat/splat.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/zfs/avl/zavl.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/avl/zavl.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/zfs/icp/icp.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/icp/icp.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/zfs/nvpair/znvpair.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/nvpair/znvpair.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/zfs/unicode/zunicode.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/unicode/zunicode.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/zfs/zcommon/zcommon.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/zcommon/zcommon.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/zfs/zfs/zfs.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/zfs/zfs.ko ln -s /lib/modules/3.10.0-862.2.3.el7.x86₆₄/extra/zfs/zpios/zpios.ko /lib/modules/3.10.0-957.1.3.el7.x86₆₄/weak-updates/zfs/zpios/zpios.ko

• Run yum update on lxfstest02. /etc/redhad-version remains at 7.5
• Putting explicit version number 7.6.1810 for the repository does not help:

cat /etc/yum.repos.d/CentOS-Base.repo

...
[base]
name=CentOS-$releasever - Base
baseurl=http://mirror.gsi.de/centos/7.6.1810/os/$basearch/
...

Upgrade lxfstest01:

• First remove all etc/yum.repos.d files except CentOS-Base.repo
  • In CentOS-Base.repo, both [base] and [updates] are enabled
  • Unconditional yum update seems to upgrade to CentOS 7.7: kernel 3.10.0-1127 gets installed (CentOS 7.7 started with kernel 3.10.0-1062).
• Lustre 2.10.8 does not work with CentOS 7.7!
  • Mentioned on the mailing list.
  • Trying to install lustre-dkms results in error message about missing header -> incompatible change in kernel, works only with Lustre 2.12 ff.
• To overcome, change CentOS-Base.repo to refer only to mirror.gsi.de/centos/7.6.1810/.
• Then install kernel 3.10.0-957.el7 (plus kernel-headers-3.10.0-957.el7.x86₆₄, kernel-devel-3.10.0-957.el7.x86₆₄).
• Remove and install the Lustre dkms packages (yum install -y lustre lustre-zfs-dkms lustre-osd-zfs-mount).

• Remove /etc/yum.repos.d/*
• Put in CentOS-Base.repo with baseurl=http://mirror.gsi.de/centos/$releasever/os/$basearch/ and baseurl=http://mirror.gsi.de/centos/$releasever/updates/$basearch/: yum update does not do anything
• Switch to baseurl=http://mirror.gsi.de/centos/7.6.1810/[os,updates]/$basearch/: yum wants to update, stumbles over leftover library /usr/lib/libibmad.so.5(.6.0) from installation of MOFED: removed manually
• Put in lustre-server.repo with baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.8/el7/patchless-ldiskfs-server
• yum update also updates the Lustre modules, they are then found in

root@lxmds17:/lib/modules# ls -l ./3.10.0-957.10.1.el7.x86_64/weak-updates/lustre/fs/lustre.ko
lrwxrwxrwx. 1 root root 64 May 26 12:18 ./3.10.0-957.10.1.el7.x86_64/weak-updates/lustre/fs/lustre.ko -> /lib/modules/3.10.0-957.1.3.el7.x86_64/extra/lustre/fs/lustre.ko

root@lxmds17:/lib/modules# modinfo /lib/modules/3.10.0-957.1.3.el7.x86_64/extra/lustre/fs/lustre.ko
filename:       /lib/modules/3.10.0-957.1.3.el7.x86_64/extra/lustre/fs/lustre.ko
license:        GPL
version:        2.10.8
description:    Lustre Client File System
author:         OpenSFS, Inc. <http://www.lustre.org/>
retpoline:      Y
rhelversion:    7.6
srcversion:     E459483EA54C83D0585ECA3
depends:        obdclass,ptlrpc,libcfs,lnet,lmv,mdc,lov
vermagic:       3.10.0-957.1.3.el7.x86_64 SMP mod_unload modversions 
root@lxmds17:/lib/modules# modinfo drbd
filename:       /lib/modules/3.10.0-957.el7_lustre.x86_64/updates/drbd.ko
alias:          block-major-147-*
license:        GPL
version:        8.4.11-1
description:    drbd - Distributed Replicated Block Device v8.4.11-1
author:         Philipp Reisner <phil@linbit.com>, Lars Ellenberg <lars@linbit.com>
retpoline:      Y
rhelversion:    7.6
srcversion:     85CDEE2271F6326BF82B3F0
depends:        libcrc32c
vermagic:       3.10.0-957.el7_lustre.x86_64 SMP mod_unload modversions 
parm:           enable_faults:int
parm:           fault_rate:int
parm:           fault_count:int
parm:           fault_devs:int
parm:           allow_oos:DONT USE! (bool)
parm:           disable_sendpage:bool
parm:           proc_details:int
parm:           minor_count:Approximate number of drbd devices (1-255) (uint)
parm:           usermode_helper:string

• umount /srv/mds leads to reboot, but machine comes up with kernel 3.10.0-957.10.1.el7
• The Lustre modules are found, the old DRBD modules cannot be reused
• DRBD has no younger version, still at 8.4.11-1. Creating the module works as before:

cd /root/drbd-8.4.11-1                                                                                                                                                                                                                              
drbd-8.4.11-1]# make km-rpm                                                                                                                                                                                                                   

• DRBD module is loaded, DRBD resource can be started, Lustre can be mounted
  • Due to missing lxmds18, the recovery is stuck

yum clean all
rm -rf /var/cache/yum
ainsl /etc/yum.conf http_caching=none
yum makecache
yum check-update
yum update

As of [2020-06-03 Wed], this will upgrade to CentOS 7.8, which is probably no good for Lustre 2.12.

Look for the package centos-release

yum info --show-duplicates centos-release

This should show centos-release-7-5.1804.[1|4|5].el7.centos.x86_64 as the installed one.

The release package has the following content:

[root@lxfs323 ~]# rpm -qpil centos-release-7-5.1804.4.el7.centos.x86_64.rpm 
warning: centos-release-7-5.1804.4.el7.centos.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Name        : centos-release
Version     : 7
Release     : 5.1804.4.el7.centos
Architecture: x86_64
Install Date: (not installed)
Group       : System Environment/Base
Size        : 40338
License     : GPLv2
Signature   : RSA/SHA256, Mon 13 Aug 2018 05:10:03 PM CEST, Key ID 24c6a8a7f4a80eb5
Source RPM  : centos-release-7-5.1804.4.el7.centos.src.rpm
Build Date  : Mon 13 Aug 2018 04:34:36 PM CEST
Build Host  : x86-01.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
Summary     : CentOS Linux release file
Description :
CentOS Linux release files
/etc/centos-release
/etc/centos-release-upstream
/etc/issue
/etc/issue.net
/etc/os-release
/etc/pki/rpm-gpg
/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Debug-7
/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Testing-7
/etc/redhat-release
/etc/rpm/macros.dist
/etc/system-release
/etc/system-release-cpe
/etc/yum.repos.d/CentOS-Base.repo
/etc/yum.repos.d/CentOS-CR.repo
/etc/yum.repos.d/CentOS-Debuginfo.repo
/etc/yum.repos.d/CentOS-Media.repo
/etc/yum.repos.d/CentOS-Sources.repo
/etc/yum.repos.d/CentOS-Vault.repo
/etc/yum.repos.d/CentOS-fasttrack.repo
/etc/yum/vars/infra
/usr/lib/systemd/system-preset/85-display-manager.preset
/usr/lib/systemd/system-preset/90-default.preset
/usr/share/centos-release/EULA
/usr/share/doc/centos-release/Contributors
/usr/share/doc/centos-release/GPL
/usr/share/doc/redhat-release
/usr/share/redhat-release

Twiddling the -release files to read something else but 7-5.1804 might do the trick, but it is obviously better to use the release package.

• While yum info only lists centos-release-7-5 as installed, it claims that other versions are already installed, e.g. when doing yum install centos-release-7-7.1908.0.el7.centos.x86₆₄

Manual install:

yum install yum-utils
yumdownloader centos-release-7-7.1908.0.el7.centos.x86_64
rpm -iv --replacefiles --force ./centos-release-7-7.1908.0.el7.centos.x86_64.rpm

Now upgrade:

yum update

After reboot, /etc/centos-release reads CentOS Linux release 7.7.1908 (Core)

The Lustre 2.12.4 branch on Whamcloud has subdirs for Centos 7.7 (and 8.1). This suggests upgrading to 7.7, by installing centos-release-7-7.1908.0.el7.centos.x86_64.rpm. This works, yum update installs the corresponding packages, kernel 3.10.0-1062 boots. However, adding the ZOL repo by yum install http://download.zfsonlinux.org/epel/zfs-release.el7_7.noarch.rpm and installing yum install zfs would put zfs 0.8x into the system.

• The Lustre repos contain the DKMS versions, so installing them would put zfs 0.7.13 together with Lustre 2.12.4

Instead, install the ZOL for 7.6:

[root@lxfs320 ~]# yum install http://download.zfsonlinux.org/epel/zfs-release.el7_6.noarch.rpm

• Make sure kmod is enabled, dkms is disabled.

[root@lxfs320 ~]# yum install zfs

[root@lxfs320 ~]# yum remove zfs zfs-kmod spl spl-kmod libzfs2 libnvpair1 libuutil1 libzpool2 zfs-release
[root@lxfs320 ~]# yum install http://download.zfsonlinux.org/epel/zfs-release.el7_6.noarch.rpm
[root@lxfs320 ~]# yum autoremove
[root@lxfs320 ~]# yum clean metadata
[root@lxfs320 ~]# yum install zfs

After rebooting in the new kernel, the zpools are available.

• Copy the Lustre packages for 2.12.4, install

[root@lxfs320 ~]# yum install ./kmod-lustre-2.12.4-1.el7.x86_64.rpm ./kmod-lustre-osd-zfs-2.12.4-1.el7.x86_64.rpm ./lustre-2.12.4-1.el7.x86_64.rpm ./lustre-osd-zfs-mount-2.12.4-1.el7.x86_64.rpm

Same procedure for the system upgrade: get the original CentOS-Base.repo, get the release package, install locally with overrides, restore the previous yum.repos.d-files, install the kernel.

yum update fails with

Transaction check error:
  file /usr/lib64/libibmad.so.5 from install of infiniband-diags-2.1.0-1.el7.x86_64 conflicts with file from package libibmad-5.0.0.MLNX20181022.0361c15-0.1.45101.x86_64

This libibmad comes from the previous installation with MOFED.

• Remove infiniband-diags
• Do yum update
• Update lustre-server.repo to the 2.12.4 location
• Install the Lustre packages:

yum install kmod-lustre.x86_64 kmod-lustre-osd-ldiskfs.x86_64 lustre.x86_64 lustre-osd-ldiskfs-mount.x86_64

• Install infiniband-diags
• Reboot: the booted system has no network: some modules were not loaded.
  • Create a file /etc/modules-load.d/ib.conf, containing the usual suspects. (effect not yet tested)

Unfortunately, the start version is not exactly know. Install lxfs322/323/324/330 with CentOS 7.4, ZFS 0.7.4-2 and Lustre 2.10.2. Filesystemname h2test

• User-IDs / LDAP connection does not work on these MDSes (lxfs322+323)
• Copy the entire /u/troth directory to h2test: /h2test/hpc/troth on mds0 with some subdirectories on mds1
  • Copy of that as /h2test/test on mds1

• CentOS remains at 7.5
• ZFS remains at 0.7.9
• zfs-server.repo now with URL baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.10.6/zfs-0.7.9/zfs-server

yum install kmod-lustre-2.10.6-1.el7.cern.x86_64 kmod-lustre-osd-zfs-2.10.6-1.el7.cern.x86_64 lustre-2.10.6-1.el7.cern.x86_64 lustre-osd-zfs-mount-2.10.6-1.el7.cern.x86_64

Reboot now - important, otherwise the modules don't load.

• Removing selinux can be quite desatrous:

lxfs322: yum remove selinux-policy-targeted

• At least the config in /etc/selinux/conf has to be set to "No", otherwise login, network and of course Lustre won't work (D-Bus errors)
• Reinstalling all kinds of selinux-packages seems to help a little.
• The output of tunefs.lustre --dryrun /dev/drbd0 contains

Persistent mount opts: user_xattr,errors=remount-ro,context="unconfined_u:object_r:user_tmp_t:s0"

After messing with selinux, the part

context="unconfined_u:object_r:user_tmp_t:s0

is no longer understood! Running

tunefs.lustre --mountfsoptions="user_xattr,errors=remount-ro" /dev/drbd0

helps.

Procedure could be avoided on lxfs323 by not removing selinux.

If a kernel module will not load, although it has been compiled with DKMS specirically for the running kernel, it might be due to remnants of older modules in some weak-updates subdir under /lib/modules/

depmod -a

might give a hint: if there is a messages as ERROR: fstatat(4, ksocklnd.ko.xz): No such file or directory Removing dead links and old stuff might make the real modules loadable.

lxfstest01:

baseurl=http://linuxsoft.cern.ch/cern/centos/$releasever/os/$basearch/

yum clean all                                                             
ainsl /etc/yum.conf http_caching=none                                                    
yumdownloader centos-release-7-8.2003.0.el7.centos
rpm -iv --replacefiles --force ./centos-release-7-8.2003.0.el7.centos.x86_64.rpm
yum makecache          
yum check-update                                                                           
yum --security update # 19m29.124s  Runs DKMS
yum update            # 0m29.792s                 

[root@lxfstest03 ~]# yum install kernel-3.10.0-1127.8.2.el7.x86_64  # 6 min

[root@lxfstest01 ~]# time rpm --rebuilddb && real 0m59.404s

[root@lxfstest01 ~]# time yum remove -y zfs zfs-kmod spl spl-kmod libzfs2 libnvpair1 libuutil1 libzpool2 zfs-release  # real 8m15.083s

[root@lxfstest01 ~]# time yum install -y zfs spl zfs-dkms spl-dkms libnvpair1 libzpool2 libzfs2  # real 12m4.744s

[root@lxfstest01 ~]# time yum install -y lustre  lustre-osd-zfs-mount lustre-zfs-dkms  # real 10m20.240s

[root@lxfstest03 ~]# time yum install -y zfs spl zfs-dkms spl-dkms libnvpair1 libzpool2 libzfs2  # real 0m9.209s

[root@lxfstest03 ~]# time yum reinstall -y lustre  lustre-osd-zfs-mount lustre-zfs-dkms real 6m45.206s

[root@lxfstest02 ~]# time yum install -y zfs spl zfs-dkms spl-dkms libnvpair1 libzpool2 libzfs2  # real 8m2.602s
[root@lxfstest02 yum.repos.d]# time yum install -y lustre  lustre-osd-zfs-mount lustre-zfs-dkms  # real 7m53.372s

root@lxmds17:~# yum clean all
root@lxmds17:~# rm -rf /var/cache/yum
root@lxmds17:~# ainsl /etc/yum.conf http_caching=none
root@lxmds17:~# yum check-update

root@lxmds17:~# time yum --security -y update  # real  30m16.032s
root@lxmds18:~# time yum --security -y update  # real  13m48.127s

root@lxmds17:~# yumdownloader centos-release-7-8.2003.0.el7.centos.x86_64                                                                                                                                                                   
root@lxmds17:~# rpm -iv --replacefiles --force ./centos-release-7-8.2003.0.el7.centos.x86_64.rpm                                                                                                                                            
root@lxmds17:~# time yum update -y    # real  2m3.565s
root@lxmds18:~# time yum update -y    # real 2m2.759s

root@lxmds18:~# cat /usr/lib/os-release                                                                                                                                                                                                     
root@lxmds18:~# cat /etc/modules-load.d/ib.conf                                                                                                                                                                                             
root@lxmds18:~# cat  /etc/sysconfig/network-scripts/ifcfg-ib0                                                                                                                                                                               
root@lxmds18:~# select-grub2-kernel.rb 0 
root@lxmds18:~# shutdown -r now                                                                                                                                                                                                             


root@lxmds17:~# yum install drbd-km-3.10.0_1127.13.1.el7.x86_64-8.4.11-1.x86_64                                                                                                                                                             
root@lxmds17:~# yum install -y e2fsprogs-1.45.6.wc1-0.el7.x86_64                                                                                                                                                                            
root@lxmds17:~# time yum install -y kmod-lustre kmod-lustre-osd-ldiskfs lustre lustre-osd-ldiskfs-mount  # real  9m3.183s
root@lxmds18:~# time yum install -y kmod-lustre kmod-lustre-osd-ldiskfs lustre lustre-osd-ldiskfs-mount  # real 11m4.864s

If DRBD is configured on a device name that might change, blkid can help:

[root@lxfs318 ~]# blkid  | grep drbd
/dev/sdcw: UUID="7cb98ee4012c3bcd" TYPE="drbd" 
/dev/sdek: UUID="7cb98ee4012c3bcd" TYPE="drbd" 
/dev/mapper/mpathct: UUID="7cb98ee4012c3bcd" TYPE="drbd" 

-> Change the device name in /etc/drbd.d/res to the correct device name.

• Load multipath module modprobe -v dm_multipath and start it systemctl start multipathd
• Run the usual three steps

update-sas-oss-disk-map
udevadm trigger --action=add
udevadm settle

• Check /dev/disk/by-location
• Enable gsi-mirror

emacs -nw /etc/yum.repos.d/CentOS-Base.repo
baseurl=http://mirror.gsi.de/centos/$releasever/os/$basearch/

yum clean all
ainsl /etc/yum.conf http_caching=none
yum makecache

• The system gets installed with kernel 3.10.0-1127.el7.x86_64
  • On the gsi-mirror, the fitting headers and devels are found:

yum install kernel-headers-3.10.0-1127.el7.x86_64 kernel-devel-3.10.0-1127.el7.x86_64 dkms

• Now install ZFS

yum install zfs spl zfs-dkms spl-dkms libnvpair1 libzpool2 libzfs2

• Create zpools with format_ost_hebe.sh.

• The system disk is not the usual hardware raid-1 on all servers.
• Comment out all explicit occurences of /dev/sda in the kickstart file (lxmgmt5:/srv/http/centos/kickstart/ibcentos78.cfg)
  • The installation stops and asks for a target device (option 5). Among the many similar devices proposed, the system disk can be found with some patience.
• After installation of CentOS, remove the chef client and node objects, run chef-client on the server, add role[hebe_testoss] to its run_list, run chef-client twice.

• Have etc/yum.repos.d/CentOS-Base.repo with the lines =baseurl=http://mirror.gsi.de/centos/$releasever…= for base and updates

yum install kernel-3.10.0-1127.19.1.el7.x86_64
yum install gsi-scripts-common ruby
select-grub2-kernel.rb

• Reboot

= For chef, create the sssd dir

mkdir /etc/sssd
chef-client

• Get Lustre

yum install kmod-lustre kmod-lustre-osd-ldiskfs lustre lustre-osd-ldiskfs-mount

• Get DRBD: files from drbd-utils cannot overwrite files from drbd-utils (OMG!)

yumdownloader drbd-utils
rpm -iv --replacefiles drbd-utils-8.9.11-1.el7.centos.x86_64.rpm
yum install drbd-km-3.10.0_1127.19.1.el7.x86_64.x86_64

Edit /etc/yum.repos.d/Centos-Base.repo to read

baseurl=http://mirror.gsi.de/centos/7.9.2009/os/$basearch/
enabled=1

and

baseurl=http://mirror.gsi.de/centos/7.9.2009/updates/$basearch/
enabled=0

yum search --show-duplicates centos-release should now show a package for CentOS 7.9

yum install centos-release-7-9.2009.1.el7.centos.x86_64

This updates /etc/centos-release.

Update chef-role hebe_test_oss: it writes only /etc/yum.repos.d/zfs-server.repo. Insert

baseurl = http://mirror.gsi.de/centos/lustre-repository/lustre-2.12.7/el7.9.2009/patchless-ldiskfs-server

Run chef-client

Run yum makecache, yum check-update, yum --security update (doesn't do anything). The upgrade itself takes quite a while:

[lxfstest01 ~] time yum update   # 35 min
[lxfstest02 ~] time yum update   # 27 min
[lxfstest03 ~] time yum update   # 22 min
[lxmds18 ~] time yum update      #  7 min

Edit /etc/yum.repos.d/Centos-Base.repo to read

baseurl=http://mirror.gsi.de/centos/7.9.2009/updates/$basearch/
enabled=1

Now, newer kernel become visible (yum search --show-duplicates kernel). Install 1160.25.1, which is used by the patched Lustre 2.12.7 version.

yum install kernel-3.10.0-1160.25.1.el7.x86_64

yum complains that the headers for this kernel cannot be found: no dkms -> Install

yum install kernel-headers-3.10.0-1160.25.1.el7.x86_64 kernel-devel-3.10.0-1160.25.1.el7.x86_64

This does not trigger any dkms actions. -> Reinstall the kernel

yum reinstall kernel-3.10.0-1160.25.1.el7.x86_64

Takes forever and results in an attempt to compile the Lustre modules, with

depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/osd_zfs.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/lfsck.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/ptlrpc.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/mdt.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/lod.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/lov.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/obdecho.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/lustre.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/obdclass.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/mdc.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/osp.ko.xz needs unknown symbol libcfs_vfree_atomic
depmod: WARNING: /lib/modules/3.10.0-1127.el7.x86_64/weak-updates/lmv.ko.xz needs unknown symbol libcfs_vfree_atomic

modinfo lustre shows already version 2.12.7 - dkms compilation has pulled in the Lustre 2.12.7 packages and compiled it.

• The warnings above refer to the running kernel 1127, which belongs to the CentOS 7.8 installation.
• Perhaps the compiled modules will work with the new kernel?

Reboot.

After booting kernel 3.10.0-1160.25.1, lnet and lustre modules can be loaded, and the OST can be mounted.

Die fertige Installation bootet in den Installationskernel 3.10.0-1127.el7. Waehrend der Installation sind auch schon die Pakete mft und kernel-mft-4.21.0-3.10.0_1127.el7 installiert worden, um am Ende der Installation set KEEP_LINK_UP_ON_BOOT_P1=0 zu setzen - das scheint nicht immer zu funktionieren.

Nachdem der Zielkernel 3.10.0-1127.19.1.el7.x86_64 installiert und gebootet ist, sollte das Modul fuer den Installationskernel eventuell nicht mehr passen.

• Tatsaechlich versucht `mst start`, die Module mst_pci und mst_pciconf zu laden. Das scheitert an den dabei mitgegebenen Parametern.
• Wenn man stattdessen manuell modprobe mst_pci und modprobe mst_pciconf ausfuehrt, werden beide Module geladen, und danach funktioniert auch mst start.
• Auch ohne die PCI-Module lassen sich die Mellanox-Tools benutzen!
  • Wenn mst start wie beschrieben fehlschlaegt, gibt mst status als Device die PCI-Adresse aus, e.g. af:00.0. Diese kann man aber auch an mlxconfig geben: mlxconfig -d af:00.0 q liefert den erwarteten Output.

Die Server bekommen die Rolle lustre_hebe_oss. Diese aendert u.a. die /etc/yum.repos.d/CentOS-Base.repo so, dass der GSI-Mirror genutzt wird.

• Dort findet sich der Kernel, der bisher von den Hebe-OSS verwendet wird.
• > =yum install kernel-3.10.0-1127.19.1.el7.x86_64, Reboot