DPM upgrade at Oxford
Contents
Objective
Upgrade and migrate DPM Head node from one SL6 physical server to another Centos7 server keeping the same hostname (t2se01 in our case)
Preparation
After the start of the GOCDB downtime, stopped all DPM related services on head node and pool nodes. Used netstat to confirm that there is no live connection to any pool node.
Run the final backup of mysql database on the old headnode
/usr/bin/mysqldump --user=$mysql_user --password=$mysql_pw --opt --all-databases --single-transaction | gzip -c > mysql-dump-final.sql.gz
Changed the name of the old headnode from t2se01 to t2se01tmp in DNS and DHCP server
New DPM head node
Installed the new machine as t2se01.
Setup mysql (MariaDB) using local puppet
cat mysql.pp $mysql_root_pass = "****" $db_user = "dpmmgr" $db_pass = "****" $db_host = "localhost" $override_options = { 'mysqld' => { 'max_connections' => '1000', 'query_cache_size' => '256M', 'query_cache_limit' => '1MB', 'innodb_flush_method' => 'O_DIRECT', 'innodb_buffer_pool_size' => '1000000000', } } class{"mysql::server": service_enabled => true, root_password => "${mysql_root_pass}", override_options => $override_options }
Run puppet locally to setup mysql
puppet apply mysql.pp
Load old mysql backup database to new db
mysql –u root –p < mysql-dump-final.sql
It is required to update tables as mysql version has changed
mysql_upgrade --upgrade-system-tables
It is worth logging into DB and compare some data from old one
Install dpm puppet modules from puppet forge on the DPM head node
puppet module install lcgdm-gridftp puppet module install lcgdm-xrootd puppet module install lcgdm-voms puppet module install puppetlabs-mysql --version=3.4 puppet module install lcgdm-dmlite puppet module install lcgdm-lcgdm puppet module install CERNOps-bdii puppet module install saz-memcached puppet module install erwbgy-limits puppet module install puppet-fetchcrl
Run puppet with final dpm manifests. Our dpm manifest is here https://gist.github.com/kashif74/499fc3678b82fbcab2229b8bbec9188e
puppet apply dpm.pp
Ideally the above command should setup DPM Head node
I have also updated DPM pool nodes rpm packages to latest dpm realease but not the OS. dpm manifest for pool node is here https://gist.github.com/kashif74/3b1a941933bc3688c896fc965859c011
Post Configuration
DNS requests
Our IT services complained that DPM Head node is making too many DNS request. Added all pool nodes in /etc/hosts file of headnode and headnode in /etc/hosts file of poolnodes
Installed nscd for local dns caching
xrootd
latest version of xrootd > 4.8.0. is not in UMD4 so it need to be pulled from EPEL (Feb 2018) . This might resolve in future.
Atlas and CMS name2name library
For CMS : https://twiki.cern.ch/twiki/bin/view/DPM/DPMComponents_Dpm-Xrootd#CMS_federation
For ATLAS : https://twiki.cern.ch/twiki/bin/view/DPM/DPMComponents_Dpm-Xrootd#ATLAS_federation. Atlas rpm is not available from Centos7 repo so install it from SL6 repo.