Hey Hey,
So it was that time again when I need to upgrade the lab environment. This environment is a Prod environment to me, it’s where I develop, Demo to customers and Demo at conferences like VMworld, it is a massive part of my job. So naturally I get a little worried. Never had a huge issue but it’s always back of mind.
Everything was going fine until I hit a snag with the upgrade of vRA. it was failing with the error:
+ echo '2018-04-16 13:14:19+10:00 /etc/bootstrap/postupdate.d/15-vcac-prepare-required-services done, succeeded.'
+ for script in '"${bootstrap_dir}"/*'
+ echo
+ '[' '!' -e /etc/bootstrap/postupdate.d/20-vcac ']'
+ '[' '!' -x /etc/bootstrap/postupdate.d/20-vcac ']'
+ log '/etc/bootstrap/postupdate.d/20-vcac starting...'
++ date '+%Y-%m-%d %H:%M:%S%:z'
+ echo '2018-04-16 13:14:19+10:00 /etc/bootstrap/postupdate.d/20-vcac starting...'
+ /etc/bootstrap/postupdate.d/20-vcac 7.3.0.536 7.4.0.645 0
NOTICE: Found extension citext and it exists in pg_catalog
Can not read the Orchestrator pid from /var/lib/vco/app-server/logs/tomcat.pid
+ res=2
+ echo 'Script /etc/bootstrap/postupdate.d/20-vcac failed, error status 2'
+ exit 2
+ rm -f /tmp/postupdate-err-log
+ exit 2
+ trapfunc
+ excode=2
+ test 2 -gt 0
+ vami_update_msg set post-install 'Post-install: failed'
+ test -x /usr/sbin/vami-update-msg
+ /usr/sbin/vami-update-msg set post-install 'Post-install: failed'
+ sleep 1
+ test 2 -gt 0 -o 0 -gt 0
+ vami_update_msg set update-status 'Update failed (code 0-2). Check logs in /opt/vmware/var/log/vami or retry update later.'
+ test -x /usr/sbin/vami-update-msg
+ /usr/sbin/vami-update-msg set update-status 'Update failed (code 0-2). Check logs in /opt/vmware/var/log/vami or retry update later.'
+ exit
16/04/2018 03:15:45 [ERROR] Failed with exit code 512
16/04/2018 03:15:45 [INFO] Update status: Running VMware tools reconfiguration
16/04/2018 03:15:45 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools
vmware-toolbox-cmd is /usr/bin/vmware-toolbox-cmd
Configuring VAMI VMware tools service wrapper.
16/04/2018 03:15:45 [INFO] Update status: Done VMware tools reconfiguration
16/04/2018 03:15:45 [INFO] Update status: Error while running post-install scripts
16/04/2018 03:15:45 [ERROR] Failure: updatecli exiting abnormally
16/04/2018 03:15:45 [INFO] Install Finished
This was not very useful since it is telling me to check the log im currently looking at for more details. There is a file /var/log/bootstrap/postupdate.log which did have more details on the error and was showing me :
Can not read the Orchestrator pid from /var/lib/vco/app-server/logs/tomcat.pid
Start configuring database settings.
Configuring database settings finished successfully.
Start updating database scheme.
ch.dunes.vso.db.check.CheckDuplicatesException:
Duplicate entries found in the Orchestrator database:
Resource element duplicates:
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'fr_FR.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_CN.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'de_DE.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'es.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_Hant.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_MO.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ja_JP.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ja.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'en.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'es_ES.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ko.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ko_KR.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'fr.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'de.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_HK.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_Hans.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_TW.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'en_US.properties'
Resolve duplicates by deleting unnecessary items.
java.lang.RuntimeException: ch.dunes.vso.db.check.CheckDuplicatesException:
Duplicate entries found in the Orchestrator database:
Resource element duplicates:
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'fr_FR.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_CN.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'de_DE.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'es.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_Hant.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_MO.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ja_JP.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ja.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'en.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'es_ES.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ko.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ko_KR.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'fr.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'de.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_HK.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_Hans.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_TW.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'en_US.properties'
Resolve duplicates by deleting unnecessary items.
at com.vmware.o11n.integration.initialization.DatabaseInitializer.initializeSchema(DatabaseInitializer.java:77)
at com.vmware.o11n.configuration.database.install.DatabaseInstallerImpl.installDatabase(DatabaseInstallerImpl.java:30)
at com.vmware.o11n.cli.configuration.commands.db.DbCommand.execute(DbCommand.java:100)
at com.vmware.o11n.cli.configuration.commands.ConfigurationCommand.executeCmd(ConfigurationCommand.java:103)
at com.vmware.o11n.cli.configuration.ConfigurationCli.executeCommand(ConfigurationCli.java:143)
at com.vmware.o11n.cli.configuration.ConfigurationCli.main(ConfigurationCli.java:121)
Caused by: ch.dunes.vso.db.check.CheckDuplicatesException:
Duplicate entries found in the Orchestrator database:
Resource element duplicates:
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'fr_FR.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_CN.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'de_DE.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'es.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_Hant.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_MO.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ja_JP.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ja.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'en.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'es_ES.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ko.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'ko_KR.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'fr.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'de.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_HK.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_Hans.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'zh_TW.properties'
1 item with ID 'ff8080815fe049c3015fe04cf88503ec' and name 'en_US.properties'
Resolve duplicates by deleting unnecessary items.
at ch.dunes.vso.db.check.ValidateDb.checkForDuplicates(ValidateDb.java:126)
at com.vmware.o11n.integration.initialization.DatabaseInitializer.initializeSchema(DatabaseInitializer.java:74)
... 5 more
Error updating the DB.
Script /etc/bootstrap/postupdate.d/20-vcac failed, error status 2
So duplicates in the database was causing the issue. As it turns out this was the horizon vRO plugin causing the issue, under resource elements it has tons of these properties files.
To check the database before upgrade for duplicates you can run this command in postgres on the appliance:
log into postgres
sudo -u postgres psql -U postgres
connect to database
\c vcac
check for duplicates
select * from (
select categoryid, name, count(*)-1, 'vmo_configelement' as type from vmo_configelement group by categoryid, name having count(*) > 1
union
select parentcategoryid, name, count(*)-1, 'vmo_configelementcategory' as type from vmo_configelementcategory group by parentcategoryid, name having count(*) > 1
union
select elementid, '' as name, count(*)-1, 'vmo_configelementcontent' as type from vmo_configelementcontent group by elementid having count(*) > 1
union
select '' as id, name, count(*)-1, 'vmo_package' as type from vmo_package group by name having count(*) > 1
union
select packageid, '' as name, count(*)-1, 'vmo_packagecontent' as type from vmo_packagecontent group by packageid having count(*) > 1
union
select policyid, '' as name, count(*)-1, 'vmo_policycontent' as type from vmo_policycontent group by policyid having count(*) > 1
union
select categoryid, name, count(*)-1, 'vmo_resourceelement' as type from vmo_resourceelement group by categoryid, name having count(*) > 1
union
select elementid, '' as name, count(*)-1, 'vmo_resourceelementcontent' as type from vmo_resourceelementcontent group by elementid having count(*) > 1
union
select scriptmodulecategoryid, name, count(*)-1, 'vmo_scriptmodule' as type from vmo_scriptmodule group by scriptmodulecategoryid, name having count(*) > 1
union
select '' as id, name, count(*)-1, 'vmo_scriptmodulecategory' as type from vmo_scriptmodulecategory group by name having count(*) > 1
union
select scriptmoduleid, '' as name, count(*)-1, 'vmo_scriptmodulecontent' as type from vmo_scriptmodulecontent group by scriptmoduleid having count(*) > 1
union
select dunesobjectid, version, count(*)-1, 'vmo_versionhistorycontent' as type from vmo_versionhistorycontent group by dunesobjectid, version having count(*) > 1
union
select workflowid, '' as name, count(*)-1, 'vmo_workflowcontent' as type from vmo_workflowcontent group by workflowid having count(*) > 1
) as u;
In my case it was pointing to the horizon plugin. I simply removed this plugin and all data including resource elements. I wasn’t using the plugin so I didn’t care. If you are using the plugin, I would not suggest you do this, but to check with support for a workaround.
once removed. the upgrade went through without a hitch.
Cheers