vCenter Server 5.1 and inventory service resourcing issues

Well this past week ive been working on upgrading one of my smaller systems or 2000 VM’s and 60 hosts from vSphere 5.0 to 5.1.

Lets just say ive had a crap load of issues as most may have experienced themselves. Due to a Database backup policy mistake I had to rebuild everything from a clean slate minus the ESXi hosts of course.

The issue I came across was strange, basically every hosts would disconnect the reconnect seconds after, this would mess up DRS moves or anything else in progress. was getting annoying. while playing around with the system I found it would only happen if the vCenter inventory service was running. These are the backtrace issues I would see:

2012-09-29T00:04:54.345+10:00 [04556 error 'Default'] SSLStreamImpl::DoClientHandshake for SSL(TCPClientSocket(this=00000000ae5ef0b0, state=CONNECTED, _connectSocket=TCP(fd=-1), error=(null)) TCPStreamWin32(socket=TCP(fd=66872) local=vcenterIP:3758, peer=hostIP:443)): SSL_connect failed with Unexpected EOF


2012-09-29T00:04:54.361+10:00 [06624 error 'Default'] SSLStreamImpl::DoClientHandshake for SSL(TCPClientSocket(this=00000000ae5fa0f0, state=CONNECTED, _connectSocket=TCP(fd=-1), error=(null)) TCPStreamWin32(socket=TCP(fd=67984) local=0.0.0.0:3757, peer=hostip:443)): SSL_connect failed with Unexpected EOF



2012-09-29T00:04:54.361+10:00 [04556 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:0000000081188820, TCP:host4.local:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: Unexpected EOF)



2012-09-29T00:04:54.361+10:00 [06624 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:000000008ac4bab0, TCP:host13.local:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: Unexpected EOF)



2012-09-29T00:04:54.361+10:00 [11868 warning 'VpxProfiler' opID=HB-host-310@1498093-59042227] [VpxdHostSync] GetChanges host:host1.local (IP Address) [GetChangesTime] took 42281 ms



2012-09-29T00:04:54.361+10:00 [11868 warning 'VpxProfiler' opID=HB-host-310@1498093-59042227] [VpxdHostSync] DoHostSync:000000000A1D5660 [DoHostSyncTime] took 42281 ms



2012-09-29T00:04:54.361+10:00 [11868 warning 'vpxdvpxdInvtHostCnx' opID=HB-host-310@1498093-59042227] [VpxdInvtHostSyncHostLRO] DoHostSync failed for host host-310



2012-09-29T00:04:54.361+10:00 [11868 warning 'vpxdvpxdInvtHostCnx' opID=HB-host-310@1498093-59042227] [VpxdInvtHostSyncHostLRO] Host sync failed to host-310



2012-09-29T00:04:54.361+10:00 [11868 error 'vpxdvpxdInvtHostCnx' opID=HB-host-310@1498093-59042227] [VpxdInvtHostSyncHostLRO] FixNotRespondingHost failed for host host-310, marking host as notResponding



2012-09-29T00:04:54.361+10:00 [07776 error 'Default'] SSLStreamImpl::DoClientHandshake for SSL(TCPClientSocket(this=000000009f5e2a20, state=CONNECTED, _connectSocket=TCP(fd=-1), error=(null)) TCPStreamWin32(socket=TCP(fd=63436) local=vcenterIP:3763, peer=HostIP:443)): SSL_connect failed with Unexpected EOF



2012-09-29T00:04:54.361+10:00 [10592 info 'vpxdvpxdVmomi' opID=HB-host-1772@1212909-1f45a976] [ClientAdapterBase::InvokeOnSoap] Invoke done (host14.local, vpxapi.VpxaService.retrieveChanges)



2012-09-29T00:04:54.361+10:00 [07776 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:0000000089de4520, TCP:host10.local:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: Unexpected EOF)



2012-09-29T00:04:54.361+10:00 [02892 error 'vpxdvpxdVmomi' opID=HB-host-1884@255600-2cfa0bd7] [VpxdClientAdapter] Got vmacore exception: SSL Exception: Unexpected EOF



2012-09-29T00:04:54.361+10:00 [02892 error 'vpxdvpxdVmomi' opID=HB-host-1884@255600-2cfa0bd7] [VpxdClientAdapter] Backtrace:



--> backtrace[00] rip 0000000180100c98



--> backtrace[01] rip 0000000180101fae



--> backtrace[02] rip 000000018008aeab



--> backtrace[03] rip 0000000180004eb4



--> backtrace[04] rip 000000018011fea2



--> backtrace[05] rip 000000018005d1fc



--> backtrace[06] rip 00000001800523dc



--> backtrace[07] rip 000000018011a0e5



--> backtrace[08] rip 00000001801a9668



--> backtrace[09] rip 00000001801a9aed



--> backtrace[10] rip 00000001801ab260



--> backtrace[11] rip 000000018019b62a



--> backtrace[12] rip 0000000078622fdf



--> backtrace[13] rip 0000000078623080



--> backtrace[14] rip 0000000077d6b71a



--> backtrace[15] rip 0000000000000000



-->



2012-09-29T00:04:54.361+10:00 [08720 error 'Default'] SSLStreamImpl::DoClientHandshake for SSL(TCPClientSocket(this=00000000a2a7fa70, state=CONNECTED, _connectSocket=TCP(fd=-1), error=(null)) TCPStreamWin32(socket=TCP(fd=62428) local=vcenterIP:3766, peer=hostip:443)): SSL_connect failed with Unexpected EOF



2012-09-29T00:04:54.361+10:00 [08720 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:000000000997f640, TCP:host16.local:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: Unexpected EOF)



2012-09-29T00:04:54.361+10:00 [07920 error 'vpxdvpxdVmomi' opID=HB-host-1740@75877-52bb6b88] [VpxdClientAdapter] Got vmacore exception: SSL Exception: Unexpected EOF



2012-09-29T00:04:54.361+10:00 [07920 error 'vpxdvpxdVmomi' opID=HB-host-1740@75877-52bb6b88] [VpxdClientAdapter] Backtrace:



--> backtrace[00] rip 0000000180100c98



--> backtrace[01] rip 0000000180101fae



--> backtrace[02] rip 000000018008aeab



--> backtrace[03] rip 0000000180004eb4



--> backtrace[04] rip 000000018011fea2



--> backtrace[05] rip 000000018005d1fc



--> backtrace[06] rip 00000001800523dc



--> backtrace[07] rip 000000018011a0e5



--> backtrace[08] rip 00000001801a9668



--> backtrace[09] rip 00000001801a9aed



--> backtrace[10] rip 00000001801ab260



--> backtrace[11] rip 000000018019b62a



--> backtrace[12] rip 0000000078622fdf



--> backtrace[13] rip 0000000078623080



--> backtrace[14] rip 0000000077d6b71a



--> backtrace[15] rip 0000000000000000



-->

I would notice that when this happened the inventory service was at 100% cpu usage, This vCenter server was a VM with 4vCPUs and 16GB RAM all reserved. After talking with VMware support they suggested the EOF error usually co insides with a network issue or congestion I knew this couldn’t be the issue I have multiple 10GB pipes out of the enclosure and 10GB min speed across the network which isn’t close to being used.