daknetworks.com

VCSA
vCenter Server with an embedded Platform Services Controller (PSC)

This has an internal ldap for single sign on (SSO); typically called VSPHERE.LOCAL. This can be changed to use SSO with another ldap or Windows Active Directory. Internally, this is called VMWare Directory or vmdir.

The VCSA system name/hostname should be in line with the domain; vcsa.company.tld

SSO: VSHPERE.LOCAL
HOSTNAME.DOMAIN.LOCAL
VCSA: v6.7.0.51000
ESXI: v6.7.0
USER: administrator@vsphere.LOCAL

=====================================
-On VCSA, new gateway, from 192.168.21.1 to 192.168.21.120
-On VCSA, changed hostname from 192.168.21.152 to HOSTNAME.DOMAIN.LOCAL
-Reboot
-Error: Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE

=====================================
Try to fix by:
-ssh in as root
-let us look at the logs

cat /var/log/vmware/messages

Failed to connect to VMware Lookup Service - https://HOSTNAME.DOMAIN.local:443/lookupservice/sdk

cd /var/log/vmware/vmdird/vmdird-syslog.log
ls -la
zcat vmdird-syslog.log.7.gz |grep 2022-07-15 |more

2022-07-15T02:20:10.930380+00:00 info vmdird  t@140164554802944: VmDir State (2)
2022-07-15T02:20:10.930719+00:00 info vmdird  t@140164554802944: Srv_RpcVmDirSetState: VmDir State (2)
2022-07-15T02:20:11.429906+00:00 info vmdird  t@140164554802944: VmDir State (3)
2022-07-15T02:20:11.430213+00:00 info vmdird  t@140164554802944: Srv_RpcVmDirSetState: VmDir State (3)
2022-07-15T02:34:26.608866+00:00 info vmdird  t@140163900499712: Add Entry (cn=HOSTNAME.DOMAIN.local,cn=Servers,cn=Pittsburgh,cn=Sites,cn=Configuration,dc=vsphere,dc=LOCAL)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=
LOCAL)(via Ext)(USN 19454,0)
2022-07-15T02:34:26.663220+00:00 info vmdird  t@140163900499712: Add Entry (cn=HOSTNAME.DOMAIN.local,ou=Domain Controllers,dc=vsphere,dc=LOCAL)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=LOCAL)(via Ext)(USN 19455,
0)
2022-07-15T02:34:26.666660+00:00 info vmdird  t@140163900499712: MOD 1,rep,member: (cn=192.168.21.152,ou=Domain Controllers,dc=vsphere,dc=LOCAL)
2022-07-15T02:34:26.666908+00:00 info vmdird  t@140163900499712: MOD 2,rep,member: (cn=HOSTNAME.DOMAIN.local,ou=Domain Controllers,DC=vsphere,DC=LOCAL)
2022-07-15T02:34:26.723039+00:00 info vmdird  t@140163900499712: Modify Entry (cn=DCAdmins,cn=Builtin,dc=vsphere,dc=LOCAL)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=LOCAL)(via Ext)(USN 19456,0)
2022-07-15T02:34:26.760656+00:00 info vmdird  t@140163900499712: Add Entry (cn=vmca/HOSTNAME.DOMAIN.local@VSPHERE.LOCAL,cn=Managed Service Accounts,dc=vsphere,dc=LOCAL)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=t
pxca)(via Ext)(USN 19457,0)
2022-07-15T02:34:26.800188+00:00 info vmdird  t@140163900499712: Add Entry (cn=ldap/HOSTNAME.DOMAIN.local@VSPHERE.LOCAL,cn=Managed Service Accounts,dc=vsphere,dc=LOCAL)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=t
pxca)(via Ext)(USN 19458,0)
2022-07-15T02:34:26.850848+00:00 info vmdird  t@140163900499712: Add Entry (cn=host/HOSTNAME.DOMAIN.local@VSPHERE.LOCAL,cn=Managed Service Accounts,dc=vsphere,dc=LOCAL)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=t
pxca)(via Ext)(USN 19459,0)
2022-07-15T02:34:26.894935+00:00 info vmdird  t@140163900499712: MOD 1,rep,vmwLKUPPropertyValue: (HOSTNAME.DOMAIN.local)
2022-07-15T02:34:26.931662+00:00 info vmdird  t@140163900499712: Modify Entry (cn=Property3,cn=06bd9be7-5a98-4435-9a40-8c11d7847b69,cn=ServiceRegistrations,cn=LookupService,cn=Pittsburgh, cn=Sites,cn=Configuration,dc=vsphere,dc=LOCAL)(fr
om 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=LOCAL)(via Ext)(USN 19460,0)
2022-07-15T02:34:26.935916+00:00 info vmdird  t@140163900499712: MOD 1,rep,serverName: (cn=HOSTNAME.DOMAIN.local,cn=Servers,cn=Pittsburgh,cn=Sites,cn=Configuration,dc=vsphere,dc=LOCAL)
2022-07-15T02:34:26.950811+00:00 info vmdird  t@140163900499712: Modify Entry (cn=DSE Root)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=LOCAL)(via Ext)(USN 19461,0)
2022-07-15T02:34:26.953615+00:00 info vmdird  t@140163900499712: MOD 1,rep,vmwDCAccountDN: (cn=HOSTNAME.DOMAIN.local,ou=Domain Controllers,dc=vsphere,dc=LOCAL)
2022-07-15T02:34:26.956569+00:00 info vmdird  t@140163900499712: Modify Entry (cn=DSE Root)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=LOCAL)(via Ext)(USN 19462,0)
2022-07-15T02:34:26.959271+00:00 info vmdird  t@140163900499712: MOD 1,rep,vmwDCAccountUPN: (HOSTNAME.DOMAIN.local@VSPHERE.LOCAL)
2022-07-15T02:34:26.962386+00:00 info vmdird  t@140163900499712: Modify Entry (cn=DSE Root)(from 127.0.0.1)(by cn=Administrator,cn=Users,dc=vsphere,dc=LOCAL)(via Ext)(USN 19463,0)
2022-07-15T02:34:26.989844+00:00 info vmdird  t@140163900499712: Add Entry (cn=Replication Agreements,cn=HOSTNAME.DOMAIN.local,cn=Servers,cn=Pittsburgh,cn=Sites,cn=Configuration,DC=vsphere,DC=LOCAL)(from 127.0.0.1)(by cn=Administrat
or,cn=Users,dc=vsphere,dc=LOCAL)(via Ext)(USN 19464,0)
2022-07-15T02:34:29.046587+00:00 err vmdird  t@140163900499712: VmDirSRPGetIdentityData (HOSTNAME.DOMAIN.local@vsphere.LOCAL) failed, (9611)
2022-07-15T02:34:29.047219+00:00 err vmdird  t@140163900499712: VmDirSRPGetIdentityData (HOSTNAME.DOMAIN.local@vsphere.LOCAL) failed, (9611)
2022-07-15T02:34:29.047458+00:00 err vmdird  t@140163900499712: SASLSessionStart: sasl error (-20)(SASL(-13): user not found: no secret in database)
2022-07-15T02:34:29.047654+00:00 err vmdird  t@140163900499712: VmDirSendLdapResult: Request (Bind), Error (49), Message ((49)(SASL start failed.)), (0) socket (127.0.0.1)
2022-07-15T02:34:29.047838+00:00 err vmdird  t@140163900499712: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "", Method: SASL

-the error is: "user not found: no secret in database"
-it is referring to the following account:
(HOSTNAME.DOMAIN.local@vsphere.LOCAL)

=====================================
Let's see if the VMWare Directory/SSO/VSPHERE.LOCAL exists and is working.

We can test using curl:
curl -v telnet://192.168.21.152:636

Or we can test using the vdcadmintool:
/usr/lib/vmware-vmdir/bin/vdcadmintool
GetVmDirState failed: VmDirGetState() failed. error(382312694)
test ldap
Please enter LDAP server host: 192.168.21.152
Please enter LDAP server port: 389
Please enter LDAP server SSL port: 636
Please enter LDAP Bind DN: dc=vsphere,dc=LOCAL
Please enter LDAP Bind UPN: administrator@vsphere.LOCAL
Please enter LDAP Bind password:

Now we know and have verified that the VMWare Directory is indeed working.

==============================================
Let's reset the password for the account found in the error from the logs: (HOSTNAME.DOMAIN.local@vsphere.LOCAL)

/usr/lib/vmware-vmdir/bin/vdcadmintool

select 3

Enter the vCenter server: HOSTNAME.DOMAIN.local@VSPHERE.LOCAL

It will generate a random password for that account, but you may need to run more than once if it puts invalid characters or blanks in the password.

Note: The tool does not filter out invalid characters from the generated password such as:
& (ampersand)
; (semicolon)
" (double quotation mark)
' (single quotation mark)
^ (circumflex)
\ (backslash)
% (percentage)

Copy that password because you have to past it in the registry.

pA$$&;^\#*2i)W}nqK!~.Jd8z

==============================================
The system mimics Windows Registry and has a registry of its own.
Windows has REGEDIT/REG QUERY but VCSA has LWREGSHELL. This can view/change the registry.

Run these commands to view the reg key:
/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
list_values

+  "Arguments"            REG_SZ          "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"
+  "dcAccount"            REG_SZ          "HOSTNAME.DOMAIN.local"
+  "dcAccountDN"          REG_SZ          "cn=HOSTNAME.DOMAIN.local,ou=Domain Controllers,dc=vsphere,dc=LOCAL"
+  "dcAccountOldPassword" REG_SZ          "pA$$&;^\#*2i)W}nqK!~.Jd8z"
+  "dcAccountPassword"    REG_SZ          "pA$$&;^\#*2i)W}nqK!~.Jd8z"
+  "DirtyShutdown"        REG_DWORD       0x00000000 (0)
+  "LduGuid"              REG_SZ          "28bf4aaa-b564-49b2-a354-abcde1234567"
+  "MachineGuid"          REG_SZ          "bcf160ba-cb01-4dc3-b574-abcde1234567"
+  "SiteGuid"             REG_SZ          "4f2b0b44-4da7-43e3-b1a9-abcde1234567"
   "Autostart"            REG_DWORD       0x00000001 (1)
   "Dependencies"         REG_SZ          "lsass dcerpc vmafd"
   "Description"          REG_SZ          "VMware Directory Service"
   "Environment"          REG_SZ          ""
   "Path"                 REG_SZ          "/usr/lib/vmware-vmdir/sbin/vmdird"
   "Type"                 REG_DWORD       0x00000001 (1)

Run these commands to update the password in the reg key:
set_value dcAccountPassword "pA$$&;^\#*2i)W}nqK!~.Jd8z"
quit

Then reboot the VCSA.

==============================================
The vSphere Client has a red bar complaining about not being able to connect to SSO.
We will need to regenerate all the certificates.

-ssh in as root
/usr/lib/vmware-vmca/bin/certificate-manager
8
-Fill in as appropriate but be sure use the ip addresswhen it says "optional" so that it will put the ip address in the Subject Alternative Name (SAN) of the certificate.
-Wait about 15 minutes.
-You should be able to access vSphere login page.

==============================================
Here are some interesting areas I found along the way:

https://192.168.21.152/sso-adminserver/sdk/vsphere.LOCAL
https://192.168.21.152:443/lookupservice/sdk
https://192.168.21.152/sts/STSService/vsphere.LOCAL
https://192.168.21.152/websso/SAML2/SLO/vsphere.LOCAL
https://192.168.21.152:443/sms/sdk
https://localhost:9090/vsphere-client/
https://192.168.21.152:443/invsvc/vmomi/sdk
https://HOSTNAME.DOMAIN.LOCAL:443/appliance/support-bundle
https://192.168.21.152:9090/vsphere-client/

/etc/vmware/vsphere-client/webclient.properties
/etc/vmware-vpx/vpxd.cfg
/etc/applmgmt/appliance/appliance.conf:
/etc/resolv.conf:search vsphere.LOCAL
/etc/systemd/network/10-eth0.network:Domains= vsphere.LOCAL
/etc/vmware/install-defaults/vmdir.domain-name

Find PNID:
During the initial configuration of the VMware vCenter Server, the system name (FQDN or IP address) is used as the PNID, Primary Network Identifier.
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
HOSTNAME.DOMAIN.local

Change the hostname can be done either via:
VAMI; or VMware Appliance Management Interface
-click NETWORKING (on left-hand side).
-find NETWORK SETTINGS (in the middle).
-click EDIT (on the right-hand side).
-run through the settings wizard.

or by ssh:
/opt/vmware/share/vami/vami_config_net

The root password has a default lockout of 3 attempts. Then you gotta wait 5 minutes to reset. To view:
pam_tally2 -u root

To reset manually:
pam_tally2 -u root -r

To shutdown/restart all the services:
service-control --all --stop
service-control --all --start

https://www.nakivo.com/blog/503-service-unavailable-error-on-the-vsphere-web-client/
(Long article but see Password Issues)
https://docs.vmware.com/en/VMware-Integrated-OpenStack/7.2/com.vmware.openstack.admin.doc/GUID-02577103-96E9-49B0-A2B1-4D6BE6B2E103.html
https://kb.vmware.com/s/article/2147280
https://communities.vmware.com/t5/vCenter-Server-Discussions/The-Reset-all-Certificates-option-in-the-certificate-manager/td-p/2247608

Failed to connect to endpoint