I had been running into issues with getting OS 10.13 running from a new virtual machine on ESXi. Running a test today we noted that upgrading from any older version (I only tried it with 10.11 and 10.12) I was able to get 10.13 running on ESXi. As a side, it was first thought that 10.13 would not run within ESXi on the XSERVE platform, but indeed upgrading a previous install of OSX, it runs just fine on the xserve platform.
I ended up doing a clean install of 10.11 (first iso I had handy) and then using Appstore to start the download of 10.13 within the OSX vm.
Once the download is complete close the screen that may have popped up to continue the installation for high sierra.
Open a terminal window and run the following
sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/startosinstall
/Applications/Install\ macOS\ High\
Sierra.app --agreetolicense --converttoapfs NO --nointeraction
(The above is all one line)
converttoapfs NO argument stops the OS from setting the drive format to the APFS.
As a note, for some reason this didnt work first time so I needed to delete the Install macOS High Sierra.app from the applications folder and then re-download it, afterwhich it took and began installation.
When the I could log back into the operating system High Sierra had indeed installed, so far so good. We are internally working to get 10.13 installable as a fresh installation instead of having to do an update.
ESXi: 6.5u1 build: 5969303
Host: Apple XServe 3,1