vSphere On-disk Metadata Analyzer

vSphere On-disk Metadata Analyzer (VOMA) is a utility which helps us in performing VMFS file system metadata checks. 

We might need to check metadata consistency of a file system when we experience problems related to storage outages or could be when we performed a disk replacement we might see errors in vmkernel.log file.

How does the error looks like well i haven't received this error in my Home Lab environment but when delivering VMware vSphere troubleshooting workshop class one particular troubleshooting topic talks about metadata consistencies and the troubleshooting steps we need to take to work with this kind of problem.

So thought of checking any related KB articles which talks about the same got one KB article 2036767 which talks about vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency.

vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 130: <START termserv2-5160fe37.vswp>
vmkernel: 25:21:39:57.861 cpu15:1047)Lock [type 10c00001 offset 52076544 v 69, hb offset 4017152
vmkernel: gen 109, mode 1, owner 4a15b3a2-fd2f4020-3625-001a64353e5c mtime 3420]
vmkernel: 25:21:39:57.861 cpu15:1047)Addr <4, 1011, 10>, gen 36, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
vmkernel: 25:21:39:57.861 cpu15:1047)len 3221225472, nb 3072 tbz 0, zla 3, bs 1048576
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 132: <END termserv2-5160fe37.vswp>vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 3586: Failed to initialize swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore_name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $

Before running VOMA ensure all virtual machines on the affected datastore are powered off or migrated to another datastore.

It's time to fetch out the name and partition number of the device which backs the VMFS datastore that we are planning to check.

It can be done using esxcli storage vmfs extent list command after connecting to your ESXi host through putty.

Now that we have got the details we require to run vSphere On-disk Metadata Analyzer it's time to run the same.

*Note: Installation/Configurations/Specifications methods used here has been Tested in My Home Lab Nested Environment.

 voma -m vmfs -f check -d /vmfs/devices/disks/eui.5adcee56739fb3ea:1

Where eui.5adcee56739fb3ea:1 refers to device name and the partition.

***** We are able to run the VOMA successfully as there is no problems with data consistency but if incase you may have metadata consistency issues in your environment you may find errors  (Error: Missing LVM Magic. Disk doesn’t have a valid LVM Device Error: Failed to Initialize LVM Metadata)

We can also run the above command by specifying the log file where we want to store the output and send it to VMware support team.

*****When the corruption is irreversible, VMware recommends us to restore the datastore files from a backup.

In my environment i stored the output in output.txt file created in temp directory by using the same command as mentioned above with the name and location of the file.

 voma -m vmfs -f check -d /vmfs/devices/disks/eui.5adcee56739fb3ea:1 -s /tmp/output.txt