Unable to Unmount/Delete VMFS Datastore: The Resource Is in Use

During the migration of virtual machines files from an old HP MSA 2000 storage to a new storage system Dell EMC Unity (both connected to ESXi hosts over SAN) I came across a problem when trying to remove an empty VMFS Datastore in VMWare vSphere. When trying to unmount the VMFS Datastore from ESXi hosts, an error message appears saying that the datastore is still in use / busy:

Unmount VMFS volume vmesxi2.contoso.com. The resource 'Datastore Name: MSA2000_LUN1 VMFS uuid: xxxxx--xxxxx-x-xxxxx-xxxx' is in use. Cannot unmount volume Datastore Name VMFS “file system is busy”.

Based on the error, it is clear that the VMFS datastore cannot be removed since ESXi hosts or vSphere are still using storage to write some data. According to the VMWare documentation, when removing LUN from vSphere you should check the following points:

• There are no virtual machine, template, snapshot or ISO image files on VMFS datastore (you must migrate your VMs to another VMFS Datastore, power off and remove them, or unregister VMs in vSphere);
• Storage I/O Control is disabled for datastore;
• The datastore is not a part of the Datastore Cluster;
• The LUN is not used as an RDM device;
• VMFS datastore is not used to store vSphere HA data, Storage DRS, logs, dumps (/vmkdump/), vSAN data (/vsantraced/), technical support data (scratch partition), or virtual machine swap files.

The virtual machines have already been moved to a new datastore using Storage vMotion. Let’s see what other files and folders remain on the VMFS datastorage:

As you can see, there are logdir (a directory with logs), sdd.sf (a directory with the description of SCSI devices), dbsData (a directory with the data of the distributed virtual switch) and naa… (VMFS metadata) folders remained.

In my case, ESXi host logs  are still located in the VMFS datastore (Scratch Location). Check out if the datastore is specified as a location of logs in the ESXi host settings. Go to Manage -> Settings -> Advanced System Settings. Find ScratchConfig.CurrentScratchLocation and Syslog.global.logDir and if your VMFS datastore is specified in it, change the path to the logs directory.

Restart the ESXi host or restart the Syslog Server service on the host.

Let’s try to unmount the datastore from the ESXi host again. Go to Manage -> Storage -> Storage Device. Find your LUN in the list, select it and click Detaches the selected device from the host.

In my case the error occurred again:
Detach SCSI LUN The resource is in use.

For a long time I was trying to find out who uses this datastore and decided that the easiest way in my case was to delete the partition table of VMFS datastore (of course, you should do it very carefully and make sure that you are removing the right datastore).

1. Connect over SSH to any ESXi host that has VMFS store mounted;
2. Copy the ID of your datastore in the vSphere interface, and check on the ESXi host, which device and path to the VMFS store it matches: esxcfg-scsidevs -c | grep naa.6001438005df0dee0000700004be0000naa.6001438005df0dee0000700004be0000 Direct-Access /vmfs/devices/disks/naa. 6001438005df0dee0000700004be0000 512000MB NMP HP Fibre Channel Disk (naa.6001438005df0dee0000700004be0000)esxcfg-scsidevs -m | grep naa.6001438005df0dee0000700004be0000naa.6001438005df0dee0000700004be0000:1 /vmfs/devices/disks/naa.6001438005df0dee0000700004be0000:1 570e5298-08f4c74e-ca3a-b4b52f5e2b38 0 MSA2000_LUN1
3. Let’s see how much space is busy on the VMFS datastore (it’s almost empty): df -h | grep MSA2000_LUN1VMFS-5 499.8G 1.4G 498.3G 0% /vmfs/volumes/MSA2000_LUN1
4. So we have made sure that the datastore name and the store ID match the LUN that we want to remove;
5. Now let’s get check information about partition table on your datastore: partedUtil getptbl /vmfs/devices/disks/naa.6001438005df0dee0000700004be0000gpt 65270 255 63 1048576000 1 2048 1048575966 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
6. In this case, the LUN has a single partition with the VMFS file system;
7. Delete the VMFS partition with the ID 1 from this disk (LUN). Please, be very attentive!!!partedUtil delete /vmfs/devices/disks/naa.6001438005df0dee0000700004be0000 1 

After deleting the partition, you can unmount the LUN in vSphere – Unmount Datastore. The ‘datastore is in use’ error won’t not appear again.

Select the ESXi hosts to unmount the datastore from.

Or you can select Detach from the list of connected storage devices on the ESXi host. The connection state will then change to Detached.

After that you can remove the VMFS store (Delete Datastore) and it will disappear from vSphere.