Need help with FreeNAS Data recovery.

[clickme.sh]

So I have a client who had a FreeNAS server in his office which was used as datastor for VMWare. A few days ago the boot SSD died and he didn't have any backup of data or Configs. His Office didnt have any IT person for a year or so.

They dont have any backup of data on NAS and don't know the FreeNAS configuration.(Pools etc0

 

I was able to restore 2 Pools(Oldboot and Oldboot1 with help of the guy working there.

We have 2 Disks(SSDs) that are not a part of pool left and his VMWare has few VM's missing. Can you guys help me with this? I am attaching the devices list and volume info here.

 

Spoiler

Device Info

[root@freenas] ~# gpart show /dev/ada0
=>        34  2051200301  ada0  GPT  (978G)
          34        2014        - free -  (1.0M)
        2048  2051196928     1  vmware-vmfs  (978G)
  2051198976        1359        - free -  (680K)

[root@freenas] ~# gpart show /dev/nvd0
=>        34  2000409197  nvd0  GPT  (954G)
          34          94        - free -  (47K)
         128  2000408448     1  vmware-vmfs  (954G)
  2000408576         655        - free -  (328K)

[root@freenas] ~# camcontrol devlist
<ATA ST2000NM0033-9ZM SN06>        at scbus0 target 0 lun 0 (pass0,da0)
<ATA ST2000NM0033-9ZM SN06>        at scbus0 target 1 lun 0 (pass1,da1)
<ATA ST2000NM0033-9ZM SN06>        at scbus0 target 2 lun 0 (pass2,da2)
<ATA ST2000NM0055-1V4 TN02>        at scbus0 target 3 lun 0 (pass3,da3)
<Crucial CT1050MX300SSD1 M0CR021>  at scbus1 target 0 lun 0 (pass4,ada0)
<WDC WD20EFRX-68EUZN0 82.00A82>    at scbus3 target 0 lun 0 (pass5,ada1)
<WDC WD20EFRX-68EUZN0 82.00A82>    at scbus4 target 0 lun 0 (pass6,ada2)
<WDC WD20EFRX-68EUZN0 82.00A82>    at scbus5 target 0 lun 0 (pass7,ada3)
<WDC WD20EFRX-68EUZN0 82.00A82>    at scbus6 target 0 lun 0 (pass8,ada4)
<64GB SSD  0701>                   at scbus8 target 0 lun 0 (pass9,da4)
<ST2000DM 006-2DM164 CC26>         at scbus9 target 0 lun 0 (da5,pass10)

[root@freenas] ~# nvmecontrol devlist
 nvme0: Samsung SSD 960 PRO 1TB
    nvme0ns1 (976762MB)

Spoiler

Volume Info

[root@freenas] ~# zpool list
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  59.5G   866M  58.7G         -      -     1%  1.00x  ONLINE  -
oldboot       7.25T  1.30T  5.95T         -    11%    17%  1.00x  ONLINE  /mnt
oldboot1      7.25T  4.00T  3.25T         -    39%    55%  1.00x  ONLINE  /mnt

 

[clickme.sh]

I found out that one of the SSD is corrupted.

I am attaching the log.

Spoiler

FS-INFO Crucial SSD
https://pastebin.com/DM2Ck3XN

Spoiler

 SAMSUNG-NVME-VMFS-Errors

https://pastebin.com/5hZ35gq8

I am planing to suggest them to send the SSD's to a professional repair/recovery agency. Any idea about how much that could cost in CAD?

[NelizMastr]

41 minutes ago, Navneet said:

I found out that one of the SSD is corrupted.

I am attaching the log.

  Reveal hidden contents

FS-INFO Crucial SSD
https://pastebin.com/DM2Ck3XN

  Reveal hidden contents

 SAMSUNG-NVME-VMFS-Errors

https://pastebin.com/5hZ35gq8

I am planing to suggest them to send the SSD's to a professional repair/recovery agency. Any idea about how much that could cost in CAD?

Recovery will be in the thousands of dollars, not tens or hundreds, depending on the amount of data.

[Velcade]

42 minutes ago, NelizMastr said:

Recovery will be in the thousands of dollars

This is true.  We recently went through this at work.  A colleague had a corrupt drive and didn't have any back ups, it ended up costing our department $2800 to recover the files.

[Lady Fitzgerald]

1 hour ago, NelizMastr said:

Recovery will be in the thousands of dollars, not tens or hundreds, depending on the amount of data.

Assuming recovery is even possible. SSDs usually fail irrecoverably. At best, professional recovery may recover only some of the data.

 

As Velcade alluded, backups are essential to ensure the safety of data.

[Razor Blade]

Data recovery varies wildly depending on what killed the drive in question. If it is just a software or firmware problem they can correct then clone the drive, recovery is going to be a lot less expensive than if they have to start swapping chips (as an example). Usually you can get an estimate from reputable data recovery services. Some services even have a "no data recovered, no charge" policy. Though like said above, even if they can recover data it may not be all of it. I feel like this is obvious....but if you're planning to send it out, don't mess with it anymore.

 

I assume that those 2 SSDs were striped and are part of the missing pool? It might be more beneficial to send out both drives and see if you can get the data recovery service to recover your entire array rather than just the one SSD because there is no guarantee FreeNAS will be able to import the disks even if you recover the corrupt drive.

 

As a side note....be careful with using iSCSI for data on the same hypervisor the NAS is hosted on. I've experienced some really...really strange problems during testing with ESXI and a FreeNAS VM. Storing ISO or other data on them seems fine but installing VMs on the drive gave me stability issues. Though to be fair it may have been a config problem on my end.

[Mikensan]

*Tongue in cheek* Restore from a backup.

 

Others have already covered what to expect - but certainly use this as a selling point to start performing backups.

[clickme.sh]

Thank you guys for your opinion. I am going to show this thread to my client and lets hope he can make an informed decision.

[clickme.sh]

Well My client didn't want to go for the professional recovery route as it was very expensive. I ended-up asking him to decide the value of data that needed to be recovered, I didn't want to risk data loss by trying manual self recovery. He decided to D.I.Y it hence I politely declined to help him further as I didn't want to be the one who screws up if something gets lost in the process. I dint charge anything for the service as I was not satisfied with the service provided. 

Edited January 20, 2019 by Navneet
Fixed typo.