Comments on: NetApp disk replacement – so easy a caveman and his tech savvy neighbor can do it

By: Replacing a Failed NetApp Drive with an Un-zeroed Spare at thinking sysadmin

Replacing a Failed NetApp Drive with an Un-zeroed Spare at thinking sysadmin — Sat, 28 May 2011 13:42:07 +0000

[...] Boche has a post on the method he used to replace a failed drive on a filer with an un-zeroed spare (transferred [...]

By: Chris Gebhardt

Chris Gebhardt — Fri, 21 May 2010 18:26:43 +0000

Sorry for the late reply, YES they were disks from a lab pull. I pulled them from my lab. And Andrew is spot on as usual. Only in cases where disks have been previously owned by a different controller will this procedure have to be performed and that was what happened in this case. Normally it is just look for the light, and pull. Now the only caveat is if you dont have auto assign enabled then you will have to assign ownership to a disk.

More on disk ownership here: http://bit.ly/bi3dDx

By: jason

jason — Sat, 15 May 2010 18:59:45 +0000

Jason Langer, the spreadsheet is really nothing special. Merely a tool I use to keep track of the storage configurations. I’ve updated the post with a screenshot.

By: jason

jason — Sat, 15 May 2010 04:41:02 +0000

I’m confident in NetApp that the replacement disk did not contain customer data. It was likely a lab pull. This drive replacement was a special circumstance.

By: Andrew Miller

Andrew Miller — Sat, 15 May 2010 02:53:32 +0000

Definitely an atypical replacement experience to be honest (speaking as someone who replaced 100+ disks as a customer and now as a vendor SE has a lot of NetApp customers).

The general process usually is….

-get an email about a failed disk (RAID rebuild starts right now)
-new disk shows up with 4 hours or next day (by which time the rebuild is usually already done)
-walk out to NetApp and pull the disk with the orange light (helps not to be colorblind)
-put in the new disk
-look in System Manager just to confirm your system sees all its disks

Almost ridicously simple to be honest most of the time…. (disk auto-assignment usually assigns the replacement disk to the same controller as the failed disk)

It is very odd that you got a spare that wasn’t zeroed and was still assigned to another system (they should always be zeroed — i.e. NetApp isn’t supposed to ship customers replacement disks with other customers data on them (even if said data is pretty much unreadable without having the WAFL patents .

By: Jason Langer

Jason Langer — Fri, 14 May 2010 23:21:42 +0000

Hey Jason,

Mind sharing that storage array spreadsheet you made the comment about?

TIA,
-Jason

By: Erick Moore

Erick Moore — Fri, 14 May 2010 01:46:30 +0000

Gotcha. When we replace a failed disk we usually see something like this in the /etc/messages file:

Mon May 3 09:53:44 MST [sanName: raid.disk.missing:info]: Disk 1b.99 Shelf 6 Bay 3 [NETAPP X268_HGEMIT75SSX A90A] S/N [P8H5MU9E] is missing from the system

Then once the replacement is inserted:

Mon May 3 10:01:27 MST [sanName: unowned.disk.reminder:info]: 1 disks are currently unowned. Use ‘disk assign’ to assign the disks to a filer.

Interesting that you didn’t get this behavior. Did you manually fail the disk?

By: jason

jason — Thu, 13 May 2010 22:43:42 +0000

Thanks for the tip Erick. Assigning of the disk failed, complaining the disk belonged to another filer.

By: Erick Moore

Erick Moore — Thu, 13 May 2010 19:50:03 +0000

Technically all you needed to do was replace the failed disk, then assign it to the SAN head that you wanted to own the disk. If you didn’t have any unowned disk you could just assign all unowned disk to that system. There is no reason to go into “priv set advanced” to make all of this happen.

san1> disk assign all -o “san1″
san1> disk zero spares

It is also best practice to pre-zero your spare disk as the second command listed shows. Also make certain you type “priv set” to get out of advanced mode.