2015年12月20日日曜日

NASの自作 Part51

OMVにComicCafeを入れてかれこれ一ヶ月ぐらい連続稼働させていて、いたって順調だったのですが、今日突然OMVから以下の様なエラーのメールが届きました。(3通)

This is an automatically generated mail message from mdadm
running on OMV-NODE804

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sdh.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdc[0] sdl[9](F) sdg[8](F) sdk[7] sdf[6] sdm[5] sdj[4] sdi[3] sdh[2](F) sdd[1]
      23441080320 blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/7] [UU_UUUUU__]

unused devices: none

(F)って多分、Failって意味だから、HDD3台故障？？？マジですか。。。本当に3台壊れたなら終わりじゃないですか。。。同時にSMARTのエラーメールも届いてました。(2通)

This email was generated by the smartd daemon running on:

   host name: OMV-NODE804
  DNS domain: local
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

Device: /dev/disk/by-id/ata-WDC_WD30EZRX-00D8PB0_WD-WCC4N3KS2AFV [SAT], unable to open device

For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
Another email message will be sent in 24 hours if the problem persists.

"unable to open device"でググってみるとどうやらHDDの故障よりケーブルなどのトラブルの可能性が高いとのこと、外出先だったので直ぐに帰って復旧作業をしました。
最初に再起動を試しましたが起動後、RAIDは認識されず。。。メールで報告されたデバイスはsdg, sdk, sdl, sdhの4つで、自作時のメモでシリアル番号を確認すると、同じケーブルに接続されている4台ではないようです。これはヤバイと思い以下のコマンドでRAIDの状態を確認すると

cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdc[0](S) sdg[9](S) sdh[8](S) sdl[7](S) sdi[6](S) sdf[5](S) sdj[4](S) sdm[3](S) sdk[2](S) sdd[1](S)
      29301355120 blocks super 1.2
       
unused devices: none

ん！？ 何故かドライブの状態が全て(S)になってます。。。調べるとどうやらSはスペアのSとのこと。さらにコマンドで状況を調べます。

mdadm -E /dev/sd[cdfgihjklm]
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : d1ec112e:6d291ee3:25845efa:d6957742

    Update Time : Sat Dec 19 22:22:17 2015
       Checksum : b585aa10 - correct
         Events : 131

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.AAAAA.. ('A' == active, '.' == missing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5740654e:ae81ac6a:eec7e781:af1c6a60

    Update Time : Sat Dec 19 22:22:17 2015
       Checksum : 2e3207a - correct
         Events : 131

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.AAAAA.. ('A' == active, '.' == missing)
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 92bf4726:5304522d:6db5ab24:f8368453

    Update Time : Sat Dec 19 22:22:17 2015
       Checksum : 33492a25 - correct
         Events : 131

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AA.AAAAA.. ('A' == active, '.' == missing)
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 33585a1d:96046870:fb9f752f:3b49e3dd

    Update Time : Sat Dec 19 17:29:20 2015
       Checksum : 2a37b3a - correct
         Events : 122

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 9
   Array State : AAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c0cdad1d:12c70ac3:bee5bd83:1ace06ca

    Update Time : Sat Dec 19 17:29:20 2015
       Checksum : 96057de4 - correct
         Events : 122

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 8
   Array State : AAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdi:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e05b06ac:92271d9d:e2e73405:afbefa7d

    Update Time : Sat Dec 19 22:22:17 2015
       Checksum : 33d2a3e0 - correct
         Events : 131

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AA.AAAAA.. ('A' == active, '.' == missing)
/dev/sdj:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 879cda62:7469e943:8dd7e491:da943dec

    Update Time : Sat Dec 19 22:22:17 2015
       Checksum : 8c65ec3d - correct
         Events : 131

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AA.AAAAA.. ('A' == active, '.' == missing)
/dev/sdk:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 264a91d5:2c04c3e4:6a5f8654:64d23e6d

    Update Time : Sat Dec 19 17:29:20 2015
       Checksum : e3a1b554 - correct
         Events : 122

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdl:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 277d4802:06ad5ffa:7d9907de:70522c67

    Update Time : Sat Dec 19 22:22:17 2015
       Checksum : a95b8ff8 - correct
         Events : 131

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 7
   Array State : AA.AAAAA.. ('A' == active, '.' == missing)
/dev/sdm:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 8002b7d3:4e1bb8d3:66b534b1:e7e9a90d
           Name : OMV-NODE804:RAID6  (local to host OMV-NODE804)
  Creation Time : Sun Sep 27 00:41:43 2015
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 23441080320 (22355.16 GiB 24003.67 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 7b8bfeed:db4b8436:0fafa8da:f59b317a

    Update Time : Sat Dec 19 22:22:17 2015
       Checksum : e0dc9c34 - correct
         Events : 131

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.AAAAA.. ('A' == active, '.' == missing)

見た感じ全く問題なさそうです。ちなみにディスクが壊れてる場合は mdadm: No md superblock detected on /dev/sda. みたいなメッセージがでるらしいです。
念のためSMARTなどの状態も確認しましたが、特に異常は見つかりませんでした。幸いバックアップは前のHDDに全て残ってるはずなので、強制的にRAIDを再構築してみることにしました。

mdadm --assemble --scan --force

コマンドは一瞬で終わり、無事にRAIDが復活しました。

cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdc[0] sdg[9] sdh[8] sdl[7] sdi[6] sdf[5] sdj[4] sdm[3] sdk[2] sdd[1]
      23441080320 blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]


cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.
# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is
# used if no RAID devices are configured.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST 

# definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 name=OMV-NODE804:RAID6 UUID=8002b7d3:4e1bb8d3:66b534b1:e7e9a90d

最近、CalcNoteの実装ばかりで、買ったUPSは放置したままだったり、OMVのアップデートも全くやってない状態だったので、時間作ってNASの運用も頑張らねばと思った次第です。でも本当にデータが消えなくて良かったです。
投稿者 時刻:

0 件のコメント:

コメントを投稿

登録: コメントの投稿 (Atom)