...

пятница, 26 января 2018 г.

ZFS: Тестируем надежность на плохих дисках

ZFS славится своей надежностью, а тут у меня под рукой собралось несколько побитых жизнью дисков. Попробуем на raidz1 закинуть некоторый объем данных, а затем проверить их целостность, а zfs на устойчивость к такой ситуации.

Диски подключены в Adaptec RAID 2805 в качестве «Simple Volume», поверх развернут raidz1 с настройками по умолчанию (для установщика FreeBSD 11-1).

Конфигурация ZFS 
zfs get all zroot
NAME   PROPERTY              VALUE                       SOURCE
zroot  type                  filesystem                  -
zroot  creation              пт янв. 26 15:40 2018  -
zroot  used                  75,5G                       -
zroot  available             10,1T                       -
zroot  referenced            128K                        -
zroot  compressratio         1.01x                       -
zroot  mounted               yes                         -
zroot  quota                 none                        default
zroot  reservation           none                        default
zroot  recordsize            128K                        default
zroot  mountpoint            /zroot                      local
zroot  sharenfs              off                         default
zroot  checksum              on                          default
zroot  compression           lz4                         local
zroot  atime                 off                         local
zroot  devices               on                          default
zroot  exec                  on                          default
zroot  setuid                on                          default
zroot  readonly              off                         default
zroot  jailed                off                         default
zroot  snapdir               hidden                      default
zroot  aclmode               discard                     default
zroot  aclinherit            restricted                  default
zroot  canmount              on                          default
zroot  xattr                 off                         temporary
zroot  copies                1                           default
zroot  version               5                           -
zroot  utf8only              off                         -
zroot  normalization         none                        -
zroot  casesensitivity       sensitive                   -
zroot  vscan                 off                         default
zroot  nbmand                off                         default
zroot  sharesmb              off                         default
zroot  refquota              none                        default
zroot  refreservation        none                        default
zroot  primarycache          all                         default
zroot  secondarycache        all                         default
zroot  usedbysnapshots       0                           -
zroot  usedbydataset         128K                        -
zroot  usedbychildren        75,5G                       -
zroot  usedbyrefreservation  0                           -
zroot  logbias               latency                     default
zroot  dedup                 off                         default
zroot  mlslabel                                          -
zroot  sync                  standard                    default
zroot  refcompressratio      1.00x                       -
zroot  written               128K                        -
zroot  logicalused           75,8G                       -
zroot  logicalreferenced     11,5K                       -
zroot  volmode               default                     default
zroot  filesystem_limit      none                        default
zroot  snapshot_limit        none                        default
zroot  filesystem_count      none                        default
zroot  snapshot_count        none                        default
zroot  redundant_metadata    all                         default


S.M.A.R.T. дисков


aacd0p4 
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E6PN673U
LU WWN Device Id: 5 0014ee 20d6399e3
Firmware Version: 82.00A82
User Capacity:    4 000 787 030 016 bytes [4,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Jan 26 15:25:30 2018 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status command failed: Input/output error
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       296
  3 Spin_Up_Time            0x0027   210   208   021    Pre-fail  Always       -       6483
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       14
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7986
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       14
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       10
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       186
194 Temperature_Celsius     0x0022   112   111   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       17
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       55


aacd1p4 
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E7LPL4AH
LU WWN Device Id: 5 0014ee 2b8213774
Firmware Version: 82.00A82
User Capacity:    4 000 787 030 016 bytes [4,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Jan 26 15:26:19 2018 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status command failed: Input/output error
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       6043
  3 Spin_Up_Time            0x0027   207   190   021    Pre-fail  Always       -       6616
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   173   173   140    Pre-fail  Always       -       803
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7964
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       10
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       212
194 Temperature_Celsius     0x0022   108   108   000    Old_age   Always       -       44
196 Reallocated_Event_Count 0x0032   049   049   000    Old_age   Always       -       151
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       6
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       101
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       232


aacd2p4 
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD40EZRX-22SPEB0
Serial Number:    WD-WCC4E4KAK52T
LU WWN Device Id: 5 0014ee 2b6900646
Firmware Version: 80.00A80
User Capacity:    4 000 787 030 016 bytes [4,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Jan 26 15:26:43 2018 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status command failed: Input/output error
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   180   173   021    Pre-fail  Always       -       7983
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       11
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       9777
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       11
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       6
193 Load_Cycle_Count        0x0032   138   138   000    Old_age   Always       -       186920
194 Temperature_Celsius     0x0022   110   109   000    Old_age   Always       -       42
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0


aacd3p4 
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD40EZRX-00SPEB0
Serial Number:    WD-WCC4E5ALXUHC
LU WWN Device Id: 5 0014ee 20c16eacf
Firmware Version: 80.00A80
User Capacity:    4 000 787 030 016 bytes [4,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Jan 26 15:27:03 2018 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status command failed: Input/output error
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       394
  3 Spin_Up_Time            0x0027   194   179   021    Pre-fail  Always       -       7258
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       16
  5 Reallocated_Sector_Ct   0x0033   195   195   140    Pre-fail  Always       -       160
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7584
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       16
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       11
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       678600
194 Temperature_Celsius     0x0022   114   112   000    Old_age   Always       -       38
196 Reallocated_Event_Count 0x0032   121   121   000    Old_age   Always       -       79
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       44
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       319
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   199   199   000    Old_age   Offline      -       530


Приступаем к мучениям


В качестве тестовых данных я закинул около 71 ГБ всякой мультимедиа,
Куча ошибок чтения 

aacd3: hard error cmd=read 40789032-40789207
aacd3: hard error cmd=read 40789032-40789207
aacd1: hard error cmd=read 40790712-40790967
aacd3: hard error cmd=read 40789560-40789735
aacd1: hard error cmd=read 40794064-40794319
aacd1: hard error cmd=read 40795208-40795287
aacd1: hard error cmd=read 40795824-40796079
aacd1: hard error cmd=read 40796088-40796343
aacd1: hard error cmd=read 40797240-40797423
aacd3: hard error cmd=read 21743840-21744015
aacd1: hard error cmd=read 28502624-28502799
aacd1: hard error cmd=read 28597680-28597855
aacd1: hard error cmd=read 28635368-28635623
aacd1: hard error cmd=read 37340776-37340951
aacd1: hard error cmd=read 37342712-37342887
aacd1: hard error cmd=read 37347808-37348063
aacd1: hard error cmd=read 37348072-37348327
aacd1: hard error cmd=read 37352168-37352343
aacd1: hard error cmd=read 37359472-37359647
aacd1: hard error cmd=read 37365576-37365831
aacd1: hard error cmd=read 37372960-37373215
aacd1: hard error cmd=read 37373488-37373743
aacd1: hard error cmd=read 37380608-37380863
aacd1: hard error cmd=read 37381136-37381391
aacd1: hard error cmd=read 37382984-37383239
aacd1: hard error cmd=read 57577976-57577999
aacd1: hard error cmd=read 4606480-4606495
aacd1: hard error cmd=read 7811867664-7811867679
aacd1: hard error cmd=read 7811868176-7811868191
aac0: COMMAND 0xfffffe0000e97690 (TYPE 502) TIMEOUT AFTER 137 SECONDS
aac0: COMMAND 0xfffffe0000e91650 (TYPE 502) TIMEOUT AFTER 137 SECONDS
aac0: COMMAND 0xfffffe0000e92d10 (TYPE 502) TIMEOUT AFTER 137 SECONDS
aac0: WARNING! Controller is no longer running! code= 0xbcc90100
aacd3: hard error cmd=read 40785088-40785343
aacd3: hard error cmd=read 40785352-40785607
aacd3: hard error cmd=read 40785616-40785871
aacd3: hard error cmd=read 40788240-40788495
aacd3: hard error cmd=read 40783592-40783847
aacd3: hard error cmd=read 40784648-40784903
aacd3: hard error cmd=read 40785176-40785431
aacd3: hard error cmd=read 40785440-40785695
aacd3: hard error cmd=read 21743928-21744103
aacd1: hard error cmd=read 25407280-25407535
aacd1: hard error cmd=read 28507712-28507967
aacd1: hard error cmd=read 37322056-37322311
aacd1: hard error cmd=read 37344208-37344383
aacd1: hard error cmd=read 37348160-37348415
aacd1: hard error cmd=read 37373488-37373743
aacd1: hard error cmd=read 37380696-37380951
aacd1: hard error cmd=read 37383072-37383327
aacd1: hard error cmd=read 37383776-37384031
aacd1: hard error cmd=read 37395312-37395487
aacd1: hard error cmd=read 37426368-37426623
aacd1: hard error cmd=read 40682424-40682679
aacd1: hard error cmd=read 40702816-40703071
aacd1: hard error cmd=read 40725472-40725647
aacd1: hard error cmd=read 40760224-40760479
aacd1: hard error cmd=read 40761280-40761535
aacd1: hard error cmd=read 40764536-40764711
aacd1: hard error cmd=read 40772144-40772399
aacd1: hard error cmd=read 40774520-40774775
aacd1: hard error cmd=read 40778304-40778559
aacd3: hard error cmd=read 40783592-40783847
aacd3: hard error cmd=read 40784648-40784903
aacd3: hard error cmd=read 40785176-40785431
aacd3: hard error cmd=read 40785440-40785695
aacd1: hard error cmd=read 40785792-40785879
aacd3: hard error cmd=read 40785792-40785871
aacd1: hard error cmd=read 40785792-40785879
aacd1: hard error cmd=read 40790624-40790879
aacd3: hard error cmd=read 40790000-40790175
aacd1: hard error cmd=read 40799280-40799535
aacd3: hard error cmd=read 41121032-41121287
aacd1: hard error cmd=read 44290824-44290999
aacd1: hard error cmd=read 44301408-44301583
aacd1: hard error cmd=read 44315680-44315935
aacd1: hard error cmd=read 44330184-44330359
aacd1: hard error cmd=read 44337224-44337399
aacd1: hard error cmd=read 44344472-44344727
aacd1: hard error cmd=read 51561672-51561927
aacd1: hard error cmd=read 51571528-51571783


Во время теста было много проблем с контроллером Adaptec RAID 2805, который падал с паникой на проблемных дисках, из-за чего, проверку запускал раз 6. Однако, проверка завершилась, и даже удалось не потерять данные.
Состояние пула 
  pool: zroot
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: scrub in progress since Fri Jan 26 16:05:37 2018
        67,0G scanned out of 101G at 71,3M/s, 0h8m to go
        2,63M repaired, 66,05% done
config:

        NAME         STATE     READ WRITE CKSUM
        zroot        DEGRADED     0     0     0
          raidz1-0   DEGRADED     0     0     0
            aacd0p4  ONLINE       0     0     0
            aacd1p4  FAULTED     40    93     7  too many errors  (repairing)
            aacd2p4  ONLINE       0     0     0
            aacd3p4  ONLINE       0     0     0

                        
  pool: zroot
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 4,54M in 0h18m with 0 errors on Fri Jan 26 16:45:35 2018
config:

        NAME         STATE     READ WRITE CKSUM
        zroot        ONLINE       0     0     0
          raidz1-0   ONLINE       0     0     0
            aacd0p4  ONLINE       0     0     0
            aacd1p4  ONLINE       1     0    11
            aacd2p4  ONLINE       0     0     0
            aacd3p4  ONLINE       0     0     2

errors: No known data errors


Для вторичного подтверждения целостности использовал torrent, который не нашел ошибок в тестовом наборе.

Итоги


ZFS показал себя хорошо, даже в заведомо плохих условиях. Против ZFS работали ушатанные диски и контроллер, прошивка которого падала из-за проблем на дисках. Что примечательно, запись на диски шла очень быстро (загружал весь гигабит сетевухи), но вот чтение, и с ошибками заняло очень много времени. Что с этим делать, решайте сами ;)

Let's block ads! (Why?)

на
Ярлыки: ,

Комментариев нет:

Отправить комментарий

Подписаться на: Комментарии к сообщению (Atom)