Ubuntu sw-raid 5 med kun 15MB/s ytelse

[m0g1e]

Har hatt problemer med mitt RAID5-array fra i litt over et år nå på min Ubuntu 9.10 server (2.6.31 kernel) som yter kun 15-25MB/s, testet på forskjellige kontrollere og hovedkort.

 

Dette er reell ytelse over samba. Og fra enkeltdisken(systemdisken) på serveren er overføringshastigheten oppi 50-60MB/s som jeg anser som normalt.

 

Har testet på s775 med Q6600 (nåværende), s939 med [email protected], og tilslutt et sw-raid PCI-E SAS til S-ATA kontroller kort, men det er ingen forskjell på disse.

 

Reinstallerte serveren og nylig fra Ubuntu 8.04 til 9.10. SW-raidet ble montert automatisk, og overføringshastigheten var den samme som før.

 

Det ser derfor ut til at dette har noe med hvordan arrayet er satt opp? har XFS filsystem på arrayet, men tviler dette har noe å si. Kan godt skrive noe ouput om arrayet for å kanskje luke ut om det er her feilen ligger...

 

 

Jeg får ganske tilsvarende lese/skriveresultater med hdparm -tT /dev/md1 til overføringshastighetene på samba også. Har testet en haug med samba-innstillinger, men problemet ligger nok i arrayet på en eller annen måte vil jeg tro.

 

Takker for alle bidrag

Endret 10. desember 2009 av m0g1e

[m0g1e]

Anyone?

[HawP]

Ingen spesielle "super-forslag", nei, men du har (selvfølgelig) sjekka de åpenbare tinga, f.eks. at arrayet ikke er "degraded", at en eller flere disker ikke har lese- og/eller skrivevansker, at dmesg/system logger ikke logger noe spesielt for arrayet, osv... ?

Kan du ha vært uheldig med stride når filsystemet ble laget (har ingen erfaring med XFS)? Selv om jeg har vanskelig for å tro at det ville utgjøre en så stor forskjell ...

Mounter du med noatime, hvis det er relevant for XFS ? Igjen: jeg har også vanskelig for å tro at dette hadde gitt så stort utslag ...

[Varj]

legg bort feilsøk via samba i første omgang, da vi ikke ønsker å blande nettverk og samba inn i feilsøkingen.

 

hdparm er en fin start, men ingen erstatning for bonnie++ og andre slike verktøy. prøv gjerne også å opprette en gigantisk fil med dd, og les den igjen etterpå.

 

dd if=/dev/zero of=/tmp/test.fil bs=1024 count=X

(hvor x styrer størrelsen på fila du ønsker å opprette.)

 

deretter kan du f.eks lese den med:

dd if=/tmp/test.fil of=/dev/null bs=1024

 

størrelsen bør være en god del større enn du har minnekapasitet, så slipper vi alt for mye cache-problematikk.

[Ueland]

Hvilke hastigheter får du internt på systemet?

 

Sitat fra oppsettet hos meg:

/dev/md0:

Timing cached reads: 1574 MB in 2.00 seconds = 787.31 MB/sec

Timing buffered disk reads: 380 MB in 3.03 seconds = 125.60 MB/sec

 

/dev/md1:

Timing cached reads: 1290 MB in 2.00 seconds = 645.28 MB/sec

Timing buffered disk reads: 440 MB in 3.01 seconds = 146.12 MB/sec

 

På mitt oppsett har jeg fordelt diskene med: 1@HK, 2@PCI-Express 4 - kontroller 1, 2@PCI-Express 4 - kontroller 2. Dette for å få ytelsen opp mest mulig "gratis".

 

Hvor stor hastighet har du på bussen til kontrolleren? Er bussen global for alle porter, eller per port?

[m0g1e]

Først, takk for alle svar

 

f.eks. at arrayet ikke er "degraded",....noatime,

 

Har ikke definert noe spesielt i /etc/fstab for montering:

/mnt/tera	   xfs rw 0 0

 

Finner ikke noe spesielt i dmesg angående raidet, men er ingen skarping på å tyde disse loggene. Hvordan sjekker jeg med garanti at arrayet ikke er degraded?

 

Her er en bonnie++ logg : [bonnie++ -n 0 -f -b -u root] (har prøvd å rydde loggen litt men det er vanskelig)

 

Version 1.03c	   ------Sequential Output------ --Sequential Input- --Random-
				-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine		Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
serverhost  8G		   56014   8   24992   3		   59786   4 119.2   0
serverhost ,8G		,,,56014,  8,  24992  ,3		,,,59786  ,4,119.2,  0,,,,,,,,,,,,,

 

noen dd tester: (har for øvrig 4GB RAM)

på systemdisken (ikke raidet):

skriving: 10240000000 bytes (10 GB) copied, 162,941 s, 62,8 MB/s

lesing: 10240000000 bytes (10 GB) copied, 171,19 s, 59,8 MB/s

 

raidet:

Skriving: 10240000000 bytes (10 GB) copied, 91,8036 s, 112 MB/s

lesing: 10240000000 bytes (10 GB) copied, 239,686 s, 42,7 MB/s

 

Testet på HK-kontroller. Dette er tilsvarende samme resultater som på annet HK også.

 

Skal kjøre dd test m PCI-E kontroller kortet også nå..

[m0g1e]

Her er litt dmesg| grep md1:

[   11.652504] raid5: allocated 3228kB for md1
[   11.652579] raid5: raid level 5 set md1 active with 3 out of 3 devices, algorithm 2
[   11.652615] md1: detected capacity change from 0 to 2000404348928
[   11.653674]  md1: unknown partition table
[   12.046361] Filesystem "md1": Disabling barriers, trial barrier write failed
[   12.046648] XFS mounting filesystem md1
[   12.342523] Ending clean XFS mount for filesystem: md1

 

Forsøkte en ny dd-test nå på PCI-E kontrolleren. Fikk opp meldinger som jeg har sett i dmesg før, men som jeg ikke har klart å tyde. Hele serveren låste seg, uten keyboard respons.

 

Nå fungerer tydeligvis ikke RAIDET helt heller, siden det ikke blir mountet. Trenger litt hjelp med å tyde dette. Kan tyde på noe driverproblematikk? Er usikker på dette da raidet fungerte fint i går uten at jeg har gjort noen endringer i konfigurasjoner eller noe.

Klikk for å se/fjerne innholdet nedenfor

[ 1.155357] sr0: scsi3-mmc drive: 40x/40x writer cd/rw xa/form2 cdda tray

[ 1.155360] Uniform CD-ROM driver Revision: 3.20

[ 1.155417] sr 4:0:0:0: Attached scsi CD-ROM sr0

[ 1.155448] sr 4:0:0:0: Attached scsi generic sg2 type 5

[ 1.159025] sdb1 sdb2 sdb3

[ 1.164458] sda1 sda2

[ 1.172014] sd 1:0:0:0: [sdb] Attached SCSI disk

[ 1.172368] sd 0:0:0:0: [sda] Attached SCSI disk

[ 1.321196] Freeing unused kernel memory: 660k freed

[ 1.321371] Write protecting the kernel read-only data: 7572k

[ 1.469890] mvsas 0000:01:00.0: mvsas: driver version 0.8.2

[ 1.469907] mvsas 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16

[ 1.469912] mvsas 0000:01:00.0: setting latency timer to 64

[ 1.472043] mvsas 0000:01:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps

[ 1.478373] md: linear personality registered for level -1

[ 1.518671] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded

[ 1.518695] r8169 0000:04:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17

[ 1.518754] r8169 0000:04:00.0: setting latency timer to 64

[ 1.518813] alloc irq_desc for 28 on node 0

[ 1.518815] alloc kstat_irqs on node 0

[ 1.518830] r8169 0000:04:00.0: irq 28 for MSI/MSI-X

[ 1.519345] eth0: RTL8168b/8111b at 0xffffc9000065c000, 00:19:db:f2:0e:36, XI D 38000000 IRQ 28

[ 1.535903] md: multipath personality registered for level -4

[ 1.538834] md: raid0 personality registered for level 0

[ 1.542222] md: raid1 personality registered for level 1

[ 1.543368] xor: automatically using best checksumming function: generic_sse

[ 1.589997] generic_sse: 8783.200 MB/sec

[ 1.589999] xor: using function: generic_sse (8783.200 MB/sec)

[ 1.590590] async_tx: api initialized (async)

[ 1.760004] raid6: int64x1 1901 MB/s

[ 1.929999] raid6: int64x2 2559 MB/s

[ 2.099996] raid6: int64x4 1817 MB/s

[ 2.270021] raid6: int64x8 1644 MB/s

[ 2.440004] raid6: sse2x1 4275 MB/s

[ 2.609999] raid6: sse2x2 4576 MB/s

[ 2.780003] raid6: sse2x4 7246 MB/s

[ 2.780005] raid6: using algorithm sse2x4 (7246 MB/s)

[ 2.782250] md: raid6 personality registered for level 6

[ 2.782253] md: raid5 personality registered for level 5

[ 2.782254] md: raid4 personality registered for level 4

[ 2.786800] md: raid10 personality registered for level 10

[ 5.840008] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 0 attach dev info is 0

[ 5.840011] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 0 attach sas addr is 0

[ 6.050022] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 1 attach dev info is 10000000

[ 6.050025] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 1 attach sas addr is 1

[ 6.260020] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 2 attach dev info is 20000

[ 6.260023] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 2 attach sas addr is 2

[ 6.470021] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 3 attach dev info is 0

[ 6.470023] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 3 attach sas addr is 3

[ 6.580006] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 4 attach dev info is 0

[ 6.580009] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 4 attach sas addr is 0

[ 6.690006] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 5 attach dev info is 0

[ 6.690008] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 5 attach sas addr is 0

[ 6.800006] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 6 attach dev info is 0

[ 6.800008] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 6 attach sas addr is 0

[ 6.910005] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 7 attach dev info is 0

[ 6.910007] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 7 attach sas addr is 0

[ 6.910012] scsi6 : mvsas

[ 6.910478] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 1 byte dmaded.

[ 6.910482] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 2 byte dmaded.

[ 6.910485] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 3 byte dmaded.

[ 6.911708] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[0:5] is gone.

[ 6.913261] ata7.00: ATA-8: ST31000340AS, SD15, max UDMA/133

[ 6.913264] ata7.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32)

[ 6.915259] ata7.00: configured for UDMA/133

[ 6.915305] scsi 6:0:0:0: Direct-Access ATA ST31000340AS SD15 PQ : 0 ANSI: 5

[ 6.916479] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[1:5] is gone.

[ 6.918056] ata8.00: ATA-8: ST31000340AS, SD15, max UDMA/133

[ 6.918059] ata8.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32)

[ 6.920073] ata8.00: configured for UDMA/133

[ 6.920111] scsi 6:0:1:0: Direct-Access ATA ST31000340AS SD15 PQ : 0 ANSI: 5

[ 6.921177] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[2:5] is gone.

[ 6.922678] ata9.00: ATA-8: ST31000340AS, SD15, max UDMA/133

[ 6.922680] ata9.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32)

[ 6.924588] ata9.00: configured for UDMA/133

[ 6.924624] scsi 6:0:2:0: Direct-Access ATA ST31000340AS SD15 PQ : 0 ANSI: 5

[ 6.946816] PM: Starting manual resume from disk

[ 6.946819] PM: Resume from partition 8:2

[ 6.946820] PM: Checking hibernation image.

[ 6.947009] PM: Resume from disk failed.

[ 6.949345] EXT4-fs (sdb1): INFO: recovery required on readonly filesystem

[ 6.949349] EXT4-fs (sdb1): write access will be enabled during recovery

[ 6.969490] EXT4-fs (sdb1): barriers enabled

[ 7.058486] kjournald2 starting: pid 401, dev sdb1:8, commit interval 5 secon ds

[ 7.058502] EXT4-fs (sdb1): delayed allocation enabled

[ 7.058505] EXT4-fs: file extents enabled

[ 7.060109] EXT4-fs: mballoc enabled

[ 7.060124] EXT4-fs (sdb1): orphan cleanup on readonly fs

[ 7.060130] EXT4-fs (sdb1): ext4_orphan_cleanup: deleting unreferenced inode 155090

[ 7.060159] EXT4-fs (sdb1): ext4_orphan_cleanup: deleting unreferenced inode 155088

[ 7.060168] EXT4-fs (sdb1): ext4_orphan_cleanup: deleting unreferenced inode 155087

[ 7.060175] EXT4-fs (sdb1): ext4_orphan_cleanup: deleting unreferenced inode 155086

[ 7.060182] EXT4-fs (sdb1): ext4_orphan_cleanup: deleting unreferenced inode 155054

[ 7.060190] EXT4-fs (sdb1): 5 orphan inodes deleted

[ 7.060192] EXT4-fs (sdb1): recovery complete

[ 7.359271] EXT4-fs (sdb1): mounted filesystem with ordered data mode

[ 7.911399] sd 6:0:0:0: Attached scsi generic sg3 type 0

[ 7.911484] sd 6:0:1:0: Attached scsi generic sg4 type 0

[ 7.911568] sd 6:0:2:0: Attached scsi generic sg5 type 0

[ 7.912736] sd 6:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/93 1 GiB)

[ 7.912774] sd 6:0:0:0: [sdc] Write Protect is off

[ 7.912777] sd 6:0:0:0: [sdc] Mode Sense: 00 3a 00 00

[ 7.912797] sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, does n't support DPO or FUA

[ 7.912911] sdc:

[ 7.913725] sd 6:0:1:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/93 1 GiB)

[ 7.913761] sd 6:0:1:0: [sdd] Write Protect is off

[ 7.913764] sd 6:0:1:0: [sdd] Mode Sense: 00 3a 00 00

[ 7.913783] sd 6:0:1:0: [sdd] Write cache: enabled, read cache: enabled, does n't support DPO or FUA

[ 7.913875] sdd:

[ 7.914733] sd 6:0:2:0: [sde] 1953525168 512-byte logical blocks: (1.00 TB/93 1 GiB)

[ 7.914845] sd 6:0:2:0: [sde] Write Protect is off

[ 7.914848] sd 6:0:2:0: [sde] Mode Sense: 00 3a 00 00

[ 7.914868] sd 6:0:2:0: [sde] Write cache: enabled, read cache: enabled, does n't support DPO or FUA

[ 7.914966] sde: sde1

[ 7.938540] sdc1

[ 7.938555] sd 6:0:2:0: [sde] Attached SCSI disk

[ 7.938698] sd 6:0:0:0: [sdc] Attached SCSI disk

[ 7.943366] sdd1

[ 7.943530] sd 6:0:1:0: [sdd] Attached SCSI disk

[ 8.429289] Adding 3903784k swap on /dev/sda2. Priority:-1 extents:1 across: 3903784k

[ 8.857651] EXT4-fs (sdb1): internal journal on sdb1:8

[ 8.860051] udev: starting version 147

[ 9.219966] udev: renamed network interface eth0 to eth2

[ 9.933269] lp: driver loaded but no devices found

[ 10.196874] s3fb 0000:05:01.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17

[ 10.198552] fb0: S3 Trio64V2/DX on 0000:05:01.0, 2 MB RAM, 60 MHz MCLK

[ 10.198558] mtrr: type mismatch for f8000000,200000 old: write-back new: writ e-combining

[ 10.482803] Console: switching to colour frame buffer device 80x30

[ 10.561350] r8169: eth2: link up

[ 10.561356] r8169: eth2: link up

[ 11.237375] md: bind<sde1>

[ 11.240621] md: bind<sdd1>

[ 11.441410] md: bind<sdc1>

[ 11.478227] raid5: md1 is not clean -- starting background reconstruction

[ 11.478232] raid5: device sdc1 operational as raid disk 2

[ 11.478234] raid5: device sdd1 operational as raid disk 1

[ 11.478235] raid5: device sde1 operational as raid disk 0

[ 11.478547] raid5: allocated 3228kB for md1

[ 11.478632] raid5: raid level 5 set md1 active with 3 out of 3 devices, algor ithm 2

[ 11.478634] RAID5 conf printout:

[ 11.478636] --- rd:3 wd:3

[ 11.478638] disk 0, o:1, dev:sde1

[ 11.478639] disk 1, o:1, dev:sdd1

[ 11.478641] disk 2, o:1, dev:sdc1

[ 11.478668] md1: detected capacity change from 0 to 2000404348928

[ 11.478994] md: resync of RAID array md1

[ 11.478997] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.

[ 11.478999] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.

[ 11.479003] md: using 128k window, over a total of 976759936 blocks.

[ 11.480094] md1: unknown partition table

[ 11.837626] SGI XFS with ACLs, security attributes, realtime, large block/ino de numbers, no debug enabled

[ 11.839319] SGI XFS Quota Management subsystem

[ 11.840145] Filesystem "md1": Disabling barriers, trial barrier write failed

[ 11.869486] XFS mounting filesystem md1

[ 21.491256] eth2: no IPv6 routers present

[ 43.040134] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_a bort_task:rc= 5

[ 43.040137] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_q uery_task:rc= 5

[ 43.040167] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_a bort_task:rc= 5

[ 43.040169] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_q uery_task:rc= 5

[ 43.040191] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_a bort_task:rc= 5

[ 43.040193] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_q uery_task:rc= 5

[ 74.040061] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_a bort_task:rc= 5

[ 74.040065] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_q uery_task:rc= 5

[ 74.040093] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_a bort_task:rc= 5

[ 74.040095] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_q uery_task:rc= 5

[ 74.040114] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_a bort_task:rc= 5

[ 74.040116] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_q

Endret 11. desember 2009 av m0g1e

[Ueland]

Legg ut hele dmesg her, i spoiler.

 

Du sjekker status på RAIDet med mdadm --detail /dev/mdX

 

Da får du se noe a-la:

 

/dev/md0:

Version : 00.90

Creation Time : Sat Jan 24 14:16:52 2009

Raid Level : raid5

Array Size : 5860553728 (5589.06 GiB 6001.21 GB)

Used Dev Size : 1465138432 (1397.26 GiB 1500.30 GB)

Raid Devices : 5

Total Devices : 5

Preferred Minor : 0

Persistence : Superblock is persistent

 

Update Time : Fri Dec 11 23:10:39 2009

State : clean

Active Devices : 5

Working Devices : 5

Failed Devices : 0

Spare Devices : 0

 

Layout : left-symmetric

Chunk Size : 128K

 

UUID : 1f095377:1c62a5f4:352a6ad4:582f9bd3 (local to host bais)

Events : 0.309124

 

Number Major Minor RaidDevice State

0 8 48 0 active sync /dev/sdd

1 8 64 1 active sync /dev/sde

2 8 176 2 active sync /dev/sdl

3 8 80 3 active sync /dev/sdf

4 8 128 4 active sync /dev/sdi

 

State = status på RAIDet.

[m0g1e]

/dev/md1:

Version : 00.90

Creation Time : Wed Feb 18 17:17:21 2009

Raid Level : raid5

Array Size : 1953519872 (1863.02 GiB 2000.40 GB)

Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)

Raid Devices : 3

Total Devices : 3

Preferred Minor : 1

Persistence : Superblock is persistent

 

Update Time : Fri Dec 11 23:41:37 2009

State : active, resyncing

Active Devices : 3

Working Devices : 3

Failed Devices : 0

Spare Devices : 0

 

Layout : left-symmetric

Chunk Size : 4K

 

Rebuild Status : 5% complete

 

UUID : 67ba08c5:1f23c5ac:8f867148:a67b3d53

Events : 0.96

 

Number Major Minor RaidDevice State

0 8 65 0 active sync /dev/sde1

1 8 49 1 active sync /dev/sdd1

2 8 33 2 active sync /dev/sdc1

 

Er raidet i rebuild?

 

 

dmsg viser en loop av noe her...

Har fortsatt PCI-E kontrolleren i bruk. RAID-volumet mountet seg ved siste reboot.

Endret 11. desember 2009 av m0g1e

[m0g1e]

Klikk for å se/fjerne innholdet nedenfor

[ 380.010618] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 380.010621] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 380.010624] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 380.010627] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 380.010630] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 380.010633] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 380.010636] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 380.010639] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 380.010641] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 380.010644] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 380.010647] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 380.010650] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 380.010653] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 380.010655] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 380.010658] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 380.010661] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 380.010664] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 380.010666] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 380.010669] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 380.010672] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 380.010674] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 380.010676] ---[ end trace 65004f6f8b820583 ]---

[ 380.010677] ------------[ cut here ]------------

[ 380.010679] WARNING: at /build/buildd/linux-2.6.31/fs/quota/dquot.c:964 dquot_claim_space+0x15d/0x170()

[ 380.010681] Hardware name: MS-7360

[ 380.010682] Modules linked in: xt_multiport iptable_filter ip_tables x_tables xfs exportfs quota_v2 quota_tree fbcon tileblit font bitblit softcursor s3fb svgalib vgastate k8temp intel_agp lp parport raid

10 raid456 raid6_pq async_xor async_memcpy async_tx r8169 mii xor raid1 raid0 multipath linear mvsas libsas scsi_transport_sas

[ 380.010703] Pid: 32, comm: pdflush Tainted: G W 2.6.31-14-server #48-Ubuntu

[ 380.010704] Call Trace:

[ 380.010707] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0

[ 380.010710] [<ffffffff8105e65f>] warn_slowpath_null+0xf/0x20

[ 380.010713] [<ffffffff8116e5cd>] dquot_claim_space+0x15d/0x170

[ 380.010715] [<ffffffff811da786>] ext4_mb_mark_diskspace_used+0x356/0x3a0

[ 380.010718] [<ffffffff811daa79>] ext4_mb_new_blocks+0x2a9/0x540

[ 380.010721] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 380.010724] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 380.010727] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 380.010730] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 380.010733] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 380.010736] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 380.010739] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 380.010742] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 380.010744] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 380.010747] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 380.010750] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 380.010753] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 380.010756] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 380.010759] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 380.010761] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 380.010764] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 380.010767] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 380.010769] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 380.010772] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 380.010775] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 380.010777] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 380.010779] ---[ end trace 65004f6f8b820584 ]---

[ 410.010056] ------------[ cut here ]------------

[ 410.010066] WARNING: at /build/buildd/linux-2.6.31/fs/quota/dquot.c:964 dquot_claim_space+0x147/0x170()

[ 410.010069] Hardware name: MS-7360

[ 410.010070] Modules linked in: xt_multiport iptable_filter ip_tables x_tables xfs exportfs quota_v2 quota_tree fbcon tileblit font bitblit softcursor s3fb svgalib vgastate k8temp intel_agp lp parport raid

10 raid456 raid6_pq async_xor async_memcpy async_tx r8169 mii xor raid1 raid0 multipath linear mvsas libsas scsi_transport_sas

[ 410.010098] Pid: 32, comm: pdflush Tainted: G W 2.6.31-14-server #48-Ubuntu

[ 410.010100] Call Trace:

[ 410.010106] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0

[ 410.010109] [<ffffffff8105e65f>] warn_slowpath_null+0xf/0x20

[ 410.010112] [<ffffffff8116e5b7>] dquot_claim_space+0x147/0x170

[ 410.010116] [<ffffffff811da786>] ext4_mb_mark_diskspace_used+0x356/0x3a0

[ 410.010119] [<ffffffff811daa79>] ext4_mb_new_blocks+0x2a9/0x540

[ 410.010123] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 410.010126] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 410.010129] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 410.010132] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 410.010136] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 410.010138] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 410.010143] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 410.010146] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 410.010150] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 410.010153] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 410.010156] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 410.010159] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 410.010162] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 410.010165] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010167] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 410.010170] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 410.010173] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010176] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 410.010180] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 410.010183] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 410.010185] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 410.010187] ---[ end trace 65004f6f8b820585 ]---

[ 410.010188] ------------[ cut here ]------------

[ 410.010191] WARNING: at /build/buildd/linux-2.6.31/fs/quota/dquot.c:964 dquot_claim_space+0x15d/0x170()

[ 410.010193] Hardware name: MS-7360

[ 410.010194] Modules linked in: xt_multiport iptable_filter ip_tables x_tables xfs exportfs quota_v2 quota_tree fbcon tileblit font bitblit softcursor s3fb svgalib vgastate k8temp intel_agp lp parport raid

10 raid456 raid6_pq async_xor async_memcpy async_tx r8169 mii xor raid1 raid0 multipath linear mvsas libsas scsi_transport_sas

[ 410.010214] Pid: 32, comm: pdflush Tainted: G W 2.6.31-14-server #48-Ubuntu

[ 410.010216] Call Trace:

[ 410.010219] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0

[ 410.010222] [<ffffffff8105e65f>] warn_slowpath_null+0xf/0x20

[ 410.010224] [<ffffffff8116e5cd>] dquot_claim_space+0x15d/0x170

[ 410.010227] [<ffffffff811da786>] ext4_mb_mark_diskspace_used+0x356/0x3a0

[ 410.010230] [<ffffffff811daa79>] ext4_mb_new_blocks+0x2a9/0x540

[ 410.010233] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 410.010236] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 410.010239] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 410.010241] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 410.010245] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 410.010247] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 410.010251] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 410.010253] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 410.010256] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 410.010259] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 410.010262] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 410.010265] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 410.010267] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 410.010270] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010273] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 410.010275] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 410.010278] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010281] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 410.010283] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 410.010286] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 410.010289] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 410.010290] ---[ end trace 65004f6f8b820586 ]---

[ 410.010343] ------------[ cut here ]------------

[ 410.010346] WARNING: at /build/buildd/linux-2.6.31/fs/quota/dquot.c:964 dquot_claim_space+0x147/0x170()

[ 410.010348] Hardware name: MS-7360

[ 410.010349] Modules linked in: xt_multiport iptable_filter ip_tables x_tables xfs exportfs quota_v2 quota_tree fbcon tileblit font bitblit softcursor s3fb svgalib vgastate k8temp intel_agp lp parport raid

10 raid456 raid6_pq async_xor async_memcpy async_tx r8169 mii xor raid1 raid0 multipath linear mvsas libsas scsi_transport_sas

[ 410.010370] Pid: 32, comm: pdflush Tainted: G W 2.6.31-14-server #48-Ubuntu

[ 410.010371] Call Trace:

[ 410.010375] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0

[ 410.010378] [<ffffffff8105e65f>] warn_slowpath_null+0xf/0x20

[ 410.010381] [<ffffffff8116e5b7>] dquot_claim_space+0x147/0x170

[ 410.010383] [<ffffffff811da786>] ext4_mb_mark_diskspace_used+0x356/0x3a0

[ 410.010386] [<ffffffff811daa79>] ext4_mb_new_blocks+0x2a9/0x540

[ 410.010389] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 410.010392] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 410.010395] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 410.010398] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 410.010401] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 410.010404] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 410.010407] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 410.010410] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 410.010413] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 410.010415] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 410.010418] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 410.010421] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 410.010424] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 410.010427] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010429] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 410.010432] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 410.010435] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010437] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 410.010440] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 410.010443] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 410.010445] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 410.010447] ---[ end trace 65004f6f8b820587 ]---

[ 410.010448] ------------[ cut here ]------------

[ 410.010451] WARNING: at /build/buildd/linux-2.6.31/fs/quota/dquot.c:964 dquot_claim_space+0x15d/0x170()

[ 410.010452] Hardware name: MS-7360

[ 410.010454] Modules linked in: xt_multiport iptable_filter ip_tables x_tables xfs exportfs quota_v2 quota_tree fbcon tileblit font bitblit softcursor s3fb svgalib vgastate k8temp intel_agp lp parport raid

10 raid456 raid6_pq async_xor async_memcpy async_tx r8169 mii xor raid1 raid0 multipath linear mvsas libsas scsi_transport_sas

[ 410.010474] Pid: 32, comm: pdflush Tainted: G W 2.6.31-14-server #48-Ubuntu

[ 410.010476] Call Trace:

[ 410.010479] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0

[ 410.010482] [<ffffffff8105e65f>] warn_slowpath_null+0xf/0x20

[ 410.010484] [<ffffffff8116e5cd>] dquot_claim_space+0x15d/0x170

[ 410.010487] [<ffffffff811da786>] ext4_mb_mark_diskspace_used+0x356/0x3a0

[ 410.010490] [<ffffffff811daa79>] ext4_mb_new_blocks+0x2a9/0x540

[ 410.010493] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 410.010496] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 410.010499] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 410.010501] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 410.010504] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 410.010507] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 410.010510] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 410.010513] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 410.010516] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 410.010519] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 410.010522] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 410.010524] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 410.010527] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 410.010530] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010533] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 410.010535] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 410.010538] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010541] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 410.010543] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 410.010546] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 410.010548] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 410.010550] ---[ end trace 65004f6f8b820588 ]---

[ 410.010578] ------------[ cut here ]------------

[ 410.010581] WARNING: at /build/buildd/linux-2.6.31/fs/quota/dquot.c:964 dquot_claim_space+0x147/0x170()

[ 410.010583] Hardware name: MS-7360

[ 410.010584] Modules linked in: xt_multiport iptable_filter ip_tables x_tables xfs exportfs quota_v2 quota_tree fbcon tileblit font bitblit softcursor s3fb svgalib vgastate k8temp intel_agp lp parport raid

10 raid456 raid6_pq async_xor async_memcpy async_tx r8169 mii xor raid1 raid0 multipath linear mvsas libsas scsi_transport_sas

[ 410.010605] Pid: 32, comm: pdflush Tainted: G W 2.6.31-14-server #48-Ubuntu

[ 410.010606] Call Trace:

[ 410.010609] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0

[ 410.010612] [<ffffffff8105e65f>] warn_slowpath_null+0xf/0x20

[ 410.010615] [<ffffffff8116e5b7>] dquot_claim_space+0x147/0x170

[ 410.010618] [<ffffffff811da786>] ext4_mb_mark_diskspace_used+0x356/0x3a0

[ 410.010621] [<ffffffff811daa79>] ext4_mb_new_blocks+0x2a9/0x540

[ 410.010624] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 410.010627] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 410.010630] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 410.010632] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 410.010635] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 410.010638] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 410.010641] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 410.010644] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 410.010647] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 410.010650] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 410.010653] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 410.010655] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 410.010658] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 410.010661] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010664] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 410.010667] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 410.010669] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010672] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 410.010674] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 410.010677] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 410.010680] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 410.010681] ---[ end trace 65004f6f8b820589 ]---

[ 410.010682] ------------[ cut here ]------------

[ 410.010685] WARNING: at /build/buildd/linux-2.6.31/fs/quota/dquot.c:964 dquot_claim_space+0x15d/0x170()

[ 410.010687] Hardware name: MS-7360

[ 410.010688] Modules linked in: xt_multiport iptable_filter ip_tables x_tables xfs exportfs quota_v2 quota_tree fbcon tileblit font bitblit softcursor s3fb svgalib vgastate k8temp intel_agp lp parport raid

10 raid456 raid6_pq async_xor async_memcpy async_tx r8169 mii xor raid1 raid0 multipath linear mvsas libsas scsi_transport_sas

[ 410.010708] Pid: 32, comm: pdflush Tainted: G W 2.6.31-14-server #48-Ubuntu

[ 410.010710] Call Trace:

[ 410.010713] [<ffffffff8105e618>] warn_slowpath_common+0x78/0xb0

[ 410.010716] [<ffffffff8105e65f>] warn_slowpath_null+0xf/0x20

[ 410.010718] [<ffffffff8116e5cd>] dquot_claim_space+0x15d/0x170

[ 410.010721] [<ffffffff811da786>] ext4_mb_mark_diskspace_used+0x356/0x3a0

[ 410.010724] [<ffffffff811daa79>] ext4_mb_new_blocks+0x2a9/0x540

[ 410.010727] [<ffffffff811d02a0>] ? ext4_ext_find_extent+0x130/0x2f0

[ 410.010730] [<ffffffff811d2124>] ext4_ext_get_blocks+0x4a4/0x5b0

[ 410.010733] [<ffffffff811b2f79>] ext4_get_blocks+0x1d9/0x210

[ 410.010735] [<ffffffff811b3957>] mpage_da_map_blocks+0xa7/0x370

[ 410.010739] [<ffffffff811e949e>] ? jbd2_journal_start+0xae/0x100

[ 410.010741] [<ffffffff811b3ec3>] ext4_da_writepages+0x2a3/0x500

[ 410.010745] [<ffffffff810e30f8>] do_writepages+0x28/0x50

[ 410.010747] [<ffffffff8113e2cc>] writeback_single_inode+0x1bc/0x450

[ 410.010750] [<ffffffff8106ad60>] ? process_timeout+0x0/0x10

[ 410.010753] [<ffffffff8113ebe8>] generic_sync_sb_inodes+0x418/0x530

[ 410.010756] [<ffffffff8113ee0b>] writeback_inodes+0x5b/0x100

[ 410.010759] [<ffffffff810e1e8c>] wb_kupdate+0xbc/0x140

[ 410.010762] [<ffffffff810e38fe>] __pdflush+0x13e/0x260

[ 410.010764] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010767] [<ffffffff810e3a68>] pdflush+0x48/0x50

[ 410.010770] [<ffffffff810e1dd0>] ? wb_kupdate+0x0/0x140

[ 410.010772] [<ffffffff810e3a20>] ? pdflush+0x0/0x50

[ 410.010775] [<ffffffff81078236>] kthread+0xa6/0xb0

[ 410.010778] [<ffffffff810130aa>] child_rip+0xa/0x20

[ 410.010780] [<ffffffff81078190>] ? kthread+0x0/0xb0

[ 410.010783] [<ffffffff810130a0>] ? child_rip+0x0/0x20

[ 410.010784] ---[ end trace 65004f6f8b82058a ]---

[Ueland]

RAIDet ditt er i resync da, det skjer hver gang strømmen går/OSet dør hardt.

 

Når det gjelder de feilmeldingene du får så er de nye for meg i farten, men det er absolutt ikke e et godt tegn. Ut fra Feilmeldingene er det ext4 som er inne i bildet her. Og har filsystemet fått seg en knekk kan du fort gå ut fra at dataene ikke "har det bra lengre"...

 

Noe spesielt du gjør for å trigge feilen?

[HawP]

cat /proc/mdstat

 

gir deg også status (inkl. evt. rebuild info) for arrayet. Du vil se [uUU], antall U avhenger av antall disker i arrayet og U = disk Up; mens _ = disk "mangler" (dvs. har blitt tatt ut av arrayet av en eller annen årsak, f.eks. pga den feilet).

Endret 12. desember 2009 av HawP

[m0g1e]

RAIDet ditt er i resync da, det skjer hver gang strømmen går/OSet dør hardt.

 

Når det gjelder de feilmeldingene du får så er de nye for meg i farten, men det er absolutt ikke e et godt tegn. Ut fra Feilmeldingene er det ext4 som er inne i bildet her. Og har filsystemet fått seg en knekk kan du fort gå ut fra at dataene ikke "har det bra lengre"...

 

Noe spesielt du gjør for å trigge feilen?

 

Raidet kjører XFS. Systemet ext4. Så feilen kan godt være relatert til systemdisken eller programmer som kjører her. Vet ext4 er rimelig nytt.

 

Det har hendt flere ganger at jeg har gjort poweroff uten at resync har klart å fullføre. Kanskje problematikken til overføringshastigheten har noe med at Raidet stadig er i resync når jeg tester det? Likevel ut ifra de resultatene jeg har fått det siste året tror jeg ikke resync er problemet.

[Ueland]

Slår du av systemet er før resync er ferdig starter den helt fra scratch, ikke veldig optimalt å gjøre det hele tiden.

 

Og ja, det vil merkes på ytelsen.

[m0g1e]

Eureka!

State : clean, degraded

 

Dette betyr i klartekst at det er feil på ene disken som ikke kan repareres?

 

 

Version : 00.90

Creation Time : Wed Feb 18 17:17:21 2009

Raid Level : raid5

Array Size : 1953519872 (1863.02 GiB 2000.40 GB)

Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)

Raid Devices : 3

Total Devices : 3

Preferred Minor : 1

Persistence : Superblock is persistent

 

Update Time : Sun Dec 13 06:51:17 2009

State : clean, degraded

Active Devices : 2

Working Devices : 2

Failed Devices : 1

Spare Devices : 0

 

Layout : left-symmetric

Chunk Size : 4K

 

UUID : 67ba08c5:1f23c5ac:8f867148:a67b3d53

Events : 0.121

 

Number Major Minor RaidDevice State

0 8 65 0 active sync /dev/sde1

1 8 49 1 active sync /dev/sdd1

2 0 0 2 removed

 

3 8 33 - faulty spare /dev/sdc1

 

Problemet nå er å finne ut "hvilken" av diskene dette er da de alle er like..

Hvordan får jeg output til nummer på S-ATA porter osv?

 

Eller behøver det ikke være feil på disken? Er det noen god måte å sjekke dette utenom å restarte server og kjøre diagnose-CD som UBCD?

Endret 13. desember 2009 av m0g1e

[Admin'c]

Number Major Minor RaidDevice State

0 8 65 0 active sync /dev/sde1

1 8 49 1 active sync /dev/sdd1

2 0 0 2 removed

 

3 8 33 - faulty spare /dev/sdc1

 

Sjekk at all info er på de to overlevende diskene, det ser ut til at de kjører. Og rebuild hot sparen

 

Triks til senere.

1. konfigurer mdadm i etc til å sende mail når arrayet sier poff

2. bruk smartmontools til å overvåke og av og til sjekke status på diskene manuelt (du kan få opp hvilken disker som er dauue ved smartctl /dev/sd* der * er disken du skal sjekke. F.eks. smartctl -a /dev/sda

3. backup backup backup. raid er ikke backup, men en metode for å få mer oppetid ettersom man kan skifte disker underveis mens serveren kjører

4. har du viktige ting på serveren bruker du ikke ubuntu server ol, bruk noe som er stabilt. Heller for stabilt enn for cutting edge. Det er ytterst sjelden du trenger de nyeste pakkene på en server.

5. planlegg før du setter opp en server slik at du kan feilsøke lettere og kjappere. Merking av id på diskene er kjekt f.eks. så slipper du å ta dem ut for å se på oversiden av dem.

6. For ekstra logsikkerhet når du ikke klarer å lese logger er programmer som logwatch praktiske.

Endret 13. desember 2009 av Admin'c

[m0g1e]

Number Major Minor RaidDevice State

0 8 65 0 active sync /dev/sde1

1 8 49 1 active sync /dev/sdd1

2 0 0 2 removed

 

3 8 33 - faulty spare /dev/sdc1

 

Sjekk at all info er på de to overlevende diskene, det ser ut til at de kjører. Og rebuild hot sparen

 

Triks til senere.

1. konfigurer mdadm i etc til å sende mail når arrayet sier poff

2. bruk smartmontools til å overvåke og av og til sjekke status på diskene manuelt (du kan få opp hvilken disker som er dauue ved smartctl /dev/sd* der * er disken du skal sjekke. F.eks. smartctl -a /dev/sda

3. backup backup backup. raid er ikke backup, men en metode for å få mer oppetid ettersom man kan skifte disker underveis mens serveren kjører

4. har du viktige ting på serveren bruker du ikke ubuntu server ol, bruk noe som er stabilt. Heller for stabilt enn for cutting edge. Det er ytterst sjelden du trenger de nyeste pakkene på en server.

5. planlegg før du setter opp en server slik at du kan feilsøke lettere og kjappere. Merking av id på diskene er kjekt f.eks. så slipper du å ta dem ut for å se på oversiden av dem.

6. For ekstra logsikkerhet når du ikke klarer å lese logger er programmer som logwatch praktiske.

 

Har ingen ekstra harddisker foreløbig. Betyr dette at den siste disken er ødelagt? Eller kan jeg forsøke å rebuild? Jeg er helt blank på dette..

[HawP]

Du kan prøve en rebuild, selvfølgelig ingenting i veien for det. Men hvor mye stoler du på den disken, iallefall uten å ha sjekket den?

 

Du kan også sjekke dmesg (dmesg | grep sdc) om det er "skumle" meldinger ang. /dev/sdc.

 

Jeg hadde tidligere en disk i et mirror som stadig ble fjernet fra arrayet. dmesg viste da feilmeldinger for denne disken; seek error eller noe slikt mener jeg å huske.

[kpolberg]

Du kan forøvrig kjøre hdparm -I /dev/sdX for å liste ut serie nr på diskene.