Avis et retour sur utilisation de stockage NetAPP

jsajous26

Bonjour,

Nous envisageons et testons une infrastructure comprenant :

deux serveurs Hyperviseurs, avec des cartes supportant le 25Gbps en SFP+ DAC pour le stockage en iSCSI, branché en Direct-Attachement. 2 ports par hyperviseur.
une baie NetAPP AFF-A30

Après plusieurs tests, nous atteignons le débit d'environ ~2/3Gbps par VHD :

--- Lecture sequentielle (1MB blocks) ---
seq-read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.35
Starting 4 processes
seq-read: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=0): [f(4)][100.0%][r=1081MiB/s][r=1081 IOPS][eta 00m:00s]
seq-read: (groupid=0, jobs=4): err= 0: pid=3279: Thu Jan 29 08:45:04 2026
  read: IOPS=1714, BW=1714MiB/s (1798MB/s)(100GiB/60001msec)
    slat (usec): min=174, max=274642, avg=2118.43, stdev=2254.11
    clat (usec): min=3, max=340205, avg=72275.65, stdev=39502.46
     lat (msec): min=2, max=342, avg=74.39, stdev=39.57
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   61], 10.00th=[   62], 20.00th=[   63],
     | 30.00th=[   64], 40.00th=[   65], 50.00th=[   65], 60.00th=[   66],
     | 70.00th=[   67], 80.00th=[   68], 90.00th=[   71], 95.00th=[   78],
     | 99.00th=[  296], 99.50th=[  300], 99.90th=[  338], 99.95th=[  338],
     | 99.99th=[  338]
   bw (  MiB/s): min=  881, max= 2044, per=100.00%, avg=1718.80, stdev=77.73, samples=476
   iops        : min=  880, max= 2044, avg=1718.15, stdev=77.74, samples=476
  lat (usec)   : 4=0.01%, 10=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.05%, 100=96.65%
  lat (msec)   : 500=3.25%
  cpu          : usr=0.14%, sys=21.80%, ctx=554669, majf=0, minf=32823
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=102868,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1714MiB/s (1798MB/s), 1714MiB/s-1714MiB/s (1798MB/s-1798MB/s), io=100GiB (108GB), run=60001-60001msec

Disk stats (read/write):
    dm-6: ios=170272/127, merge=0/0, ticks=84737/61, in_queue=84798, util=80.03%, aggrios=661787/139, aggrmerge=2511/21, aggrticks=352428/91, aggrin_queue=352520, aggrutil=77.94%
  xvda: ios=661787/139, merge=2511/21, ticks=352428/91, in_queue=352520, util=77.94%

--- Ecriture sequentielle (1MB blocks) ---
seq-write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.35
Starting 4 processes
Jobs: 4 (f=4): [W(4)][100.0%][eta 00m:00s]
seq-write: (groupid=0, jobs=4): err= 0: pid=3319: Thu Jan 29 08:46:05 2026
  write: IOPS=1140, BW=1141MiB/s (1196MB/s)(68.0GiB/61049msec); 0 zone resets
    slat (usec): min=220, max=8635, avg=1058.26, stdev=1250.23
    clat (usec): min=7, max=3092.6k, avg=108622.83, stdev=431088.98
     lat (usec): min=470, max=3095.8k, avg=109681.09, stdev=431047.23
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   17], 20.00th=[   24],
     | 30.00th=[   26], 40.00th=[   29], 50.00th=[   31], 60.00th=[   33],
     | 70.00th=[   36], 80.00th=[   41], 90.00th=[   66], 95.00th=[   87],
     | 99.00th=[ 2635], 99.50th=[ 2735], 99.90th=[ 3037], 99.95th=[ 3071],
     | 99.99th=[ 3104]
   bw (  MiB/s): min=   54, max= 5504, per=100.00%, avg=2532.47, stdev=352.22, samples=220
   iops        : min=   54, max= 5504, avg=2532.45, stdev=352.21, samples=220
  lat (usec)   : 10=0.01%, 20=0.01%
  lat (msec)   : 10=6.40%, 20=8.05%, 50=72.33%, 100=9.71%, 250=0.48%
  lat (msec)   : 2000=0.01%, >=2000=3.03%
  cpu          : usr=1.55%, sys=26.90%, ctx=636361, majf=0, minf=48
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,69636,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1141MiB/s (1196MB/s), 1141MiB/s-1141MiB/s (1196MB/s-1196MB/s), io=68.0GiB (73.0GB), run=61049-61049msec

Disk stats (read/write):
    dm-6: ios=145/746281, merge=0/0, ticks=50/505091, in_queue=505141, util=67.51%, aggrios=163/1015432, aggrmerge=0/28896, aggrticks=58/597902, aggrin_queue=597960, aggrutil=56.54%
  xvda: ios=163/1015432, merge=0/28896, ticks=58/597902, in_queue=597960, util=56.54%

Est-ce une limitation côté hyperviseur (Xen) ?

Comment voyez-vous l'architecture avec une ou plusieurs baies NetAPP ?

Merci de votre réponse,
Cordialement.

olivierlambert

@jsajous26 said in Avis et retour sur utilisation de stockage NetAPP:

Bonjour,

1798MB/s en lecture, fait environ 15Gbits/s
1196MB/s en écriture fait environ 10Gbits/s

C'est tout à fait dans la norme pour un benchmark sur un seul disque. À noter que cela passe à l'échelle avec le nombre de VHD, ce qui fait que dans un contexte de production réaliste (des dizaines de VHD qui écrivent/lisent en même temps), on arrive à saturer un lien 25G.

Nous travaillons en parallèle sur 2 axes :

amélioration (multiqueue) de tapdisk pour la performance sur un seul disque
utilisation de l'équivalent de Vvols en parlant directement à la baie pour éviter des conversion de format et bénéficier des performances natives (ou proche) de la baie de stockage.

jsajous26

@olivierlambert

Je corrige le test FIO qui ne reflète pas la réalité. Le débit est d'environ 5.5Gbps/s

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=2452: Fri Feb 20 10:02:44 2026
  read: IOPS=10.3k, BW=646MiB/s (677MB/s)(75.7GiB/120002msec)
    slat (nsec): min=0, max=230384k, avg=183401.87, stdev=705796.53
    clat (usec): min=1227, max=252507, avg=11943.41, stdev=5670.46
     lat (usec): min=1357, max=252514, avg=12126.82, stdev=5722.82
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   15], 95.00th=[   16],
     | 99.00th=[   19], 99.50th=[   20], 99.90th=[   23], 99.95th=[  241],
     | 99.99th=[  247]
   bw (  KiB/s): min=347392, max=848640, per=100.00%, avg=661471.87, stdev=15968.62, samples=956
   iops        : min= 5428, max=13260, avg=10335.50, stdev=249.51, samples=956
  write: IOPS=10.3k, BW=646MiB/s (677MB/s)(75.7GiB/120002msec); 0 zone resets
    slat (nsec): min=0, max=230979k, avg=199173.27, stdev=642430.39
    clat (usec): min=1057, max=257260, avg=12446.58, stdev=5587.92
     lat (usec): min=1349, max=257596, avg=12645.76, stdev=5633.93
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   11], 20.00th=[   11],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   16], 95.00th=[   17],
     | 99.00th=[   20], 99.50th=[   21], 99.90th=[   24], 99.95th=[   40],
     | 99.99th=[  247]
   bw (  KiB/s): min=339968, max=829184, per=100.00%, avg=661765.89, stdev=15966.89, samples=956
   iops        : min= 5312, max=12956, avg=10340.09, stdev=249.48, samples=956
  lat (msec)   : 2=0.01%, 4=0.01%, 10=10.97%, 20=88.57%, 50=0.40%
  lat (msec)   : 250=0.05%, 500=0.01%
  cpu          : usr=1.26%, sys=13.37%, ctx=2934238, majf=0, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1239575,1240247,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=646MiB/s (677MB/s), 646MiB/s-646MiB/s (677MB/s-677MB/s), io=75.7GiB (81.2GB), run=120002-120002msec
  WRITE: bw=646MiB/s (677MB/s), 646MiB/s-646MiB/s (677MB/s-677MB/s), io=75.7GiB (81.3GB), run=120002-120002msec

Disk stats (read/write):
    dm-3: ios=1238448/1239080, merge=0/0, ticks=1874753/2464217, in_queue=4338970, util=99.99%, aggrios=2479429/2480865, aggrmerge=0/48, aggrticks=3199165/4352700, aggrin_queue=7551881, aggrutil=81.90%
  xvda: ios=2479429/2480865, merge=0/48, ticks=3199165/4352700, in_queue=7551881, util=81.90%

olivierlambert

Difficile de parler de « réalité » avec les benchmarks. Vérifiez aussi l'iodepth (qui dépend du type matériel que vous avez, sur du flash/NVMe vous pouvez monter à 128 ou 256), la latence entre en compte aussi bien sûr.

Le bottleneck principal est le monothreading de tapdisk, si vous testez sur plusieurs VMs différentes la somme va monter de manière assez régulière.