Docu review done: Mon 03 Jul 2023 16:34:19 CEST

Table of content

• Installation
• Testing IOPS
  • RW performance
  • Random read performance
  • Random write performance
  • Benchmark disk sample
  • Benchmark test db storage

Installation

$ apt install fio

Testing IOPS

RW performance

The first test is for measuring random read/write performances. In a terminal, execute the following command:

$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

During the test, the terminal window will display an output like the following one:

test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.2.8
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [0.1% done] [447KB/131KB/0KB /s] [111/32/0 iops] [eta 01h:Jobs: 1 (f=1): [m(1)] [0.1% done] [383KB/147KB/0KB /s] [95/36/0 iops] [eta 01h:4Jobs: 1 (f=1): [m(1)] [0.1% done] [456KB/184KB/0KB /s] [114/46/0 iops] [eta 01h:Jobs: 1 (f=1): [m(1)] [0.1% done] [624KB/188KB/0KB /s] [156/47/0 iops] [eta 01h:Jobs: 1 (f=1): [m(1)] [0.1% done] [443KB/115KB/0KB /s] [110/28/0 iops] [eta 01h:Jobs: 1 (f=1): [m(1)] [0.1% done] [515KB/95KB/0KB /s] [128/23/0 iops] [eta 01h:4Jobs: 1 (f=1): [m(1)] [0.1% done] [475KB/163KB/0KB /s] [118/40/0 iops] [eta 01h:Jobs: 1 (f=1): [m(1)] [0.2% done] [451KB/127KB/0KB /s] [112/31/0 iops]

So, the program will create a 4GB file (--size=4G), and perform 4KB reads and writes using three reads for every write ratio (75%/25%, as specified with option --rwmixread=75), split within the file, with 64 operations running at a time. The RW ratio can be adjusted for simulating various usage scenarios. At the end, it will display the final results:

test: (groupid=0, jobs=1): err= 0: pid=4760: Thu Mar  2 13:23:28 2017
  read : io=7884.0KB, bw=864925B/s, iops=211, runt=  9334msec
  write: io=2356.0KB, bw=258468B/s, iops=63, runt=  9334msec
  cpu          : usr=0.46%, sys=2.35%, ctx=2289, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=1971/w=589/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=7884KB, aggrb=844KB/s, minb=844KB/s, maxb=844KB/s, mint=9334msec, maxt=9334msec
  WRITE: io=2356KB, aggrb=252KB/s, minb=252KB/s, maxb=252KB/s, mint=9334msec, maxt=9334msec

Disk stats (read/write):
    dm-2: ios=1971/589, merge=0/0, ticks=454568/120101, in_queue=581406, util=98.44%, aggrios=1788/574, aggrmerge=182/15, aggrticks=425947/119120, aggrin_queue=545252, aggrutil=98.48%
  sda: ios=1788/574, merge=182/15, ticks=425947/119120, in_queue=545252, util=98.48%

Note from the author: I ran fio on my laptop, so the last output was obtained running the test with a 10MB file; as can be seen above, the 4GB option would have taken more than 1 hour.

Random read performance

In this case, the command is:

$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read.fio --bs=4k --iodepth=64 --size=4G --readwrite=randread

The output will be similar to the RW case, just specialized in the read case.

Random write performance

$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randwrite

As above, in random write case.

Write test sample

$ fio --time_based --filename=./fio-bench-file --direct=1 --rw=write --refill_buffers --ioengine=libaio --bs=128k --size=4G --filesize=4G --iodepth=16  --numjobs=24  --runtime=600 --group_reporting --name=fio_secwrite

Benchmark disk sample

#!/bin/bash
#
# - Required Debian packages: bonnie++ fio libaio1
#

CPUS=$(cat /proc/cpuinfo | grep processor | wc -l)
RAM=$(free -m | grep 'Mem:' | perl -ne '/Mem:\s+(\d+)/; print $1')
BLOCK_COUNT=$(echo "scale=0;(${RAM} * 1024) * 4.25 / 8"|bc -l) # (RAM = in MB, so * 1024) * 4.25 / 8k blocksize
RAM_BYTES=$(echo "scale=0; $RAM * 1024 * 1024" | bc -l)
FIO_BENCH_SIZE=$(echo "scale=0; $RAM_BYTES * 4.25" | bc -l)
FIO_CPUS=$(echo "scale=0; $CPUS * 2" | bc -l)

logfile="bench_disk_$$.log"

openlog() {
    exec 6>&1
    exec 7>&2
    exec >> $logfile 2>&1
}
closelog() {
    # Close the file descriptors
    exec 1>&6 6>&-
    exec 2>&7 7>&-
}

# disable oom killer for this process
echo -17 > /proc/$$/oom_adj

openlog
echo "Number of CPUs: $CPUS"
echo "Available RAM : $RAM"
echo "Available RAM : $RAM_BYTES bytes"
echo "Block Size    : $BLOCK_COUNT"
closelog

TEST_DIR=${1:-/data/postgres/testdir}

BENCH1="bench01"
BENCH2="bench02"
BENCH3="bench03"
BENCH4="bench04"
BENCH5="bench05"

openlog
echo "Test directory: $TEST_DIR"
closelog

mkdir -p "${TEST_DIR}"
mkdir -p "logs/$date"
cd       "logs/$date"

# Test 0: fio benchtest
echo "test 0: FIO benchtests"
openlog
echo "test 0.1: FIO seq write benchtest"
date
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=write --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --refill_buffers --end_fsync=1 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_seqwrite
echo "test 0.2: FIO random write benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=randwrite --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_randomwrite
echo "test 0.3: FIO seq read benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=read --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_seqread
echo "test 0.4: FIO random read benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=randread --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_randomread
echo "test 0.5: FIO random read/write benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --rwmixread=60 --iodepth=16 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_randomreadwrite
date
closelog

echo "finished bench_disk"

Benchmark test db storage

#!/bin/bash
#
# - Required Debian packages: bonnie++ fio libaio1
# - Postgresql must be installed, tuned by Ansible playbook and started
#

running_user=$(id | egrep -E -o "uid=[0-9]+\([a-z]+\)" | cut -d'(' -f2 | cut -d ')' -f1)
if [ $running_user != "postgres" ]; then
    echo "You must run this script as 'postgres' user"
    exit 1
fi

CPUS=$(cat /proc/cpuinfo | grep processor | wc -l)
RAM=$(free -m | grep 'Mem:' | perl -ne '/Mem:\s+(\d+)/; print $1')
BLOCK_COUNT=$(echo "scale=0;(${RAM} * 1024) * 4.25 / 8"|bc -l) # (RAM = in MB, so * 1024) * 4.25 / 8k blocksize
RAM_BYTES=$(echo "scale=0; $RAM * 1024 * 1024" | bc -l)
FIO_BENCH_SIZE=$(echo "scale=0; $RAM_BYTES * 4.25" | bc -l)
FIO_CPUS=$(echo "scale=0; $CPUS * 2" | bc -l)

logfile="bench_db_$$.log"

openlog() {
    exec 6>&1
    exec 7>&2
    exec >> $logfile 2>&1
}
closelog() {
    # Close the file descriptors
    exec 1>&6 6>&-
    exec 2>&7 7>&-
}

# disable oom killer for this process
echo -17 > /proc/$$/oom_adj

openlog
echo "Number of CPUs: $CPUS"
echo "Available RAM : $RAM"
echo "Available RAM : $RAM_BYTES bytes"
echo "Block Size    : $BLOCK_COUNT"
closelog

DB_DIR=${1:-/data/postgres}
TEST_DIR=${1:-/data/postgres/testdir}
PG_DIR="/opt/postgresql"
pgbench="$PG_DIR/bin/pgbench "
createdb="$PG_DIR/bin/createdb"
dropdb="$PG_DIR/bin/dropdb"
initdb="$PG_DIR/bin/initdb"

BENCH1="bench01"
BENCH2="bench02"
BENCH3="bench03"
BENCH4="bench04"
BENCH5="bench05"

postgres_uid=$(id postgres|sed -e 's/uid=\([0-9]*\).*/\1/')
postgres_gid=$(id postgres|sed -e 's/.*gid=\([0-9]*\).*/\1/')

openlog
echo "Test directory: $DB_DIR"
echo "Postgres UID  : $postgres_uid"
echo "Postgres GID  : $postgres_gid"
closelog

mkdir -p "${TEST_DIR}"
mkdir -p "logs/$date"
cd       "logs/$date"

# Test 0: fio benchtest
echo "test 0: FIO benchtests"
openlog
echo "test 0.1: FIO seq write benchtest"
date
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=write --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --refill_buffers --end_fsync=1 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_seqwrite
echo "test 0.2: FIO random write benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=randwrite --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_randomwrite
echo "test 0.3: FIO seq read benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=read --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_seqread
echo "test 0.4: FIO random read benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=randread --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --filesize=$FIO_BENCH_SIZE --iodepth=256 --sync=0 --randrepeat=0 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_randomread
echo "test 0.5: FIO random read/write benchtest"
fio --time_based --filename=$TEST_DIR/fio-bench-file2 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --size=$FIO_BENCH_SIZE --rwmixread=60 --iodepth=16 --numjobs=$FIO_CPUS --runtime=600 --group_reporting --name=fio_randomreadwrite
date
closelog

#echo "test 1: dd write tests"
# Test 1.: dd write test
#openlog
#echo "test 1.1: dd write test oflag=direct"
#date
#time bash -c "dd if=/dev/zero of=${TEST_DIR}/dd_benchmark_file bs=8k count=$BLOCK_COUNT conv=fdatasync oflag=direct && sync" 2>&1
#date
#echo "test 1.2: dd write test oflag=dsync"
#time bash -c "dd if=/dev/zero of=${TEST_DIR}/dd_benchmark_file bs=8k count=$BLOCK_COUNT conv=fdatasync oflag=dsync && sync" 2>&1
#date
#echo "test 1.3: dd write test no oflag"
#time bash -c "dd if=/dev/zero of=${TEST_DIR}/dd_benchmark_file bs=8k count=$BLOCK_COUNT conv=fdatasync && sync" 2>&1
#date
#echo
#closelog

#echo "test 2: dd read test"
# Test 2.: dd read test
# Redirect output
#openlog
#echo "test 2: dd read test"
#date
#time dd if=${TEST_DIR}/dd_benchmark_file of=/dev/null bs=8k 2>&1
#date
#rm ${TEST_DIR}/dd_benchmark_file
#closelog

# Test 3: bonnie++
#echo "test 3: bonnie test"
#openlog
#echo "test 3: bonnie test"
#date
#/usr/sbin/bonnie++ -n 0 -u ${postgres_uid}:${postgres_gid} -r $(free -m | grep 'Mem:' | awk '{print $2}') -s $(echo "scale=0;`free -m | grep 'Mem:' | awk '{print $2}'`*4.25" | bc -l) -f -b -d ${TEST_DIR}
#date
#closelog

# Test 4: pgbench buffer test
echo "test 4: pgbench buffer test"
openlog
echo "test 4: pgbench buffer test"
$dropdb $BENCH1
$createdb $BENCH1
$pgbench -i -s 15 $BENCH1
date
$pgbench -c 24 -j 12 -T 600 $BENCH1
date
closelog

# Test 5: pgbench mostly cache test
echo "test 5: pgbench mostly cache test"
openlog
echo "test 5: pgbench mostly cache test"
$dropdb $BENCH2
$createdb $BENCH2
$pgbench -i -s 70 $BENCH2
date
$pgbench -c 24 -j 12 -T 600 $BENCH2
date
closelog

# Test 6: pgbench on-disk test
echo "test 6: pgbench on-disk test"
openlog
echo "test 6: pgbench on-disk test"
$dropdb $BENCH3
$createdb $BENCH3
$pgbench -i -s 600 $BENCH3
date
$pgbench -c 24 -j 12 -T 600 $BENCH3
date
closelog

# Test 7: pgbench Read-Only Test
echo "test 7: pgbench read-only test"
openlog
echo "test 7: pgbench read-only test"
date
$pgbench -c 24 -j 12 -T 600 -S $BENCH2
date
closelog

# Test 8: pgbench simple write test
echo "test 8: pgbench simple write test"
openlog
echo "test 8: pgbench simple write test"
date
$pgbench -c 24 -j 12 -T 600 -N $BENCH2
date
closelog

# Test 9: pgbench prepared read-write
echo "test 9: pgbench prepared read-write"
openlog
echo "test 9: pgbench prepared read-write"
date
$pgbench -c 24 -j 12 -T 600 -M prepared $BENCH2
date
closelog

# Test 10: pgbench prepared read-only
echo "test 10: pgbench prepared read-only"
openlog
echo "test 10: pgbench prepared read-only"
date
$pgbench -c 24 -j 12 -T 600 -M prepared -S $BENCH2
date
closelog

# Test 11: connection test
echo "test 11: pgbench connection test"
openlog
echo "test 11: pgbench connection test"
echo " - fill up database (+-73GB)"
$dropdb $BENCH4
$createdb $BENCH4
$pgbench -i -s 5000 $BENCH4
echo " - fill up filesystem caches"
tar cvf - ${DB_DIR} > /dev/null
echo " - warmup postgres cache"
$pgbench -j 6 -c 6 -T 1800 -S $BENCH4
#for clients in 1 5 10 20 30 40 50 60 80 100 150 200 250 300 350 400 450 500
for clients in 1 5 10 20 30 40 50 60 80 100 150 200
do
    THREADS=${clients}
    if [ $clients > $CPUS ]
    then
        THREADS=10
    else
        THREADS=${clients}
    fi
    echo " -- Number of Clients: ${clients} | THREADS: ${THREADS}"
    $pgbench -j ${THREADS} -c ${clients} -T 180 -S $BENCH4
done
closelog

# cleanup

#pg_ctl -D ${DB_DIR} -m fast stop
#rm -fr ${DB_DIR}