Automating Performance Tests with Vdbench and JenkinsPerformance testing storage systems can be tedious if done manually. Combining Vdbench — a powerful, scriptable I/O workload generator — with Jenkins — a flexible automation server — lets you run repeatable, scheduled, and reportable storage benchmarks as part of your CI/CD pipeline. This article explains how to design, implement, and maintain an automated performance testing pipeline using Vdbench and Jenkins, including sample configurations, best practices, and troubleshooting tips.
What is Vdbench and why use it?
Vdbench is a command-line tool developed by Oracle for generating configurable I/O workloads against local disks, SAN, NAS, or virtualized storage devices. It excels at:
- High configurability of I/O patterns (sequential/random, read/write mix, block sizes, alignment).
- Scriptable test profiles that can be version controlled.
- Detailed metrics and latency distribution reporting.
- Running on a variety of platforms (Linux, Solaris, Windows).
Use Vdbench when you need deterministic, repeatable storage benchmarks that mirror production workloads.
Why integrate with Jenkins?
Jenkins automates task execution and orchestration. Integrating Vdbench with Jenkins provides:
- Scheduled and triggered test runs (on commits, nightly, or before releases).
- Centralized logs and artifact storage.
- Notifications on test success/failure and performance regressions.
- Easy scaling via build agents (run tests in parallel on multiple systems).
Architecture and Workflow
A typical automated pipeline includes:
- Jenkins orchestrator (master) scheduling builds and collecting artifacts.
- Build agents (nodes) where Vdbench runs against the target storage.
- A version-controlled repository (Git) holding Vdbench parameter files and test scripts.
- An artifacts storage (Jenkins workspace, S3, or network share) for Vdbench logs and results.
- Optional visualization/reporting tools (Grafana, Elasticsearch) for long-term metrics.
Workflow example:
- Developer updates Vdbench profile in Git.
- Jenkins detects change and triggers a job.
- Jenkins prepares the agent, deploys the profile, and runs Vdbench.
- Vdbench outputs results; Jenkins archives artifacts and fails the build if thresholds are breached.
- Notifications are sent; historical results are pushed to a dashboard.
Preparing Vdbench test profiles
Vdbench uses parameter files (.vdbench) that describe drives, files, threads, and workloads. Keep profiles modular:
- drives file: list of devices or mountpoints
- workload files: named run definitions with different I/O patterns
- include files for shared settings (duration, warmup, output options)
Example minimal profile (save as workload.vdbench):
* drives definition sd=sd1,lun=/dev/sdb,openflags=direct sd=sd2,lun=/dev/sdc,openflags=direct * define fileset wd=wd1,sd=sd1,xfersize=4k,seekpct=100 wd=wd2,sd=sd2,xfersize=64k,seekpct=0 * run definitions rd=run1,wd=wd1,iorate=max,elapsed=300,interval=1,threads=8 rd=run2,wd=wd2,iorate=1000,elapsed=300,threads=4
Tips:
- Use openflags=direct to bypass cache when needed.
- Set warmup with “warmup=” before measured intervals.
- Use meaningful names and comments for maintainability.
Jenkins job types and configuration
Choose between freestyle jobs, Pipeline (Declarative or Scripted), or multibranch pipelines. Pipelines are recommended for reproducibility and version control.
Example Declarative pipeline (Jenkinsfile):
pipeline { agent { label 'vdbench-agent' } environment { VD_HOME = '/opt/vdbench' PROFILE = 'workload.vdbench' } stages { stage('Checkout') { steps { checkout scm } } stage('Prepare') { steps { sh 'mkdir -p results' sh 'cp ${PROFILE} results/' } } stage('Run Vdbench') { steps { sh ''' cd results ${VD_HOME}/vdbench -f ${PROFILE} -o output ''' } post { always { archiveArtifacts artifacts: 'results/output/**', fingerprint: true } } } stage('Analyze') { steps { sh 'python3 ci/parse_vdbench_results.py results/output' } } } post { success { mail to: '[email protected]', subject: 'Vdbench CI: Success', body: 'Performance test passed.' } failure { mail to: '[email protected]', subject: 'Vdbench CI: Failure', body: 'Performance test failed — check artifacts.' } } }
Notes:
- Label agents with capabilities (access to test hardware, required OS).
- Use credentials and secure environment variables for sensitive settings.
Parsing and asserting results
Vdbench produces detailed output including histograms and summary files. Automate result parsing to detect regressions.
Common approach:
- Write a parsing script (Python recommended) that extracts IOPS, throughput, average/percentile latencies from Vdbench summary files.
- Compare measured values against thresholds (absolute or relative to a baseline).
- Fail the Jenkins build if thresholds are exceeded.
Simple parsing example (conceptual):
# parse_vdbench_results.py import sys, json, re def parse_summary(path): # open and extract lines with "aggregate" or "overall" metrics # return structured dict with iops, avg_lat_ms, p99_ms, throughput_mb pass results = parse_summary(sys.argv[1]) print(json.dumps(results)) # exit with non-zero if thresholds violated
Store baselines in the repo or a central database. Use percentage-based regression checks to avoid false positives from small variability.
Reporting and visualization
Short-term: archive raw Vdbench output and a parsed JSON summary in Jenkins artifacts.
Long-term: push metrics to time-series stores:
- Push parsed metrics to Prometheus via a pushgateway or to InfluxDB.
- Visualize with Grafana dashboards showing IOPS, latency percentiles, and throughput across runs.
Include trend analysis:
- Plot moving averages and standard deviation bands.
- Annotate commit IDs or test parameters to correlate changes.
Scaling and distributed testing
For larger environments, you may need distributed Vdbench runs:
- Run multiple Vdbench instances across agents, each targeting different hosts or volumes.
- Use Jenkins matrix or parallel stages to coordinate.
- Collect and aggregate outputs centrally; ensure timestamps and workload IDs are consistent.
Network and orchestration considerations:
- Ensure agents have network access to storage targets.
- Use synchronized clocks (NTP) for comparing timelines.
- Limit concurrent runs against the same storage target to avoid resource contention.
Best practices
- Version-control all Vdbench profiles and parsing scripts.
- Keep test hardware and environment stable; document configuration.
- Run warmup phases before measurement to avoid transient behavior.
- Use meaningful thresholds informed by historical runs.
- Tag Jenkins artifacts with commit hashes and build numbers for traceability.
- Rotate or prune old artifacts to save storage.
- Secure access to test systems and credentials.
Common pitfalls and troubleshooting
- Unexpectedly low performance: check caching, openflags, alignment, and multipath settings.
- High variance: increase run times, ensure isolation, increase threads for better statistical confidence.
- Permission or device access errors: run Jenkins agent with appropriate privileges or use sudo carefully.
- Time drift between hosts: synchronize time with NTP to correlate logs.
Example: End-to-end checklist
- Create modular Vdbench profiles and store in Git.
- Provision Jenkins agents with Vdbench installed and device access.
- Implement a Jenkins Pipeline to checkout, run, archive, and parse results.
- Define pass/fail thresholds and baseline comparisons.
- Push metrics to a dashboard for trend analysis.
- Schedule regular runs and enable on-demand triggers (PRs, releases).
- Review and adjust tests after major infra changes.
Automating Vdbench with Jenkins turns manual storage benchmarking into a reproducible, traceable, and actionable process. With modular profiles, robust parsing, and clear thresholds, teams can detect regressions early, correlate performance changes with code or infrastructure changes, and keep storage performance predictable as systems evolve.