Useful Commands

A collection of practical Linux commands for system administration and testing

About This Page

This page contains a curated collection of useful Linux commands for system administration, performance testing, and troubleshooting. Click on any command section below to expand and view detailed examples and explanations.

FIO (Flexible I/O Tester)

▼

FIO is a versatile tool for benchmarking and testing storage devices. It provides comprehensive I/O performance testing capabilities with extensive configuration options.

Installation

# Install FIO on Ubuntu/Debian
sudo apt update
sudo apt install fio

# Install FIO on RHEL/CentOS
sudo yum install fio

# Verify installation
fio --version

Fill a Disk

Use FIO to fill a disk with data for testing purposes. This is useful for initializing storage devices, testing write performance, or preparing disks for benchmarking.

Sequential Write (Fastest)

# Fill entire disk with sequential writes
sudo fio --name=filltest --filename=/dev/nvme0n1 --rw=write --bs=1M --direct=1 --size=100% --numjobs=1

# Fill with larger block size for maximum speed
sudo fio --name=filltest --filename=/dev/nvme0n1 --rw=write --bs=4M --direct=1 --size=100% --numjobs=1

Random Write (More Realistic)

# Fill disk with random data (more realistic workload)
sudo fio --name=filltest --filename=/dev/nvme0n1 --rw=randwrite --bs=4k --direct=1 --size=100% --numjobs=1

# Fill with multiple jobs for parallel writes
sudo fio --name=filltest --filename=/dev/nvme0n1 --rw=randwrite --bs=4k --direct=1 --size=100% --numjobs=4

Fill Specific Partition

# Fill a specific partition instead of entire disk
sudo fio --name=filltest --filename=/dev/nvme0n1p1 --rw=write --bs=1M --direct=1 --size=100%

# Fill specific size (e.g., 50GB)
sudo fio --name=filltest --filename=/dev/nvme0n1 --rw=write --bs=1M --direct=1 --size=50G

⚠️ Data Loss Warning

CRITICAL: The commands above will PERMANENTLY OVERWRITE ALL DATA on the specified device. Always double-check the device path before running these commands.

Use lsblk or fdisk -l to verify device names before proceeding.

FIO Parameters Explained

  • --name: Job name for identification in output
  • --filename: Device or file to test (use full path, e.g., /dev/nvme0n1)
  • --rw: I/O pattern - write, randwrite, read, randread, readwrite, etc.
  • --bs: Block size - 4k (4 kilobytes), 1M (1 megabyte), etc.
  • --direct: Use direct I/O (1 = bypass OS cache, 0 = use cache)
  • --size: Amount of data to write - 100% (entire device), 50G (50 gigabytes), etc.
  • --numjobs: Number of parallel jobs (threads) to run
  • --ioengine: I/O engine to use (libaio, io_uring, sync, etc.)

Verify Device Before Filling

Always verify the device path and ensure no mounted partitions exist before filling a disk:

# List all block devices and their mount points
lsblk

# Show detailed partition information
sudo fdisk -l /dev/nvme0n1

# Check if device or partitions are mounted
mount | grep nvme0n1

# If mounted, unmount first (replace with your partition)
sudo umount /dev/nvme0n1p1

# Verify device is not in use
sudo fuser -v /dev/nvme0n1

Monitor Fill Progress

FIO provides real-time progress updates. You can also monitor from another terminal:

# Watch I/O statistics in real-time
iostat -x 2

# Monitor specific device
iostat -x 2 nvme0n1

# Watch FIO progress with bandwidth and IOPS
watch -n 1 'iostat -x | grep nvme0n1'

Additional FIO Examples

Random Read Performance Test

# Test random read performance
sudo fio --name=randread --filename=/dev/nvme0n1 --rw=randread --bs=4k --direct=1 --numjobs=4 --time_based --runtime=60

Sequential Read/Write Test

# Test sequential read and write
sudo fio --name=seqrw --filename=/dev/nvme0n1 --rw=readwrite --bs=1M --direct=1 --numjobs=1 --time_based --runtime=60

Mixed Workload Test

# 70% read, 30% write mixed workload
sudo fio --name=mixed --filename=/dev/nvme0n1 --rw=randrw --rwmixread=70 --bs=4k --direct=1 --numjobs=4 --time_based --runtime=60

✓ Best Practices

  • Always verify device paths using lsblk before running write operations
  • Unmount any mounted partitions before testing
  • Use --direct=1 to bypass OS cache for accurate results
  • Start with small test sizes before running full disk operations
  • Monitor system resources during tests to avoid overload

Smartmontools - Disk Health Monitoring

▼

Smartmontools is a suite of utilities for monitoring and analyzing the health of storage devices using SMART (Self-Monitoring, Analysis and Reporting Technology). It can automatically detect all devices, perform periodic scans, and send email alerts when errors are detected.

Installation

# Install on Ubuntu/Debian
sudo apt update
sudo apt install smartmontools

# Install on RHEL/CentOS
sudo yum install smartmontools

# Verify installation
smartctl --version

Setting Up Automatic Monitoring

Configure smartd (the SMART daemon) to automatically detect all devices, perform periodic scans, and send email alerts for any errors detected.

Step 1: Configure Email Settings

First, ensure your system can send emails. Install and configure a mail transfer agent:

# Install mailutils (Debian/Ubuntu)
sudo apt install mailutils

# Install mailx (RHEL/CentOS)
sudo yum install mailx

# Test email sending
echo "Test email from smartmontools" | mail -s "Test Subject" darren@soothill.com

Step 2: Configure smartd Daemon

Edit the smartd configuration file to enable automatic device detection and email alerts:

# Backup original configuration
sudo cp /etc/smartd.conf /etc/smartd.conf.backup

# Edit configuration file
sudo nano /etc/smartd.conf

Step 3: Add Configuration Directives

Add the following line to /etc/smartd.conf to monitor all devices automatically:

# Monitor all devices, run short self-test daily, long test weekly
# Send email alerts to darren@soothill.com on any SMART errors
DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m darren@soothill.com -M exec /usr/share/smartmontools/smartd-runner

Configuration Parameters Explained

  • DEVICESCAN: Automatically detect and monitor all SMART-capable devices
  • -a: Monitor all SMART attributes and report changes
  • -o on: Enable automatic offline testing
  • -S on: Enable automatic attribute autosave
  • -s (S/../.././02|L/../../6/03): Schedule tests
    • Short self-test daily at 2 AM
    • Long self-test weekly on Saturday at 3 AM
  • -m: Email address to send alerts to
  • -M exec: Script to execute when sending email alerts

Step 4: Configure Email Subject Line (Optional)

Customize the email alert subject line for better filtering:

# More advanced configuration with custom email subject
DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m darren@soothill.com -M exec /usr/share/smartmontools/smartd-runner -M diminishing -M test

Additional Email Options

  • -M diminishing: Reduces frequency of repeated error notifications (sends alert, then waits 24h, 48h, 72h, etc.)
  • -M test: Send a test email on smartd startup to verify email configuration
  • -M once: Send only one email per problem (no repeat notifications)
  • -M daily: Send reminder email daily if problem persists

Step 5: Enable and Start smartd Service

# Enable smartd to start on boot
sudo systemctl enable smartd

# Start smartd service
sudo systemctl start smartd

# Check service status
sudo systemctl status smartd

# Restart service after configuration changes
sudo systemctl restart smartd

Step 6: Verify Configuration

Test your smartd configuration and verify it's monitoring devices:

# Test configuration file syntax
sudo smartd -q onecheck

# Check which devices are being monitored
sudo smartctl --scan

# View smartd log
sudo journalctl -u smartd -f

# Check syslog for smartd messages
sudo tail -f /var/log/syslog | grep smartd

Manual Device Testing

While smartd monitors devices automatically, you can also run manual tests:

Check Device SMART Status

# Check if SMART is available and enabled
sudo smartctl -i /dev/sda

# Check overall health status
sudo smartctl -H /dev/sda

# View all SMART attributes
sudo smartctl -A /dev/sda

# View detailed SMART information
sudo smartctl -a /dev/sda

Run Manual Self-Tests

# Start a short self-test (takes ~2 minutes)
sudo smartctl -t short /dev/sda

# Start a long self-test (takes 1-2 hours)
sudo smartctl -t long /dev/sda

# View self-test results
sudo smartctl -l selftest /dev/sda

# View error log
sudo smartctl -l error /dev/sda

Monitor Multiple Devices

# Scan for all SMART devices
sudo smartctl --scan

# Check health of all detected devices
for device in $(sudo smartctl --scan | awk '{print $1}'); do
    echo "Checking $device"
    sudo smartctl -H $device
    echo "---"
done

# Run short test on all devices
for device in $(sudo smartctl --scan | awk '{print $1}'); do
    echo "Starting short test on $device"
    sudo smartctl -t short $device
done

Advanced Configuration Examples

Monitor Specific Device Types

Configure monitoring for specific device types with tailored settings:

# In /etc/smartd.conf - Monitor NVMe devices separately
DEVICESCAN -d nvme -a -o on -S on -s (S/../.././02|L/../../6/04) -m darren@soothill.com

# Monitor SATA/SAS devices
DEVICESCAN -d sat -a -o on -S on -s (S/../.././02|L/../../6/03) -m darren@soothill.com

# Monitor specific device with custom schedule
/dev/sda -a -s (S/../.././01|L/../../7/02) -m darren@soothill.com -M daily

Monitor Temperature Changes

# Alert if temperature exceeds thresholds
DEVICESCAN -a -W 4,40,45 -m darren@soothill.com

# Parameters: -W diff,info,crit
# diff: Alert on 4°C change since last check
# info: Informational alert at 40°C
# crit: Critical alert at 45°C

⚠️ Important Notes

Email Configuration: Ensure your system's mail configuration is working before relying on email alerts. Test with the -M test option.

Firewall/SMTP: If emails aren't being delivered, check that your firewall allows outbound SMTP connections (port 25, 465, or 587).

Self-Tests: Long self-tests can impact system performance. Schedule them during maintenance windows or low-usage periods.

✓ Best Practices

  • Always test email functionality with -M test before relying on alerts
  • Use -M diminishing to avoid alert fatigue from repeated errors
  • Schedule long tests during off-peak hours (weekends/nights)
  • Monitor smartd logs regularly: sudo journalctl -u smartd
  • Keep a backup of your smartd.conf before making changes
  • Document your monitoring configuration for future reference

SSH Connection Fix - Legacy Algorithms

▼

Fix SSH connection issues with legacy servers that use outdated key exchange and host key algorithms. Modern SSH clients disable these older algorithms by default for security reasons, but they can be re-enabled when needed for legacy equipment.

Common Error Messages

You may encounter these error messages when connecting to older SSH servers:

Unable to negotiate with 10.10.201.2 port 22: no matching key exchange method found.
Their offer: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1

Unable to negotiate with 10.10.201.2 port 22: no matching host key type found.
Their offer: ssh-rsa

Quick Fix (One-time Connection)

Use this command to connect immediately with both legacy algorithms enabled. Replace user with your actual username:

ssh -oKexAlgorithms=+diffie-hellman-group14-sha1 -oHostKeyAlgorithms=+ssh-rsa user@10.10.201.2

Alternative with Additional Legacy Support

# Include both diffie-hellman algorithms and ssh-rsa
ssh -oKexAlgorithms=+diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 \
    -oHostKeyAlgorithms=+ssh-rsa \
    -oPubkeyAcceptedKeyTypes=+ssh-rsa \
    user@10.10.201.2

Permanent Fix (SSH Config)

For servers you connect to regularly, add configuration to your ~/.ssh/config file for automatic handling:

For Specific Host

# Add to ~/.ssh/config
Host 10.10.201.2
    KexAlgorithms +diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa

For Entire Subnet

# Add to ~/.ssh/config - applies to all hosts in 10.10.201.* subnet
Host 10.10.201.*
    KexAlgorithms +diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa

Using Hostname Pattern

# Add to ~/.ssh/config - applies to legacy-* hostnames
Host legacy-*
    KexAlgorithms +diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa

After adding this configuration, connect normally:

ssh user@10.10.201.2

If You Still Have Issues

Very old servers might require additional legacy cipher support:

# Add to ~/.ssh/config with legacy cipher support
Host 10.10.201.2
    KexAlgorithms +diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa
    Ciphers +aes256-cbc,aes128-cbc,3des-cbc

MAC Algorithm Issues

If you get errors about MAC (Message Authentication Code) algorithms:

# Add legacy MAC algorithms
Host 10.10.201.2
    KexAlgorithms +diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa
    MACs +hmac-sha1,hmac-sha1-96

Creating/Editing SSH Config File

If you don't have an SSH config file yet, create one:

# Create .ssh directory if it doesn't exist
mkdir -p ~/.ssh

# Set proper permissions
chmod 700 ~/.ssh

# Create or edit config file
nano ~/.ssh/config

# Set proper permissions on config file
chmod 600 ~/.ssh/config

Checking What Algorithms a Server Supports

To see what algorithms a remote SSH server offers:

# Use nmap to scan SSH algorithms
nmap --script ssh2-enum-algos -sV -p 22 10.10.201.2

# Or use ssh with verbose mode to see negotiation
ssh -vv user@10.10.201.2

Configuration Parameters Explained

  • KexAlgorithms: Key exchange algorithms - methods for securely exchanging encryption keys
  • HostKeyAlgorithms: Algorithms used to verify the server's identity
  • PubkeyAcceptedKeyTypes: Algorithms accepted for public key authentication
  • Ciphers: Encryption algorithms used for the actual data transfer
  • MACs: Message Authentication Codes for verifying data integrity
  • + prefix: Appends algorithms to the default list instead of replacing it

⚠️ Security Warning

Important: These algorithms are disabled by default in modern SSH clients because they have known security weaknesses:

  • diffie-hellman-group1-sha1: Vulnerable to logjam attacks, uses weak 1024-bit key
  • ssh-rsa with SHA-1: SHA-1 hash collisions have been demonstrated
  • Legacy ciphers: CBC-mode ciphers are vulnerable to certain attacks

Only re-enable these if:

  • You're on a trusted, isolated network (not over the internet)
  • You need to connect to legacy equipment that cannot be updated
  • You understand and accept the security risks

Best practice: The proper long-term solution is to upgrade the SSH server to support modern algorithms.

✓ Best Practices

  • Use SSH config file instead of command-line options for regular connections
  • Limit legacy algorithm support to specific hosts, not globally
  • Document why you need legacy algorithm support (compliance, old hardware, etc.)
  • Regularly check if legacy servers can be upgraded to modern SSH versions
  • Use VPN or jump hosts when accessing legacy systems over untrusted networks
  • Keep your SSH config file backed up and version controlled

📌 More Commands Coming Soon

Additional useful commands will be added to this page. Check back regularly for updates including networking tools, monitoring commands, and more storage utilities.