Monitoring & Troubleshooting Guide

1. Monitoring with Unix Commands

System Health

1. Displays real-time processes, CPU, and memory usage.

top

2. An interactive version of top with colors and easier navigation.

htop

3. Shows memory, CPU, and I/O statistics every second.

vmstat 1

4. Reports CPU load and detailed disk I/O statistics every second.

iostat -x 1

5. Displays system memory usage in megabytes.

free -m

6. Shows disk space usage of mounted filesystems in human-readable format.

df -h

7. Shows how long the system has been running, number of users, and load averages.

uptime

Processes & Networking

8. Lists all processes and filters for those containing “trading”.

ps -ef | grep trading

9. Shows active network connections and listening ports, filtering for port 5000.

netstat -anp | grep 5000

10. Summarizes socket statistics (TCP/UDP connections, states).

ss -s

11. Tests connectivity and latency to exchange.com.

ping exchange.com

12. Shows the network path packets take to reach exchange.com.

traceroute exchange.com

13. Captures and inspects network packets on interface eth0 filtered by port 5000.

tcpdump -i eth0 port 5000

Logs

14. Continuously monitors the latest lines of the trading engine log in real time.

tail -f /var/log/trading_engine.log

15. Searches the log for entries containing the word “ERROR”.

grep "ERROR" /var/log/trading_engine.log

16. Extracts the 5th column of trade.log, counts unique values, and sorts them by frequency.

awk '{print $5}' trade.log | sort | uniq -c | sort -nr

2. Automation with Shell Script

#!/bin/bash
# monitor_trading.sh

LOG="/var/log/trading_engine.log"
EMAIL="ops-team@example.com"

# Check if trading engine is running
if ! pgrep trading_engine > /dev/null; then
  echo "Trading engine DOWN!" | mail -s "ALERT: Trading Engine Down" $EMAIL
fi

# Check for high CPU usage
CPU=$(top -bn1 | grep "trading_engine" | awk '{print $9}')
if (( $(echo "$CPU > 80" | bc -l) )); then
  echo "High CPU usage: $CPU%" | mail -s "ALERT: High CPU" $EMAIL
fi

# Check logs for errors
if grep -q "ERROR" $LOG; then
  echo "Errors detected in trading logs" | mail -s "ALERT: Trading Errors" $EMAIL
fi

This script automates health checks for a trading system:

Make it executable:

chmod +x monitor_trading.sh

Schedule with cron:

*/5 * * * * /path/to/monitor_trading.sh

3. Troubleshooting Checklist