3 fundamental tools to troubleshoot Linux performance problems

In this article and video, you'll learn how to collect information about your 
Linux system's performance.

Posted: March 6, 2023 | %t min read |
by
Nathan Lager (Sudoer, Red Hat)

A row of old tools for cutting and carving

Photo by Philip Swinburn on Unsplash

Performance is one of those things that many system administrators dread. 
Sometimes you don't even think about it when things are running well. Then, 
suddenly you get a call from an end user, or even worse, lots of end users, that
the application you're responsible for "feels slow." Or maybe it's outright 
unavailable. Now you have to go into troubleshooting mode.



Great Linux resources

Advanced Linux commands cheat sheet
 Download RHEL 9 at no charge through the Red Hat Developer program
 A guide to installing applications on Linux
 Linux system administration skills assessment
 How well do you know Linux? Take a quiz and get a badge

Where do you start, though? In this article, I'll walk through some basics, but
first, I'll take a step back. It's important to have some baseline information 
about your system before trying to identify problems. Maybe your system is 
doing something it shouldn't be, and that is contributing to the high load. Or 
perhaps it's doing exactly what it always has, and it's just under some 
additional load. Performance troubleshooting starts before a problem occurs by 
keeping good documentation and historical performance data.

A recent episode of Into The Terminal examines performance troubleshooting. This
article summarizes three utilities that provide great initial performance 
information and introduces the concepts covered by the video. If you'd rather, 
you can skip straight to the video.


Start at the to 


The first go-to performance troubleshooting tool is top. The top utility gives 
you a great, constantly updated process and performance dashboard. How much 
memory is in use? What's the load average? What processes are using the most 
resources? All of this is ready at a moment's notice in top.




top - 10:55:50 up 6 min,  1 user,  load average: 0.03, 0.18, 0.10


Tasks: 188 total,   1 running, 187 sleeping,   0 stopped,   0 zombie


%Cpu(s):  0.6 us,  0.0 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st


MiB Mem :   3654.6 total,   1477.3 free,    700.3 used,   1752.4 buff/cache


MiB Swap:   3280.0 total,   3280.0 free,      0.0 used.   2954.3 avail Mem

PID
USER
PR
NI
VIRT
RES
SHR
S
%CPU
%MEM
TIME+
COMMAND

3
root
0
-20
0
0
0
I
0.0
0.0
0:00.00
rcu_gp

4
root
0
-20
0
0
0
I
0.0
0.0
0:00.00
rcu_par_gp

5
root
0
-20
0
0
0
I
0.0
0.0
0:00.00
netns

6
root
20
0
0
0
0
I
0.0
0.0
0:00.01
kworker/0:0-xfs-c+

7
root
0
-20
0
0
0
I
0.0
0.0
0:00.00
kworker/0:0H-even+

8
root
20
0
0
0
0
I
0.0
0.0
0:00.16
kworker/u8:0-flus+

9
root
0
-20
0
0
0
I
0.0
0.0
0:00.00
mm_percpu_wq

10
root
20
0
0
0
0
I
0.0
0.0
0:00.36
kworker/u8:1-even+

11
root
20
0
0
0
0
I
0.0
0.0
0:00.00
rcu_tasks_kthre

12
root
20
0
0
0
0
I
0.0
0.0
0:00.00
rcu_tasks_rude_

13
root
20
0
0
0
0
I
0.0
0.0
0:00.00
rcu_tasks_trace

14
root
20
0
0
0
0
S
0.0
0.0
0:00.03
ksoftirqd/0

15
root
20
0
0
0
0
I
0.0
0.0
0:00.41
rcu_preempt

16
root
rt
0
0
0
0
S
0.0
0.0
0:00.00
migration/0

17
root
20
0
0
0
0
I
0.0
0.0
0:00.00
kworker/0:1-xfs-c+

18
root
20
0
0
0
0
S
0.0
0.0
0:00.00
cpuhp/0

19
root
20
0
0
0
0
S
0.0
0.0
0:00.00
cpuhp/1


Additionally, you can filter top, search for processes, or even view the 
utilization of each core in your system. So, your first stop should be top!

[ Learn how to manage your Linux environment for success. ]


Are you free? 


The free tool displays the system's current memory consumption, and it's a great
second stop in your journey to discover what's eating at your system.



total
used
free
shared
buff/cache
available

Mem:
3742352
717408
1512544
5312
1794520
3024944

Swap:
3358716
0
3358716


Remember that the Linux kernel will take available memory and allocate it to 
disk buffers and cache. Linux will release those resources when the system needs
them, so if free suggests that you have no available memory, be sure to check 
that buffers/cache column before you conclude that you're out of memory.


Check vmstat


Using vmstat is another great way to look at your memory consumption.


procs
-----------memory----------
---swap--
-----io----
-system--
------cpu-----

r
b
swpd
free
buff
cache
si
so
bi
bo
in
cs
us
sy
id
wa
st

0
0
0
1511816
9684
1784948
0
0
233
676
256
262
3
2
95
0
0


It breaks down the buffers and cache, and you'll also get information about your
swap utilization. Swapping is generally a sign that your system is exhausting 
its higher-performing memory, and it must resort to swapping to disk. This 
process can have a devastating effect on some applications.

[ Keep your most commonly used commands handy with the Linux commands cheat sheet. ]