As you might know by now, I was debugging memory related problems on one of my systems recently and concluded that normal output from Linux commands are more or less inaccurate. If you want to know why, take a look at Matt Mackall presentation at ELC2009: Visualizing Process Memory or watch following video:
Convinced? So, hop at smem page, compile user-land part and start really tracking your memory usage, let's compare:
dpavlin@t61p:/rest/cvs/smem$ free
total used free shared buffers cached
Mem: 4081400 3882476 198924 0 142904 2731480
-/+ buffers/cache: 1008092 3073308
Swap: 8209172 7492 8201680
dpavlin@t61p:/rest/cvs/smem$ ./smem -w -t
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 2927016 2845456 81560
userspace memory 954900 119368 835532
free memory 199484 199484 0
----------------------------------------------------------
4081400 3164308 917092
Just a few quick notes if you didn't watched whole video carefully:
- needs kernel 2.6.27 or newer
- it can work on archived data (from cron in my example usage)
- userspace cache is backed by file on disk
- it's a python script which requires matplotlib to create graphs so it's for local reporting