Why isn’t “Used memory MBytes” in performance counter of Azure Monitor equal to “used” got via free command?

This entry is written on June 1, 2021.
Originally published at https://logico-jp.io/2021/06/01/used-memory-mbytes-from-performance-counter-is-not-equal-to-used-by-free-command/ in Japanese on June 1, 2021.

Someone asked me about the following question.

We monitor memory consumption trend of each Linux node using “Used Memory MBytes” performance counter, which is available in Azure Monitor (Log Analytics). This data is not equal to “used” data got via free command. Could you please tell me the reason?

Regarding “Performance Counter” of Log Analytics, please read the following document.

You can query several metrics with kusto in Azure portal.

On the other hand, all of you might know the command of free...

You can easily find Used Memory MBytes (677 MiB) in Log Analytics performance counter is not equal to used memory (369 MiB) got via free command. Why?

Why one data is not equal to the other?

Indeed, both data are based on /proc/meminfo , but each data is derived from different equation. Therefore, one data is not equal to the other. The following screenshot is /proc/meminfo in my Linux box (Ubuntu 18.04). For comparison, output of free is also available in the same screenshot.

Performance counter (Log Analytics)

“Used Memory MBytes” is derived from the following equation.

UsedMemoryMBytes = MemTotal - MemAvailable 

based on /proc/meminfo, calculated value of “Used Memory MBytes” is 678 (MiB).

UsedMemoryMBytes = 8,152,804 - 7,457,416
= 695,388 (KiB)
= 678 (MiB)

“Used Memory MBytes” from performance counter is 678 (MiB), too.

free command

Documents in each distribution elaborates data got via free command and equations to derive these data.

In case of Red Hat Enterprise Linux,

In case of Ubuntu,

Ubuntu 18.04 runs on my Linux box. In case of Ubuntu 18.04, equations to derive “used” are listed below. The 1st one is used when using data collected via free command, while the 2nd one is used when using data collected from /proc/meminfo.

used = total - free - buffers - cache
= (MemTotal + SwapTotal) - (MemFree + SwapFree) - (Buffers) - (Cached + SReclaimable)

You can easily recognize data derived from this equation is not equal to “Used memory MBytes” in performance counter. To make sure, let me calculate “used” data.

used = total - free - buffers - cache
= (MemTotal + SwapTotal) - (MemFree + SwapFree) - (Buffers) - (Cached + SReclaimable)
= (8,152,804 + 0) - (5,842,952 + 0) - 112,500 - (1,591,036 + 225,580)
= 380,736 (KiB)
= 371 (MiB)

Data derived from the equation is 371 (MiB), which is equal to output data from free command (see the screenshot above).

Conclusion

Even if the same word of “used” is used, one data is different from another because each equation to derive data is not the same. Before using metric(s), you should check how to derive metric(s) carefully.

Last but not least, minimum interval to collect performance counter is 10 seconds. Unlike free command, therefore, you cannot collect performance metric(s) on demand through performance counter.

Cloud Solution Architect @Microsoft, focusing on Application Development. ❤️Java (JVM/GraalVM) and open source technologies. All views are my own. Ex-🥑.

Cloud Solution Architect @Microsoft, focusing on Application Development. ❤️Java (JVM/GraalVM) and open source technologies. All views are my own. Ex-🥑.