Sometimes on a Linux web server you may see memory allocation failures in your /var/log/syslog file. This can happen even when there is spare RAM and swap. On a server running Apache or Nginx, they may look something like this:
Jul 15 07:10:30 hostname kernel: swapper/1: page allocation failure: order:3, mode:0x20
Jul 15 07:10:30 hostname kernel: Pid: 0, comm: swapper/1 Not tainted 3.4.2-kernelversion1234 #1
Jul 15 07:10:30 hostname kernel: Call Trace:
Jul 15 07:10:30 hostname kernel: [] ? warn_alloc_failed+0x98/0x100
Jul 15 07:10:30 hostname kernel: [] ? __alloc_pages_nodemask+0x4d8/0x6e0
Jul 15 07:10:30 hostname kernel: [] ? T.874+0x31/0xe0
Jul 15 07:10:30 hostname kernel: [] ? T.871+0x91/0x250
Jul 15 07:10:30 hostname kernel: [] ? cache_alloc_refill+0x24e/0x290
Jul 15 07:10:30 hostname kernel: [] ? __kmalloc+0xbe/0xd0
Jul 15 07:10:30 hostname kernel: [] ? pskb_expand_head+0x12e/0x240
Jul 15 07:10:30 hostname kernel: [] ? kmem_cache_free+0x42/0x60
Jul 15 07:10:30 hostname kernel: [] ? __pskb_pull_tail+0x4d/0x2a0
Jul 15 07:10:30 hostname kernel: [] ? netif_skb_features+0xaf/0xc0
Jul 15 07:10:30 hostname kernel: [] ? dev_hard_start_xmit+0x1ed/0x410
Jul 15 07:10:30 hostname kernel: [] ? nf_iterate+0x6c/0x90
Jul 15 07:10:30 hostname kernel: [] ? sch_direct_xmit+0xba/0x180
Jul 15 07:10:30 hostname kernel: [] ? ip_finish_output2+0x280/0x280
Jul 15 07:10:30 hostname kernel: [] ? dev_queue_xmit+0xff/0x340
Jul 15 07:10:30 hostname kernel: [] ? ip_local_out+0x18/0x20
Jul 15 07:10:30 hostname kernel: [] ? tcp_transmit_skb+0x395/0x660
Jul 15 07:10:30 hostname kernel: [] ? tcp_write_xmit+0x1dd/0x500
Jul 15 07:10:30 hostname kernel: [] ? __tcp_push_pending_frames+0x24/0x90
Jul 15 07:10:30 hostname kernel: [] ? tcp_rcv_established+0xfb/0x610
Jul 15 07:10:30 hostname kernel: [] ? tcp_v4_do_rcv+0xbb/0x190
Jul 15 07:10:30 hostname kernel: [] ? tcp_v4_rcv+0x6bd/0x7a0
Jul 15 07:10:30 hostname kernel: [] ? ip_local_deliver_finish+0x97/0x220
Jul 15 07:10:30 hostname kernel: [] ? ip_rcv+0x330/0x330
Jul 15 07:10:30 hostname kernel: [] ? ip_rcv_finish+0xd7/0x340
Jul 15 07:10:30 hostname kernel: [] ? __netif_receive_skb+0x2c3/0x350
Jul 15 07:10:30 hostname kernel: [] ? netif_receive_skb+0x1f/0x70
Jul 15 07:10:30 hostname kernel: [] ? handle_incoming_queue+0x1a1/0x270
Jul 15 07:10:30 hostname kernel: [] ? xennet_poll+0x224/0x570
Jul 15 07:10:30 hostname kernel: [] ? _raw_spin_unlock_irqrestore+0x11/0x20
Jul 15 07:10:30 hostname kernel: [] ? net_rx_action+0xea/0x1a0
Jul 15 07:10:30 hostname kernel: [] ? __do_softirq+0x7c/0x110
Jul 15 07:10:30 hostname kernel: [] ? irq_enter+0x70/0x70
Jul 15 07:10:30 hostname kernel: [] ? irq_exit+0x66/0x90
Jul 15 07:10:30 hostname kernel: [] ? xen_evtchn_do_upcall+0x1d/0x30
Jul 15 07:10:30 hostname kernel: [] ? xen_do_upcall+0x7/0xc
Jul 15 07:10:30 hostname kernel: [] ? hypercall_page+0x3a7/0x1000
Jul 15 07:10:30 hostname kernel: [] ? xen_safe_halt+0xf/0x20
Jul 15 07:10:30 hostname kernel: [] ? default_idle+0x1c/0x40
Jul 15 07:10:30 hostname kernel: [] ? cpu_idle+0x4a/0x80
Jul 15 07:10:30 hostname kernel: Mem-Info:
Jul 15 07:10:30 hostname kernel: DMA per-cpu:
Jul 15 07:10:30 hostname kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jul 15 07:10:30 hostname kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jul 15 07:10:30 hostname kernel: CPU 2: hi: 0, btch: 1 usd: 0
Jul 15 07:10:30 hostname kernel: CPU 3: hi: 0, btch: 1 usd: 0
Jul 15 07:10:30 hostname kernel: Normal per-cpu:
Jul 15 07:10:30 hostname kernel: CPU 0: hi: 186, btch: 31 usd: 129
Jul 15 07:10:30 hostname kernel: CPU 1: hi: 186, btch: 31 usd: 85
Jul 15 07:10:30 hostname kernel: CPU 2: hi: 186, btch: 31 usd: 28
Jul 15 07:10:30 hostname kernel: CPU 3: hi: 186, btch: 31 usd: 164
Jul 15 07:10:30 hostname kernel: active_anon:12649 inactive_anon:40482 isolated_anon:0
Jul 15 07:10:30 hostname kernel: active_file:37627 inactive_file:19347 isolated_file:0
Jul 15 07:10:30 hostname kernel: unevictable:0 dirty:11 writeback:0 unstable:0
Jul 15 07:10:30 hostname kernel: free:8971 slab_reclaimable:2680 slab_unreclaimable:1874
Jul 15 07:10:30 hostname kernel: mapped:10980 shmem:8204 pagetables:651 bounce:0
Jul 15 07:10:30 hostname kernel: DMA free:2088kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:8kB active_file:3232kB inactive_file:1764kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 15 07:10:30 hostname kernel: lowmem_reserve[]: 0 500 500 500
Jul 15 07:10:30 hostname kernel: Normal free:33796kB min:2816kB low:3520kB high:4224kB active_anon:50596kB inactive_anon:161920kB active_file:147276kB inactive_file:75624kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:44kB writeback:0kB mapped:43900kB shmem:32816kB slab_reclaimable:10680kB slab_unreclaimable:7496kB kernel_stack:1000kB pagetables:2604kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 15 07:10:30 hostname kernel: lowmem_reserve[]: 0 0 0 0
Jul 15 07:10:30 hostname kernel: DMA: 38*4kB 154*8kB 44*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2088kB
Jul 15 07:10:30 hostname kernel: Normal: 4229*4kB 1802*8kB 152*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 33796kB
Jul 15 07:10:30 hostname kernel: 68212 total pagecache pages
Jul 15 07:10:30 hostname kernel: 3021 pages in swap cache
Jul 15 07:10:30 hostname kernel: Swap cache stats: add 315367, delete 312346, find 239079/265626
Jul 15 07:10:30 hostname kernel: Free swap = 477752kB
Jul 15 07:10:30 hostname kernel: Total swap = 524284kB
Jul 15 07:10:30 hostname kernel: 133104 pages RAM
Jul 15 07:10:30 hostname kernel: 0 pages HighMem
Jul 15 07:10:30 hostname kernel: 5882 pages reserved
Jul 15 07:10:30 hostname kernel: 72851 pages shared
Jul 15 07:10:30 hostname kernel: 98932 pages non-shared
Jul 15 07:10:30 hostname kernel: SLAB: Unable to allocate memory on node 0 (gfp=0x20)
Jul 15 07:10:30 hostname kernel: cache: size-32768, object size: 32768, order: 3
Jul 15 07:10:30 hostname kernel: node 0: slabs: 1/1, objs: 1/1, free: 0
Your problem may involve a virtual memory setting that is too low. To check for this, at a Linux shell prompt execute:
cat /proc/sys/vm/min_free_kbytes
You can also check this with the sysctl program:
sysctl vm.min_free_kbytes
If you see a value under about 8192, there is a good chance the page allocation failures can be reduced or eliminated by raising this setting. Some virtual private server (VPS) providers are recommending even larger settings such as 16384. Some server tuning guides recommend even higher values, 656636 or larger, on systems with gigabytes of RAM. However, such large settings are probably not suitable for small dedicated server or VPS installations with less than 1GB RAM.
To change the setting, edit the /etc/sysctl.conf file to add the following line:
Continue reading →