The Linux and Open Source Community Website

  • vibrant gnu/linux user group
  • linux blog
  • events pictures
  • contact us
  • feedback
Home › Forums › VGLUG - Vibrant GNU/Linux User Group

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Fri, 07/29/2011 - 11:29

Hi

I am getting below error messages in my /var/log/messages
Swap gets out. and then processes are killed one by one.. sh,httpd2-
prefork,osiris.cgi repeating the same logs.. as below
Please somebody help why this is happening, what could be the cause of
the problem? what is oom-killer and why it gets invoked?

Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:
gfp_mask=0x1201d2, order=0, oomkilladj=0
Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted
2.6.27.19-5-xen #1
Jul 28 01:42:10 APP0001 kernel:
Jul 28 01:42:10 APP0001 kernel: Call Trace:
Jul 28 01:42:10 APP0001 kernel: []
show_trace_log_lvl+0x41/0x58
Jul 28 01:42:10 APP0001 kernel: [] dump_stack
+0x69/0x6f
Jul 28 01:42:10 APP0001 kernel: [] oom_kill_process
+0x5c/0x227
Jul 28 01:42:10 APP0001 kernel: [] out_of_memory
+0xdd/0x10d
Jul 28 01:42:10 APP0001 kernel: []
__alloc_pages_internal+0x2ef/0x3f7
Jul 28 01:42:10 APP0001 kernel: []
__do_page_cache_readahead+0x94/0x194
Jul 28 01:42:10 APP0001 kernel: []
force_page_cache_readahead+0x57/0x7e
Jul 28 01:42:10 APP0001 kernel: []
sys_fadvise64_64+0x11e/0x190
Jul 28 01:42:10 APP0001 kernel: []
system_call_fastpath+0x16/0x1b
Jul 28 01:42:10 APP0001 kernel: [<00007f2ac4a576dd>] 0x7f2ac4a576dd
Jul 28 01:42:10 APP0001 kernel:
Jul 28 01:42:10 APP0001 kernel: Mem-Info:
Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:
Jul 28 01:42:10 APP0001 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:
Jul 28 01:42:10 APP0001 kernel: CPU 0: hi: 186, btch: 31 usd: 176
Jul 28 01:42:10 APP0001 kernel: CPU 1: hi: 186, btch: 31 usd: 179
Jul 28 01:42:10 APP0001 kernel: CPU 2: hi: 186, btch: 31 usd: 158
Jul 28 01:42:10 APP0001 kernel: CPU 3: hi: 186, btch: 31 usd: 94
Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0
writeback:0 unstable:0
Jul 28 01:42:10 APP0001 kernel: free:4576 slab:2988 mapped:1
pagetables:0 bounce:0
Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0
all_unreclaimable? ye
s
Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016
Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:
10112kB high:12136kB active:2012168kB inactive:2020092kB present:
4112640kB pages_scanned:
7556254 all_unreclaimable? yes
Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0
Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB
1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB
Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB
1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB
Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages
Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache
Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete
4269236, find 8801268/8935257
Jul 28 01:42:10 APP0001 kernel: Free swap = 0kB
Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB
Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM
Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved
Jul 28 01:42:10 APP0001 kernel: 1899 pages shared
Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared
Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)
score 48420 or a child
Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)

Thanks
Niyati

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

‹ Understanding Open Source Licensing Codecracker 2012 @NIT Durgapur ›
  • VGLUG - Vibrant GNU/Linux User Group
  • Login to post comments

Squid big log file:

.bashrc — Thu, 01/26/2012 - 04:30

Hello,


I am running squid-3.2 and in squid.conf I put lot of restrictions. Because of that I am getting so many TCP_DENIED entries in access.log, which is overloading my log file making it bulky unnecessarily. Is there any way so that I can prevent such entries from writing to access.log.

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Alok Thaker — Fri, 07/29/2011 - 12:30

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati





--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Fri, 07/29/2011 - 12:30

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....


Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again

On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:
Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati





--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

kandarp thaker — Fri, 07/29/2011 - 12:30
Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a
leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u
more insight. Are you using rhel5. Then after some time it goes on
killing other processes to free up the memory.



Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof

M:-9978926593




On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again



On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati





--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG




--

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Fri, 07/29/2011 - 13:30

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.





On Fri, Jul 29, 2011 at 12:04 PM, kandarp thaker <kandarp.inferno@gmail.com> wrote:

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a
leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u
more insight. Are you using rhel5. Then after some time it goes on
killing other processes to free up the memory.



Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof

M:-9978926593




On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niyati277@gmail.com> wrote:
Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again



On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati





--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Nirmal Pathak — Fri, 07/29/2011 - 16:30



On 29 July 2011 12:37, Niyati Dave <niyati277@gmail.com> wrote:

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.


From the oom-kill messages I can see that the system has exhausted Swap Memory

--
Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB
--

I would suggest to increase Swap Space on the system prior to setting any memory related tunable on the system.


It is difficult from logs to find exact process which triggered the oom-kill.

Out of Memory occurs when the system memory is exhausted on the system and kernel tries to recover the memory by killing various process on the system which is consuming more memory.


You have to monitor your system continuously to find the cause!
 



On Fri, Jul 29, 2011 at 12:04 PM, kandarp thaker <kandarp.inferno@gmail.com> wrote:

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a
leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u
more insight. Are you using rhel5. Then after some time it goes on
killing other processes to free up the memory.



This does not seem to be httpd issue to me. Issue occurred as system had exhausted memory including the swap memory.

 
Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof

M:-9978926593




On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?

It could be both! :)
 
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


As mentioned above when system is under heavy memory load, the kernel starts killing processes which are consuming high memory on the system.

 


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again


On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati



Have FuN!
--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Fri, 07/29/2011 - 18:30

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?


Thanks
Niyati

On Fri, Jul 29, 2011 at 3:32 PM, Nirmal Pathak <nirmal.pathak@gmail.com> wrote:



On 29 July 2011 12:37, Niyati Dave <niyati277@gmail.com> wrote:

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.


From the oom-kill messages I can see that the system has exhausted Swap Memory

--

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

--

I would suggest to increase Swap Space on the system prior to setting any memory related tunable on the system.


It is difficult from logs to find exact process which triggered the oom-kill.

Out of Memory occurs when the system memory is exhausted on the system and kernel tries to recover the memory by killing various process on the system which is consuming more memory.


You have to monitor your system continuously to find the cause!
 



On Fri, Jul 29, 2011 at 12:04 PM, kandarp thaker <kandarp.inferno@gmail.com> wrote:

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a
leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u
more insight. Are you using rhel5. Then after some time it goes on
killing other processes to free up the memory.



This does not seem to be httpd issue to me. Issue occurred as system had exhausted memory including the swap memory.

 
Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof

M:-9978926593




On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?

It could be both! :)
 
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


As mentioned above when system is under heavy memory load, the kernel starts killing processes which are consuming high memory on the system.

 


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again


On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati



Have FuN!
--
Nirmal D Pathak.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Squid big log file:

.bashrc — Wed, 02/01/2012 - 02:30

Sorry Nishith for being late. Actually, access.log is critical for me. I parse it on daily basis so I cant rotate logs multiple times a day. I just wanted to skip the log entries with particular method(like None) so that it does not gets bulky so early.



On Tue, Jan 31, 2012 at 5:04 PM, Nishith Vyas <nishithnvyas@gmail.com> wrote:

Any Updates?




On Fri, Jan 27, 2012 at 5:09 PM, Nishith Vyas <nishithnvyas@gmail.com> wrote:

It's simple. If you don't want to monitor "access.log" for longer period of time, just make it empty by executing a script.......!!!!!

OR

type "> /var/spool/squid/access.log" command, which will empty your access.log file.

OR

take a backup of "access.log" & make a new file on the same location you have given in "squid.conf"
 
- Nishith



On Thu, Jan 26, 2012 at 4:19 AM, dinesh sohu <dineshsohu@gmail.com> wrote:

Hello,


I am running squid-3.2 and in squid.conf I put lot of restrictions. Because of that I am getting so many TCP_DENIED entries in access.log, which is overloading my log file making it bulky unnecessarily. Is there any way so that I can prevent such entries from writing to access.log.

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG




--
This e-mail is composed by-
Dinesh Sohu
Final Year Undergraduate,
Department of Computer Engineering,

Malaviya National Institute Of Technology, Jaipur.
Contact: +91-9414348496


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Squid big log file:

Nirmal Pathak — Wed, 02/01/2012 - 12:30



On Wednesday, February 1, 2012 1:39:55 AM UTC+5:30, .bashrc wrote:

Sorry Nishith for being late. Actually, access.log is critical for me. I parse it on daily basis so I cant rotate logs multiple times a day. I just wanted to skip the log entries with particular method(like None) so that it does not gets bulky so early.


You can try rotating logs using cron job. Check following link,
http://wiki.squid-cache.org/SquidFaq/SquidLogs#My_log_files_get_very_big.21

You can not simply filter the TCP_DENIED logs from log files. You will have to change the code of squid. :)
 
On Tue, Jan 31, 2012 at 5:04 PM, Nishith Vyas <nishit...@gmail.com> wrote:

Any Updates?




On Fri, Jan 27, 2012 at 5:09 PM, Nishith Vyas <nishit...@gmail.com> wrote:

It's simple. If you don't want to monitor "access.log" for longer period of time, just make it empty by executing a script.......!!!!!

OR

type "> /var/spool/squid/access.log" command, which will empty your access.log file.

OR

take a backup of "access.log" & make a new file on the same location you have given in "squid.conf"
 
- Nishith



On Thu, Jan 26, 2012 at 4:19 AM, dinesh sohu <dines...@gmail.com> wrote:

Hello,


I am running squid-3.2 and in squid.conf I put lot of restrictions. Because of that I am getting so many TCP_DENIED entries in access.log, which is overloading my log file making it bulky unnecessarily. Is there any way so that I can prevent such entries from writing to access.log.

--


Have FuN!
- Nirmal D Pathak.

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Squid big log file:

.bashrc — Thu, 02/02/2012 - 12:30

Or I seem to get to a solution. I can write a cron job by which I am parsing access.log periodically, say 2 hrs skipping the lines having None or/and TCP_DENIED in 4th and 7th field. Am I wrong anywhere(inspite a little overhead :) )?



On Wed, Feb 1, 2012 at 11:57 AM, Nirmal <nirmal.pathak@gmail.com> wrote:



On Wednesday, February 1, 2012 1:39:55 AM UTC+5:30, .bashrc wrote:
Sorry Nishith for being late. Actually, access.log is critical for me. I parse it on daily basis so I cant rotate logs multiple times a day. I just wanted to skip the log entries with particular method(like None) so that it does not gets bulky so early.


You can try rotating logs using cron job. Check following link,
http://wiki.squid-cache.org/SquidFaq/SquidLogs#My_log_files_get_very_big.21


You can not simply filter the TCP_DENIED logs from log files. You will have to change the code of squid. :)
 

On Tue, Jan 31, 2012 at 5:04 PM, Nishith Vyas <nishit...@gmail.com> wrote:

Any Updates?




On Fri, Jan 27, 2012 at 5:09 PM, Nishith Vyas <nishit...@gmail.com> wrote:

It's simple. If you don't want to monitor "access.log" for longer period of time, just make it empty by executing a script.......!!!!!
OR

type "> /var/spool/squid/access.log" command, which will empty your access.log file.

OR

take a backup of "access.log" & make a new file on the same location you have given in "squid.conf"
 
- Nishith



On Thu, Jan 26, 2012 at 4:19 AM, dinesh sohu <dines...@gmail.com> wrote:

Hello,


I am running squid-3.2 and in squid.conf I put lot of restrictions. Because of that I am getting so many TCP_DENIED entries in access.log, which is overloading my log file making it bulky unnecessarily. Is there any way so that I can prevent such entries from writing to access.log.

--


Have FuN!
- Nirmal D Pathak.

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG




--
This e-mail is composed by-
Dinesh Sohu
Final Year Undergraduate,
Department of Computer Engineering,

Malaviya National Institute Of Technology, Jaipur.
Contact: +91-9414348496


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Squid big log file:

nehal dattani — Thu, 02/02/2012 - 20:30

Hi,

This is what you have asked for.

"Is there any way so that I can prevent such entries from writing to access.log.
"

Therefore  we are focused on preventing unwanted entries in log. Of-cousre you can parse it periodically with certain conditions  but these entries are still there in log file.

So its your choice, how to deal with log files. For a busy server, avoiding unwanted I/O is a good step to increase performance.




On Thu, Feb 2, 2012 at 12:02 PM, dinesh sohu <dineshsohu@gmail.com> wrote:

Or I seem to get to a solution. I can write a cron job by which I am parsing access.log periodically, say 2 hrs skipping the lines having None or/and TCP_DENIED in 4th and 7th field. Am I wrong anywhere(inspite a little overhead :) )?



With best regards,
Nehal Dattani

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Squid big log file:

nehal dattani — Thu, 02/02/2012 - 11:30

Hi,

There is a little hack.

You can specify "none" for all those acl's  which you do not want to log.

Example:

access_log none [acl acl ...]



Please refer to description of "TAG: access_log" in squid.conf file for more details.

Regards,
Nehal Dattani


On Wed, Feb 1, 2012 at 11:57 AM, Nirmal <nirmal.pathak@gmail.com> wrote:



On Wednesday, February 1, 2012 1:39:55 AM UTC+5:30, .bashrc wrote:
Sorry Nishith for being late. Actually, access.log is critical for me. I parse it on daily basis so I cant rotate logs multiple times a day. I just wanted to skip the log entries with particular method(like None) so that it does not gets bulky so early.

You can try rotating logs using cron job. Check following link,
http://wiki.squid-cache.org/SquidFaq/SquidLogs#My_log_files_get_very_big.21


You can not simply filter the TCP_DENIED logs from log files. You will have to change the code of squid. :)
 
On Tue, Jan 31, 2012 at 5:04 PM, Nishith Vyas <nishit...@gmail.com> wrote:

Any Updates?




On Fri, Jan 27, 2012 at 5:09 PM, Nishith Vyas <nishit...@gmail.com> wrote:

It's simple. If you don't want to monitor "access.log" for longer period of time, just make it empty by executing a script.......!!!!!
OR

type "> /var/spool/squid/access.log" command, which will empty your access.log file.

OR

take a backup of "access.log" & make a new file on the same location you have given in "squid.conf"
 
- Nishith



On Thu, Jan 26, 2012 at 4:19 AM, dinesh sohu <dines...@gmail.com> wrote:

Hello,


I am running squid-3.2 and in squid.conf I put lot of restrictions. Because of that I am getting so many TCP_DENIED entries in access.log, which is overloading my log file making it bulky unnecessarily. Is there any way so that I can prevent such entries from writing to access.log.

--


Have FuN!
- Nirmal D Pathak.

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG




--
The production of too many useful things results in too many useless people.

With best regards,
Nehal Dattani

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

ElectroMech — Mon, 08/01/2011 - 09:30

Hi,


On Fri, Jul 29, 2011 at 5:49 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?


Can you give us more idea what services are provided by server ?


Mostly 4 G Memory is good, still depends on what application you served. Still we can change some parameters within application ( for memory utilization) so OOM-KILLER should not occurs.
 


Thanks
Niyati



Thanks and Regards.
--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner

404, Maulik arcade, Above Karnavati Pagarakha Bazar,
Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Mon, 08/01/2011 - 11:30

Hello all

I have read the above discussion, so I would like to clarify a few things.. the kernel is a latest one as provided by the suselinux enterprise server and version 11 - so the latest one. The server is a xen guest and applications running on the server is just apache, postfix. But yes it has a good usage of apache application and there are few cron scripts related to emailing, cgi. And as per the logs too after giving out of memory error related to swap then oom-killer started killing sh, httpd-prefork, top, the script.


Thanks again
Niyati

On Mon, Aug 1, 2011 at 8:39 AM, ElectroMech <electromech@electromech.info> wrote:

Hi,


On Fri, Jul 29, 2011 at 5:49 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?



Can you give us more idea what services are provided by server ?


Mostly 4 G Memory is good, still depends on what application you served. Still we can change some parameters within application ( for memory utilization) so OOM-KILLER should not occurs.
 


Thanks
Niyati



Thanks and Regards.
--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation

Redhat Channel Partner and Training Partner

404, Maulik arcade, Above Karnavati Pagarakha Bazar,
Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info




--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Mon, 08/01/2011 - 14:30

Hi all

I am continuously facing the same logs and problem again. I want to try out increasing the swap. What I know is creating a swap file or a new swap partition. Is there any way to increase the present swap which is already of 2G.

Waiting for resonse

Thanks
Niyati
 


On Mon, Aug 1, 2011 at 11:09 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hello all

I have read the above discussion, so I would like to clarify a few things.. the kernel is a latest one as provided by the suselinux enterprise server and version 11 - so the latest one. The server is a xen guest and applications running on the server is just apache, postfix. But yes it has a good usage of apache application and there are few cron scripts related to emailing, cgi. And as per the logs too after giving out of memory error related to swap then oom-killer started killing sh, httpd-prefork, top, the script.


Thanks again
Niyati



On Mon, Aug 1, 2011 at 8:39 AM, ElectroMech <electromech@electromech.info> wrote:

Hi,


On Fri, Jul 29, 2011 at 5:49 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?


Can you give us more idea what services are provided by server ?


Mostly 4 G Memory is good, still depends on what application you served. Still we can change some parameters within application ( for memory utilization) so OOM-KILLER should not occurs.
 


Thanks
Niyati



Thanks and Regards.
--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation

Redhat Channel Partner and Training Partner

404, Maulik arcade, Above Karnavati Pagarakha Bazar,
Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info




--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG





--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Lan-port configuration problem:

.bashrc — Thu, 01/12/2012 - 12:30



Hello,

I have IBM x3450 rack server with two identical racks with 3 ethernet interfaces interfaces on each. I just tried to configure them on 1000baseT-FD mode using mii-tools and ethtool. It worked on all except one that refuses to come up on 1000baseT mode. It, then, goes in 100baseT mode and denies to set up on 1000Mbps mode. Any help appreciated.

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

RE: Lan-port configuration problem:

Arvind Patel — Thu, 01/12/2012 - 12:30




Hi dinesh
 
plaese check with your network switch , it is supported up to 1Gbps or not?
 
Regards..
 
Arvind
 

From: dineshsohu@gmail.com
Date: Thu, 12 Jan 2012 11:52:54 +0530
Subject: [VGLUG] Lan-port configuration problem:
To: vglug@googlegroups.com



Hello,

I have IBM x3450 rack server with two identical racks with 3 ethernet interfaces interfaces on each. I just tried to configure them on 1000baseT-FD mode using mii-tools and ethtool. It worked on all except one that refuses to come up on 1000baseT mode. It, then, goes in 100baseT mode and denies to set up on 1000Mbps mode. Any help appreciated.


--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Lan-port configuration problem:

.bashrc — Thu, 01/12/2012 - 13:30

The network switch is configured to 1Gbps on all ports coz the other machine is connected on same switch. Also I crosschecked with switch admin. One thing, how can I find out h/w capabilities of a ethernet port.(I am using debian)



On Thu, Jan 12, 2012 at 12:02 PM, Arvind Patel <arvind_cmc@hotmail.com> wrote:

Hi dinesh
 
plaese check with your network switch , it is supported up to 1Gbps or not?
 
Regards..
 
Arvind
 

From: dineshsohu@gmail.com

Date: Thu, 12 Jan 2012 11:52:54 +0530
Subject: [VGLUG] Lan-port configuration problem:
To: vglug@googlegroups.com





Hello,

I have IBM x3450 rack server with two identical racks with 3 ethernet interfaces interfaces on each. I just tried to configure them on 1000baseT-FD mode using mii-tools and ethtool. It worked on all except one that refuses to come up on 1000baseT mode. It, then, goes in 100baseT mode and denies to set up on 1000Mbps mode. Any help appreciated.


--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG




-- 

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

ElectroMech — Mon, 08/01/2011 - 16:30

Hi,


On Mon, Aug 1, 2011 at 1:52 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hi all

I am continuously facing the same logs and problem again. I want to try out increasing the swap. What I know is creating a swap file or a new swap partition. Is there any way to increase the present swap which is already of 2G.

Waiting for resonse

Can you send me the output of following command.

[root@nilesh ~]# top -b -u apache -n 1

Might suggest you soemthing valueable.

Thanks
Niyati

 --

--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner

404, Maulik arcade, Above Karnavati Pagarakha Bazar,
Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Mon, 08/01/2011 - 16:30

Below is the output of top -b -u wwwrun -n 1

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38
Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st

Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

16057 wwwrun    20   0 97364  27m 6300 R   28  0.7   0:02.52 osiris.cgi
12918 wwwrun    20   0  154m  87m 6516 R   26  2.1   4:29.78 osiris.cgi
32376 wwwrun    20   0  839m 456m  860 R   26 11.1  29:21.22 osiris.cgi

 2111 wwwrun    20   0  778m 456m  860 R   24 11.1  25:33.46 osiris.cgi
 3033 wwwrun    20   0  757m 459m  860 R   24 11.2  24:19.25 osiris.cgi
11107 wwwrun    20   0  273m 206m 6516 R   24  5.0   8:17.03 osiris.cgi

11543 wwwrun    20   0  154m  87m 6516 R   24  2.1   7:15.97 osiris.cgi
12162 wwwrun    20   0  154m  87m 6516 R   24  2.1   5:43.74 osiris.cgi
12911 wwwrun    20   0  154m  87m 6516 R   24  2.1   4:29.74 osiris.cgi

16105 wwwrun    20   0 97336  17m 6172 R   24  0.4   0:01.20 osiris.cgi
12742 wwwrun    20   0  154m  87m 6516 R   22  2.1   4:36.37 osiris.cgi
10224 wwwrun    20   0  456m 389m 6516 R   20  9.5  11:18.80 osiris.cgi

12740 wwwrun    20   0  154m  87m 6516 R   20  2.1   3:59.65 osiris.cgi
14356 wwwrun    20   0  140m  73m 6516 R   20  1.8   1:57.56 osiris.cgi
16015 wwwrun    20   0  143m  76m 6348 R   20  1.9   0:06.58 osiris.cgi

 1870 wwwrun    20   0  773m 464m  860 R   18 11.3  25:29.98 osiris.cgi
 9615 wwwrun    20   0  540m 445m 6516 R   16 10.9  13:34.17 osiris.cgi
15622 wwwrun    20   0 91940 2784 1832 S    4  0.1   0:00.08 httpd2-prefork

  834 wwwrun    20   0 92168  352    4 S    0  0.0   0:00.30 httpd2-prefork
 1532 wwwrun    20   0 92196 2664 1492 S    0  0.1   0:00.10 httpd2-prefork
 1545 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.08 httpd2-prefork

 2076 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.00 httpd2-prefork
 2389 wwwrun    20   0 91924  352    4 S    0  0.0   0:03.66 httpd2-prefork
 2554 wwwrun    20   0 92024  352    4 S    0  0.0   0:00.04 httpd2-prefork

 4047 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.08 httpd2-prefork
 4065 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.04 httpd2-prefork
 4069 wwwrun    20   0 91936  352    4 S    0  0.0   0:00.06 httpd2-prefork

 4089 wwwrun    20   0 43608    8    4 S    0  0.0   0:00.00 osiris.cgi
 4167 wwwrun    20   0 91804  352    4 S    0  0.0   0:00.00 httpd2-prefork
 4176 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.02 httpd2-prefork

 5449 wwwrun    20   0 94500  352    4 S    0  0.0   0:00.20 httpd2-prefork
 9022 wwwrun    20   0 94652 2432 1868 S    0  0.1   0:00.14 httpd2-prefork
 9221 wwwrun    20   0 92168 2740 1888 S    0  0.1   0:00.22 httpd2-prefork

 9733 wwwrun    20   0 94744 5720 1952 S    0  0.1   0:00.34 httpd2-prefork
11196 wwwrun    20   0 91940 2784 1824 S    0  0.1   0:00.06 httpd2-prefork
11612 wwwrun    20   0 92192 3184 1920 S    0  0.1   0:00.20 httpd2-prefork

12322 wwwrun    20   0 92988 3308 1432 S    0  0.1   0:00.96 httpd2-prefork
12388 wwwrun    20   0 92060 2880 1788 S    0  0.1   0:00.06 httpd2-prefork
12870 wwwrun    20   0 91804 2428 1588 S    0  0.1   0:00.00 httpd2-prefork

12881 wwwrun    20   0 91804 2388 1564 S    0  0.1   0:00.00 httpd2-prefork
13992 wwwrun    20   0 91960 3120 2016 S    0  0.1   0:00.30 httpd2-prefork
14202 wwwrun    20   0 91924 2692 1756 S    0  0.1   0:00.02 httpd2-prefork

15199 wwwrun    20   0 92168 3152 1864 S    0  0.1   0:00.10 httpd2-prefork
15300 wwwrun    20   0 92668 3524 1864 S    0  0.1   0:00.08 httpd2-prefork
15308 wwwrun    20   0 91948 2952 1872 S    0  0.1   0:00.08 httpd2-prefork

15444 wwwrun    20   0 91936 2936 1876 S    0  0.1   0:00.06 httpd2-prefork
15581 wwwrun    20   0 92112 3036 1868 S    0  0.1   0:00.08 httpd2-prefork
15617 wwwrun    20   0 92056 2916 1772 S    0  0.1   0:00.04 httpd2-prefork

15839 wwwrun    20   0 91968 2920 1844 S    0  0.1   0:00.04 httpd2-prefork
15864 wwwrun    20   0 91924 2544 1612 S    0  0.1   0:00.02 httpd2-prefork
15978 wwwrun    20   0 91940 2544 1624 S    0  0.1   0:00.00 httpd2-prefork

16001 wwwrun    20   0     0    0    0 Z    0  0.0   0:00.00 httpd2-prefork <defunct>
16017 wwwrun    20   0 91928 2684 1748 S    0  0.1   0:00.02 httpd2-prefork
31900 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.04 httpd2-prefork




On Mon, Aug 1, 2011 at 4:15 PM, ElectroMech <electromech@electromech.info> wrote:

Hi,


On Mon, Aug 1, 2011 at 1:52 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hi all

I am continuously facing the same logs and problem again. I want to try out increasing the swap. What I know is creating a swap file or a new swap partition. Is there any way to increase the present swap which is already of 2G.

Waiting for resonse

Can you send me the output of following command.

[root@nilesh ~]# top -b -u apache -n 1

Might suggest you soemthing valueable.

Thanks
Niyati

 --

--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner

404, Maulik arcade, Above Karnavati Pagarakha Bazar,
Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info


--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

ElectroMech — Mon, 08/01/2011 - 19:30

Hi,




On Mon, Aug 1, 2011 at 4:23 PM, Niyati Dave <niyati277@gmail.com> wrote:

Below is the output of top -b -u wwwrun -n 1

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38
Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st

Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

Can you reduce

<IfModule prefork.c>

StartServers       8  
MinSpareServers    5
MaxSpareServers   20 <============== check this value try to reduce it to half from actual value
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

</IfModule>



 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

16057 wwwrun    20   0 97364  27m 6300 R   28  0.7   0:02.52 osiris.cgi
12918 wwwrun    20   0  154m  87m 6516 R   26  2.1   4:29.78 osiris.cgi
32376 wwwrun    20   0  839m 456m  860 R   26 11.1  29:21.22 osiris.cgi

 2111 wwwrun    20   0  778m 456m  860 R   24 11.1  25:33.46 osiris.cgi
 3033 wwwrun    20   0  757m 459m  860 R   24 11.2  24:19.25 osiris.cgi
11107 wwwrun    20   0  273m 206m 6516 R   24  5.0   8:17.03 osiris.cgi

11543 wwwrun    20   0  154m  87m 6516 R   24  2.1   7:15.97 osiris.cgi
12162 wwwrun    20   0  154m  87m 6516 R   24  2.1   5:43.74 osiris.cgi
12911 wwwrun    20   0  154m  87m 6516 R   24  2.1   4:29.74 osiris.cgi

16105 wwwrun    20   0 97336  17m 6172 R   24  0.4   0:01.20 osiris.cgi
12742 wwwrun    20   0  154m  87m 6516 R   22  2.1   4:36.37 osiris.cgi
10224 wwwrun    20   0  456m 389m 6516 R   20  9.5  11:18.80 osiris.cgi

12740 wwwrun    20   0  154m  87m 6516 R   20  2.1   3:59.65 osiris.cgi
14356 wwwrun    20   0  140m  73m 6516 R   20  1.8   1:57.56 osiris.cgi
16015 wwwrun    20   0  143m  76m 6348 R   20  1.9   0:06.58 osiris.cgi

 1870 wwwrun    20   0  773m 464m  860 R   18 11.3  25:29.98 osiris.cgi
 9615 wwwrun    20   0  540m 445m 6516 R   16 10.9  13:34.17 osiris.cgi
15622 wwwrun    20   0 91940 2784 1832 S    4  0.1   0:00.08 httpd2-prefork

  834 wwwrun    20   0 92168  352    4 S    0  0.0   0:00.30 httpd2-prefork
 1532 wwwrun    20   0 92196 2664 1492 S    0  0.1   0:00.10 httpd2-prefork
 1545 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.08 httpd2-prefork

 2076 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.00 httpd2-prefork
 2389 wwwrun    20   0 91924  352    4 S    0  0.0   0:03.66 httpd2-prefork
 2554 wwwrun    20   0 92024  352    4 S    0  0.0   0:00.04 httpd2-prefork

 4047 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.08 httpd2-prefork
 4065 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.04 httpd2-prefork
 4069 wwwrun    20   0 91936  352    4 S    0  0.0   0:00.06 httpd2-prefork

 4089 wwwrun    20   0 43608    8    4 S    0  0.0   0:00.00 osiris.cgi
 4167 wwwrun    20   0 91804  352    4 S    0  0.0   0:00.00 httpd2-prefork
 4176 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.02 httpd2-prefork

 5449 wwwrun    20   0 94500  352    4 S    0  0.0   0:00.20 httpd2-prefork
 9022 wwwrun    20   0 94652 2432 1868 S    0  0.1   0:00.14 httpd2-prefork
 9221 wwwrun    20   0 92168 2740 1888 S    0  0.1   0:00.22 httpd2-prefork

 9733 wwwrun    20   0 94744 5720 1952 S    0  0.1   0:00.34 httpd2-prefork
11196 wwwrun    20   0 91940 2784 1824 S    0  0.1   0:00.06 httpd2-prefork
11612 wwwrun    20   0 92192 3184 1920 S    0  0.1   0:00.20 httpd2-prefork

12322 wwwrun    20   0 92988 3308 1432 S    0  0.1   0:00.96 httpd2-prefork
12388 wwwrun    20   0 92060 2880 1788 S    0  0.1   0:00.06 httpd2-prefork
12870 wwwrun    20   0 91804 2428 1588 S    0  0.1   0:00.00 httpd2-prefork

12881 wwwrun    20   0 91804 2388 1564 S    0  0.1   0:00.00 httpd2-prefork
13992 wwwrun    20   0 91960 3120 2016 S    0  0.1   0:00.30 httpd2-prefork
14202 wwwrun    20   0 91924 2692 1756 S    0  0.1   0:00.02 httpd2-prefork

15199 wwwrun    20   0 92168 3152 1864 S    0  0.1   0:00.10 httpd2-prefork
15300 wwwrun    20   0 92668 3524 1864 S    0  0.1   0:00.08 httpd2-prefork
15308 wwwrun    20   0 91948 2952 1872 S    0  0.1   0:00.08 httpd2-prefork

15444 wwwrun    20   0 91936 2936 1876 S    0  0.1   0:00.06 httpd2-prefork
15581 wwwrun    20   0 92112 3036 1868 S    0  0.1   0:00.08 httpd2-prefork
15617 wwwrun    20   0 92056 2916 1772 S    0  0.1   0:00.04 httpd2-prefork

15839 wwwrun    20   0 91968 2920 1844 S    0  0.1   0:00.04 httpd2-prefork
15864 wwwrun    20   0 91924 2544 1612 S    0  0.1   0:00.02 httpd2-prefork
15978 wwwrun    20   0 91940 2544 1624 S    0  0.1   0:00.00 httpd2-prefork

16001 wwwrun    20   0     0    0    0 Z    0  0.0   0:00.00 httpd2-prefork <defunct>
16017 wwwrun    20   0 91928 2684 1748 S    0  0.1   0:00.02 httpd2-prefork
31900 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.04 httpd2-prefork




On Mon, Aug 1, 2011 at 4:15 PM, ElectroMech <electromech@electromech.info> wrote:
Hi,


On Mon, Aug 1, 2011 at 1:52 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hi all

I am continuously facing the same logs and problem again. I want to try out increasing the swap. What I know is creating a swap file or a new swap partition. Is there any way to increase the present swap which is already of 2G.

Waiting for resonse

Can you send me the output of following command.

[root@nilesh ~]# top -b -u apache -n 1

Might suggest you soemthing valueable.



Thanks and Regards.

--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner
404, Maulik arcade, Above Karnavati Pagarakha Bazar,

Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Mon, 08/01/2011 - 19:30

I have the value of MaxSpareServers 10. Should I still try changing.


On Mon, Aug 1, 2011 at 7:01 PM, ElectroMech <electromech@electromech.info> wrote:

Hi,




On Mon, Aug 1, 2011 at 4:23 PM, Niyati Dave <niyati277@gmail.com> wrote:

Below is the output of top -b -u wwwrun -n 1

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38
Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st

Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

Can you reduce

<IfModule prefork.c>

StartServers       8  
MinSpareServers    5
MaxSpareServers   20 <============== check this value try to reduce it to half from actual value
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

</IfModule>



 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

16057 wwwrun    20   0 97364  27m 6300 R   28  0.7   0:02.52 osiris.cgi
12918 wwwrun    20   0  154m  87m 6516 R   26  2.1   4:29.78 osiris.cgi
32376 wwwrun    20   0  839m 456m  860 R   26 11.1  29:21.22 osiris.cgi

 2111 wwwrun    20   0  778m 456m  860 R   24 11.1  25:33.46 osiris.cgi
 3033 wwwrun    20   0  757m 459m  860 R   24 11.2  24:19.25 osiris.cgi
11107 wwwrun    20   0  273m 206m 6516 R   24  5.0   8:17.03 osiris.cgi

11543 wwwrun    20   0  154m  87m 6516 R   24  2.1   7:15.97 osiris.cgi
12162 wwwrun    20   0  154m  87m 6516 R   24  2.1   5:43.74 osiris.cgi
12911 wwwrun    20   0  154m  87m 6516 R   24  2.1   4:29.74 osiris.cgi

16105 wwwrun    20   0 97336  17m 6172 R   24  0.4   0:01.20 osiris.cgi
12742 wwwrun    20   0  154m  87m 6516 R   22  2.1   4:36.37 osiris.cgi
10224 wwwrun    20   0  456m 389m 6516 R   20  9.5  11:18.80 osiris.cgi

12740 wwwrun    20   0  154m  87m 6516 R   20  2.1   3:59.65 osiris.cgi
14356 wwwrun    20   0  140m  73m 6516 R   20  1.8   1:57.56 osiris.cgi
16015 wwwrun    20   0  143m  76m 6348 R   20  1.9   0:06.58 osiris.cgi

 1870 wwwrun    20   0  773m 464m  860 R   18 11.3  25:29.98 osiris.cgi
 9615 wwwrun    20   0  540m 445m 6516 R   16 10.9  13:34.17 osiris.cgi
15622 wwwrun    20   0 91940 2784 1832 S    4  0.1   0:00.08 httpd2-prefork

  834 wwwrun    20   0 92168  352    4 S    0  0.0   0:00.30 httpd2-prefork
 1532 wwwrun    20   0 92196 2664 1492 S    0  0.1   0:00.10 httpd2-prefork
 1545 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.08 httpd2-prefork

 2076 wwwrun    20   0 91940  352    4 S    0  0.0   0:00.00 httpd2-prefork
 2389 wwwrun    20   0 91924  352    4 S    0  0.0   0:03.66 httpd2-prefork
 2554 wwwrun    20   0 92024  352    4 S    0  0.0   0:00.04 httpd2-prefork

 4047 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.08 httpd2-prefork
 4065 wwwrun    20   0 91928  352    4 S    0  0.0   0:00.04 httpd2-prefork
 4069 wwwrun    20   0 91936  352    4 S    0  0.0   0:00.06 httpd2-prefork

 4089 wwwrun    20   0 43608    8    4 S    0  0.0   0:00.00 osiris.cgi
 4167 wwwrun    20   0 91804  352    4 S    0  0.0   0:00.00 httpd2-prefork
 4176 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.02 httpd2-prefork

 5449 wwwrun    20   0 94500  352    4 S    0  0.0   0:00.20 httpd2-prefork
 9022 wwwrun    20   0 94652 2432 1868 S    0  0.1   0:00.14 httpd2-prefork
 9221 wwwrun    20   0 92168 2740 1888 S    0  0.1   0:00.22 httpd2-prefork

 9733 wwwrun    20   0 94744 5720 1952 S    0  0.1   0:00.34 httpd2-prefork
11196 wwwrun    20   0 91940 2784 1824 S    0  0.1   0:00.06 httpd2-prefork
11612 wwwrun    20   0 92192 3184 1920 S    0  0.1   0:00.20 httpd2-prefork

12322 wwwrun    20   0 92988 3308 1432 S    0  0.1   0:00.96 httpd2-prefork
12388 wwwrun    20   0 92060 2880 1788 S    0  0.1   0:00.06 httpd2-prefork
12870 wwwrun    20   0 91804 2428 1588 S    0  0.1   0:00.00 httpd2-prefork

12881 wwwrun    20   0 91804 2388 1564 S    0  0.1   0:00.00 httpd2-prefork
13992 wwwrun    20   0 91960 3120 2016 S    0  0.1   0:00.30 httpd2-prefork
14202 wwwrun    20   0 91924 2692 1756 S    0  0.1   0:00.02 httpd2-prefork

15199 wwwrun    20   0 92168 3152 1864 S    0  0.1   0:00.10 httpd2-prefork
15300 wwwrun    20   0 92668 3524 1864 S    0  0.1   0:00.08 httpd2-prefork
15308 wwwrun    20   0 91948 2952 1872 S    0  0.1   0:00.08 httpd2-prefork

15444 wwwrun    20   0 91936 2936 1876 S    0  0.1   0:00.06 httpd2-prefork
15581 wwwrun    20   0 92112 3036 1868 S    0  0.1   0:00.08 httpd2-prefork
15617 wwwrun    20   0 92056 2916 1772 S    0  0.1   0:00.04 httpd2-prefork

15839 wwwrun    20   0 91968 2920 1844 S    0  0.1   0:00.04 httpd2-prefork
15864 wwwrun    20   0 91924 2544 1612 S    0  0.1   0:00.02 httpd2-prefork
15978 wwwrun    20   0 91940 2544 1624 S    0  0.1   0:00.00 httpd2-prefork

16001 wwwrun    20   0     0    0    0 Z    0  0.0   0:00.00 httpd2-prefork <defunct>
16017 wwwrun    20   0 91928 2684 1748 S    0  0.1   0:00.02 httpd2-prefork
31900 wwwrun    20   0 91924  352    4 S    0  0.0   0:00.04 httpd2-prefork




On Mon, Aug 1, 2011 at 4:15 PM, ElectroMech <electromech@electromech.info> wrote:

Hi,


On Mon, Aug 1, 2011 at 1:52 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hi all

I am continuously facing the same logs and problem again. I want to try out increasing the swap. What I know is creating a swap file or a new swap partition. Is there any way to increase the present swap which is already of 2G.

Waiting for resonse

Can you send me the output of following command.

[root@nilesh ~]# top -b -u apache -n 1

Might suggest you soemthing valueable.



Thanks and Regards.



--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner

404, Maulik arcade, Above Karnavati Pagarakha Bazar,

Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

ElectroMech — Mon, 08/01/2011 - 20:30

Hi,


On Mon, Aug 1, 2011 at 7:19 PM, Niyati Dave <niyati277@gmail.com> wrote:

I have the value of MaxSpareServers 10. Should I still try changing.


Always check top and free -m.

Check it should not utilize swap Memory, if used should temporary.

Check also uptime for load average should not be greater than total logical cpu.


If you want change for performance should change following value without increasing usage of memory.

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections

# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process

# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2  
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25

MaxRequestsPerChild  0
</IfModule>

Let us know the impact and check the log.
 
On Mon, Aug 1, 2011 at 7:01 PM, ElectroMech <electromech@electromech.info> wrote:
Hi,




On Mon, Aug 1, 2011 at 4:23 PM, Niyati Dave <niyati277@gmail.com> wrote:

Below is the output of top -b -u wwwrun -n 1


Can you provide again the output of above ?

Remember to restart the apache.
 

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38

Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st

Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

Can you reduce


--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner
404, Maulik arcade, Above Karnavati Pagarakha Bazar,

Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Niyati Dave — Mon, 08/01/2011 - 20:30

Hii

This is the free -m

            total       used       free     shared    buffers     cached
Mem:          4102       4069         32          0          4         39
-/+ buffers/cache:       4025         76

Swap:         2055       2038         17

Today it has remained fully utilised- swap and also sending the output of top

top - 16:15:45 up 130 days,  5:03,  2 users,  load average: 12.20, 12.14, 13.94
Tasks: 115 total,  10 running, 105 sleeping,   0 stopped,   0 zombie

Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   4201096k total,  4161232k used,    39864k free,    11604k buffers
Swap:  2104472k total,  2093824k used,    10648k free,    51360k cached


  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10271 wwwrun    20   0 97740  28m 6052 R   40  0.7   0:02.64 osiris.cgi
10277 wwwrun    20   0 97376  24m 6296 R   40  0.6   0:02.06 osiris.cgi

10281 wwwrun    20   0 83064  12m 5284 R   40  0.3   0:01.30 osiris.cgi
 5729 wwwrun    20   0  378m 273m  860 R   38  6.7  10:10.27 osiris.cgi
21345 wwwrun    20   0  941m 434m  328 R   38 10.6  40:34.26 osiris.cgi

30444 wwwrun    20   0  807m 430m  860 R   38 10.5  27:06.21 osiris.cgi
14356 wwwrun    20   0  926m 529m  324 D   36 12.9  45:46.06 osiris.cgi
10264 wwwrun    20   0  124m  57m 6204 R   32  1.4   0:04.60 osiris.cgi

12162 wwwrun    20   0  943m 480m  324 R   32 11.7  49:07.30 osiris.cgi
25712 wwwrun    20   0  924m 603m  964 R   32 14.7  34:36.30 osiris.cgi
28097 wwwrun    20   0  893m 525m  964 D   32 12.8  31:20.03 osiris.cgi

 5578 wwwrun    20   0 92068  804    4 S    0  0.0   0:00.14 httpd2-prefork
 6915 wwwrun    20   0 92952 3548 1620 S    0  0.1   0:00.14 httpd2-prefork
 8131 wwwrun    20   0 94892 5752 1956 S    0  0.1   0:00.56 httpd2-prefork

 8371 wwwrun    20   0 92364 3388 1932 S    0  0.1   0:00.36 httpd2-prefork
 8644 wwwrun    20   0 92336 3404 1924 S    0  0.1   0:00.28 httpd2-prefork
 8683 wwwrun    20   0 92348 3420 1940 S    0  0.1   0:00.34 httpd2-prefork

 9093 wwwrun    20   0 94908 5676 1916 S    0  0.1   0:00.30 httpd2-prefork
 9094 wwwrun    20   0 92840 3852 1908 S    0  0.1   0:00.26 httpd2-prefork
 9352 wwwrun    20   0 92276 3248 1900 S    0  0.1   0:00.18 httpd2-prefork

 9866 wwwrun    20   0 92688 3172 1848 S    0  0.1   0:00.08 httpd2-prefork
 9982 wwwrun    20   0 92680 3548 1852 S    0  0.1   0:00.06 httpd2-prefork
10108 wwwrun    20   0 92068 2712 1620 S    0  0.1   0:00.00 httpd2-prefork

10121 wwwrun    20   0 92656 3292 1636 S    0  0.1   0:00.00 httpd2-prefork
10210 wwwrun    20   0 92068 2684 1632 S    0  0.1   0:00.00 httpd2-prefork
10268 wwwrun    20   0 97952  23m 5848 S    0  0.6   0:02.66 osiris.cgi

12322 wwwrun    20   0 92988   52    4 S    0  0.0   0:00.96 httpd2-prefork
14202 wwwrun    20   0 91924   56    4 S    0  0.0   0:00.02 httpd2-prefork
21059 wwwrun    20   0 91936   64    4 S    0  0.0   0:00.04 httpd2-prefork

24808 wwwrun    20   0 92364   84    4 S    0  0.0   0:00.20 httpd2-prefork
27685 wwwrun    20   0 94852  140    4 S    0  0.0   0:00.14 httpd2-prefork
30437 wwwrun    20   0 92068  652    4 S    0  0.0   0:00.00 httpd2-prefork

32242 wwwrun    20   0 92308  632    4 S    0  0.0   0:00.04 httpd2-prefork



On Mon, Aug 1, 2011 at 7:31 PM, ElectroMech <electromech@electromech.info> wrote:

Hi,


On Mon, Aug 1, 2011 at 7:19 PM, Niyati Dave <niyati277@gmail.com> wrote:

I have the value of MaxSpareServers 10. Should I still try changing.


Always check top and free -m.

Check it should not utilize swap Memory, if used should temporary.

Check also uptime for load average should not be greater than total logical cpu.


If you want change for performance should change following value without increasing usage of memory.

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections

# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process

# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2  
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25

MaxRequestsPerChild  0
</IfModule>

Let us know the impact and check the log.
 

On Mon, Aug 1, 2011 at 7:01 PM, ElectroMech <electromech@electromech.info> wrote:
Hi,




On Mon, Aug 1, 2011 at 4:23 PM, Niyati Dave <niyati277@gmail.com> wrote:

Below is the output of top -b -u wwwrun -n 1


Can you provide again the output of above ?

Remember to restart the apache.

 

top - 12:58:50 up 130 days,  1:46,  1 user,  load average: 22.64, 22.90, 20.38

Tasks: 127 total,  19 running, 107 sleeping,   0 stopped,   1 zombie
Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st

Mem:   4201096k total,  4061640k used,   139456k free,    12768k buffers
Swap:  2104472k total,  1126172k used,   978300k free,    69968k cached

Can you reduce


--
--
Nilesh Vaghela
(RHCSA RHCE)
ElectroMech Corporation
Redhat Channel Partner and Training Partner
404, Maulik arcade, Above Karnavati Pagarakha Bazar,

Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.
www.electromech.info

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

ElectroMech — Tue, 08/02/2011 - 11:30

Hi,


On Mon, Aug 1, 2011 at 7:39 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hii

This is the free -m

            total       used       free     shared    buffers     cached
Mem:          4102       4069         32          0          4         39
-/+ buffers/cache:       4025         76

Swap:         2055       2038         17

Today it has remained fully utilised- swap and also sending the output of top

Still it seems you have around 22 apache processes it should be 11 only, if we refer last config.


Please check the setting MaxSpareServers 10 is ok and you have restarted apache.

One more observation

 

top - 16:15:45 up 130 days,  5:03,  2 users,  load average: 12.20, 12.14, 13.94
Tasks: 115 total,  10 running, 105 sleeping,   0 stopped,   0 zombie

Cpu(s):  9.7%us,  0.4%sy,  0.0%ni, 89.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:   4201096k total,  4161232k used,    39864k free,    11604k buffers
Swap:  2104472k total,  2093824k used,    10648k free,    51360k cached




  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

10271 wwwrun    20   0 97740  28m 6052 R   40  0.7   0:02.64 osiris.cgi
10277 wwwrun    20   0 97376  24m 6296 R   40  0.6   0:02.06 osiris.cgi

10281 wwwrun    20   0 83064  12m 5284 R   40  0.3   0:01.30 osiris.cgi
 5729 wwwrun    20   0  378m 273m  860 R   38  6.7  10:10.27 osiris.cgi
21345 wwwrun    20   0  941m 434m  328 R   38 10.6  40:34.26 osiris.cgi

30444 wwwrun    20   0  807m 430m  860 R   38 10.5  27:06.21 osiris.cgi
14356 wwwrun    20   0  926m 529m  324 D   36 12.9  45:46.06 osiris.cgi
10264 wwwrun    20   0  124m  57m 6204 R   32  1.4   0:04.60 osiris.cgi

12162 wwwrun    20   0  943m 480m  324 R   32 11.7  49:07.30 osiris.cgi
25712 wwwrun    20   0  924m 603m  964 R   32 14.7  34:36.30 osiris.cgi
28097 wwwrun    20   0  893m 525m  964 D   32 12.8  31:20.03 osiris.cgi


There are some problem in osiris.cgi almost consuming all memory and processor through apache.

What is the process ? please check or let us have some information.

 5578 wwwrun    20   0 92068  804    4 S    0  0.0   0:00.14 httpd2-prefork
 6915 wwwrun    20   0 92952 3548 1620 S    0  0.1   0:00.14 httpd2-prefork
 8131 wwwrun    20   0 94892 5752 1956 S    0  0.1   0:00.56 httpd2-prefork

 8371 wwwrun    20   0 92364 3388 1932 S    0  0.1   0:00.36 httpd2-prefork
 8644 wwwrun    20   0 92336 3404 1924 S    0  0.1   0:00.28 httpd2-prefork
 8683 wwwrun    20   0 92348 3420 1940 S    0  0.1   0:00.34 httpd2-prefork

 9093 wwwrun    20   0 94908 5676 1916 S    0  0.1   0:00.30 httpd2-prefork
 9094 wwwrun    20   0 92840 3852 1908 S    0  0.1   0:00.26 httpd2-prefork
 9352 wwwrun    20   0 92276 3248 1900 S    0  0.1   0:00.18 httpd2-prefork

 9866 wwwrun    20   0 92688 3172 1848 S    0  0.1   0:00.08 httpd2-prefork
 9982 wwwrun    20   0 92680 3548 1852 S    0  0.1   0:00.06 httpd2-prefork
10108 wwwrun    20   0 92068 2712 1620 S    0  0.1   0:00.00 httpd2-prefork

10121 wwwrun    20   0 92656 3292 1636 S    0  0.1   0:00.00 httpd2-prefork
10210 wwwrun    20   0 92068 2684 1632 S    0  0.1   0:00.00 httpd2-prefork
10268 wwwrun    20   0 97952  23m 5848 S    0  0.6   0:02.66 osiris.cgi

12322 wwwrun    20   0 92988   52    4 S    0  0.0   0:00.96 httpd2-prefork
14202 wwwrun    20   0 91924   56    4 S    0  0.0   0:00.02 httpd2-prefork
21059 wwwrun    20   0 91936   64    4 S    0  0.0   0:00.04 httpd2-prefork

24808 wwwrun    20   0 92364   84    4 S    0  0.0   0:00.20 httpd2-prefork
27685 wwwrun    20   0 94852  140    4 S    0  0.0   0:00.14 httpd2-prefork
30437 wwwrun    20   0 92068  652    4 S    0  0.0   0:00.00 httpd2-prefork

32242 wwwrun    20   0 92308  632    4 S    0  0.0   0:00.04 httpd2-prefork



On Mon, Aug 1, 2011 at 7:31 PM, ElectroMech <electromech@electromech.info> wrote:

Hi,


On Mon, Aug 1, 2011 at 7:19 PM, Niyati Dave <niyati277@gmail.com> wrote:

I have the value of MaxSpareServers 10. Should I still try changing.


Always check top and free -m.

Check it should not utilize swap Memory, if used should temporary.

Check also uptime for load average should not be greater than total logical cpu.


If you want change for performance should change following value without increasing usage of memory.

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections

# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process

# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2  
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25

MaxRequestsPerChild  0
</IfModule>

Let us know the impact and check the log.
 


Thanks and Regards.
--
--
Nilesh Vaghela
(RHCSA RHCE)

ElectroMech Corporation
Redhat Channel Partner and Training Partner
404, Maulik arcade, Above Karnavati Pagarakha Bazar,
Nr. Mansi cross Road,
Satellite Rd, Ahmedabad
25, The Emperor, Fatehgunj, Baroda.

www.electromech.info

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Nirmal Pathak — Mon, 08/01/2011 - 14:30

On 1 August 2011 13:52, Niyati Dave wrote:
> Hi all
>
> I am continuously facing the same logs and problem again. I want to try out
> increasing the swap. What I know is creating a swap file or a new swap
> partition. Is there any way to increase the present swap which is already of
> 2G.
> Waiting for resonse
>
If your existing swap filesystem is on LVM then you can extend the
Logical Volume.

If not, then you need to create a file or partition on the system and
create new swap space.
Check out this, http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/

> Thanks
> Niyati
>
>
> On Mon, Aug 1, 2011 at 11:09 AM, Niyati Dave wrote:
>>
>> Hello all
>>
>> I have read the above discussion, so I would like to clarify a few
>> things.. the kernel is a latest one as provided by the suselinux enterprise
>> server and version 11 - so the latest one. The server is a xen guest and
>> applications running on the server is just apache, postfix. But yes it has a
>> good usage of apache application and there are few cron scripts related to
>> emailing, cgi. And as per the logs too after giving out of memory error
>> related to swap then oom-killer started killing sh, httpd-prefork, top, the
>> script.
>>
>> Thanks again
>> Niyati
>> On Mon, Aug 1, 2011 at 8:39 AM, ElectroMech
>> wrote:
>>>
>>> Hi,
>>>
>>> On Fri, Jul 29, 2011 at 5:49 PM, Niyati Dave wrote:
>>>>
>>>> Hello Sir,
>>>>
>>>> thanks for the response and explaination:
>>>>
>>>> 1 query I have the physical memory : 4102 M
>>>> and swap space : 2055 M
>>>> Is it recommended to increase the swap more?
>>>>
>>>
>>> Can you give us more idea what services are provided by server ?
>>>
>>> Mostly 4 G Memory is good, still depends on what application you served.
>>> Still we can change some parameters within application ( for memory
>>> utilization) so OOM-KILLER should not occurs.
>>>
>>>>
>>>> Thanks
>>>> Niyati
>>>
>>> Thanks and Regards.
>>> --

Have FuN!
--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Nirmal Pathak — Fri, 07/29/2011 - 18:30



On 29 July 2011 17:49, Niyati Dave <niyati277@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?


Yes, as Alok mentioned in old days when Physical Memory was less & costly we used to put swap as double of Physical memory.


Now days we have huge physical memory on servers like 64GB, 128GB, etc there we do not require swap double  of Physical Memory. Linux kernel hardly uses Swap space more than 20 GB, no matter how large Swap Space you keep on the servers.


 

Thanks
Niyati

On Fri, Jul 29, 2011 at 3:32 PM, Nirmal Pathak <nirmal.pathak@gmail.com> wrote:



On 29 July 2011 12:37, Niyati Dave <niyati277@gmail.com> wrote:

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.


From the oom-kill messages I can see that the system has exhausted Swap Memory

--

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

--

I would suggest to increase Swap Space on the system prior to setting any memory related tunable on the system.


It is difficult from logs to find exact process which triggered the oom-kill.

Out of Memory occurs when the system memory is exhausted on the system and kernel tries to recover the memory by killing various process on the system which is consuming more memory.


You have to monitor your system continuously to find the cause!
 



On Fri, Jul 29, 2011 at 12:04 PM, kandarp thaker <kandarp.inferno@gmail.com> wrote:

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a
leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u
more insight. Are you using rhel5. Then after some time it goes on
killing other processes to free up the memory.



This does not seem to be httpd issue to me. Issue occurred as system had exhausted memory including the swap memory.

 
Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof

M:-9978926593




On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?

It could be both! :)
 
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


As mentioned above when system is under heavy memory load, the kernel starts killing processes which are consuming high memory on the system.

 


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again


On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati



Have FuN!
--
Nirmal D Pathak.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG




--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Alok Thaker — Fri, 07/29/2011 - 19:30

Yes dear nirmal i.e very true..&  she is using suselinux what I found is that there is a leakage somewhere, niyati are you doing some kind of scanning on webserver like PCI or Audit scan...that can cause memory to be consumed also....


Thanks & Regards,
Alok 


On Fri, Jul 29, 2011 at 6:13 PM, Nirmal Pathak <nirmal.pathak@gmail.com> wrote:



On 29 July 2011 17:49, Niyati Dave <niyati277@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?


Yes, as Alok mentioned in old days when Physical Memory was less & costly we used to put swap as double of Physical memory.


Now days we have huge physical memory on servers like 64GB, 128GB, etc there we do not require swap double  of Physical Memory. Linux kernel hardly uses Swap space more than 20 GB, no matter how large Swap Space you keep on the servers.


 

Thanks
Niyati

On Fri, Jul 29, 2011 at 3:32 PM, Nirmal Pathak <nirmal.pathak@gmail.com> wrote:


On 29 July 2011 12:37, Niyati Dave <niyati277@gmail.com> wrote:

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.


From the oom-kill messages I can see that the system has exhausted Swap Memory

--

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

--

I would suggest to increase Swap Space on the system prior to setting any memory related tunable on the system.


It is difficult from logs to find exact process which triggered the oom-kill.

Out of Memory occurs when the system memory is exhausted on the system and kernel tries to recover the memory by killing various process on the system which is consuming more memory.


You have to monitor your system continuously to find the cause!
 



On Fri, Jul 29, 2011 at 12:04 PM, kandarp thaker <kandarp.inferno@gmail.com> wrote:

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a
leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u
more insight. Are you using rhel5. Then after some time it goes on
killing other processes to free up the memory.



This does not seem to be httpd issue to me. Issue occurred as system had exhausted memory including the swap memory.

 
Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof

M:-9978926593




On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?

It could be both! :)
 
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


As mentioned above when system is under heavy memory load, the kernel starts killing processes which are consuming high memory on the system.

 


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again


On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati



Have FuN!
--
Nirmal D Pathak.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG




--
Nirmal D Pathak.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What I want is freedom, not life, nor pleasure, nor good!

FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Alok Thaker — Fri, 07/29/2011 - 18:30

Hi,

It is always recommended to have swap memory as twice of physical memory as per linux fundatmentals , in your case it looks it has less swap memory as compared to physical memory.

Thanks & Regards,

Alok


On Fri, Jul 29, 2011 at 5:49 PM, Niyati Dave <niyati277@gmail.com> wrote:

Hello Sir,

thanks for the response and explaination:

1 query I have the physical memory : 4102 M
and swap space : 2055 M
Is it recommended to increase the swap more?


Thanks
Niyati

On Fri, Jul 29, 2011 at 3:32 PM, Nirmal Pathak <nirmal.pathak@gmail.com> wrote:



On 29 July 2011 12:37, Niyati Dave <niyati277@gmail.com> wrote:

Thanks again

bt this is not rhel , I have the server suselinux version 11
IN the logs the processes which invoked oom-killer are sshd, then top, httpd2-prefork and then osiris.cgi.
After a few minutes the server was normal so that is why I am not able to make out what actully may have caused the problem.


From the oom-kill messages I can see that the system has exhausted Swap Memory

--

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

--

I would suggest to increase Swap Space on the system prior to setting any memory related tunable on the system.


It is difficult from logs to find exact process which triggered the oom-kill.

Out of Memory occurs when the system memory is exhausted on the system and kernel tries to recover the memory by killing various process on the system which is consuming more memory.


You have to monitor your system continuously to find the cause!
 



On Fri, Jul 29, 2011 at 12:04 PM, kandarp thaker <kandarp.inferno@gmail.com> wrote:

Hi Niyati,

Yes it shows that apache i.e http2 consumes more memory and has a
leakage https://bugzilla.redhat.com/show_bug.cgi?id=497077 this gives u
more insight. Are you using rhel5. Then after some time it goes on
killing other processes to free up the memory.



This does not seem to be httpd issue to me. Issue occurred as system had exhausted memory including the swap memory.

 
Thanks,
Kandarp
IT Admin-Consultant
Inferno Solutions
Go with the best
www.inferno.co.in
All Linux-Java-Web-Voip Solutions-Training under one Roof

M:-9978926593




On Fri, Jul 29, 2011 at 11:55 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi

thanks for your response

just a few queries.. does this occur because process start utilising high memory or the number of processes running increase on the server?

It could be both! :)
 
I could see the above logs continuously: like at one instance it says :

 
kernel: httpd2-prefork invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


then after a few seconds

 kernel: osiris.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0

....

Kernel: cron invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
....


As mentioned above when system is under heavy memory load, the kernel starts killing processes which are consuming high memory on the system.

 


How can i interprete which process is actully causing the memory issue and triggering oom-killer? cron, httpd2-prefork, sshd.. ?


Thanks again


On Fri, Jul 29, 2011 at 11:38 AM, Alok Thaker <alok.akki@gmail.com> wrote:

Hi,

The problem with Linux is that by default it overcommits memory –
basically when you do a malloc on linux it doesn’t reserve ane swap
areay (memory + swap disk) and alway returns as successful. Then if you
have couple of programs and all of them actually do want to use the
memory the problem starts as system is running out of memory but there
is basically no interface to tell it to applications as it already told
all of them that there is enough of it… so it starts killing application
in order to avoid complete lock so there oom-killer is invoked when linux overcommits the memory for the processes whereas in case of solaris that's not the case.

To prevent it you can do echo 2 > /proc/sys/vm/overcommit_memory


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:29 AM, Niyati Dave <niyati277@gmail.com> wrote:

Hi



I am getting below error messages in my /var/log/messages

Swap gets out. and then processes are killed one by one.. sh,httpd2-

prefork,osiris.cgi repeating the same logs.. as below

Please somebody help why this is happening, what could be the cause of

the problem? what is oom-killer and why it gets invoked?



Jul 28 01:42:10 APP0001 kernel: sshd invoked oom-killer:

gfp_mask=0x1201d2, order=0, oomkilladj=0

Jul 28 01:42:10 APP0001 kernel: Pid: 16401, comm: sshd Not tainted

2.6.27.19-5-xen #1

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Call Trace:

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020c597>]

show_trace_log_lvl+0x41/0x58

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8046357f>] dump_stack

+0x69/0x6f

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276608>] oom_kill_process

+0x5c/0x227

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80276e54>] out_of_memory

+0xdd/0x10d

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027990d>]

__alloc_pages_internal+0x2ef/0x3f7

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b70c>]

__do_page_cache_readahead+0x94/0x194

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8027b8c4>]

force_page_cache_readahead+0x57/0x7e

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff80277022>]

sys_fadvise64_64+0x11e/0x190

Jul 28 01:42:10 APP0001 kernel:  [<ffffffff8020b3b8>]

system_call_fastpath+0x16/0x1b

Jul 28 01:42:10 APP0001 kernel:  [<00007f2ac4a576dd>] 0x7f2ac4a576dd

Jul 28 01:42:10 APP0001 kernel:

Jul 28 01:42:10 APP0001 kernel: Mem-Info:

Jul 28 01:42:10 APP0001 kernel: DMA per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jul 28 01:42:10 APP0001 kernel: DMA32 per-cpu:

Jul 28 01:42:10 APP0001 kernel: CPU    0: hi:  186, btch:  31 usd: 176

Jul 28 01:42:10 APP0001 kernel: CPU    1: hi:  186, btch:  31 usd: 179

Jul 28 01:42:10 APP0001 kernel: CPU    2: hi:  186, btch:  31 usd: 158

Jul 28 01:42:10 APP0001 kernel: CPU    3: hi:  186, btch:  31 usd:  94

Jul 28 01:42:10 APP0001 kernel: Active:503042 inactive:505023 dirty:0

writeback:0 unstable:0

Jul 28 01:42:10 APP0001 kernel:  free:4576 slab:2988 mapped:1

pagetables:0 bounce:0

Jul 28 01:42:10 APP0001 kernel: DMA free:10340kB min:28kB low:32kB

high:40kB active:0kB inactive:0kB present:16128kB pages_scanned:0

all_unreclaimable? ye

s

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 4016 4016 4016

Jul 28 01:42:10 APP0001 kernel: DMA32 free:7964kB min:8092kB low:

10112kB high:12136kB active:2012168kB inactive:2020092kB present:

4112640kB pages_scanned:

7556254 all_unreclaimable? yes

Jul 28 01:42:10 APP0001 kernel: lowmem_reserve[]: 0 0 0 0

Jul 28 01:42:10 APP0001 kernel: DMA: 1*4kB 2*8kB 1*16kB 0*32kB 3*64kB

1*128kB 1*256kB 1*512kB 1*1024kB 2*2048kB 1*4096kB = 10340kB

Jul 28 01:42:10 APP0001 kernel: DMA32: 37*4kB 5*8kB 4*16kB 3*32kB

1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7964kB

Jul 28 01:42:10 APP0001 kernel: 52 total pagecache pages

Jul 28 01:42:10 APP0001 kernel: 0 pages in swap cache

Jul 28 01:42:10 APP0001 kernel: Swap cache stats: add 4269236, delete

4269236, find 8801268/8935257

Jul 28 01:42:10 APP0001 kernel: Free swap  = 0kB

Jul 28 01:42:10 APP0001 kernel: Total swap = 2104472kB

Jul 28 01:42:10 APP0001 kernel: 1048576 pages RAM

Jul 28 01:42:10 APP0001 kernel: 23652 pages reserved

Jul 28 01:42:10 APP0001 kernel: 1899 pages shared

Jul 28 01:42:10 APP0001 kernel: 1018508 pages non-shared

Jul 28 01:42:10 APP0001 kernel: Out of memory: kill process 8623 (sh)

score 48420 or a child

Jul 28 01:42:10 APP0001 kernel: Killed process 12844 (osiris.cgi)





Thanks

Niyati



Have FuN!
--
Nirmal D Pathak.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I want is freedom, not life, nor pleasure, nor good!
FOSS gives that.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG



--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Shakthi Kannan — Sat, 07/30/2011 - 00:30

Hi,

--- On Fri, Jul 29, 2011 at 5:52 PM, Alok Thaker wrote:
| It is always recommended to have swap memory as twice of physical memory as
| per linux fundatmentals ,
\--

No. It was due to expensive RAM on the old architectures in the early
days. Linux memory management is quite proficient since 2.6, and this
false notion doesn't apply today. You can run GNU/Linux even without
any swap.

SK

--
Shakthi Kannan
http://www.shakthimaan.com

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Alok Thaker — Sat, 07/30/2011 - 00:30

Hi,

Yes linux can be installed with or without swap but definiitely it impacts heavy applications like running big servers with big deployments of jboss,tomcat,apache,mod_jk,clustering things etc..definitely swap is helpful and here kernel version wasn't discussed but the problem of application leakage was discussed by niyati.


Thanks & Regards,
Alok


On Fri, Jul 29, 2011 at 11:30 PM, Shakthi Kannan <shakthimaan@gmail.com> wrote:

Hi,



--- On Fri, Jul 29, 2011 at 5:52 PM, Alok Thaker <alok.akki@gmail.com> wrote:

| It is always recommended to have swap memory as twice of physical memory as

| per linux fundatmentals ,

\--



No. It was due to expensive RAM on the old architectures in the early

days. Linux memory management is quite proficient since 2.6, and this

false notion doesn't apply today. You can run GNU/Linux even without

any swap.



SK



--

Shakthi Kannan

http://www.shakthimaan.com



--

Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.

You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".

To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com

To post to this group, send email to VGLUG@googlegroups.com

For more options, visit this group at http://groups.google.com/group/VGLUG


--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

help to understand kernel logs: sshd invoked oom-killer:

Shakthi Kannan — Sat, 07/30/2011 - 08:30

Hi,

--- On Fri, Jul 29, 2011 at 11:33 PM, Alok Thaker wrote:
| and here kernel version wasn't discussed
\--

Linux is the kernel. You only gave (incorrect) information regarding
the Linux kernel which I quoted and replied. Now, you are saying
kernel version was not discussed. Read the replies, slowly, carefully
before replying.

=== QUOTED ===

On Fri, Jul 29, 2011 at 5:52 PM, Alok Thaker
wrote:
| It is always recommended to have swap memory as twice of physical memory
| as per linux fundatmentals ,
\--

=== END ===

Please don't send HTML e-mail to the list. Please change your e-mail
client settings to send plain ASCII text. It is already mentioned in
the VGLUG mail footers:

http://www.catb.org/~esr/faqs/smart-questions.html#formats

SK

--
Shakthi Kannan
http://www.shakthimaan.com

--
Please read http://www.catb.org/~esr/faqs/smart-questions.html before posting.
You received this message because you are subscribed to the "Vibrant GNU/Linux User Group".
To stop receiving emails from this group, mail to VGLUG+unsubscribe@googlegroups.com
To post to this group, send email to VGLUG@googlegroups.com
For more options, visit this group at http://groups.google.com/group/VGLUG

  • Login to post comments

Search

Supporters

  • Ask4 IT Solutions
  • Electromech Corporation
  • VGLUG Website

New forum topics

  • WWW1, WWW2, WWW3
  • need suggestion on FIREWALL
  • Mysqlbackup Error + Out of resources Problem
  • CentOS hard drive space issue in VM
  • Best shell script pdf
more

Active forum topics

  • WWW1, WWW2, WWW3
  • need suggestion on FIREWALL
  • Oracle Installation on Centos
  • Mysqlbackup Error + Out of resources Problem
  • Setting up password less login in linux servers i.e. logging in linux machine without password .
more

  • vibrant gnu/linux user group
  • linux blog
  • events pictures
  • contact us
  • feedback


Sitemap | Ask4 IT Solutions | Planet VGLUG