[Community] goldelico/gta04/openphoenux server issues found

Dr. H. Nikolaus Schaller hns at goldelico.com
Tue Jun 12 22:29:34 CEST 2012


Hi,

Am 08.06.2012 um 10:23 schrieb Neil Jerram:

> Jason Cawood <openmoko at jasoncawood.com> writes:
> 
>> Is gta04.org working for you guys?  When I try to go there I get a 504
>> error. 
> 
> Normally those are all fine, so I guess Golden Delicious must be having
> a spot of technical trouble.  Hopefully it won't last long.

the problem has been identified. It is that the source code display function
the Indefero tool (http://www.indefero.net/) we use is calling git in a
subshell to do this.

Especially for a git repo with the Linux kernel, some git commands
may need a lot of time and therefore building a HTML answer
may require 30-60 seconds at 100% CPU utilization (it is a 2.5 GHz
Dual Core) and use up to 500 MByte of RAM for the git process.

This is not a problem if one or two human users browse through
the tree since there are 2 CPUs... And it is not a problem if a web
robot is scanning the whole tree one-file-after-the-other.

But it becomes a problem if several spiders are accessing this
simultaneously plus maybe one or two users. Then, there may
be several git commands running in parallel. This will fill the
Server's RAM until swapping starts. If in this case the spider
or the user simply restarts another request, the number of
parallel httpd and git instances may grow even further. One
example this afternoon has shown approx. 100 httpd and more
than 200 git commands running... It was difficult to get answers
from a ssh+bash... 1-2 minutes response time :)

So the system is thrashing and *very* unresponsive. It still
responds to ping but nothing else because clients tend to
timeout before the server has swapped in the server processes.

It will take a little time to install a solution (either improve Indefero,
or limit memory utilization by git or something else. Adding more
memory does not solve the issue, it will just delay it and make it a
little less likely).

Sorry for the server-down-times hitting these mailing lists as well,

Nikolaus


 



More information about the Community mailing list