[Gta04-owner] recent server timeouts
Dr. H. Nikolaus Schaller
hns at goldelico.com
Wed Aug 15 23:06:13 CEST 2012
you may have seen persistent server timeouts
on www.gta04.org or www.openphoenux.org
since last Sunday morning.
This has been due to a hickup of the server 
which is now solved.
Web access, mailing lists etc. should now work
 if you are interested in details:
* there is kernel tree project <http://projects.goldelico.com/p/gta04-kernel/>
* the web pages are served through the Indefero (open source) tool
* the kernel tree for the hw-validation kernel is hosted on the same server
* Indefero provides its own graphical representation of the git tree - similar to gitweb
* but using its own code incl. some python scripts and heavily calling git log etc.
* in some rare situation (some very old code) git log needs 300 MByte of server space and up to 30 seconds to complete (this high computation time was even discussed by git developers)
* this are pages you usually would never manually navigate to
* but a web robot is doing because it is stupid...
* now, when web robots are scanning the Indefero representation of our hw-validation kernel, this spawns several potentially memory and computation intensive git subprocesses
* this can overload the server if several such pages are scanned in parallel and the physical memory (4 GB) is reached
* in this case, the server will start to swap out git commands making their completion time grow rapidly beyond 30 seconds
* now, if the robot repeats or continues its work, we will get a exploding number of git processes
Analysis has shown that the server did have approx. 1000 simultaneuos git commands running, occupying 30 GB swap space.
So you can imagine that the server is completely saturated by swapping and is
no longer serving web pages, ssh or the mailing lists in a time that peer
We simply did disconnect the server from the public Internet and did wait 3 days until
all git commands where worked through and swapping did end. Then, the server
was easily accessible again and ready for analyses.
The solution is that the kernel is no longer available through the Indefero web interface,
but only gitweb (and a mirror on github).
For details of the new links, please see:
More information about the Gta04-owner