lacie-nas.org system administration

Although the virtual machine hosting lacie-nas.org has 1GB of RAM, it suffers from memory shortage on a daily basis. The causes were analyzed and it was concluded that it will migrate to a larger (4GB and 4CPUs) host. GRUB2 was installed and configured to boot on linux 2.6.38.

git server

The lacie-orion git repository is a derivative of the linux kernel. When checked out using

git clone git://lacie-nas.org/lacie-orion.git

it first calls

git pack-objects --revs --thin --stdout --progress --delta-base-offset

which takes a few minutes to complete and does not deliver any useful content while it runs.

Initialized empty Git repository in /tmp/c/lacie-orion/.git/
remote: Counting objects: 1895586, done.

It uses around 150MB of allocated space in the heap. When examining the process with top(1) it shows about 400MB in virtual (VIRT) size, that is including the git .pack files for the most part. The pmap(1) command displays [anon] memory blocs totaling 150MB : they are distinct from the [stack] and cannot be shared with other processes : they show under the RES column of top.

lacie-nas:~# pmap 1068
1068:   git pack-objects --revs --thin --stdout --progress --delta-base-offset
...
08124000    268K rwx--    [ anon ]
08c4a000  35696K rwx--    [ anon ]
9ab9a000  76808K rwx--    [ anon ]
a0762000  26036K r-x--  /home/git/lacie-orion.git/objects/pack/pack-4a066193714044bd9d40fd9032432afc3f4e7213.pack
...
b7103000    180K r-x--  /home/git/lacie-orion.git/objects/pack/pack-53b8c87ebc3526a45f7f9dd3875f9ec12dcb3d6a.idx
b7130000   2824K r-x--  /home/git/lacie-orion.git/objects/pack/pack-b88030889e04b11427a39d30f30bca5f7e41646a.idx
...
bf985000    132K rw---    [ stack ]
 total   423492K

Some memory pages used by the process can be shared with other processes and show in the SHR column of top. We speculate that these pages are allocated as a consequence of visited a file mapped with mmap(2) : although they are necessary for the process to run and cannot be freed, they can be shared by any other process mapping the same file and reading the same area.
When compressing objects, it uses around 300MB.

Remote: Compressing objects:  27% (106652/395005)

This memory footprint will not be reduced until the clone is complete, which will take as long as it needs to transfert over 500MB of information.

Receiving objects:  13% (253848/1895586), 103.92 MiB | 285 KiB/s

git repacking and configuration

In an attempt to reduce the memory footprint of ”git pack-objects” the repository was repacked using

git repack -a -f -d

and

git repack -a -f -d --max-pack-size=50m

but it did not show any improvement. The following git-config(1) parameters were set but it made no visible difference:

git config pack.deltaCacheSize 32m
git config pack.windowMemory 32m

limiting the number of simultaneous git servers

The audience of lacie-nas.org is not high and limiting the number of simultaneous connections to the git server can be limited

lacie-nas:/home/git# cat /etc/service/git-daemon/run
#!/bin/sh
exec 2>&1
echo 'git daemon starting.'
exec chpst -uwww-data git daemon --verbose --max-connections=1 --export-all --base-path=/home/gitweb /home/gitweb
#exec chpst -uwww-data git daemon --verbose --export-all --base-path=/home/gitweb /home/gitweb

and restarted

sv restart git-daemon

http git

When cloning the lacie-orion repository using http://git.lacie-nas.org/lacie-orion.git, the memory footprint (around 50MB when preparing and less than 10MB while sending data) is much lower than when using the git server. This advantage is only true for the clone. When fetching, the http protocol will require the copy of entire pack files and be much less efficient than asking the git server.

git clone http://git.lacie-nas.org/lacie-orion.git

apache memory footprint

A number of apache processes have a fairly large memory footprint:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  929 www-data  20   0  117m 103m 3104 S  0.0  5.1   0:00.52 /usr/sbin/apache2 -k start
  940 www-data  20   0  116m 102m 3028 S  0.0  5.1   0:00.33 /usr/sbin/apache2 -k start
  852 www-data  20   0  116m 102m 2812 S  0.0  5.0   0:00.27 /usr/sbin/apache2 -k start
  939 www-data  20   0  115m 100m 2632 S  0.0  5.0   0:00.30 /usr/sbin/apache2 -k start
  994 www-data  20   0  114m 100m 2068 S  0.0  4.9   0:00.40 /usr/sbin/apache2 -k start

The reason is still unknown. It is likely to be a consequence of triggering memory hungry wiki pages at http://lacie-nas.org/doku.php. A workaround was to modify /etc/apache2/apache2.conf

diff --git a/apache2/apache2.conf b/apache2/apache2.conf
index 1e97b4e..38496ed 100644
--- a/apache2/apache2.conf
+++ b/apache2/apache2.conf
@@ -100,7 +100,7 @@ KeepAliveTimeout 15
     MinSpareServers       5
     MaxSpareServers      10
     MaxClients          150
-    MaxRequestsPerChild   0
+    MaxRequestsPerChild  10
 

 # worker MPM

Setting MaxRequestsPerChild to 10 so that the apache process expires. If it never expires (as it was the case because it was set to 0) this amount of memory is locked for ever, for all 10 processes apache will fork because MaxSpareServers is 10.

installing grub2

The virtual machine was installed on a dedicated LVM volume formatted with a MSDOS partition containing a single ext3 partition. It was installed in paravirtualized mode which means it uses a kernel found on the host instead of a kernel found on the disk of the virtual machine itself. It is better when a virtual machine is self contained and does not rely on external resources to boot. It makes it easier to migrate and backup. For this reason a kernel was installed and an attempt to install grub was done:

lacie-nas:~# grub-install --no-floppy --root-directory=/boot /dev/vda
/usr/sbin/grub-setup: warn: This msdos-style partition label has no post-MBR gap; embedding won't be possible!.
/usr/sbin/grub-setup: warn: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
/usr/sbin/grub-setup: error: will not proceed with blocklists.

It failed because there was only 1 bloc available as shown by:

lacie-nas:~# cat /sys/block/vda/vda1/start
1

On a virtual machine installed with a Debian GNU/Linux squeeze, the first partition starts at block 2048.

root@seeks:~# cat /sys/block/vda/vda1/start
2048

Furthermore, this specific version of grub (found in Debian GNU/Linux sid at the time) is 0.99 and does not propose to use the –force option to create the blocklists. It was downgraded to the 0.98 version found in Debian GNU/Linux squeeze and the –force option used to successfully install grub.

grub-install --force --no-floppy --root-directory=/boot /dev/vda

The version 0.99 was then re-installed so that the configuration files ( /boot/grub/grub.cfg ) are recognized.

This entry was posted in lacie-nas. Bookmark the permalink.

One Response to lacie-nas.org system administration

  1. Loic Dachary says:

    The kernel.org modified gitweb can be found at http://git.kernel.org/?p=git/warthog9/gitweb.git;a=summary

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>