Although the virtual machine hosting lacie-nas.org has 1GB of RAM, it suffers from memory shortage on a daily basis. The causes were analyzed and it was concluded that it will migrate to a larger (4GB and 4CPUs) host. GRUB2 was installed and configured to boot on linux 2.6.38.
git clone git://lacie-nas.org/lacie-orion.git
it first calls
git pack-objects --revs --thin --stdout --progress --delta-base-offset
which takes a few minutes to complete and does not deliver any useful content while it runs.
Initialized empty Git repository in /tmp/c/lacie-orion/.git/ remote: Counting objects: 1895586, done.
It uses around 150MB of allocated space in the heap. When examining the process with top(1) it shows about 400MB in virtual (VIRT) size, that is including the git .pack files for the most part. The pmap(1) command displays [anon] memory blocs totaling 150MB : they are distinct from the [stack] and cannot be shared with other processes : they show under the RES column of top.
lacie-nas:~# pmap 1068 1068: git pack-objects --revs --thin --stdout --progress --delta-base-offset ... 08124000 268K rwx-- [ anon ] 08c4a000 35696K rwx-- [ anon ] 9ab9a000 76808K rwx-- [ anon ] a0762000 26036K r-x-- /home/git/lacie-orion.git/objects/pack/pack-4a066193714044bd9d40fd9032432afc3f4e7213.pack ... b7103000 180K r-x-- /home/git/lacie-orion.git/objects/pack/pack-53b8c87ebc3526a45f7f9dd3875f9ec12dcb3d6a.idx b7130000 2824K r-x-- /home/git/lacie-orion.git/objects/pack/pack-b88030889e04b11427a39d30f30bca5f7e41646a.idx ... bf985000 132K rw--- [ stack ] total 423492K
Some memory pages used by the process can be shared with other processes and show in the SHR column of top. We speculate that these pages are allocated as a consequence of visited a file mapped with mmap(2) : although they are necessary for the process to run and cannot be freed, they can be shared by any other process mapping the same file and reading the same area.
When compressing objects, it uses around 300MB.
Remote: Compressing objects: 27% (106652/395005)
This memory footprint will not be reduced until the clone is complete, which will take as long as it needs to transfert over 500MB of information.
Receiving objects: 13% (253848/1895586), 103.92 MiB | 285 KiB/s
git repacking and configuration
In an attempt to reduce the memory footprint of ”git pack-objects” the repository was repacked using
git repack -a -f -d
git repack -a -f -d --max-pack-size=50m
but it did not show any improvement. The following git-config(1) parameters were set but it made no visible difference:
git config pack.deltaCacheSize 32m git config pack.windowMemory 32m
limiting the number of simultaneous git servers
The audience of lacie-nas.org is not high and limiting the number of simultaneous connections to the git server can be limited
lacie-nas:/home/git# cat /etc/service/git-daemon/run #!/bin/sh exec 2>&1 echo 'git daemon starting.' exec chpst -uwww-data git daemon --verbose --max-connections=1 --export-all --base-path=/home/gitweb /home/gitweb #exec chpst -uwww-data git daemon --verbose --export-all --base-path=/home/gitweb /home/gitweb
sv restart git-daemon
When cloning the lacie-orion repository using http://git.lacie-nas.org/lacie-orion.git, the memory footprint (around 50MB when preparing and less than 10MB while sending data) is much lower than when using the git server. This advantage is only true for the clone. When fetching, the http protocol will require the copy of entire pack files and be much less efficient than asking the git server.
git clone http://git.lacie-nas.org/lacie-orion.git
apache memory footprint
A number of apache processes have a fairly large memory footprint:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 929 www-data 20 0 117m 103m 3104 S 0.0 5.1 0:00.52 /usr/sbin/apache2 -k start 940 www-data 20 0 116m 102m 3028 S 0.0 5.1 0:00.33 /usr/sbin/apache2 -k start 852 www-data 20 0 116m 102m 2812 S 0.0 5.0 0:00.27 /usr/sbin/apache2 -k start 939 www-data 20 0 115m 100m 2632 S 0.0 5.0 0:00.30 /usr/sbin/apache2 -k start 994 www-data 20 0 114m 100m 2068 S 0.0 4.9 0:00.40 /usr/sbin/apache2 -k start
The reason is still unknown. It is likely to be a consequence of triggering memory hungry wiki pages at http://lacie-nas.org/doku.php. A workaround was to modify /etc/apache2/apache2.conf
diff --git a/apache2/apache2.conf b/apache2/apache2.conf index 1e97b4e..38496ed 100644 --- a/apache2/apache2.conf +++ b/apache2/apache2.conf @@ -100,7 +100,7 @@ KeepAliveTimeout 15 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 - MaxRequestsPerChild 0 + MaxRequestsPerChild 10 # worker MPM
Setting MaxRequestsPerChild to 10 so that the apache process expires. If it never expires (as it was the case because it was set to 0) this amount of memory is locked for ever, for all 10 processes apache will fork because MaxSpareServers is 10.
The virtual machine was installed on a dedicated LVM volume formatted with a MSDOS partition containing a single ext3 partition. It was installed in paravirtualized mode which means it uses a kernel found on the host instead of a kernel found on the disk of the virtual machine itself. It is better when a virtual machine is self contained and does not rely on external resources to boot. It makes it easier to migrate and backup. For this reason a kernel was installed and an attempt to install grub was done:
lacie-nas:~# grub-install --no-floppy --root-directory=/boot /dev/vda /usr/sbin/grub-setup: warn: This msdos-style partition label has no post-MBR gap; embedding won't be possible!. /usr/sbin/grub-setup: warn: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged.. /usr/sbin/grub-setup: error: will not proceed with blocklists.
It failed because there was only 1 bloc available as shown by:
lacie-nas:~# cat /sys/block/vda/vda1/start 1
On a virtual machine installed with a Debian GNU/Linux squeeze, the first partition starts at block 2048.
root@seeks:~# cat /sys/block/vda/vda1/start 2048
Furthermore, this specific version of grub (found in Debian GNU/Linux sid at the time) is 0.99 and does not propose to use the –force option to create the blocklists. It was downgraded to the 0.98 version found in Debian GNU/Linux squeeze and the –force option used to successfully install grub.
grub-install --force --no-floppy --root-directory=/boot /dev/vda
The version 0.99 was then re-installed so that the configuration files ( /boot/grub/grub.cfg ) are recognized.