Setting up redundant file- and mediaserver with Debian GNU/Linux 7.5 and RAID-5

This post is effectively a continuation of a saga I started now 5 years ago.

Times have moved on, electricity price has gone up and at some point I changed my Library box setup, swapping out the four 320GB hard drives with total of 960GB of RAID-5 storage for two 500GB hard drives in RAID-1, so retaining 500GB of redundant space.

I though back then that I would make the box consume less power that way. In some ways it worked, the power use went down about a dozen watts. In addition, I removed the videocard, as that motherboard had integrated graphics and I also removed a gigabit network card, as my intention was to put the machine into a closet, where I only had 100Mbit network connection anyway. With all those changes I managed to reduce the power usage from the original 70W down to 50W at idle (and in text mode).

As I now needed more redundant space, I swapped out the hard drives once again. In addition to a 1TB Western Digital Black drive, which I already owned, I bought additional two 1TB Western Digital Green drives (49€ a piece) to set up the system on RAID-5 again, giving me almost 2TB of redundant storage.

The other machine specs remained the same:

• AMD Athlon 64 3700+ San-Diego (clocks between 1.0 and 2.2GHz)
• Zalman CNPS7000B-Cu
• ASRock 939NF6G-VSTA (nForce 430)
• 1.0 GiB Kingston HyperX
• Tagan 420W i-Xeye TC PSU
• Antec Three-Hundred case

Server installation

As before, my choice was Debian GNU/Linux, though this time around the newest version 7.5, "Wheezy".

Partitioning

On each of the drives I created two partitions for software RAID.
The first partition is 998GB for the main data and the remaining 2GB will be used for swap.
NB! It is important to enable the boot flag on all of the three disks!
The bigger partitions were then combined into RAID-5 array to form the root partition of about 2TB using EXT4 and the smaller partitions formed the swap space of about 4GB, also using RAID-5.
This way, not only the data would survive a failure of any one of the drives, but also the whole setup should withstand such a failure.

After partitioning and loading some packages, the Software categories choice came up. I decided to leave the desktop choice, though I will later switch off the automatic boot into graphic mode.
I chose also web server, SQL server, file server and ssh server. I left also the print server, though I'm pretty sure I will not be using it for printer serving.

As I was using a "netinst" image, the installation took quite a while, which was no doubt dependent on my network speed.

When the time came to set up grub, I already knew in advance that the installer only installs grub on the first drive and I also need to install grub on the other two disks.
The necessary commands, to be entered before you allow the installer to install the grub into /dev/sda and call update-grub, are:

chroot /target /bin/bash
grub-install /dev/sdb
grub-install /dev/sdc

The install will finish shortly after that.

Reduce the boot manager delay:

mcedit /etc/default/grub 

and change the timeout value

GRUB_TIMEOUT=5

to something smaller, like 1, and then execute:

update-grub

Enable sudo for the main user:

adduser username sudo

this takes effect only after logging out and back in.

Change Gnome Classic to be default graphical shell:

update-alternatives --config x-session-manager

and selec the fallback line, usually number 2.

Change the IP to be static:

In Gnome Classic, in the right at the upper bar, right-click the network icon and choose Edit to configure the network interface.
In the window that opens, select your network interface and click Edit in the right.
In the following window, at IPv4 settings tab, switch the method from DHCP into Manual, click Add, enter your static IP, netmask and gateway, also enter the DNS below and click Save.

Make Debian boot into console mode

update-rc.d gdm3 disable

This will retain the possibility to start the graphical UI with the startx command.

Setting up the environment:

mcedit ~/.bashrc

add the following lines:

stty -ixon  # enables forward bash command history search CTRL-S
export LC_COLLATE="en_US.UTF-8"  # fixes the broken Estonian character map which breaks regexp [a-z]

File server setup

Creating a directory for the data to be served and group for users:

Create a new group called "share":

addgroup share

Add your user into that new group:

adduser username share 

Create and fix the rights to the /Data directory:

cd /
mkdir Data
chown nobody:share Data
chmod 775 Data

Set up an anonymous readonly Samba account:

mcedit /etc/samba/smb.conf

find the section "Share Definitions" and add the following beneath it:

[library]
   comment = Library
   path = /Data
   browseable = yes
   guest ok = yes
   read only = yes

Set up an NFS share:

mcedit /etc/exports

Add the following line:

/Data 192.168.10.0/24(rw,root_squash,async,no_subtree_check)

Making sure the file owner is displayed correctly over NFS:

mcedit /etc/idmapd.conf

Uncomment the "Domain = xxx" line, and enter your domain.
NB! Be sure to set that same domain name on the client aswell.
Reload the NFS configuration on the server:

/etc/init.d/nfs-kernel-server restart

Probably a better way for all of the changes to take effect is to reboot the machine now.

Media server setup

Setting up Plex Media Server

echo "deb http://shell.ninthgate.se/packages/debian wheezy main" > /etc/apt/sources.list.d/plexmediaserver.list
wget http://shell.ninthgate.se/packages/shell-ninthgate-se-keyring.key | apt-key add -
apt-get update
apt-get install plexmediaserver

This will also add the PMS to be automatically started when the server starts.

Other optional steps

Fixing the locale problem:

Somehow I managed to make all locales, except my own, to disappear. The fix:

dpkg-reconfigure locales

Setting mcedit as the default editor:

update-alternatives --config editor

How to make Avidemux work under Ubuntu 13.10

Ubuntu 13.10 Saucy Salamander
Avidemux 2.5.4

I was quite disappointed in Ubuntu 13.10, when I first tried it last year. Several apps were broken and many things did not function.

One of those apps was Avidemux, which I am using regularly at home to encode videos. Note that I'm talking about avidemux2-gtk version.

In Ubuntu 13.10, Avidemux would run, but after selecting some menu item which opens a dialogue window and then closing that dialogue would always hang Avidemux.

For example Help / About will open the About window and when you click OK, the window will hang with the button still appearing pressed and after a moment the main window will fade into black-and-white indicating an unresponsive window. Only xkill will help out and put Avidemux out of its misery.


I recently decided to try again. At my workplace I had upgraded to 13.10 a while ago, as I did not need Avidemux in there. A lot of time had passed and I thought perhaps it had been fixed already and Avidemux would now work.

I installed Avidemux and surprise - it worked. No more hangs. But as I had upgraded my machine at work from 13.04 to 13.10, I was not sure as of why it worked. I decided to try it out also in fresh installation into VirtualBox.

I installed a fresh 13.10 64-bit image into a new VM, then, without installing the VirtualBox additions, upgraded the packages and then installed Avidemux. The result - it hung, like before.

I started to look for differences between the two machines. Into my work machine, I had installed a mainline 3.12 kernel - maybe that was the solution? Nope, that did not help, the Avidemux in VirtualBox would still hang.

Then I noticed that the Avidemux in my main machine was displaying its menus inside the window, and not in the global menu bar. I also noticed, that gnome-terminal, for example, was also displaying its menus inside the main window, so I ran dpkg --get-selections|sort in both machines and compared the selections. And there was the answer - my main machine was missing the following packages:

unity-gtk2-module
unity-gtk3-module
unity-gtk-module-common

I removed these packages from my VirtualMachine installation and confirmed that Avidemux would no longer hang. Problem solved.

How I got the Precise Pangolin to play ball.

Ubuntu 12.04 Precise Pangolin

At first I thought that it would be impossible to get Ubuntu 12.04 to be usable in my workplace environment, but after a lot of tweaking and time, I have finally made the Precise Pangolin work exactly the way I wanted.

In the following I bring a set of changes I had to do to make it work.

1. The language problem (see my original post about this)
   The main problem is, that by choosing Estonian as your language, then in addition to the LANG environment variable, all of the LC_* variables are set to "et_EE.UTF-8" as well. This, as I wrote earlier, makes regexp [a-z] to exclude the letters coming after "s". The original solution I used previously was to set LANG back to English "en_US.UTF-8" locale in .profile, but this has the side-effect of turning everything into English.
   
   My current solution is much more elegant and will solve the regexp problem, while retaining the Estonian language - add these two lines into .profile:
   export LC_CTYPE="en_US.UTF-8"
export LC_COLLATE="en_US.UTF-8"