Setting up file- and mediaserver with Debian GNU/Linux 7.8 and 6TB of RAID-5 space

This post a continuing saga about my home-server started in this post now more than 6 years ago.

Times have moved along and more and more I have realized that not only is file redundancy important, but having half-a-thousand CD's and DVD's lying around is not the nicest of one's room decoration.
Due to the need to clear one wall where currently most of the mentioned CD's and DVD's are located, I decided to move the more important content into my library server and eliminate the physical media. This, however, needs a lot of storage space. My Library server previously had three 1.0TB Western Digital Green hard drives in RAID-5, which amounted to 2.0TB of space. I decided to triple that to 6.0TB with three 3.0TB Western Digital Green drives. One funny note to make is that the new 3.0TB Green drive also weighs subjectively about three times as much as the previous 1.0TB Green drive.

In the end, the power consumption of the whole system went up as expected, but not that much - from about 58W average to 65W average at idle.

Along with the change of hard drives I also switched the processor and thus also the motherboard and memory. This decision was primarily caused by the inability of the old Athlon 64 3700+ to fully facilitate the Plex media server, especially the transcoding was unacceptably slow, but also the Plex client interface itself was lagging badly both in browser and in my Samsung TV. The one single Mebibyte of memory in the old server might've also played a part on the bad performance.

So the specs of the new server are as follows:

• AMD A8-6500T APU (4 cores, 45W, 2.1GHz (3.1GHz Turbo), 1.4GHz idle)
• ASUS A88XM-E
• 4.0GiB Crucial Ballistix Sport
• Tagan 420W i-Xeye TCF PSU
• Antec Three-Hundred case

Server installation

OS choice
The operating system was again a tried and true Debian GNU/Linux, this time around the version 7.8, "Wheezy".

Partitioning
As before, I decided to split the space up to three parts. An adequately sized (30GiB) root partition at the beginning, a 4GiB swap partition at the end and everything between them as the Data partition.

The first time I did the installation I ended up with an error at the point in the installer, where the grub was to be installed. A bit of digging in the net revealed that with 3.0TB drives, GPT must be used and for a boot drive, there has to be a small reserved space for the boot code left on the drive, with the "bios_grub" flag set

Since this kind of partitioning cannot be done from within the Debian installer, use an Ubuntu live CD, open up three terminals in order to be sure to do the exact same steps on all three disks and use the following steps in each:

sudo parted /dev/sda         # use sdb and sdc in the other windows.
mklabel gpt                  # create a new blank GUID partition table.
unit MiB                     # change the units to Mebibytes (notice the 'i' the middle).
print                        # prints the disk information, including the maximum MiB position.
mkpart GRUB 1 2              # create 1 MiB partition between 1 MiB and 2 MiB positions for BIOS boot code and set
set 1 bios_grub on           #   the "bios_grub" flag for that partition.
mkpart ROOT 2 15362          # create 15360 MiB (15 GiB) partition between 2 MiB and 15362 MiB positions for root.
mkpart HOME 15362 2859536    # create 2.7 TiB partition for Data; as the second position, use the max position you got
                             #   above, but subtract 2048 MiB for the swap partition and an additional 2 MiB for some
                             #   spare space at the end.
mkpart SWAP 2859536 2861584  # finally, create a 2 GiB partition for the swap at the end.
quit                         # exit from parted.

Notice that these partitions are used in RAID-5, therefore doubling their sizes, so root will be 30 GiB, swap will be 4 GiB and the Data partition will be 5.4 TiB.

Now, start the Debian installer and in the partitioning step...

• be sure to leave the first 1.0 MiB partition as it is, do not change it!
• mark the latter three partitions on each disk as volumes for RAID,
• combine the corresponding partitions on all three disks into RAID-5 arrays.
• use ext4 on the 30 GiB partition and use it as the / (root) partition with the noatime flag.
• also use ext4 on the ~5.4 TiB partition, but set the mount point to be /Data, also use the noatime flag.
• set the last 4 GiB partition as the swap partition.

After partitioning, at software selection step, I included the web server, file server and ssh server.

At the grub setup step, the Debian installer only installs grub into the first disk, therefore it is important to install grub manually into the other disks first! Switch to one of the other consoles, eg. with ALT+F2, and enter the following commands:

chroot /target /bin/bash
grub-install /dev/sdb
grub-install /dev/sdc

The installer will finish soon thereafter.

Server setup

Enable sudo for your user:

adduser username sudo

Note that this takes effect only after logging out and back in.

Reduce the boot manager delay:

sudo mcedit /etc/default/grub

Find and change the GRUB_TIMEOUT value from 5 to 1 and execute:

sudo update-grub

Change Gnome Classic to be the default graphical shell:

sudo update-alternatives --config x-session-manager

And select the fallback line, usually number 2.

Change the dynamic IP to be static:
In Gnome Classic, in the right at the upper bar, right-click the network icon and choose Edit to configure the network interface. In the window that opens, select your network interface and click Edit in the right. In the following window, at IPv4 settings tab, switch the method from DHCP into Manual, click Add, enter your static IP (eg. 192.168.10.2), netmask (255.255.255.0) and gateway (192.168.10.1), also enter the DNS below and click Save.

Make Debian boot into console mode:

sudo update-rc.d gdm3 disable

Setting up the environment:

mcedit ~/.bashrc

Add the following lines:

stty -ixon  # enables forward bash command history search CTRL-S
export LC_COLLATE="en_US.UTF-8"  # fixes the broken Estonian character map which breaks regexp [a-z]

File server setup

Setting up the Data directory:

sudo addgroup share
sudo adduser username share
cd /
chown nobody:share /Data
chmod 777 /Data
cd /Data
mkdir .transcodetemp  # this directory is for Plex Media Server
chmod 777 .transcodetemp

Setting up anonymous read-only Samba account:

sudo mcedit /etc/samba/smb.conf

Find the section "Share Definitions" and add the following right beneath it:

[library]
   comment = Library
   path = /Data
   browseable = yes
   guest ok = yes
   read only = yes

Setting up NFS share:

sudo mcedit /etc/exports

Add the following line:

/Data 192.168.10.0/24(rw,root_squash,async,no_subtree_check)

Making sure the file owner is displayed correctly over NFS:

sudo mcedit /etc/idmapd.conf

Uncomment the "Domain = xxxx" line and enter your domain in place of the xxxx.
Make sure to use the same domain in your other servers that connect here over NFS!

Reboot the server to make the changes take effect.

Media server setup

Setting up Plex Media Server:

echo "deb http://shell.ninthgate.se/packages/debian wheezy main" | sudo tee /etc/apt/sources.list.d/plexmediaserver.list
wget http://shell.ninthgate.se/packages/shell-ninthgate-se-keyring.key
sudo apt-key add shell-ninthgate-se-keyring.key
sudo apt-get update
sudo apt-get install plexmediaserver

This will also set the Plex Media Server to be automatically started when the server starts.

Other optional steps

Repairing the locales:
When you select a different language at the start of the Debian installer, it configures only the locale for this language. However, for many reasons the English locale should also exist, therefore:

sudo dpkg-reconfigure locales

And check also the en_US.UTF-8 locale in addition to your language locale.

Setting up redundant file- and mediaserver with Debian GNU/Linux 7.5 and RAID-5

This post is effectively a continuation of a saga I started now 5 years ago.

Times have moved on, electricity price has gone up and at some point I changed my Library box setup, swapping out the four 320GB hard drives with total of 960GB of RAID-5 storage for two 500GB hard drives in RAID-1, so retaining 500GB of redundant space.

I though back then that I would make the box consume less power that way. In some ways it worked, the power use went down about a dozen watts. In addition, I removed the videocard, as that motherboard had integrated graphics and I also removed a gigabit network card, as my intention was to put the machine into a closet, where I only had 100Mbit network connection anyway. With all those changes I managed to reduce the power usage from the original 70W down to 50W at idle (and in text mode).

As I now needed more redundant space, I swapped out the hard drives once again. In addition to a 1TB Western Digital Black drive, which I already owned, I bought additional two 1TB Western Digital Green drives (49€ a piece) to set up the system on RAID-5 again, giving me almost 2TB of redundant storage.

The other machine specs remained the same:

• AMD Athlon 64 3700+ San-Diego (clocks between 1.0 and 2.2GHz)
• Zalman CNPS7000B-Cu
• ASRock 939NF6G-VSTA (nForce 430)
• 1.0 GiB Kingston HyperX
• Tagan 420W i-Xeye TC PSU
• Antec Three-Hundred case

Server installation

As before, my choice was Debian GNU/Linux, though this time around the newest version 7.5, "Wheezy".

Partitioning

On each of the drives I created two partitions for software RAID.
The first partition is 998GB for the main data and the remaining 2GB will be used for swap.
NB! It is important to enable the boot flag on all of the three disks!
The bigger partitions were then combined into RAID-5 array to form the root partition of about 2TB using EXT4 and the smaller partitions formed the swap space of about 4GB, also using RAID-5.
This way, not only the data would survive a failure of any one of the drives, but also the whole setup should withstand such a failure.

After partitioning and loading some packages, the Software categories choice came up. I decided to leave the desktop choice, though I will later switch off the automatic boot into graphic mode.
I chose also web server, SQL server, file server and ssh server. I left also the print server, though I'm pretty sure I will not be using it for printer serving.

As I was using a "netinst" image, the installation took quite a while, which was no doubt dependent on my network speed.

When the time came to set up grub, I already knew in advance that the installer only installs grub on the first drive and I also need to install grub on the other two disks.
The necessary commands, to be entered before you allow the installer to install the grub into /dev/sda and call update-grub, are:

chroot /target /bin/bash
grub-install /dev/sdb
grub-install /dev/sdc

The install will finish shortly after that.

Reduce the boot manager delay:

mcedit /etc/default/grub 

and change the timeout value

GRUB_TIMEOUT=5

to something smaller, like 1, and then execute:

update-grub

Enable sudo for the main user:

adduser username sudo

this takes effect only after logging out and back in.

Change Gnome Classic to be default graphical shell:

update-alternatives --config x-session-manager

and selec the fallback line, usually number 2.

Change the IP to be static:

In Gnome Classic, in the right at the upper bar, right-click the network icon and choose Edit to configure the network interface.
In the window that opens, select your network interface and click Edit in the right.
In the following window, at IPv4 settings tab, switch the method from DHCP into Manual, click Add, enter your static IP, netmask and gateway, also enter the DNS below and click Save.

Make Debian boot into console mode

update-rc.d gdm3 disable

This will retain the possibility to start the graphical UI with the startx command.

Setting up the environment:

mcedit ~/.bashrc

add the following lines:

stty -ixon  # enables forward bash command history search CTRL-S
export LC_COLLATE="en_US.UTF-8"  # fixes the broken Estonian character map which breaks regexp [a-z]

File server setup

Creating a directory for the data to be served and group for users:

Create a new group called "share":

addgroup share

Add your user into that new group:

adduser username share 

Create and fix the rights to the /Data directory:

cd /
mkdir Data
chown nobody:share Data
chmod 775 Data

Set up an anonymous readonly Samba account:

mcedit /etc/samba/smb.conf

find the section "Share Definitions" and add the following beneath it:

[library]
   comment = Library
   path = /Data
   browseable = yes
   guest ok = yes
   read only = yes

Set up an NFS share:

mcedit /etc/exports

Add the following line:

/Data 192.168.10.0/24(rw,root_squash,async,no_subtree_check)

Making sure the file owner is displayed correctly over NFS:

mcedit /etc/idmapd.conf

Uncomment the "Domain = xxx" line, and enter your domain.
NB! Be sure to set that same domain name on the client aswell.
Reload the NFS configuration on the server:

/etc/init.d/nfs-kernel-server restart

Probably a better way for all of the changes to take effect is to reboot the machine now.

Media server setup

Setting up Plex Media Server

echo "deb http://shell.ninthgate.se/packages/debian wheezy main" > /etc/apt/sources.list.d/plexmediaserver.list
wget http://shell.ninthgate.se/packages/shell-ninthgate-se-keyring.key | apt-key add -
apt-get update
apt-get install plexmediaserver

This will also add the PMS to be automatically started when the server starts.

Other optional steps

Fixing the locale problem:

Somehow I managed to make all locales, except my own, to disappear. The fix:

dpkg-reconfigure locales

Setting mcedit as the default editor:

update-alternatives --config editor

How to make Avidemux work under Ubuntu 13.10

Ubuntu 13.10 Saucy Salamander
Avidemux 2.5.4

I was quite disappointed in Ubuntu 13.10, when I first tried it last year. Several apps were broken and many things did not function.

One of those apps was Avidemux, which I am using regularly at home to encode videos. Note that I'm talking about avidemux2-gtk version.

In Ubuntu 13.10, Avidemux would run, but after selecting some menu item which opens a dialogue window and then closing that dialogue would always hang Avidemux.

For example Help / About will open the About window and when you click OK, the window will hang with the button still appearing pressed and after a moment the main window will fade into black-and-white indicating an unresponsive window. Only xkill will help out and put Avidemux out of its misery.


I recently decided to try again. At my workplace I had upgraded to 13.10 a while ago, as I did not need Avidemux in there. A lot of time had passed and I thought perhaps it had been fixed already and Avidemux would now work.

I installed Avidemux and surprise - it worked. No more hangs. But as I had upgraded my machine at work from 13.04 to 13.10, I was not sure as of why it worked. I decided to try it out also in fresh installation into VirtualBox.

I installed a fresh 13.10 64-bit image into a new VM, then, without installing the VirtualBox additions, upgraded the packages and then installed Avidemux. The result - it hung, like before.

I started to look for differences between the two machines. Into my work machine, I had installed a mainline 3.12 kernel - maybe that was the solution? Nope, that did not help, the Avidemux in VirtualBox would still hang.

Then I noticed that the Avidemux in my main machine was displaying its menus inside the window, and not in the global menu bar. I also noticed, that gnome-terminal, for example, was also displaying its menus inside the main window, so I ran dpkg --get-selections|sort in both machines and compared the selections. And there was the answer - my main machine was missing the following packages:

unity-gtk2-module
unity-gtk3-module
unity-gtk-module-common

I removed these packages from my VirtualMachine installation and confirmed that Avidemux would no longer hang. Problem solved.

How I got the Precise Pangolin to play ball.

Ubuntu 12.04 Precise Pangolin

At first I thought that it would be impossible to get Ubuntu 12.04 to be usable in my workplace environment, but after a lot of tweaking and time, I have finally made the Precise Pangolin work exactly the way I wanted.

In the following I bring a set of changes I had to do to make it work.

1. The language problem (see my original post about this)
   The main problem is, that by choosing Estonian as your language, then in addition to the LANG environment variable, all of the LC_* variables are set to "et_EE.UTF-8" as well. This, as I wrote earlier, makes regexp [a-z] to exclude the letters coming after "s". The original solution I used previously was to set LANG back to English "en_US.UTF-8" locale in .profile, but this has the side-effect of turning everything into English.
   
   My current solution is much more elegant and will solve the regexp problem, while retaining the Estonian language - add these two lines into .profile:
   export LC_CTYPE="en_US.UTF-8"
export LC_COLLATE="en_US.UTF-8"

A quote to remember.

Babylon 5, “A Late Delivery from Avalon”, Marcus Cole:

 I used to think it was awful that life was so unfair. Then I thought, wouldn’t it be much worse if life were fair, and all the terrible things that happen to us come because we actually deserve them? So, now I take great comfort in the general hostility and unfairness of the universe.

WD My Book Live - a good external HDD and NAS

Western Digital My Book Live
AnandTech review

As I have previously posted, I have one of my older PC's (an Athlon 64 3200+ box) acting as a sort of NAS box, a Library, with four 320GB hard drives in RAID-5 configuration, resulting in approx 875GiB of useful redundant storage.
In addition to that, a few years ago, for media storing purposes, I put together a SATA-to-USB box and a 500GB WD Green drive in it.

However, now that I have bought Samsung's 46" LED-TV (EH5450W), which is also a PVR, I realized that I need an external HDD for the PVR functionality, and since it does not really have to be very big, the above mentioned 500GB box suit the purpose best.

I needed to move the stuff on that drive to somewhere else. The Library machine was out of the question, as my digital photos, documents and source code takes a lot of space there already and the RAID-5 redundancy is more important for this kind of content. It was therefore time to get a new external HDD.

At first I thought of getting WD My Book Essential drive, as I had some experience with it previously. Deciding to go with 3TB version, I was a bit worried about the max 30GiB/s transfer speeds inherent to USB2, as my machine lacks any USB3 ports.

Fortunately though, the Essential drives were currently out of circulation and I was offered a My Book Live instead. Back then, I only knew that the Essential is accessed through USB and the Live through the network. But in fact, they are very different beasts indeed. While the My Book Essential has only SATA-to-USB bridge in the box beside the drive, the Live is actually a whole independent PC, with 800MHz PowerPC 464 processor, 256MiB of memory and a Gigabit LAN port. It is running a ppc version of Debian Linux.

In addition to exposing the contents of the 3TB WD Green drive via Samba, also NFS is supported (though it is considerably slower apparently thanks to the NFS inherent encrypted data transfers). SSH access into the machine is also available. It also has web server, which includes nice UI for changing the settings of the NAS. And it has a media server, so my Smart TV picked it up automatically and was able to play back the media I had already copied there.

I was really surprised by this device.  The transfer speeds are much higher than USB2, reaching an average of 65MiB/s and going sometimes as high as 80MiB/s. The whole device is also almost totally silent, save some HDD head moving sound.
A bit worrying are the temperature readings, as after long coping stuff into the device, the hddtemp reports up to 65 degrees Celsius. Under normal operation, however, the drive is mostly sleeping and the temperature drops considerably.


I highly recommend this drive!

Setting up Flash Builder in Ubuntu 10.10

Adobe Flash Builder 4
Ubuntu 10.10 Maverick Meerkat

In a very sad news, Adobe has officially terminated the Linux Flex Builder project, which has stagnated in alpha status for more than three years now, olny receiving updates to extend the termination date. Regardless of the stated alpha status, the Linux Flex Builder plugin was very usable and quite stable, provided, of course, that you used it with Eclipse 3.3.

Since the current alpha 5 of the Linux Flex Builder plugin terminates at the end of this year and as Adobe has no intentions to allow anyone to use it past that termination date, I had to find an alternative.

Although JetBrains is welcoming all Flex programmers in Linux to switch to IntelliJ IDEA, I didn't want to learn yet an another IDE, especially one that is not open source nor free.

And finally I found the fb4linux project which works great (including the debugging).

The following is mostly a copy of the project wiki with only some additions to make it work with the Eclipse installed from Ubuntu repository.

1. Install Eclipse 3.5.x from Ubuntu repository;
   
2. Download the four FB4Linux* files from the fb4linux project download section, concatenate them to form FB4Linux.tar.bz2 and extract it somewhere.
   
3. Run Eclipse as root!
   sudo su; cd; eclipse
   
4. Select Window / Preferences and open the General / Capabilities section in the Preferences window.
   
5. Verify that Classic Update option is checked and close the preferences window.
   
6. Select Help / Software Updates / Manage Configuration
   
7. Right-click, select Add / Extension location... and browse to the Adobe Flash Builder 4 directory you extracted above.
   
8. Restart Eclipse as per its demands.
   

That's it, you can now open Eclipse from your user account and start coding Flex and AIR applications!