Cortex setup

Click here to go back to the main Cortex article.

This page explains the setup of this cluster in detail.

Network setup

Connectivity

Cortex machines are connected to Cortex Switch, that links to VisLab switch with a fiber optic connection of 4Gbit/s.

Cortex nodes

Cortex server and clients have the following IPs and domain names:

• Server: 10.10.1.240, server.visnet
• Client 1: 10.10.1.1, cortex1.visnet
• Client 2: 10.10.1.2, cortex2.visnet
• Client 3: 10.10.1.3, cortex3.visnet
• Client 4: 10.10.1.4, cortex4.visnet
• Client 5: 10.10.1.5, cortex5.visnet
• Client 6: 10.10.1.6, cortex6.visnet

For further details, see VisLab network.

Boot procedure

The clients boot via the network, using the PXE system. Each machine determines its own identity and asks the server for a kernel image and an initial ram disk. Kernel images and initial ram disks are stored on the server in the /tftpboot/ directory. Kernels can be stock kernels, but the initial ram disk must be created in a way that enables booting from the network. This is not as bad as it sounds. It involves invoking the command mkinitramfs.

The server decides which kernel and initramdisk to send to each machine based on the information stored in the two files: /tftpboot/pxelinux.cfg/default(for cortexes 1-5) and /tftpboot/pxelinux.cfg/01-00-19-d1-9e-e9-53 (for cortex6).

The roots and the homes of the users are also stored on the server machine, so they are mounted by each client at boot time.

Mounting of root directory

We are not sure which mechanism mounts the root filesystem, exactly. Here is the relevant line from /etc/fstab:

 # <file system> <mount point>   <type>  <options>       <dump>  <pass>
 /dev/nfs        /               nfs     defaults        1       1

Mounting of home directory

The home directory is mounted via the Upstart system a few seconds after booting. When rebooting the system, it is possible to login while /home is still not mounted. In that case, log out and log in again, in order for your environment variables to be set correctly.

In November 2010, we created a file called /etc/init/mountHome-net.conf containing:

 description     "Mount network filesystems"
 start on started networking or runlevel 2
 exec /usr/local/bin/mountHome.sh

and /usr/local/bin/mountHome.sh containing:

 MOUNTED=$(mount | grep home)
 while [ -z "$MOUNTED" ]
 do
       su icub -c 'mount /home' &> /var/tmp/mountHomeUpstartOut.txt
       MOUNTED=$(mount | grep home)
       echo $MOUNTED
       sleep 1
 done
 ls /home > /var/tmp/mountHomeUpstartLs.txt
 runlevel > /var/tmp/mountHomeUpstartRunlevel.txt

After a successful mount, we should see something like

 $ mount | grep home
 10.10.1.240:/nfsroot.home on /home type nfs (rw,user=icub,addr=10.10.1.240)

If /home is wrongly mounted with the noexec flag, users won't be able to execute binaries located inside it.

Server machine

The server has:

• a boot folder for the clients at /tftpboot/pxelinux.cfg. It contains the files:
  • default - default boot file;
  • <mac_address> - specific for a machine with the given mac address.
• startup scripts for each machine at /nfsroot/app

Client machines

The clients have:

• A superuser account (compurack) to administer system-wide settings (configurations, libraries, etc.)
• Normal user accounts. By default, the login script runs the contents of file $HOME/.bash_env, where users can set their environment variables, e.g., export ICUB_ROOT=$HOME/iCub. This works for both interactive shell sessions and non-interactive ones (i.e., commands remotely invoked by yarprun).
• A yarp account to update and install the YARP library. Variable YARP_ROOT is set by default to /home/yarp/yarp2 for all users (in /etc/bash.bashrc) <-- change this policy
• An icub account with sudo privileges (created with sudo adduser icub admin on 2009-06-30) <-- change this policy
• cortex6's /etc/hosts file can include the following line:

 127.0.0.1	cortex6