Cortex/Archive: Difference between revisions
(→Specifications: archive) |
m (→Customization: auto mounting of blackhole) |
||
(2 intermediate revisions by the same user not shown) | |||
Line 102: | Line 102: | ||
= Additional setup = | = Additional setup = | ||
== Network tuning == | |||
sysctl -w net.core.rmem_max=8388608 | |||
sysctl -w net.core.wmem_max=8388608 | |||
sysctl -w net.core.rmem_default=65536 | |||
sysctl -w net.core.wmem_default=65536 | |||
sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608' | |||
sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608' | |||
sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608' | |||
sysctl -w net.ipv4.route.flush=1 | |||
== Other libraries, installed with Ubuntu packages == | == Other libraries, installed with Ubuntu packages == | ||
Line 145: | Line 156: | ||
to | to | ||
SET(IPOPT_LIB ${IPOPT_LIB} gfortran) | SET(IPOPT_LIB ${IPOPT_LIB} gfortran) | ||
= Customization = | |||
== Prompt ($PS1) == | |||
The prompt is set to <code>user@cortex?:pwd$</code> in <code>/etc/bash.bashrc</code>. With those settings, if you log in to Cortex1, the prompt will be <code>user@cortex1:~$</code>. | |||
We chose to do so because sometimes it's convenient to have the number of the Cortex machine you're working on embedded in the prompt. | |||
By default, though, this configuration is overridden in the users' <code>~/.bashrc</code> file, and the prompt is set to <code>user@source</code> regardless of the Cortex machine you log in to.<br> | |||
If you want to inhibit this behaviour in <code>~/.bashrc</code> and thus have a prompt like <code>user@cortex?:pwd</code>, just comment these lines in your <code>~/.bashrc</code>: | |||
# set a fancy prompt (non-color, unless we know we "want" color) | |||
case "$TERM" in | |||
xterm-color) | |||
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' | |||
;; | |||
*) | |||
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ ' | |||
;; | |||
esac | |||
However, for users created after 2009-05-07, the prompt is already set to <code>user@cortex?:pwd$</code> by default. | |||
== Mounting network folders == | |||
The folder "database" on BlackHole is mounted automatically to /media/blackhole. | |||
For details, here is the content of /etc/fstab: | |||
#MATTEO: removed because it wouldn't work at startup: file systems are mounted before the network connection is available. | |||
#//10.0.3.118/database /media/blackhole smbfs username=vislab,password=vislab,iocharset=utf8,sec=ntlm 0 0 | |||
= Helper commands = | = Helper commands = |
Latest revision as of 14:01, 14 July 2017
Note: these methods are obsolete and kept here for historic reference only. Most probably, you may ignore this page and go back to Cortex.
Specifications
During 2009-2012, the setup was the following.
From an end-user perspective:
- each machine hosts an Intel Pentium Dual-Core processor (2 x E2180 @ 2.00GHz);
- memory: 2GB for each machine.
More specifically:
- each of the client machines has the following specs: Intel Desktop DG965SS motherboard (info1, info2); Intel DualCore E2180 CPU @ 2 GHz with cache 1 MB; 2 x RAM 1GB DDR2 677 MHz; rackmount 19" 2U case from Chieftec (UNC-210S-B), with ATX 300W power supply;
- the server machine has similar specs plus Intel PRO 1000GTLP/ENet gigabit ethernet card; 2 x WD2500JS 250GB SATAII 8MB disks for software RAID-1.
The Cortex computation rack actually contains 7 machines:
- 1 server that manages startup, shutdown and the file system of the clients;
- 6 clients (named
cortex1
...cortex6
) that run user processes.
All clients numbered 1 to 5 mount the same file system. Therefore, performing changes in the file system of cortex[1-5] will reflect to all other four clients. Beware, though, that because of the way the file systems are mounted, there is some caching going on. This improves disk access performance, but strange phenomena might happen, i.e., after a file is modified and saved on one client, other clients can continue to see the old version of it for some time (probably less than one minute).
The client cortex6
is separate for now, because it runs a 64 bit Linux.
For further details, see the Detailed setup section below.
Detailed setup
Network setup
Connectivity
Cortex machines are connected to Cortex Switch, that links to VisLab switch with a fiber optic connection of 4Gbit/s.
Cortex nodes
Cortex server and clients have the following IPs and domain names:
- Server: 10.10.1.240, server.visnet
- Client 1: 10.10.1.1, cortex1.visnet
- Client 2: 10.10.1.2, cortex2.visnet
- Client 3: 10.10.1.3, cortex3.visnet
- Client 4: 10.10.1.4, cortex4.visnet
- Client 5: 10.10.1.5, cortex5.visnet
- Client 6: 10.10.1.6, cortex6.visnet
For further details, see VisLab network.
Boot procedure
The clients boot via the network, using the PXE system. Each machine determines its own identity and asks the server for a kernel image and an initial ram disk. Kernel images and initial ram disks are stored on the server in the /tftpboot/
directory. Kernels can be stock kernels, but the initial ram disk must be created in a way that enables booting from the network. This is not as bad as it sounds. It involves invoking the command mkinitramfs
.
The server decides which kernel and initramdisk to send to each machine based on the information stored in the two files: /tftpboot/pxelinux.cfg/default
(for cortexes 1-5) and /tftpboot/pxelinux.cfg/01-00-19-d1-9e-e9-53
(for cortex6).
The roots and the homes of the users are also stored on the server machine, so they are mounted by each client at boot time.
Mounting of root directory
We are not sure which mechanism mounts the root filesystem, exactly. Here is the relevant line from /etc/fstab
:
# <file system> <mount point> <type> <options> <dump> <pass> /dev/nfs / nfs defaults 1 1
Mounting of home directory
The home directory is mounted via the Upstart system a few seconds after booting. When rebooting the system, it is possible to login while /home
is still not mounted. In that case, log out and log in again, in order for your environment variables to be set correctly.
In November 2010, we created a file called /etc/init/mountHome-net.conf
containing:
description "Mount network filesystems" start on started networking or runlevel 2 exec /usr/local/bin/mountHome.sh
and /usr/local/bin/mountHome.sh
containing:
MOUNTED=$(mount | grep home) while [ -z "$MOUNTED" ] do su icub -c 'mount /home' &> /var/tmp/mountHomeUpstartOut.txt MOUNTED=$(mount | grep home) echo $MOUNTED sleep 1 done ls /home > /var/tmp/mountHomeUpstartLs.txt runlevel > /var/tmp/mountHomeUpstartRunlevel.txt
After a successful mount, we should see something like
$ mount | grep home 10.10.1.240:/nfsroot.home on /home type nfs (rw,user=icub,addr=10.10.1.240)
If /home
is wrongly mounted with the noexec
flag, users won't be able to execute binaries located inside it.
Server machine
The server has:
- a boot folder for the clients at
/tftpboot/pxelinux.cfg
. It contains the files:default
- default boot file;- <mac_address> - specific for a machine with the given mac address.
- startup scripts for each machine at
/nfsroot/app
Client machines
The clients have:
- A superuser account (
compurack
) to administer system-wide settings (configurations, libraries, etc.) - Normal user accounts. By default, the login script runs the contents of file
$HOME/.bash_env
, where users can set their environment variables, e.g.,export ICUB_ROOT=$HOME/iCub
. This works for both interactive shell sessions and non-interactive ones (i.e., commands remotely invoked byyarprun
). - A
yarp
account to update and install the YARP library. VariableYARP_ROOT
is set by default to/home/yarp/yarp2
for all users (in/etc/bash.bashrc
) <-- change this policy - An
icub
account with sudo privileges (created withsudo adduser icub admin
on 2009-06-30) <-- change this policy - cortex6's
/etc/hosts
file can include the following line:
127.0.0.1 cortex6
Additional setup
Network tuning
sysctl -w net.core.rmem_max=8388608 sysctl -w net.core.wmem_max=8388608 sysctl -w net.core.rmem_default=65536 sysctl -w net.core.wmem_default=65536 sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608' sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608' sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608' sysctl -w net.ipv4.route.flush=1
Other libraries, installed with Ubuntu packages
These packages were installed with sudo apt-get install
:
libgtk2.0-dev libgtkmm-2.4-dev libglademm-2.4-dev glew-utils libglew1.5-dev libglut-dev git-core
OpenCV:
libcv-dev libhighgui-dev libcvaux-dev
System-wide libraries and repositories
YARP
Presently (November 2010), the yarp2 SVN repository is installed under user yarp
(with sudo make install
), last updated on 13-July-2010. <-- change this policy?
iCub
Presently (November 2010), the iCub SVN repository is installed under user icub
(with sudo make install
), last updated on 13-July-2010. <-- change this policy?
There was a conflict with iKin, which could not find libipopt.so.0
, but it is now fixed thanks to setting the environment variable
LD_LIBRARY_PATH=/opt/Ipopt-3.5.5-linux-x86_32-gcc4.2.4/lib/
into /home/icub/.bash_env
.
One module has been disabled in the CMakeList.txt file, because it was not compiling properly: crawling.
User repositories
you should add <iCub>/bin to your PATH by editing your ~/.bashrc like this:
PATH=$PATH:~/iCub/bin/ ICUB_DIR=~/iCub/ <-- needs to change export ICUB_DIR ICUB_ROOT=$ICUB_DIR export ICUB_ROOT
You should also edit ~/.bash_env adding these lines:
export ICUB_DIR=$HOME/iCub <-- needs to change export ICUB_ROOT=$ICUB_DIR
this is needed when you connect non-interactively via ssh to a Cortex computer, for instance when execute a "yarp run ..." on a Cortex, from Chico2.
Be aware that Ubuntu 7.10 (the version currently installed on the cluster) has a conflict with iKin, specifically with iCub/conf/FindIPOPT.cmake (used by iKin): for now, in order to compile iKin, change the following line of FindIPOPT.cmake from
SET(IPOPT_LIB ${IPOPT_LIB} gfortranbegin gfortran)
to
SET(IPOPT_LIB ${IPOPT_LIB} gfortran)
Customization
Prompt ($PS1)
The prompt is set to user@cortex?:pwd$
in /etc/bash.bashrc
. With those settings, if you log in to Cortex1, the prompt will be user@cortex1:~$
.
We chose to do so because sometimes it's convenient to have the number of the Cortex machine you're working on embedded in the prompt.
By default, though, this configuration is overridden in the users' ~/.bashrc
file, and the prompt is set to user@source
regardless of the Cortex machine you log in to.
If you want to inhibit this behaviour in ~/.bashrc
and thus have a prompt like user@cortex?:pwd
, just comment these lines in your ~/.bashrc
:
# set a fancy prompt (non-color, unless we know we "want" color) case "$TERM" in xterm-color) PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' ;; *) PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ ' ;; esac
However, for users created after 2009-05-07, the prompt is already set to user@cortex?:pwd$
by default.
Mounting network folders
The folder "database" on BlackHole is mounted automatically to /media/blackhole. For details, here is the content of /etc/fstab:
#MATTEO: removed because it wouldn't work at startup: file systems are mounted before the network connection is available. #//10.0.3.118/database /media/blackhole smbfs username=vislab,password=vislab,iocharset=utf8,sec=ntlm 0 0
Helper commands
These generic Linux commands should be written somewhere else, as they are not Cortex-specific
- Check the kernel:
uname -m
- Check the file versions:
file
- Set bash shell in
/etc/passwd
- Check disk space:
du –sh /home
- Check per-user processes:
ps -U <user>