|
|
| (45 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| Cortex is a cluster of 6 servers used by VisLab for '''development''' purposes (the other cluster, [[iCubBrain server configuration | iCubBrain]], is used for demos). | | Cortex is a server used by VisLab for running simulations (not the ones with the iCub robot). |
|
| |
|
| ''Old information can be consulted at [[Cortex/Archive]].'' | | ''Old information can be consulted at [[Cortex/Archive]].'' |
| Line 5: |
Line 5: |
| = Specifications = | | = Specifications = |
|
| |
|
| From an end-user perspective:
| | As of 2017, there is one machine (cortex1) with these specs: |
| * each machine hosts an Intel Pentium Dual-Core processor (2 x [http://ark.intel.com/products/31733/Intel-Pentium-Processor-E2180-(1M-Cache-2_00-GHz-800-MHz-FSB) E2180] @ 2.00GHz); | | * 8 x [http://ark.intel.com/products/65523/Intel-Core-i7-3770K-Processor-(8M-Cache-up-to-3_90-GHz) i7-3770K] @ 3.50GHz processor |
| * memory: 2GB for each machine (<code>sudo dmidecode --type 17</code> to see RAM speed and type). | | * 16GB of memory (<code>sudo dmidecode --type 17</code> to see RAM speed and type) |
| | | * 112GB SSD drive + 1TB HDD drive |
| More specifically:
| | * NVidia [http://www.geforce.com/hardware/desktop-gpus/geforce-gtx-670 GeForce GTX 670] graphics card (CUDA) |
| * each of the client machines has the following specs: Intel Desktop DG965SS motherboard ([http://www.intel.com/support/motherboards/desktop/dg965ss/sb/CS-026600.htm info1], [http://processormatch.intel.com/CompDB/SearchResult.aspx?Boardname=dg965ss info2]); Intel DualCore E2180 CPU @ 2 GHz with cache 1 MB; 2 x RAM 1GB DDR2 677 MHz; rackmount 19" 2U case from Chieftec (UNC-210S-B), with ATX 300W power supply; | | * internal ISR IP address: 10.10.1.1 |
| * the server machine has similar specs plus Intel PRO 1000GTLP/ENet gigabit ethernet card; 2 x WD2500JS 250GB SATAII 8MB disks for software RAID-1.
| |
| | |
| = Setup =
| |
| | |
| The Cortex computation rack actually contains 7 machines:
| |
| * 1 server that manages startup, shutdown and the file system of the clients;
| |
| * 6 clients (named <code>cortex1</code>...<code>cortex6</code>) that run user processes.
| |
| All clients numbered 1 to 5 mount the same file system. Therefore, performing changes in the file system of cortex[1-5] will reflect to all other four clients. Beware, though, that because of the way the file systems are mounted, there is some '''caching''' going on. This improves disk access performance, but strange phenomena might happen, i.e., after a file is modified and saved on one client, other clients can continue to see the old version of it for some time (probably less than one minute).
| |
| The client <code>cortex6</code> is separate for now, because it runs a 64 bit Linux.
| |
| | |
| == Network setup ==
| |
| | |
| === Connectivity ===
| |
| | |
| Cortex machines are connected to Cortex Switch, that links to VisLab switch with a fiber optic connection of 4Gbit/s.
| |
| | |
| === Cortex nodes ===
| |
| | |
| Cortex server and clients have the following IPs and domain names:
| |
| * Server: 10.10.1.240, server.visnet | |
| * Client 1: 10.10.1.1, cortex1.visnet
| |
| * Client 2: 10.10.1.2, cortex2.visnet
| |
| * Client 3: 10.10.1.3, cortex3.visnet
| |
| * Client 4: 10.10.1.4, cortex4.visnet
| |
| * Client 5: 10.10.1.5, cortex5.visnet
| |
| * Client 6: 10.10.1.6, cortex6.visnet
| |
| | |
| ''For further details, see [[VisLab network]].''
| |
| | |
| == Boot procedure ==
| |
| | |
| The clients boot via the network, using the PXE system. Each machine determines its own identity and asks the server for a kernel image and an initial ram disk. Kernel images and initial ram disks are stored on the server in the <code>/tftpboot/</code> directory. Kernels can be stock kernels, but the initial ram disk must be created in a way that enables booting from the network. This is not as bad as it sounds. It involves invoking the command <code>mkinitramfs</code>.
| |
| | |
| The server decides which kernel and initramdisk to send to each machine based on the information stored in the two files: <code>/tftpboot/pxelinux.cfg/default</code>(for cortexes 1-5) and <code>/tftpboot/pxelinux.cfg/01-00-19-d1-9e-e9-53</code> (for cortex6).
| |
| | |
| The roots and the homes of the users are also stored on the server machine, so they are mounted by each client at boot time.
| |
| | |
| === Mounting of root directory ===
| |
| | |
| We are not sure which mechanism mounts the root filesystem, exactly. Here is the relevant line from <code>/etc/fstab</code>:
| |
| # <file system> <mount point> <type> <options> <dump> <pass>
| |
| /dev/nfs / nfs defaults 1 1
| |
| | |
| === Mounting of home directory ===
| |
| | |
| The home directory is mounted via the Upstart system a few seconds after booting. When rebooting the system, it is possible to login while <code>/home</code> is still not mounted. In that case, log out and log in again, in order for your environment variables to be set correctly.
| |
| | |
| In November 2010, we created a file called <code>/etc/init/mountHome-net.conf</code> containing:
| |
| description "Mount network filesystems"
| |
| start on started networking or runlevel 2
| |
| exec /usr/local/bin/mountHome.sh
| |
| | |
| and <code>/usr/local/bin/mountHome.sh</code> containing:
| |
| MOUNTED=$(mount | grep home)
| |
| while [ -z "$MOUNTED" ]
| |
| do
| |
| su icub -c 'mount /home' &> /var/tmp/mountHomeUpstartOut.txt
| |
| MOUNTED=$(mount | grep home)
| |
| echo $MOUNTED
| |
| sleep 1
| |
| done
| |
| ls /home > /var/tmp/mountHomeUpstartLs.txt
| |
| runlevel > /var/tmp/mountHomeUpstartRunlevel.txt
| |
| | |
| After a successful mount, we should see something like
| |
| $ mount | grep home
| |
| 10.10.1.240:/nfsroot.home on /home type nfs (rw,user=icub,addr=10.10.1.240)
| |
| | |
| If <code>/home</code> is wrongly mounted with the <code>noexec</code> flag, users won't be able to execute binaries located inside it.
| |
| | |
| == Server machine ==
| |
| | |
| The server has:
| |
| * a boot folder for the clients at <code>/tftpboot/pxelinux.cfg</code>. It contains the files:
| |
| ** <code>default</code> - default boot file;
| |
| ** <mac_address> - specific for a machine with the given mac address.
| |
| * startup scripts for each machine at <code>/nfsroot/app</code>
| |
| | |
| == Client machines ==
| |
| | |
| The clients have:
| |
| * A superuser account (<code>compurack</code>) to administer system-wide settings (configurations, libraries, etc.)
| |
| * Normal user accounts. By default, the login script runs the contents of file <code>$HOME/.bash_env</code>, where users can set their environment variables, e.g., <code>export ICUB_ROOT=$HOME/iCub</code>. This works for both interactive shell sessions and non-interactive ones (i.e., commands remotely invoked by <code>yarprun</code>).
| |
| * A <code>yarp</code> account to update and install the YARP library. Variable <code>YARP_ROOT</code> is set by default to <code>/home/yarp/yarp2</code> for all users (in <code>/etc/bash.bashrc</code>) <-- change this policy
| |
| * An <code>icub</code> account with sudo privileges (created with <code>sudo adduser icub admin</code> on 2009-06-30) <-- change this policy
| |
| * cortex6's <code>/etc/hosts</code> file can include the following line:
| |
| 127.0.0.1 cortex6
| |
| | |
| = Operating system installation =
| |
| | |
| In general, follow the guidelines at [[VisLab machines configuration]]. Below are some special notes.
| |
| | |
| = Additional software =
| |
| | |
| == YARP ==
| |
| | |
| yarp2 is installed for user icub, similarly to [[iCubBrain server configuration]].
| |
| | |
| For now, we don't use system-wide installation (<code>sudo make install</code>). We could use it again after we verify that a user can easily override global settings.
| |
| | |
| == iCub ==
| |
| | |
| iCub is installed for user icub, similarly to [[iCubBrain server configuration]].
| |
| | |
| For now, we don't use system-wide installation (<code>sudo make install</code>). We could use it again after we verify that a user can easily override global settings.
| |
| | |
| == Other libraries, manually installed ==
| |
| | |
| Please list here the system-wide libraries and applications that were installed by the superuser, especially the ones that do not have a clean 'make install' procedure but were manually installed into <code>/opt</code>:
| |
| | |
| * ARToolKit
| |
| * Ipopt-3.5.5-linux-x86_32-gcc4.2.4
| |
| | |
| CMake 2.6 does not come with the version of Ubuntu currently installed, but it is needed by the latest version of yarp, so we installed it via this [http://www.cmake.org/files/v2.6/cmake-2.6.4-Linux-i386.sh archive].
| |
| * cmake 2.6
| |
| | |
| == Other libraries, installed with Ubuntu packages ==
| |
| | |
| These packages were installed with <code>apt-get install</code>
| |
| libncurses5-dev
| |
| libace-dev
| |
| libgsl0-dev
| |
| libgtk2.0-dev libgtkmm-2.4-dev libglademm-2.4-dev
| |
| glew-utils libglew1.5-dev
| |
| libglut-dev
| |
| git-core
| |
| | |
| OpenCV:
| |
| libcv-dev libhighgui-dev libcvaux-dev
| |
| | |
| == User repositories ==
| |
| | |
| Each user should manage his own yarp2 and iCub repositories. <-- ''then we shouldn't have done sudo make install here :)''
| |
| | |
| We recommend to set your environment variables in a new file, called <code>~/.bash_env</code> and containing:
| |
| export YARP_ROOT=~/yarp2
| |
| export YARP_DIR=$YARP_ROOT/build
| |
| export ICUB_ROOT=~/iCub
| |
| export ICUB_DIR=$ICUB_ROOT/main/build
| |
| export PATH=$PATH:$YARP_DIR/bin:$ICUB_DIR/bin
| |
| | |
| Refer to the [[RobotCub software]] article for further instructions.
| |
| | |
| = Customization =
| |
| | |
| == Network tuning ==
| |
| | |
| sysctl -w net.core.rmem_max=8388608
| |
| sysctl -w net.core.wmem_max=8388608
| |
| sysctl -w net.core.rmem_default=65536
| |
| sysctl -w net.core.wmem_default=65536
| |
| sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608'
| |
| sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608'
| |
| sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'
| |
| sysctl -w net.ipv4.route.flush=1
| |
| | |
| == Prompt ($PS1) ==
| |
| | |
| The prompt is set to <code>user@cortex?:pwd$</code> in <code>/etc/bash.bashrc</code>. With those settings, if you log in to Cortex1, the prompt will be <code>user@cortex1:~$</code>.
| |
| We chose to do so because sometimes it's convenient to have the number of the Cortex machine you're working on embedded in the prompt.
| |
| By default, though, this configuration is overridden in the users' <code>~/.bashrc</code> file, and the prompt is set to <code>user@source</code> regardless of the Cortex machine you log in to.<br>
| |
| If you want to inhibit this behaviour in <code>~/.bashrc</code> and thus have a prompt like <code>user@cortex?:pwd</code>, just comment these lines in your <code>~/.bashrc</code>:
| |
| # set a fancy prompt (non-color, unless we know we "want" color)
| |
| case "$TERM" in
| |
| xterm-color)
| |
| PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
| |
| ;;
| |
| *)
| |
| PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
| |
| ;;
| |
| esac
| |
| However, for users created after 2009-05-07, the prompt is already set to <code>user@cortex?:pwd$</code> by default.
| |
| | |
|
| |
|
| [[Category:Vislab]] | | [[Category:Vislab]] |