(1) Get yourself a few cans of compressed air and a vacuum cleaner and take a look inside those old boxes. Make sure they have a fan on the CPU, the cables are all there and the spiders have vacated the premises.
(2) Decide what version of linux you want to use. I chose RedHat 7.3, the size of your hard drives (HD's) might make that decision for you however! If you have tiny HD's then go with Debian. I have 8 boxes, 5 pentium II's and 3 pentiums.
fdisk your HD's and I made the following partitions:
Server or main node:
hda1 /boot 100MB
hda2 / 1500MB
hda3 SWAP 256MB
hda4 extended (rest fo drive)
hda5 /usr 1600MB (need at least 1200MB here)
hda6 /usr/local 500MB
hda7 /home (rest of disk)
4.3GB disk with floppy and CDROM. Use twice your availbale RAM for SWAP space. I had 128MB of RAM so I used 256MB swap. I installed two NICs (Network Interface Cards) one to goto a switch and the rest of the cluster and one to access the outside world. Choose one of the better machines for the main node (server for the cluster) install a server and use a graphical interface. This machine is the only one with bells and whistles since you will be interfacing with it and not the others. I did not use YPbind or the server settings, instead we shall use the /etc/hosts file for all our name serving needs. It is recommended you set up one node first, get it working then worry about the other nodes. All nodes will be the same, or at least have a very similar setup, so once you know how to do one the rest is cake!
DO NOT install any kind of firewall. This will mess up the nfs mounts later. Say NO FIREWALL.
For the nodes I used the following partitions:
hda1 /boot 100MB
hda2 / 2000MB
hda3 SWAP 256MB
hda4 extended (rest fo drive)
hda5 /usr 1500MB (need at least 1200MB here)
hda6 /tmp (rest of disk)
Note there are no /home directories or /usr/local they will be nfs mounted from the master node. I bought a netgear FS108 switch with 8 ports, and 9 ethernet cables, one long eight short.
(rest of world)--eth0----(Masternode)--eth1--------(switch)-----All 7 nodes
The Masternode needs 2 ethernet cards, eth0 was setup to go to the outside world. eth1 was setup to go to the switch and all the other 7 nodes were attached directly to the switch.
All the nodes have one ethernet card, eth0 all going to the switch. The easiest way to get all the nodes working smoothly, is to set them ALL up as servers, that way you can dleete things you don't want and will not have to load missing items later. For the message passing interface to work (MPI availbale off the net free) you need all the nodes and master to be running the daemon rshd this is automatically done if you load the nodes as servers.. you can turn off all the yp stuff. Otherwise, you need to select individual packages and make sure ssh and rsh are turned on and of course gc++ c and g77 etc which are also automatically loaded for a server! and not for a workstation.
The /etc/hosts file on the masternode hostname (shall we say) fred, looks like this:
127.0.0.1 localhost localhost.localdomain
123.456.x.x fred fred.y.y
10.0.0.1 node1
10.0.0.2 node2
10.0.0.3 node3
10.0.0.4 node4
10.0.0.5 node5
10.0.0.6 node6
10.0.0.7 node7
10.0.0.8 node8
The x.x stands for numbers of course and the y.y for words.
I did not bother with a domainname for the cluster, more trouble than its
worth.
The destination network for the switch is 10.0.0.0, the broadcast is 10.0.0.255, the netmask is 255.255.255.0 and leave the gateway blank. The FS108 netgear switch cannot be configured for a IP number and therefore there is no gateway as such.
You may want to look into kickstart to load all the nodes once you've done one, but with only a few nodes doing them by hand only takes a couple of hours, each!
To do the NFS mount of /home and /usr/local:
On the Master node, edit a file /etc/exports as follows:
/usr/local 10.0.0/255.255.255.0 (ro)
/home 10.0.0/255.255.255.0 (rw,sync)
This allows all nodes in the 10.0.0.0 nework to access the /usr/local and /home
directories of the master node.
On the nodes you need to edit /etc/fstab as follows:
node1:/usr/local /usr/local nfs defaults 0 0
node1:/home /home nfs defaults 0 0
if you loaded any kind of firewall this WILL NOT WORK!
MPI is very fussy about dates and timing.. so to ensure all the nodes are in sync with the Master create a file in /etc/cron.daily which will be executed every day,
cd /etc/cron.daily
vi check_time
#!/bin/sh
rdate -s node1
shift zz (to exit the editor)
this create a file called check_time which will look at node1's clock once a day
and reset the time on each node. You must make this file executable by issuing
the command
chmod 755 check_time
That should do the trick.
Also on the master node, it needs to list all the "trusted machines" under the
file /etc/hosts.equiv
This file should look like this:
fred
node1
node2
node3
node4
node5
node6
node7
node8
You might also put the same list of machines in a file called .rhosts in the home dirtectory. This will ensure that rsh knows that all these mahcines are trusted.
I would recommend only having root and one user on each machine. Use the same
name and passwords on all machines.
You need to go under the directory /etc/xinetd.d and make sure all the daemons
you need are listed as "disable=no" and not "disable=yes". Edit the files,
rshd rlogind wu_ftpd and any others you want in that directory.
u might find the nodes are running too much junk. You can switch off daemons
using /sbin/chkconfig --list to list what you have running and then
/usr/sbin/ntsysv to turn things on and off. You need to run these as root (su).
You should leave on the following:
anacron
atd
crond
keytable
netfs
network
nfslock
random
rawdevices
sshd
syslog
xinetd (rsh and rlogin )
If you leave on kudzu that will automatically detect new hardware, which is nice
on your masternode but probably not needed on the slave nodes. You can use
nmap node? to check what is running on node?. For example
nmap of node1 (or fred) looks like this:
ftp
ssh
telnet
sunrpc
exec (rexec)
login (rlogin )
shell (rsh)
unknown
rsync
kdm
listen (portmapper)
x11
I am using a text interface on all the nodes, they do not need to waste cpu time
on the graphical interface.
You need to download MPI from http://www.lam-mpi.org/ I got the test suite aswell
to test to see if my cluster was working properly.
Installing the lam-6.5.9 package, read # as the prompt,
# gunzip -c lam-6* |tar xvf -
# cd lam*
# ./configure (this uses rsh by default, check documents if you want to use ssh
instead)
# make
# make install
# make examples
Before you try to use lamboot you must make sure you can rsh into each node
without any messages appearing on the screen. The login usually wants
to type when you last logged in and the date so to turn that off, go to
the home directory and issue the command,
# touch .hushlogin
this creates an empty file called .hushlogin which basically shuts up the .login
script and there will be no messages to the screen when you rsh into it.
Then to get started you need to tell lamboot how many nodes you have and what their
addresses are,
Goto the top of the lam directory and find the file /etc/lam-bhost.def
You need to edit this file and tell lamboot how many nodes you have and their
names. The /etc/lam-bhost.def file should read like this:
node1
node2
node3
node4
node5
node6
node7
node8
Now you should be ready.. try the test script
# recon -dv
If that runs with no error messages you're all set, congratulations you have a parallel processing cluster!
lamboot and lamhalt to stop, good luck!
There's a nice tutorial teach yourself course at the site
http://webct.ncsa.uiuc.edu:8900/public/MPI/
click here
Return to:
