LDoms on Sun SPARC Enterprise T5240

What

It is the third generation Chip Multithreading Technology (CMT) Ultra SPARC 2T Plus CPU. I did work on this configuration : 128 virtul CPUs, 64GB RAM, 16 cryptographic units (MAU) and total of 16 hard disks.

Why we need them

Drive up utilization and consolidate workloads with built-in virtualization technology - Logical Domains LDoms.

How to install LDoms - control one

Get the file LDoms_Manager-1_1.zip from Sun, unzip it and you'll get directory LDoms_Manager-1_1
Go to LDoms_Manager-1_1/Install and run the command install-ldm -d none
In this way, you'll not harden your system using Solaris Security Toolkit drive, otherwise you may not be able to login agan.
And now you'll have 2 new dirs in /opt - SUNWldm and SUNWjass.

Also don't forget to add /opt/SUNWldm/bin in PATH and /opt/SUNWldm/man in MANPATH
Make sure service svc:/ldoms/ldmd:default is running.
Okay, so 'primary' is now your control domain and will take care of all other LDoms.

• Virtual service for disk server : ldm add-vds vds-${ldom_name} primary
• Virtual network terminal server : ldm add-vcc port-range=5000-5010 vcc-console primary
• Virtual service for network switch : ldm add-vsw net-dev=${eth_name} vsw-${sub_name} primary
• example of ${eth_name} is nxge0
• example of ${sub_name} is dmz or sub1
• Verify what you did : ldm list-services

• Cryptographic engine : ldm set-mau < number > primary
• Virtual CPU : ldm set-vcpu < number > primary
• Memory : ldm set-memory < number >g primary

• Check configuration : ldm list-spconfig
• Add configuration : ldm add-spconfig name
• verify resources are allocated to primary : ldm list
• Power off : init 0
• One note here : "init 5" behaives as "init 6", strange. So I "init 0" and from ALOM do poweroff

How to install guest LDoms

okay, so now we need to install guest LDoms. Here is one option:

The zfs root is installed on first 2 HDDs (mirror zpool, disks are 146G).
Additional 6 zpool are created (mirror of 2 HDDs, each 73G) and 2 disks are spare. So it will be 4 LDoms, and 2 of them will use 2 zpool(s).

So variables used in this setup are for example:
LDOM = WEB, MAIL, QA, DEV
ZPOOL = poolweb, poolwebdata, poolmail, poolmaildata, poolqa, pooldev
ZVOLUME = poolweb_vol, poolwebdata_vol, poolmail_vol, poolmaildata_vol, poolqa_vol, pooldev_vol
Note: Coresponding {$ZVOLUME} is zfs volume in ${ZPOOL}
SUB_NAME = dmz, sub1, sub2, bk (T5240 has builtin NIC with 4 ports nxge0-3)

• Create guest LDom : ldm add-domain ${LDOM}
• Add vcpu : ldm add-vcpu 16 ${LDOM}
• Add RAM : ldm add-memory 8g ${LDOM}
• Add MAU : ldm add-mau 2 ${LDOM}
• Add virtual network devices on net switch (vnet0 for dmz/sub1/sub2 and vnet1 for bk): ldm add-vnet vnet<number> vsw-${SUB_NAME} ${LDOM}
• Add DEVICE (zvol) to be exported by VDS as virtual disk to LDOM : ldm add-vdsdev /dev/zvol/dsk/${ZPOOL}/${ZVOLUME} zvol0@vds-${LDOM}
• Add (assign) virtual DISK to LDom : ldm add-vdisk zdisk0 zvol0@vds-${LDOM} ${LDOM}
• Note: 2 LDOMs (WEB/MAIL) will have second device (zvol1) and second virtual disk (zdisk1)"
• Set autoboot to false : ldm set-variable auto-boot\?=false ${LDOM}
• Point to virtual disk device as boot device : ldm set-variable boot-device=/virtual-devices@100/channel-devices@200/disk@0 ${LDOM}
• Bind LDom : ldm bind-domain ${LDOM}
• Start LDom : ldm start-domain ${LDOM}

Now we have LDoms, there are many ways to install the OS. Say use jumpstart installation ( boot net -v - install from 'ok') or create zfs snapshot from 'reference' system, boot from cdrom and restore zfs snapshot to LDom.

Let's try second option, so having zfs snapshot, booting from cdrom and restoring zfs snapshot.

• Get solaris dvd image, save to /var/tmp
• Add cdrom for LDom you want to install : ldm add-vdsdev /var/tmp/sol-10-u7-ga-sparc-dvd.iso cdrom@vds-misc
• This will require poweroff/poweron from "sc>"
• Add virtual cdrom : ldm add-vdisk cdrom cdrom@vds-misc ${LDOM}
• Save spconfiguration : ldm add-spconfig intial-new
• Again poweroff/poweron
• Stop domain : ldm stop-domain ${LDOM}
• Unbind domain : ldm unbind ${LDOM}
• Copy previously saved zfs to new filesystem : zfs send space/reference@installed | zfs receive -F space/${LDOM}
• Make sure this service is running svc:/ldoms/vntsd:default
• Start domain : ldm start-domain ${LDOM}
• Bind domain : ldm bind ${LDOM}
• Go to LDoms console : telnet localhost 5001
• You'll get to 'ok' so boot in single user mode from cdrom : boot cdrom -s
• See all zfs you have available for import : zpool import
• Import : zpool import -f ${ZPOOL}
• Reboot : init 6
• If needed change hostname/IP