Friday, September 26, 2008

Inside the Linux boot process

The process of booting a Linux® system consists of a number of stages. But whether you're booting a standard x86 desktop or a deeply embedded PowerPC® target, much of the flow is surprisingly similar. This article explores the Linux boot process from the initial bootstrap to the start of the first user-space application. Along the way, you'll learn about various other boot-related topics such as the boot loaders, kernel decompression, the initial RAM disk, and other elements of Linux boot.

Figure 1 gives you the 20,000-foot view.


Figure 1. The 20,000-foot view of the Linux boot process
High-level view of the Linux kernel boot

When a system is first booted, or is reset, the processor executes code at a well-known location. In a personal computer (PC), this location is in the basic input/output system (BIOS), which is stored in flash memory on the motherboard. The central processing unit (CPU) in an embedded system invokes the reset vector to start a program at a known address in flash/ROM. In either case, the result is the same. Because PCs offer so much flexibility, the BIOS must determine which devices are candidates for boot. We'll look at this in more detail later.

When a boot device is found, the first-stage boot loader is loaded into RAM and executed. This boot loader is less than 512 bytes in length (a single sector), and its job is to load the second-stage boot loader.

When the second-stage boot loader is in RAM and executing, a splash screen is commonly displayed, and Linux and an optional initial RAM disk (temporary root file system) are loaded into memory. When the images are loaded, the second-stage boot loader passes control to the kernel image and the kernel is decompressed and initialized. At this stage, the second-stage boot loader checks the system hardware, enumerates the attached hardware devices, mounts the root device, and then loads the necessary kernel modules. When complete, the first user-space program (init) starts, and high-level system initialization is performed.

That's Linux boot in a nutshell. Now let's dig in a little further and explore some of the details of the Linux boot process.




System startup

The system startup stage depends on the hardware that Linux is being booted on. On an embedded platform, a bootstrap environment is used when the system is powered on, or reset. Examples include U-Boot, RedBoot, and MicroMonitor from Lucent. Embedded platforms are commonly shipped with a boot monitor. These programs reside in special region of flash memory on the target hardware and provide the means to download a Linux kernel image into flash memory and subsequently execute it. In addition to having the ability to store and boot a Linux image, these boot monitors perform some level of system test and hardware initialization. In an embedded target, these boot monitors commonly cover both the first- and second-stage boot loaders.

Extracting the MBR

To see the contents of your MBR, use this command:

# dd if=/dev/hda of=mbr.bin bs=512 count=1
# od -xa mbr.bin

The dd command, which needs to be run from root, reads the first 512 bytes from /dev/hda (the first Integrated Drive Electronics, or IDE drive) and writes them to the mbr.bin file. The od command prints the binary file in hex and ASCII formats.

In a PC, booting Linux begins in the BIOS at address 0xFFFF0. The first step of the BIOS is the power-on self test (POST). The job of the POST is to perform a check of the hardware. The second step of the BIOS is local device enumeration and initialization.

Given the different uses of BIOS functions, the BIOS is made up of two parts: the POST code and runtime services. After the POST is complete, it is flushed from memory, but the BIOS runtime services remain and are available to the target operating system.

To boot an operating system, the BIOS runtime searches for devices that are both active and bootable in the order of preference defined by the complementary metal oxide semiconductor (CMOS) settings. A boot device can be a floppy disk, a CD-ROM, a partition on a hard disk, a device on the network, or even a USB flash memory stick.

Commonly, Linux is booted from a hard disk, where the Master Boot Record (MBR) contains the primary boot loader. The MBR is a 512-byte sector, located in the first sector on the disk (sector 1 of cylinder 0, head 0). After the MBR is loaded into RAM, the BIOS yields control to it.



Stage 1 boot loader

The primary boot loader that resides in the MBR is a 512-byte image containing both program code and a small partition table (see Figure 2). The first 446 bytes are the primary boot loader, which contains both executable code and error message text. The next sixty-four bytes are the partition table, which contains a record for each of four partitions (sixteen bytes each). The MBR ends with two bytes that are defined as the magic number (0xAA55). The magic number serves as a validation check of the MBR.


Figure 2. Anatomy of the MBR
Anatomy of the MBR

The job of the primary boot loader is to find and load the secondary boot loader (stage 2). It does this by looking through the partition table for an active partition. When it finds an active partition, it scans the remaining partitions in the table to ensure that they're all inactive. When this is verified, the active partition's boot record is read from the device into RAM and executed.






Stage 2 boot loader

The secondary, or second-stage, boot loader could be more aptly called the kernel loader. The task at this stage is to load the Linux kernel and optional initial RAM disk.

GRUB stage boot loaders

The /boot/grub directory contains the stage1, stage1.5, and stage2 boot loaders, as well as a number of alternate loaders (for example, CR-ROMs use the iso9660_stage_1_5).

The first- and second-stage boot loaders combined are called Linux Loader (LILO) or GRand Unified Bootloader (GRUB) in the x86 PC environment. Because LILO has some disadvantages that were corrected in GRUB, let's look into GRUB. (See many additional resources on GRUB, LILO, and related topics in the Resources section later in this article.)

The great thing about GRUB is that it includes knowledge of Linux file systems. Instead of using raw sectors on the disk, as LILO does, GRUB can load a Linux kernel from an ext2 or ext3 file system. It does this by making the two-stage boot loader into a three-stage boot loader. Stage 1 (MBR) boots a stage 1.5 boot loader that understands the particular file system containing the Linux kernel image. Examples include reiserfs_stage1_5 (to load from a Reiser journaling file system) or e2fs_stage1_5 (to load from an ext2 or ext3 file system). When the stage 1.5 boot loader is loaded and running, the stage 2 boot loader can be loaded.

With stage 2 loaded, GRUB can, upon request, display a list of available kernels (defined in /etc/grub.conf, with soft links from /etc/grub/menu.lst and /etc/grub.conf). You can select a kernel and even amend it with additional kernel parameters. Optionally, you can use a command-line shell for greater manual control over the boot process.

With the second-stage boot loader in memory, the file system is consulted, and the default kernel image and initrd image are loaded into memory. With the images ready, the stage 2 boot loader invokes the kernel image.






Kernel

Manual boot in GRUB

From the GRUB command-line, you can boot a specific kernel with a named initrd image as follows:

grub> kernel /bzImage-2.6.14.2
[Linux-bzImage, setup=0x1400, size=0x29672e]

grub> initrd /initrd-2.6.14.2.img
[Linux-initrd @ 0x5f13000, 0xcc199 bytes]

grub> boot

Uncompressing Linux... Ok, booting the kernel.

If you don't know the name of the kernel to boot, just type a forward slash (/) and press the Tab key. GRUB will display the list of kernels and initrd images.

With the kernel image in memory and control given from the stage 2 boot loader, the kernel stage begins. The kernel image isn't so much an executable kernel, but a compressed kernel image. Typically this is a zImage (compressed image, less than 512KB) or a bzImage (big compressed image, greater than 512KB), that has been previously compressed with zlib. At the head of this kernel image is a routine that does some minimal amount of hardware setup and then decompresses the kernel contained within the kernel image and places it into high memory. If an initial RAM disk image is present, this routine moves it into memory and notes it for later use. The routine then calls the kernel and the kernel boot begins.

When the bzImage (for an i386 image) is invoked, you begin at ./arch/i386/boot/head.S in the start assembly routine (see Figure 3 for the major flow). This routine does some basic hardware setup and invokes the startup_32 routine in ./arch/i386/boot/compressed/head.S. This routine sets up a basic environment (stack, etc.) and clears the Block Started by Symbol (BSS). The kernel is then decompressed through a call to a C function called decompress_kernel (located in ./arch/i386/boot/compressed/misc.c). When the kernel is decompressed into memory, it is called. This is yet another startup_32 function, but this function is in ./arch/i386/kernel/head.S.

In the new startup_32 function (also called the swapper or process 0), the page tables are initialized and memory paging is enabled. The type of CPU is detected along with any optional floating-point unit (FPU) and stored away for later use. The start_kernel function is then invoked (init/main.c), which takes you to the non-architecture specific Linux kernel. This is, in essence, the main function for the Linux kernel.


Figure 3. Major functions flow for the Linux kernel i386 boot
Major Functions in Linux Kernel i386 Boot Process

With the call to start_kernel, a long list of initialization functions are called to set up interrupts, perform further memory configuration, and load the initial RAM disk. In the end, a call is made to kernel_thread (in arch/i386/kernel/process.c) to start the init function, which is the first user-space process. Finally, the idle task is started and the scheduler can now take control (after the call to cpu_idle). With interrupts enabled, the pre-emptive scheduler periodically takes control to provide multitasking.

During the boot of the kernel, the initial-RAM disk (initrd) that was loaded into memory by the stage 2 boot loader is copied into RAM and mounted. This initrd serves as a temporary root file system in RAM and allows the kernel to fully boot without having to mount any physical disks. Since the necessary modules needed to interface with peripherals can be part of the initrd, the kernel can be very small, but still support a large number of possible hardware configurations. After the kernel is booted, the root file system is pivoted (via pivot_root) where the initrd root file system is unmounted and the real root file system is mounted.

decompress_kernel output

The decompress_kernel function is where you see the usual decompression messages emitted to the display:

Uncompressing Linux... Ok, booting the kernel.

The initrd function allows you to create a small Linux kernel with drivers compiled as loadable modules. These loadable modules give the kernel the means to access disks and the file systems on those disks, as well as drivers for other hardware assets. Because the root file system is a file system on a disk, the initrd function provides a means of bootstrapping to gain access to the disk and mount the real root file system. In an embedded target without a hard disk, the initrd can be the final root file system, or the final root file system can be mounted via the Network File System (NFS).






Init

After the kernel is booted and initialized, the kernel starts the first user-space application. This is the first program invoked that is compiled with the standard C library. Prior to this point in the process, no standard C applications have been executed.

In a desktop Linux system, the first application started is commonly /sbin/init. But it need not be. Rarely do embedded systems require the extensive initialization provided by init (as configured through /etc/inittab). In many cases, you can invoke a simple shell script that starts the necessary embedded applications.





Summary

Much like Linux itself, the Linux boot process is highly flexible, supporting a huge number of processors and hardware platforms. In the beginning, the loadlin boot loader provided a simple way to boot Linux without any frills. The LILO boot loader expanded the boot capabilities, but lacked any file system awareness. The latest generation of boot loaders, such as GRUB, permits Linux to boot from a range of file systems (from Minix to Reiser).


Thursday, September 25, 2008

Port Numbers

http             80/tcp    World Wide Web HTTP
https 443/tcp http protocol over TLS/SSL
smtp 25/tcp Simple Mail Transfer
snmp 161/tcp SNMP
ftp-data 20/tcp File Transfer [Default Data]
ftp 21/tcp File Transfer [Control]
ftps-data 989/tcp ftp protocol, data, over TLS/SS
ftps 990/tcp ftp protocol, control, over
TLS/SSL
telnet 23/tcp Telnet
telnets 992/tcp telnet protocol over TLS/SSL
nfs 2049/tcp Network File System - Sun
Microsystems
pop2 109/tcp Post Office Protocol - Version 2
pop3 110/tcp Post Office Protocol - Version 3
imap 143/tcp Internet Message Access Protocol
imaps 993/tcp imap4 protocol over TLS/SSL
tftp 69/tcp Trivial File Transfer
tftps 3713/tcp TFTP over TLS
domain 53/tcp Domain Name Server
nntp 119/tcp Network News Transfer Protocol
dhcpv6-client 546/tcp DHCPv6 Client
dhcpv6-server 547/tcp DHCPv6 Server
ldap 389/tcp Lightweight Directory Access
Protocol
sftp 115/tcp Simple File Transfer Protocol
ssh 22/tcp SSH Remote Login Protocol
msft-gc 3268/tcp Microsoft Global Catalog
msft-gc-ssl 3269/tcp Microsoft Global Catalog with
LDAP/SSL
.........................................................................
There r 65,535port numbers.out of that
0-1023 are well known port numbers.
1024-49,151 are Registered.
49,152-65,535 are Dynamic or private ports.

Below r some of the well known ports:-
Port Description status
0/TCP,UDP Reserved; do not use (but is a permissible source
port value if the sending process does not expect messages
in response) 1/TCP,UDP TCPMUX (TCP port service
multiplexer) Official
5/TCP,UDP RJE (Remote Job Entry)
7/TCP,UDP ECHO protocol
9/TCP,UDP DISCARD protocol
11/TCP,UDP SYSTAT protocol
13/TCP,UDP DAYTIME protocol
17/TCP,UDP QOTD (Quote of the Day) protocol
18/TCP,UDP Message Send Protocol
19/TCP,UDP CHARGEN (Character Generator) protocol
20/TCP FTP - data port
21/TCP FTP - control (command) port
22/TCP,UDP SSH (Secure Shell) - used for secure logins,
file transfers (scp, sftp) and port forwarding
23/TCP,UDP Telnet protocol - unencrypted text
communications
25/TCP,UDP SMTP - used for e-mail routing between
mailservers E-mails
26/TCP,UDP RSFTP - A simple FTP-like protocol
35/TCP,UDP QMS Magicolor 2 printer
37/TCP,UDP TIME protocol
38/TCP,UDP Route Access Protocol
39/TCP,UDP Resource Location Protocol
41/TCP,UDP Graphics
42/TCP,UDP Host Name Server
42/TCP,UDP WINS [1]
43/TCP WHOIS protocol
49/TCP,UDP TACACS Login Host protocol
53/TCP,UDP DNS (Domain Name System)
57/TCP MTP, Mail Transfer Protocol
67/UDP BOOTP (BootStrap Protocol) server; also used by DHCP
(Dynamic Host Configuration Protocol)
68/UDP BOOTP client; also used by DHCP
69/UDP TFTP (Trivial File Transfer Protocol)
70/TCP Gopher protocol
79/TCP Finger protocol
80/TCP HTTP (HyperText Transfer Protocol) - used for
transferring web pages
81/TCP HTTP Alternate (HyperText Transfer Protocol)
81/TCP Torpark - Onion routing ORport
82/UDP Torpark - Control Port Unofficial
88/TCP Kerberos - authenticating agent
101/TCP HOSTNAME
102/TCP ISO-TSAP protocol
107/TCP Remote Telnet Service
109/TCP POP, Post Office Protocol, version 2
110/TCP POP3 (Post Office Protocol version 3) - used for
retrieving E-mails
111/TCP,UDP SUNRPC protocol
113/TCP ident - old server identification system, still
used by IRC servers to identify its users
115/TCP SFTP, Simple File Transfer Protocol
117/TCP UUCP-PATH
118/TCP,UDP SQL Services
119/TCP NNTP (Network News Transfer Protocol) - used for
retrieving newsgroups messages
123/UDP NTP (Network Time Protocol) - used for time
synchronization
135/TCP,UDP EPMAP (End Point Mapper) / Microsoft RPC
Locator Service
137/TCP,UDP NetBIOS NetBIOS Name Service
138/TCP,UDP NetBIOS NetBIOS Datagram Service
139/TCP,UDP NetBIOS NetBIOS Session Service
143/TCP,UDP IMAP4 (Internet Message Access Protocol 4) -
used for retrieving E-mails
152/TCP,UDP BFTP, Background File Transfer Program
153/TCP,UDP SGMP, Simple Gateway Monitoring Protocol
156/TCP,UDP SQL Service
158/TCP,UDP DMSP, Distributed Mail Service Protocol
161/TCP,UDP SNMP (Simple Network Management Protocol)
162/TCP,UDP SNMPTRAP
170/TCP Print-srv
179/TCP BGP (Border Gateway Protocol)
194/TCP IRC (Internet Relay Chat)
201/TCP,UDP AppleTalk Routing Maintenance
209/TCP,UDP The Quick Mail Transfer Protocol
213/TCP,UDP IPX
218/TCP,UDP MPP, Message Posting Protocol
220/TCP,UDP IMAP, Interactive Mail Access Protocol, version
3
259/TCP,UDP ESRO, Efficient Short Remote Operations
264/TCP,UDP BGMP, Border Gateway Multicast Protocol
311/TCP Apple Server-Admin-Tool, Workgroup-Manager-Tool
318/TCP,UDP TSP, Time Stamp Protocol
323/TCP,UDP IMMP, Internet Message Mapping Protocol
383/TCP,UDP HP OpenView HTTPs Operations Agent
366/TCP,UDP SMTP, Simple Mail Transfer Protocol. ODMR, On-
Demand Mail Relay
369/TCP,UDP Rpc2portmap
371/TCP,UDP ClearCase albd
384/TCP,UDP A Remote Network Server System
387/TCP,UDP AURP, AppleTalk Update-based Routing Protocol
389/TCP,UDP LDAP (Lightweight Directory Access Protocol)
401/TCP,UDP UPS Uninterruptible Power Supply
411/TCP Direct Connect Hub port
412/TCP Direct Connect Client-To-Client port
427/TCP,UDP SLP (Service Location Protocol)
443/TCP HTTPS - HTTP Protocol over TLS/SSL (encrypted
transmission)
444/TCP,UDP SNPP, Simple Network Paging Protocol
445/TCP Microsoft-DS (Active Directory, Windows shares,
Sasser worm, Agobot, Zobotworm) Official
445/UDP Microsoft-DS SMB file sharing Official
464/TCP,UDP Kerberos Change/Set password Official
465/TCP SMTP over SSL - CONFLICT with registered Cisco
protocol Conflict
500/TCP,UDP ISAKMP, IKE-Internet Key Exchange Official
512/TCP exec, Remote Process Execution-- 

Manipulate process priority with nice

Modern operating systems are multi-user and multitasking, which means that multiple users and multiple tasks can be using the computer at any given time. Typically you’ll have one person using a desktop system running any number of applications or many users using many applications on a server.

The amount of time devoted to tasks largely depends on how intensive the task is. Some tasks require higher priority than others; for instance, if you were compiling a large software package you didn’t need immediately, that priority should probably be lower than your Web browser or e-mail client.

Each process has a niceness value associated with it, which is what the kernel uses to determine which processes require more processor time than others. The higher the nice value, the lower the priority of the process. In other words, the “nicer” the program, the less CPU it will try to take from other processes; programs that are less nice tend to demand more CPU time than other programs that are nicer.

The priority is noted by a range of -20 (the highest) to 20 (the lowest). Using ps, you can see the current nice value of all programs:

$ ps axl
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4     0     1     0  16   0   2648   112 -      S    ?          0:01 init [3]
1     0     2     1  34  19      0     0 ksofti SN   ?          0:02 [ksoftirqd/0]
5     0     3     1  10  -5      0     0 worker S<   ?          0:00 [events/0]
...

You can see that init has a nice value of 0, while other kernel tasks associated with PID 2 and 3 have a nice value of 19 and -5 respectively.

Typically, a program inherits its nice value from its parent; this prevents low priority processes from spawning high priority children. Having said that, you can use the nice command (as root or via sudo) with the command you wish to execute in order to alter its nice value. Here is a short illustration:

# ps axl | grep axl | grep -v grep
4     0 30819 30623  15   0   4660   772 -      R+   pts/0      0:00 ps axl
# nice -10 ps axl | grep axl | grep -v grep
4     0 30822 30623  30  10   4660   772 -      RN+  pts/0      0:00 ps axl

You can see there that the nice value, represented by column six, has been altered. You can also use the renice command to alter running processes. In the following example, vim was started to edit the file foo and began with a default nice value of 0. Using renice, we can change its priority:

# ps axl | grep vim | grep -v grep
0     0 30832 30623  16   0  15840  3140 -      S+   pts/0      0:00 vim foo
# renice -5 30832
30832: old priority 0, new priority -5
# ps axl | grep vim | grep -v grep
0     0 30832 30623  15  -5  15840  3140 -      S<+  pts/0      0:00 vim foo

Here, we have adjusted the priority of vim, giving it a slightly higher priority. Renice operates on the process ID, so using grep, we determined that vim is process ID 30832 and saw that the nice value was 0. After executing renice, the nice value is now -5.

Standard caveats apply: Only root can alter the nice priority of programs. So if you find that your compilation is taking too much CPU from other activities, consider renicing the parent process via root. Subsequent children should have a better nice value, or you can even start the compilation (or any other activity) with nice, specifying an appropriate nice value. You can also use renice to renice all programs belonging to a process group or user name/ID.

Monday, September 15, 2008

XEN3.2 Installation-Ubuntu 8.04

XEN3.2 Installation-Ubuntu 8.04
Disable Apparmor:
/etc/init.d/apparmor stop
update-rc.d -f apparmor remove

AppArmor is a kernel enhancement to confine programs to a limited set of resources. AppArmor’s unique security model is to bind access control attributes to programs rather than to users.

Install XEN:
apt-get install ubuntu-xen-server
After installation of xen you should edit xorg.conf file.
vim /etc/X11/ xorg.conf

Your original xorg.conf files looks
Section "Device"
Identifier "Configured Video Device"

Now add the following line in xorg.conf

Section "Device"
Identifier "Configured Video Device"
Driver "i810"
Note: Here i810 refers to graphics chipset.So check out your system graphics chipset.

Download 2.6.24-16-xen kernel To avoid network problem:
wget http://www.il.is.s.u-tokyo.ac.jp/~hiranotaka/linux-image-2.6.24-16-xen_2.6.24-16.30zng1_i386.deb
Replace the Ubuntu 2.6.24-16-xen kernel:
dpkg -i linux-image-2.6.24-16-xen_2.6.24-16.30zng1_i386.deb
Reboot your system

Move tls file
mv /lib/tls /lib/tls.disabled
Modify Modules:
Add the line in vim /etc/modules
loop max_loop=64
Directory for XEN
mkdir /home/xen
Modify xen-tools.conf
vim /etc/xen-tools/xen-tools.conf
install-method = debootstrap # method will change with respect to type of OS
dir = /home/xen
dhcp = 1
dist = hardy # Type of guest os
passwd = 1
mirror = http://192.168.1.36:9999/ubuntu/
Reboot your system

Now Run
jayapal@kcs:~$ uname -r
2.6.24-16-xen
jayapal@kcs:~$
Create Guest OS:
jayapal@kcs:~$ xen-create-image --hostname=Guestos --ide --passwd
Note: You can give size, swap, Network settings, etc... to override settings in xen-tools.conf.For more details jayapal@kcs:~$ man xen-create-image
After creating image file,Terminal will show message as :
Logfile produced at:
/var/log/xen-tools/Guestos.log
Guestos.cfg
vim /etc/xen/Guestos.cfg
kernel = '/boot/vmlinuz-2.6.24-16-xen'
ramdisk = '/boot/initrd.img-2.6.24-16-xen'
memory = '128'
vcpus = '2' #For dual-core CPU
root = '/dev/hda2 ro'
disk = [
'tap:aio:/home/xen/domains/Guestos/swap.img,hda1,w',
'tap:aio:/home/xen/domains/Guestos/disk.img,hda2,w',
]
vif = [ 'ip=192.168.2.10 mac=00:16:3E:C9:C2:6C' ]

To start the virtual machine:
xm create /etc/xen/Guestos.cfg
Using config file "/etc/xen/Guestos.cfg".
Started domain Guestos
xm console Guestos
Guestos started and it will prompt for user login.
To Stop the virtual machine:
xm shutdown Guestos
for more command see xm man
For Centos:
Download initrd.img and vmlinuz, then move in to /boot/ directory
wget http://mirror.centos.org/centos/5/os/i386/images/xen/initrd.img
wget http://mirror.centos.org/centos/5/os/i386/images/xen/vmlinuz
Mirror
http://mirror.centos.org/centos/5/os/i386/images/xen/
Source:
http://www.howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositories