bear has been The Editor Gal's teddy bear for some time now. He has
only gotten into Linux recently, as he has been looking up to
xteddy and how
much he enjoyed the desktop, bear enjoys Fvwm button bars best
(though large dock icons are nice too) and has a snug $HOME in
/usr/share/xteddy.
Heather is Linux Gazette's Technical Editor and The Answer Gang's Editor
Gal.
Heather got started in computing before she quite got started learning
English. By 8 she was a happy programmer, by 15 the system administrator
for the home... Dad had finally broken down and gotten one of those personal
computers, only to find it needed regular care and feeding like any other
pet. Except it wasn't a Pet: it was one of those brands we find most
everywhere today...
Heather is a hardware agnostic, but has spent more hours as a tech in
Windows related tech support than most people have spent with their computers.
(Got the pin, got the Jacket, got about a zillion T-shirts.) When she
discovered Linux in 1993, it wasn't long before the home systems ran Linux
regardless of what was in use at work.
By 1995 she was training others in using Linux - and in charge of all the
"strange systems" at a (then) 90 million dollar company. Moving onwards, it's
safe to say, Linux has been an excellent companion and breadwinner... She
took over the HTML editing for "The Answer Guy" in issue 28, and has been
slowly improving the preprocessing scripts she uses ever since.
Linux Gazette 111: The Answer Gang (TWDT)The Answer Gang 111:
...making Linux just a little more fun!
The Answer Gang By Jim Dennis, Karl-Heinz Herrmann, Breen, Chris, and...
(meet the Gang) ...
the Editors of Linux Gazette...
and
You!
We have guidelines for asking and answering questions. Linux questions only, please.
We make no guarantees about answers, but you can be anonymous on request. See also: The Answer Gang's
Knowledge Base
and the LGSearch Engine
Greetings, folks, and welcome once more to the world of The Answer Gang.
By the way, if you've any particularly marvelous answers (might we say, rants
of the enlightening and useful variety) - feel free to mail them to us - you
too could be part of The Answer Gang, yourself.
I often use this space to talk about whatever it is I'm up to. I have not
had a great month, really - you'd rather not know. Suffice to say I'm looking
forward to Chinese New Year on February 9th, so that I can declare the awful
mess as belonging to the previous year, and move on.
My Star Trek crew is expecting to run the Internet Lounge at my nearby music
convention,
Consonance, first weekend of March.
I do believe that our "Songs In The Key of Tux" series will be a big hit there
- thanks Jimmy!
Here's to hoping that you and yours are faring better than me and mine have.
If not, fear not - things do improve. Even if it's up to us at the last to
make it so - this is a world of choices, and malleable tools like Linux allow
us the freedom to make the most of them. 'Til next month -- Heather
Update vs Install, how best to manage /home?
From Edgar Howell
Answered By: Neil Youngman, Thomas Adam, Mike Orr, Benjamin Okopnik.
Before I go any further, here is the environment on the machine
in question, SuSE 9.2 on both drives, no other OS:
/dev/hda (non-Internet drive, system doesn't even know about a modem, /etc/fstab mounts /dev/hdb2)
1 swap
2 /
3 /home
/dev/hdb (the drive booted for Internet access, /etc/fstab has no information about /dev/hda)
1 swap
2 /
a) update vs install
In part because I tend to omit a couple of releases instead of just
blindly installing successive releases but also because I used to
install new software into a new partition and play with it for a
while before removing the previous version, in the past I have
always done a clean installation. With all that entailed, creating
the users again, /etc/hosts* (SOHO network) and the like.
[Thomas]
When you do a new install a distro (albeit a new one, or an upgrade to a
new point release of one you currently have) you can instruct it not to
touch certain partition (like /home). This means you don't have to
worry about loss of data. You mentioned UIDs. Backup
/etc/{passwd,shadow} beforehand.
Recently I experimented with update. It worked well, avoided lots
of questions and seemed really neat. But I had again skipped a
couple of releases and ultimately discovered some problems.
[Thomas]
I can't see where the contention is. An upgrade saves a lot of time,
since all you're doing is upgrading the software, nothing more.
[Mike]
I haven't used SuSE much, but that's the general problem you get when
updating through multiple OS releases. Debian has special scripts
you're supposed to run to switch releases; they try to automate the
tricky stuff but sometimes they can't foresee what your configuration
has turned into. And they have to be run in order without skipping. If
you don't update packages frequently and don't follow the user
forums/newsletters where potential breakages are discussed, I would
stick with the clean install and copy your configuration. That way you
know everything has the latest recommended configuration, whatever that
is. It also provides a chance to clear out the cruft that has
accumulated from packages you installed but never used; cruft that may
leave droppings in the filesystem when uninstalled since it wasn't built
by a clueful developer.
Alternatively, back up your entire system onto your second drive and
make sure you can boot into it, then update your primary system. That
way if something breaks you can just blow it away and go back to where
you were.
/home isn't a big deal. If you have it on a separate partition like you
do, just let the fresh install create its own home directory in the
root partition. You'll have to do everything as root anyway while
you're installing, so just pretend home doesn't exist. Then when
everything's settled, delete the bogus home and mount your real /home
partition. Same for /usr/local if you store stuff in there. I keep a
/mnt/data partition with my home and local stuff, and use symlinks to
get to them. That also lets me have multiple OS versions installed, all
sharing the same local data. And I can unmount the local data when I'm
afraid an upgrade might hurt it.
Under the old version the first user ID was 500 and under 9.2 it is
1000. That of course caused problems in the above environment:
/dev/hdb under a completely new installation got new user IDs,
/dev/hda under the update inherited the old ones. It was fun to
re-boot into /dev/hdbafter I wrote to it having booted from
/dev/hda...
[Mike]
The easiest way is to recreate the users with the same UIDs and GIDs
they previously had. You may have to run "useradd" manually to do it.
(Or "adduser" on some systems.) If your UID overlaps with an existing
UID on the new system, you'll have to compromise somehow. If you give
each user their own group rather than putting them all into "users",
you'll have to create the group first. On my Gentoo:
This is the best way if you want to boot back and forth between OS
versions, you have files with unexpected owners inside the home
directories, or you have programs that refer to users by UID rather than
name.
Alternatively, you can just go with the new UIDs and switch the existing
home directories with "chown -R USER.GROUP /home/USER". (Note that
chown is going through a syntax change from "USER.GROUP" to
"USER:GROUP"; you'll have to see which syntax your version supports.)
[Ben]
Being the tool-using critter that I am, things like this (the word
"manually", specifically) bring a shudder to the spine and a script to
mind.
while read user x uid gid foo
do
adduser --uid $uid --gid $gid $user
done < /mnt/hda<N>/etc/passwd
I suppose you could always put in a line that says
((uid+=1000))
right below the 'do'.
[Thomas]
This presupposes that the users were added with "adduser" to begin with
(note that UIDs from 1000+ are indicative of this). But on some
systems, UIDs > 499 are used as a valid starting place for normal user
IDs.
[Ben]
I was walking past a haberdashery and happened to see a nice hat in the
window, so I extracted a '1000' from it. };> The number would have to
come from examining the password file on the current system and adapting
the existing number range to what is available - obviously, there's no
single 'right' answer. Sorry I didn't make that more explicit.
Oh, and the '<N>' in the '/mnt/hda<N>' isn't the explicit version
either. :)
What does the Answer Gang recommend, update or clean installation?
[Thomas]
An update.
[Neil]
As you've noted a clean install requires you to set the whole system up again,
whereas a good update should be seamless. Again as you note, there may be
circumstances where the update leaves the seams showing. A clean install will
normally leave you with a nice consistent system, with any cruft that was in
your configuration cleaned out and everything shiny and sparkly.
Obviously, if you're changing distributions, rather than just changing
versions of the same distribution then a clean install is the way to go.
Personally I incline towards doing a clean install every so often. If you're
only taking every 3rd release or so, then a clean install may be worth the
effort, but if you're putting every release on, then I would alternate
upgrades and clean installs, or even keep the clean installs to every 3rd
release.
In practice, I tend to have a number of old releases lying around in separate
partitions, so I wipe an old partition and install there, then when I'm happy
I've got it set up the way I like it, I copy /home across and change my
default boot. This means I also have a number of old copies of my home
directory left lying around.
b) managing /home etc.
I have read recommendations about distributing the various
directories but assume that they only apply to environments with
different physical drives (load-balancing). In this specific
installation there is only one hard drive (at a time) involved.
[Thomas]
This "load balancing" concept is a marketing myth, a band-wagon of
terminology that's thrown around the place which people latch on to.
[Neil]
Generally, I think it's about reliability more than load balancing. The point
being that if some eejit fills up one partition, e.g. /home, with junk,
there's still space in the other partitions, e.g. /var and /tmp, for the
system to go on about it's business until the problem is rectified. If it's
all in one big partition the the whole system is likely to fail.
In practice that's more applicable to big IT departments than simple home
systems. At home I install everything in one big partition. It keeps things
simple and I've had no problems with reliability, but I wouldn't recommend it
for my work systems.
How can one best deal with update or install in order to avoid
having to back up /home, waste the drive, install the software
and then restore /home?
[Thomas]
(see above about partitions and installation)
[Mike]
For ease of use, put everything in one partition. To guard against disk
I/O errors or stupid admins who don't look before they "rm -rf", put
/boot, /home/ and /usr on separate partitions, make /boot and /usr
readonly, and don't mount /boot by default. The reason is that if a
disk sector goes bad, it may take the entire filesystem with it, but it
won't disturb other partitions. Likewise, users can't mess with stuff
that's mounted readonly. The down side is managing the partitions and
predicting how big to make them. If one gets full but another is mostly
empty, you'll have to repartition or use symlinks.
Here /home is on a different partition than the partition with the
software. Will users (IDs) be created with respect to a partition
/home?
[Thomas]
An update doesn't concern itself with changing critical information such
as that -- the only way {U,G}IDs would be affected is if shadow or login
were updated -- and even then, the config files are not touched as a
result.
[Mike]
User IDs (UIDs) are created in /etc/passwd according to the existing
entries in /etc/passwd. The smallest unused number is assigned, subject
to a minimum and maximum. Where /home is mounted doesn't matter. /home
doesn't have to be mounted at all if you don't use the -m option
(useradd), which creates the home directory.
OK, I've been wanting to do a completely new installation on
/dev/hda, let's just try it...
A little later: it turns out that things can be quite simple
after all.
YaST was so unhappy at being told not to format / that I backed
up the one non-system-directory on it and let YaST reformat it.
Other than that and my dictating partitioning, a vanilla install,
basically just accepted whatever was suggested.
[Thomas]
Of course it would be unhappy about that. /etc, /lib, /bin./sbin -- are
all directories the installer will need access to. It's highly
unlikely they're as their own partition (you wouldn't need nor want them
to be so they're under "/".
[Mike]
I've never seen a Linux installer that sniffled and sulked if it
couldn't format something, but I guess there's always a first.
Usually they have options to let you mount a preformatted partition.
What was really neat is that YaST had absolutely no trouble using
a previously available /home! True, I had to re-create the users,
but that is normal and in the process YaST notes the presence of
corresponding home directories and asks whether they are to be used.
That pretty much solves that problem, for me at least. The Germans
have a saying, roughly: trying something out is better than studying
it.
But I'd still appreciate comments. Is there a gotcha somewhere?
Hmmm, this SOHO doesn't have many users. And what about all the
settings in /etc?/boot? Like would it have been possible to copy
/etc/passwd and /etc/shadow from a backup? Sounds like that
particular ice might be getting a bit thin...
[Neil]
Settings in /etc and /boot are best created from scratch when doing a clean
install IMO, especially when any of them are maintained by automated tools
like YaST. There's always the possibly of significant changes between
versions and just copying back your old settings can be a bit risky, although
9 times out of 10 you won't have a problem.
All IMO. Unlike many in the gang, I don't sysadmin, so others may have more
authoritative answers. NOTE: We do not guarantee a consensus ;-)
[Mike]
Copying the user entries from /etc/passwd is fine, as long as the
numbers don't overlap. Just make sure nothing else is editing
/etc/passwd simultaneously, or use "vipw" to lock it while you're
editing. /etc/shadow is probably fine too, just be aware that the
other distribution may have a different file location and different
syntax. If it
doesn't recognize the new password, you may have to restart... something.
(Actually, the UIDs can overlap if you really want two usernames treated
the same. Some people use this to have a second root login with a
static shell (/bin/sash). This is useful if you hose your dynamic
libraries; with a static shell you can repair the damage. Just copy
the root line, leave the UID 0, change the username to something else,
and set the password.)
after installing new kernel running lilo crushes system
From Ridder, Peter, AGU
Answered By: Neil Youngman, John Karns
Hallo,
I have a Knoppix 3.3 HD installation on a Dell Latitude 610.
The HD holds the following partitions:
hda1 Primér FAT16
hda5 Logisch NTFS
hda6 Logisch Linux ext2
hda7 Logisch Linux swap
hda8 Logisch Linux ext3
hda9 Logisch NTFS
I use the NT loader placed in /dev/hda1
/dev/hda1 contains a file bootsec.lin which is a copy.
This file is the result of the Knoppix HD installation
After adding a new compiled kernel in /boot (/boot/bzImage-2.4.24 for example)
adding lines in /etc/lilo.conf like:
image=/boot/bzImage-2.4.24
label=Li_new
and running /sbin/lilo
results after a restart in: L 99 99 99 etc.
And the file /boot/boot-menu.b doesn't exist! Also there is no /boot/boot.b
Why does it work until I try to install another kernel?
[Neil]
According to the LILO man page
...............
"Errors 99 and 9A usually mean the map file(1,n) (-m or map=) is not readable,
likely because LILO was not re-run after some system change, or there is a
geometry mis-match between what LILO used (lilo(5,8) -v3 to display) and what
is actually being used by the BIOS (one of the lilo(5,8) diagnostic disks,
available in the source distribution, may be needed to diagnose this
problem)."
...............
[John]
I haven't taken the time to trace through the init scripts to track down
the specifics, but my hd install of knoppix 3.4 doesn't maintain the boot
partition mounted under the /boot dir, presumably unmounting it at some
point during system boot. It is accessible as an automounted partition
thereafter. The /boot directory you see is the mountpoint referenced by
lilo.conf.
What's on your hda6 partition? My guess is that's where your boot menu
files are. You can verify by issuing the command (as root):
mount /dev/hda6 /boot
Or by clicking on the hda6 desktop icon and looking at the files that come
up in konqueror as a result.
If you then see the "missing" files, then after having mounted the
partition under /boot as above, you can copy your compiled kernel there
and edit lilo.conf. It's usually a good idea to make a new entry in
lilo.conf for the new kernel, leaving the old on untouched for a fall-back
in case for some reason the system won't boot from the new kernel. Just
copy the existing lines referring the original kernel, and paste them in a
new stanza above or below the original. Then edit them making name
changes as necessary to refer to the new kernel. After saving, you can
then run 'lilo' (also as root).
Assuming this all works for you, then 'umount /boot' and delete the kernel
you had put there before. It isn't seen when there is a partition mounted
at /boot, and just takes up disk space.
finding then catting
From Greg Messer
Answered By: Neil Youngman, Jason Creighton, Thomas Adam, Benjamin Okopnik
Answer Gang,
I run a smallish mail server and am using Squirelmail for web-based email. I
use spamassassin/procmail to move emails that are borderline spam to
/home/username/Trash.
My users have been instructed to occasionaly log into squirelmail and look
through their Trash folder and empty it occassionaly even if they're popping
in with the dreaded Outlook Express. They of course don't, and it's becoming a
problem. I need to run a command that will find all the files called Trash in
the user's home directories and empty them. I can't simply delete this file as
it causes Squiremail to generate an error and I would get many many phone
calls even though squirelmail will fix this problem on their next login.
This is my third attempt at automating this procedure and my third failure.
That will empty anything called Trash in subdirectories as well as in the
login directories. To only hit files in the login directories you should use
a for loop, e.g.
for file in /home/*/Trash
do
echo -n > $file
done
Before trying this put another echo in front of echo echo -n > $file, so you
can see the commands it will run and sanity check them before running it for
real.
What errors are you getting? Do you have permissions to write to these files?
or this:
find /home -name Trash | xargs cat > /dev/null
[Neil]
That wouldn't work. You're just listing the files and directing the output
to /dev/null, which won't achieve what you want.
so obviously it didn't work but I didn't get any errors.
Your "for" script worked great and is short and sweet. I'm very greatful,
however, for my own information, I'd still like to understand what's wrong
with my find syntax/structure. If you guys post this solution on the website
you should put in the keywords "empty files". I've googled for all kinds of
crazy things and never found a solution.
This runs "find /hometest -name -Trash -exec cat /dev/null" and
redirects the output to a file named "{}".
Quoting the '>' doesn't help since find doesn't use the shell to expand
commands given with -exec. (That is, if you quoted the ">", cat would be
run with three arguments. The first would be a file named "/dev/null".
The second would a file named ">", which cat would probably complain
doesn't exist. It is possible you might actually have a file named ">",
but it's such a weird and confusing name that you probably don't. And
the third would be the name of the file you're trying to truncate.)
If, for some reason, you needed to use "find" (perhaps to only truncate
files with a certain mtime, or whatever), you could use a script like
this:
#! /bin/sh
for file in "$@"; do
[ -f "$file" ] && echo -n > "$file"
done
name it truncate.sh or something, make it executable, and save it
somewhere. Then you could do:
find /path/to/files -exec truncate.sh {} \;
...or use xargs, or whatever.
[Thomas]
There's nothing wrong in your implimentation, but it is worthy of note
that the test is simply going to add another "thing" for the script to
do. If the number of files are vast, this is just going to slow it down.
You could remove [1.] entirely and let find match the files beforehand:
find . -type f -exec ./truncate {} \;
[Jason]
Oh! I didn't think of that. That is better than silently dropping
non-existent and non-regular files.
[Thomas]
I could hash this argument out in any number of combinations involving
xargs, -exec, etc, with arguments as to whether you should use a shell
script, etc., etc.
[Jason]
Yes, and you probably would be wanting to use xargs if the number of
files is vast.
[Thomas]
Maybe. But that will still fail where a filename has spaces in it.
Example:
[n6tadam@station fi]$ ls -lFb
total 8
-rw-r--r-- 1 n6tadam n6tadam 0 Jan 11 11:18 foo
drwxr-xr-x 2 n6tadam n6tadam 4096 Jan 11 11:11 ignore/
-rw-r--r-- 1 n6tadam n6tadam 120 Jan 11 11:08 this\ has\ spaces
Ignoring the "ignore/" directory, I've got a file with spaces in the
filename [1], as well as a 'normal' file. If I wanted to truncate the
files in the CWD above, I might use:
find . -type f -maxdepth 1 -exec sh -c 'cat /dev/null > {}' \;
... which is fine, for the file with no spaces. Of course, the
truncate.sh script you wrote is fine for handling that (you actually
quoted the variable -- thousands do not). But just what is wrong with
that command above? Well, for each file that find finds, it has to spawn
a separate non-interactive shell to process it. That's slow.
xargs might improve things (I'll leave this as an exercise to the reader
to use 'time'):
find . -type f -maxdepth 1 -print0 | xargs -0i sh -c "cat /dev/null > \"{}\""
Note the quoting. It's paramount that this is done, because even though
the '-print0' option to find splits file names ending '\0' (and xargs
re-interprets them again at the other end), we're still having to
quote the filename (this will still fail if the filename contains a
'"' character, though). Why? Because by the time it gets passed through to
the shell to handle it, we're back to the our old tricks of: '"\"use\" more
quo\"t\"es'.
So is using find(1) any better than using a plain shell script that
globs a given directory for files to truncate? No. Because find blindly
exec()'s whatever we pass to it (and we're having to use shell
redirection) we must invoke the shell for it to work. The only advantage
to using find is that it would handle some strange files, nothing more
(in this particular application of it, anyway).
I suppose you could make that find command more efficient:
find . -type f -maxdepth 1 -not -empty -print0 | xargs -0i sh -c "cat /dev/null > \"{}\""
... which just ensures that the files we pass to it have a filesize
greater than zero. The "best" solution that I personally can see,
is using the following:
This obliterates the need to fork a subshell to perform any redirection
-- and as with any "find .. | xargs" combination, it'll be quite fast,
too. But the main reason for using it is that by avoiding any
shell-redirection-mangle-filename techniques, we don't have to worry
about quoting. The delimiter of '\0' via find and xargs should be enough
to protect it.
Also note that cat'ting /dev/null is nonsensical in this instance.
[1] Remember that there is nothing "illegal" about using such
characters. Any character is a valid one for filenames at the filesystem
level. What defines them as being a pain is the shell. Nothing more.
[Ben]
Not quite; '/' can't be used as a filename. Although "\n" can, which
(along with any high-bit characters) can create lots of pain for anyone
trying to work with them...
[Jason]
But ASCII NUL is an illegal character, right? So this will always work?
find -print0 | xargs -0 command
Jason Creighton
[Ben]
Right; you can't use a NUL or a '/'. Other than those two, anything is
fair game... well, not really.
Mostly, it's a REALLY good way to
screw yourself up; in general, it's not a good idea to use anything outside
of [a-zA-Z0-9_] as part of a filename.
But then, we're talking about us.
"What do you mean, I can't jump off
this cliff? It doesn't look all that high!"
ben@Fenrir:/tmp/foo$ for n in `seq 0 255`; do a=$(printf "\x`printf "%x" $n`"); >"a${a}"; done
bash: a/: Is a directory
ben@Fenrir:/tmp/foo$ ls -b
a a{ a\214 a\243 a\272 a\321 a\350 a\377 a\031 ad aO
a` a} a\215 a\244 a\273 a\322 a\351 a\002 a\032 aD ap
a^ a@ a\216 a\245 a\274 a\323 a\352 a\003 a\033 ae aP
a~ a$ a\217 a\246 a\275 a\324 a\353 a\004 a\034 aE aq
a< a* a\220 a\247 a\276 a\325 a\354 a\005 a\035 af aQ
a= a\\ a\221 a\250 a\277 a\326 a\355 a\006 a\036 aF ar
a> a& a\222 a\251 a\300 a\327 a\356 a\a a\037 ag aR
a| a# a\223 a\252 a\301 a\330 a\357 a\b a0 aG as
a\ a% a\224 a\253 a\302 a\331 a\360 a\t a1 ah aS
a_ a+ a\225 a\254 a\303 a\332 a\361 a\v a2 aH at
a- a\001 a\226 a\255 a\304 a\333 a\362 a\f a3 ai aT
a, a\200 a\227 a\256 a\305 a\334 a\363 a\r a4 aI au
a; a\201 a\230 a\257 a\306 a\335 a\364 a\016 a5 aj aU
a: a\202 a\231 a\260 a\307 a\336 a\365 a\017 a6 aJ av
a! a\203 a\232 a\261 a\310 a\337 a\366 a\020 a7 ak aV
a? a\204 a\233 a\262 a\311 a\340 a\367 a\021 a8 aK aw
a. a\205 a\234 a\263 a\312 a\341 a\370 a\022 a9 al aW
a' a\206 a\235 a\264 a\313 a\342 a\371 a\023 aa aL ax
a" a\207 a\236 a\265 a\314 a\343 a\372 a\024 aA am aX
a( a\210 a\237 a\266 a\315 a\344 a\373 a\025 ab aM ay
a) a\211 a\240 a\267 a\316 a\345 a\374 a\026 aB an aY
a[ a\212 a\241 a\270 a\317 a\346 a\375 a\027 ac aN az
a] a\213 a\242 a\271 a\320 a\347 a\376 a\030 aC ao aZ
tar and find
From anonymous
Answered By: Thomas Adam
I'd like to tar up the contents of /var/www but I'd like to exclude a couple
of directories.
I usually use
tar -zcvf www.tar.gz /var/www
but that does everything.
Idea's please
[Thomas]
Tar allows for you to have an exclude wildcard, rather than using a file
from which exclusions are stored:
tar -czvf foo.tar.gz --exclude='*foo*' /var/www
.. would allow you to specify a wildcard from which a list of files
and/or directories could be excluded.
Of course, if you're going to do that, this is where you really want to
use find. Here's an example. I have a directory "tar" which has some
files, and two directories:
[n6tadam@station tar]$ ls -lFh
total 20K
-rw-r--r-- 1 n6tadam n6tadam 4 Jan 17 15:05 a
-rw-r--r-- 1 n6tadam n6tadam 34 Jan 17 15:31 b
-rw-r--r-- 1 n6tadam n6tadam 32 Jan 17 15:31 c
drwxr-xr-x 2 n6tadam n6tadam 4.0K Jan 17 15:05 foo/
drwxr-xr-x 2 n6tadam n6tadam 4.0K Jan 17 15:04 foo2/
Now let us assume that I only want to tar the files a,b,c and exclude
the ./foo{,2} stuff. What you really want is to preprocess your results
with find. You can exclude one directory from a list. Here's an example:
find . -path './foo' -prune -o -print
.. and note the syntax. The "." assumes that we're already in the same
directory that we want the search to start from. In this case the
"-path" option to find matches a pattern, treating "." and "/" as
literal characters to match against. The -prune option excludes it (it
assumes a -depth level, and doesn't descend into the path given. Then
"-o" says to match everything else, and -print the results [1].
... you'll see ./foo has been excluded. But how do you match more than
one exclusion? I might not want ./fooor./foo2 to be in my tar
archive. Ok, here's how:
I've encapsulated the order that find should place when looking for the
files, but it's not necessary in this example. It's just an aggregation
of the command we saw earlier.
Now the fun stuff. How do you get tar to use the results given to
produce a tar file? For ease of use, we'll modify our find command to
show the filenames with the full path name, rather than "./" (which
isn't at all helpful to us):
... but in fact that won't work, since what that does is runs the
command like this:
tar -czvf ./foofile.tgz /tmp/tar
tar -czvf ./foofile.tgz /tmp/tar/a
tar -czvf ./foofile.tgz /tmp/tar/b
tar -czvf ./foofile.tgz /tmp/tar/c
... but, there are two things wrong with this. One, is that it's
specifying "/tmp/tar" as a valid entry to our tar file. That's not what
we want -- we *don't* want that recursive nature to tar -- so already
that's put pay to the whole of the find command (more about that in a
minute).
The second problem is that each time that tar command runs, it's
replacing the tar file with the new file, rather than appending it.
Ouch! So if you were to look at that tar file now, all you would see is
"/tmp/tar/c" since that was the last file created in the tar file.
Tar supports the "-A" option -- to append to a tar file. But that
presupposes that the tar file is already in existence -- and the
assumption here is that it isn't. So we can't use it.
Also, using -exec on find is a terrible idea in this case, since it runs
a copy of the same command (tar in this case) for every file
encountered, and since the tar file is never created...
So, we'll use xargs. That builds up command-line input on a chain so
that when it is run, we'll see something like this:
tar -czvf ./foofile.tar /tmp/tar /tmp/tar/a /tmp/tar/b /tmp/tar/c
Which is exactly what we want. But we first have to ensure that we
further disclude that "/tmp/tar" entry. And there's an option to tar to
do that: "--no-recursion".
The other consideration to take into account are filenames. Even if
you're sure that the filenames are valid, etc., it is still good
practise to assume the worst. Modifying our initial find command, we can
tell it to split filenames based on '\0' (rather than what $IFS defines
it as). The "print0" option to find defines this:
Which by itself is useless. But in this situation, we can tell xargs
to reinterpret that via "xargs -0", so that's not a problem. It's just
a means of protecting the filenames so that they're not mangled.
So if we piece my ramblings together the actual command you'll want to
use is:
Submitters, send your News Bytes items in
PLAIN TEXT
format. Other formats may be rejected without reading. You have been
warned! A one- or two-paragraph summary plus URL gets you a better
announcement than an entire press release. Submit items to
bytes@lists.linuxgazette.net
Legislation and More Legislation
Software Patents
The proposed European Union directive on the patentability
of computer-implemented inventions (software patents
directive) is still in more or less the same limbo it was in
when last discussed here in November. As mentioned in
November, Poland has expressed serious reservations about
the proposed measures and
has blocked the European Council of Ministers
from adopting the relatively pro-patent proposal
currently before it. This would have have occurred, for
reasons of expediency, at a
meeting of the Council of Agriculture and Fisheries.
Tom Chance, writing at LWN,
has given
a clear and useful overview
of how these developments fit within the decision making
structure of the EU. He also presents a neat précis
of the realpolitik which will likely lead to Poland shelving
its objections in order to achieve agreement on matters
closer to its economic self-interest. Should this happen,
meaning the measure is adopted by the Council, it could
again be rejected by a Parliament keen to reinstate its
original amendments. However, under the relevant voting
rules this would require two thirds of all MEPs to vote
accordingly. This is unlikely to occur.
As we approach the time when decisions on the future of this
directive will be made, many individuals and organisations
are attempting to raise their political representatives'
awareness of these matters. Although it is valuable in this
process to achieve some understanding of the issues and
consequences of the proposed policy, it is perhaps even more
useful to impress on elected representatives that they will
be called to account in their own countries for their voting
and policy in Europe.
Useful sites for those opposed to software patents:
The Open Grapics Project aims to develop 2D and 3D graphics
hardware based on open standards and documentation that will
allow full operation with GNU/Linux and other open source
software.
The principles behind this development are outlined
by Timothy Miller
in an interview he gave to KernelTrap.org.
The hardware part of this venture is Miller's employer,
Tech Source. A
mailing list has been established
to discuss the project.
Linux Kernel
As of Christmas Eve 2004, the latest version of the stable
2.6.x series of
Linux Kernels is
2.6.10.
The new year brought an update to the older 2.4 series,
which has now been updated to version
2.4.29.
The
Debian Project
has announced the release of the fourth update for the
current stable release of their GNU/Linux distribution.
Debian Woody GNU/Linux 3.0, (r4), released on January
1st 2005, comprises for the most part a collection of
security fixes accumulated (and addressed) over the past
months since the r3 release in October.
Screenshot tour
of
Games Knoppix, a Knoppix variant that comes loaded with
a selection of some of the best games available for
GNU/Linux, all on a live, bootable, CD.
There is a review of this distribution
at Linux.com.
LFS
The Linux From Scratch
2nd edition book is currently
on pre-order sale. This means you can order this book
for USD13.99, whereas in a couple of weeks when it starts
to ship it will be available at USD19.99.
Linux From Scratch provides a set of instructions allowing
you to build your GNU/Linux system entirely from source.
Even if you do not plan to use such a system, it is an
interesting exercise, and this book could provide useful
information and background to many non LFS uses.
Spidertools.com has
published a brief review of
Ubuntu Linux. This looks both at the live-CD and
hard drive install versions included in the standard Ubuntu
CD package.
Looking towards the next release of this distribution,
it has been announced
that the first milestone live-CD preview of the next Ubuntu
release (Hoary Hedgehog) has been let loose.
Xandros
NewsForge has
reviewed
Xandros Desktop OS 3 Deluxe Edition.
This Debian-based GNU/Linux distribution aims to provide a
comfortable and familiar experience to users more accustomed
to the use of GUI environments, and perhaps new to GNU/Linux.
IBM is to launch
a new low-end Power-based server
running GNU/Linux, making a definite marker with regard to
its relationship with chip-making rivals AMD and Intel.
Acrobat
Adobe has
released
version 7 of its
Acrobat software
for GNU/Linux.
The stand-alone Acrobat Reader has also
been updated to version 7.0, and this update too is
available for GNU/Linux as a no-cost download.
Originally hailing from Ireland, Michael is currently living in Baden,
Switzerland. There he works with ABB Corporate Research as a
Marie-Curie fellow, developing software for the simulation and design
of electrical power-systems equipment.
Before this, Michael worked as a lecturer in the Department of
Mechanical Engineering, University College Dublin; the same
institution that awarded him his PhD. The topic of this PhD research
was the use of Lamb waves in nondestructive testing. GNU/Linux has
been very useful in his past work, and Michael has a strong interest
in applying free software solutions to other problems in engineering.
In a word, No. No machine connected to the internet is
100% secure. This doesn't mean that you are helpless. You can take
measures to avoid hacks, but you cannot avoid them completely. This
is like a house — when the windows and doors are open then
the probability of a thief coming in is high, but if the doors and
windows are closed and locked the probability of being robbed is
less, but still not nil.
1 What is Information Security?
For our purposes, Information Security means the methods we use
to protect sensitive data from unauthorized users.
2 Why do we need Information Security?
The entire world
is rapidly becoming IT enabled. Wherever you look, computer
technology has revolutionized the way things operate. Some examples
are airports, seaports, telecommunication industries, and TV
broadcasting, all of which are thriving as a result of the use of
IT. "IT is everywhere."
A lot of sensitive information passes through the Internet, such
as credit card data, mission critical server passwords, and
important files. There is always a chance of some one viewing
and/or modifying the data while it is in transmission. There are
countless horror stories of what happens when an outsider gets
someone's credit card or financial information. He or she can use
it in any way they like and could even destroy you and your
business by taking or destroying all your assets. As we all know
"An ounce of prevention beats a pound of cure," so to avoid such
critical situations, it is advisable to have a good security policy
and security implementation.
3 Security Framework
The following illustrates the framework needed to implement a
functioning security implementation:
[ Risk Analysis ] [ Business Requirements ]
|
[ Security Policy ]
|
[ Security Service, Mechanisms, and Objects ]
|
[ Security Management, Monitoring, Detection and Response ]
This framework shows the basic steps in the life cycle of
securing a system. "Risk Analysis" deals with the risk associated
with the data in the server to be secured. "Business Requirements"
is the study which deals with the actual requirements for
conducting business. These two components cover the business
aspects of the security implementation.
The "Security Policy" covers 8 specific areas of the security
implementation, and is discussed in more detail in section 4
below. "Security Service, Mechanisms and Objects" is actually the
implementation part of security. "Security Management, Monitoring,
Detection and Response" is the operational face of security, where
we cover the specifics of how we find a security breach, and how we
react if a breach is found.
4 Security Policy
The Security Policy is a document which addresses the following
areas:
Authentication: This section deals with what methods are used
to determine if a user is real or not, which users can or cannot
access the system, the minimum length of password allowed, how long
can a user be idle before he is logged out, etc.
Authorization: This area deals with classifying user levels and
what each level is allowed to do on the system, which users can
become root, etc.
Data Protection: Data protection deals with the details like
what data should be protected and who can access which levels of
data on the system.
Internet Access: This area deals with the details of the users
having access to the internet and what they can do there.
Internet Services: This section deals with what services on the
server are accessible from the internet and which are not.
Security Audit: This area addresses how audit and review of
security related areas and processes will be done.
Incident Handling: This area addresses the steps and measures
to be taken if there is a breach of security. This also covers the
steps to find out the actual culprit and the methods to prevent
future incidents.
Responsibilities: This part covers who will be contacted at any
given stage of an incident and the responsibilities of the
administrator(s) during and after the incident. This is a very
important area, since the operation of the incident handling
mechanism is dependent on it.
5 Types of Information Security
There are 2 types of security. (1) Physical security / Host
Security and (2) Network security. Each of these sections has 3
parts:
Protection: Slow down or stop intrusions or damage
Detection: Alert someone if a breach (or attempted breach) of
security occurs, and quantify and qualify what sort of damage
occurred or would have occurred.
Recovery: Re-secure the system or data after the breach or
damage and where possible, undo whatever damage occurred
5.1 Host Security / Physical Security
Host Security / Physical Security means securing the server from
unauthorized access. For that we can password protect the box with
such steps as setting up a bios password, placing the computer box
in a locked room where only authorized users have access, applying
OS security patches, and checking logs on regular basis for any
intrusion and attacks. In Host security we check and correct the
permissions on all OS related files.
5.2 Network security
Network security is one of the most important aspects of overall
security. As I mentioned earlier, no machine connected to the
internet is completely secure, so security administrators and server
owners need to be alert, and make sure that they are informed of all
new bugs and exploits that are discovered. Failure to keep up with
these may leave you at the mercy of some script kiddy.
5.3 Which operating system is the most secure?
Every OS has its own pros and cons. There are ways to make
Windows more secure, but the implementation is quite costly. Linux
is stable and reasonably secure, but many companies perceive it as
having little vendor support. My vote for the best OS for security
purposes goes to FreeBSD, another free Unix-like OS, but not many
people are aware of its existence.
6 Is a firewall the final solution to the Network Security problem?
No, a firewall is just a part of the security implementation.
Again, we will use the example of a house. In a house all the
windows and doors can be closed but if the lock on the front door
of the house is so bad that someone can put just any key-like thing
in and open it, then what is the use of the house being all closed
up? Similarly, if we have a strong firewall policy, it will
restrict unauthorized access, but if the software running on the
box is outdated or full of bugs then crackers can use it to intrude
into the server and gain root access. This shows that a firewall is
not the final solution. A planned security implementation is the
only real quality solution to this issue.
7 Security is a continuous process
Continuing security is a on-going process. Security
administrators can only conduct their work on the basis of the
alerts and bugfixes released up to the date of securing, so in
order to accommodate all of the fixes for the latest bugs, security
work has to be done on a regular basis.
8 Does Security implementation create overhead and/or reduce
performance?
Yes, Security implementation creates a small amount of overhead,
but it need not reduce overall performance drastically. In order to
take care of such things, a well done security implementation has
an optimization section where the security administration gives
priority to both performance and security. While securing any
software, we should secure it in such a way that it provides
maximum performance.
9 Security Audits - What Should be Checked
A security audit is a part of security implementation where we
try to find out the vulnerabilities of the system and suggest actions
to improve the security. In a normal audit, the points below should
be checked, and a report with the results of that audit should be
created.
Check intrusion detection. Use chkrootkit or rkhunter for this
purpose.
Check for known bugs in the software installed on the server -
the kernel, openssl, openssh, etc.
Scan all network ports and find out which ports are open.
Report the ports that should not be open and what program is
listening on them.
Check whether /tmp is secured.
Check for hidden processes.
Check for bad disk blocks in all partitions. (This is just to
make sure that the system is reasonably healthy.)
Check for unsafe file permissions.
Check whether the kernel has a ptrace vulnerability.
Check the memory (Another system health check.)
Check if the server is an open e-mail relay.
Check if the partitions have enough free space.
Check the size of the log files. It's better that the log size
remains in megabytes.
10 How to know if you are being hacked?
To find out if your box is compromised or not, follow these
steps. These are the steps which I used to do and will be handy in
most of the situations.
10.1 Check your box to see if your performance has degraded or
if your machine is being over used.
For that, use the commands
vmstat
Displays information about memory, cpu and disk.
Ex: bash#vmstat 1 4 (where 1 is
delay and 4 is count)
mpstat
Displays statistics about cpu utilization. This will help us to
see if your cpu is over worked or not.
Ex: bash#mpstat 1 4 (where 1 is
delay and 4 is count)
iostat
This command displays statistics about the disk system.
Useful options:
-d - Gives the device utilization report.
-k - Display statistics in kilobytes per
second.
Ex: bash#iostat -dk 1 4 (where 1 is
delay and 4 is count)
sar
Displays overall system performance.
10.2 Check to see if your server has any hidden processes
running.
ps
Displays the status of all known processes.
lsof
List all open files. In Linux everything is considered a file,
so you will be able to see almost all of the activity on your
system with this command.
If the uptime is less than it should be, this can mean that your
machine's resources are being used by someone. Linux doesn't crash
or reboot under normal conditions because it is such a stable OS.
If your machine has been rebooted try to find out the actual reason
behind it.
10.5 Determine what your unknown processes are and what they are
doing.
10.5.0.1 Use commands like the following to take apart unknown
programs
readelf
This command will display what the executable's program is
performing.
ldd
This command will show the details of libraries used by a
executable.
string
This command will display the strings in the binary.
strace
This command will display the system calls a program makes as
it runs.
11 Hardening Methodology
Read all security related sites and keep up to date. This is
one of the main things a security administrator or server owner
should do. Server owners should be made aware of security and its
importance. Security training is an important part of an overall
security package.
Create a good security policy. Conduct security audits on the
basis of this policy.
Keep your OS updated by applying all patches.
Install a custom kernel with all unwanted services removed and
patched with either grsecurity or openwall.
Disable all unwanted services and harden the services you leave
running; Change file and directory permissions so that security is
tightened.
Install a firewall and create good rule sets.
Test and audit the server on regular basis
Install an intrusion detection system, log monitor, all of the
Apache security modules, bfd, faf and tmp monitor. Make your
partitions secure.
Run a good backup system to recover data in case of an
intrusion, crash, or other destructive incident.
Install a log analyzer and check your logs for any suspicious
entries.
Install scripts to send out mail or enable notifications when a
security breach occurs.
After a security breach try to find out how, when and through
what the breach occurred. When you find a fix for it, document the
details for future reference.
12 Summary
Now lets conclude by covering the main steps by which a
hosting server can be secured.
12.1 Determine the business requirements and risk factors
which are applicable to this system
12.2 Devise a security policy with the above data in mind.
Get management's approval and signoff on this security
policy.
12.3 On approval of the policy, do a security audit on any
existing systems to determine the current vulnerabilities and
submit a report regarding this to the management.
The report should also cover the methods needed to improve existing
security.
A quick checklist:
Software Vulnerabilities.
Kernel Upgrades and vulnerabilities.
Check for any Trojans.
Run chkrootkit.
Check ports.
Check for any hidden processes.
Use audittools to check system.
Check logs.
Check binaries and RPMS.
Check for open email relays.
Check for malicious cron entries.
Check /dev /tmp /var directories.
Check whether backups are maintained.
Check for unwanted users, groups, etc. on the system.
Check for and disable any unneeded services.
Locate malicious scripts.
Querylog in DNS.
Check for the suid scripts and nouser scripts.
Check valid scripts in /tmp.
Use intrusion detection tools.
Check the system performance.
Check memory performance (run memtest).
12.4 Implement the security policy
12.4.1 Correct all known existing software vulnerabilities
either by applying patches or by upgrading the software.
12.4.2 Implement host security
Protect your systems with passwords
Check the file systems and set correct permissions and
ownerships on all directories and files
Using iptables, allow only pings from a specific locations (for
monitoring systems to work).
Take preventive measures against DOS, "ping of death" attacks, etc.
Install a firewall (eg: apf and iptables) and only allow ports
to operate which the box needs for its normal functions; block all
other ports to prevent mischief.
Restrict ssh to specific IP addresses and specific users (I
suggest key authentication using passphrase).
Install logcheck to check the logs.
Install tmpwatch to delete the unused files from /tmp
directory.
Install and setup portsentry and configure it to use iptables
to block IPs.
Install mod_security and mod_dosevasive to safe guard
apache.
Delete files with nouser and nogroup.
Deleted unwanted files/folders in htdocs, disable directory
indexing.
Check for unwanted scripts in /root, /usr/local,
/var/spool/mbox.
Install BFD and FAF for additional security.
Disable open email relaying.
Submit a status report to management detailing all discovered
vulnerabilities and fixes.
12.5 Testing phase
Use tools like nessus, nikto, and nmap to do a penetration test
and see how well your server is secured. Also do a stress test.
Security is of utmost importance to a server, compromising
security is compromising the server itself. Hence, an understanding
of the same is a prerequisite to server ownership and
administration.
About this document...
This document was generated using the LaTeX2HTML
translator Version 2002 (1.62)
My name is Blessen and I prefer people calling me Bless. I got
interested in Linux when I joined the software firm, Poornam Info Vision Pvt Ltd also known as Bobcares. They gave me exposure to linux.
I am a B.Tech in Computer Science from the College of Engineering,
Chengannur. I passed out in the year 2001 and got into the company that
year. During my work, I was passionate with Linux security and I look
forward to grow in that field.
My hobbies are browsing net, learning new technologies and helping
others. In my free time I also develop open source softwares and one of
them is a scaled down version of formmail. The project is called "Smart
Mail" which is more secure than formmail.
Free as in Freedom: Part Two: Linux for the "Rest of Us"
[NOTE: all dialog is culled from email conversations unless otherwise
noted --AE].
I was fortunate to have found the perfect guide for my journey through
the politics of GNU/Linux past and present,
Ben Okopnik,
Editor-in-Chief of the Linux Gazette (LG). Yoga instructor and
practitioner, Unix instructor and practitioner, writer, editor and
Linux aficionado, Ben was both open to new ideas, familiar with "old"
ones, and willing and able to point me in the various directions I
needed to go to "explain the GNU/Linux model" to mostly non-technical
"left" and "progressive" readers.
Okopnik wrote, "Linux is inextricably political - and deliberately so,
from its very inception. The OS itself is a tool, as sharp, bright, and
beautiful as it may be; creating a better world, in which human beings
cooperate rather than fight each other 'to achieve the same exact ends'
is, from my perspective, the goal."
It was Okopnik who urged me to publish this article under a GPL-like
license and pointed me to the website where this could be done in
minutes.
Okopnik wrote,"Please consider releasing this interview under the Open
Publication License (the OPL is available at http://www.opencontent.org/openpub/),
or something similar. It's not a condition of the interview, but I'd
strongly prefer it. This license respects your commercial rights and can be
"adjusted" to suit your exact purposes; it's the license under which all LG
articles are published," he wrote.
[ This interview was originally slated for publication
in political venues rather than LG; however, I found the end result
interesting enough that I asked Adam to release it here as well. Besides,
I've never seen my name in print before, and wanted to seize a unique
opportunity. :) -- Ben ]
"If you release this interview under the OPL, you can
define whatever restrictions on distribution you choose; as an example,
Cory Doctorow, an excellent and highly-popular writer (see http://craphound.com/) recently released
several of his books under the OPL. He talks about his experience on the
site, and as a result of that experience has actually eased off on the
restrictions he originally imposed. Nor is he the first by far; take a look
at, e.g., MIT's OpenCourseWare site: http://ocw.mit.edu/index.html.
(Good resource to bookmark in any case.)"
"In order to license this article, I just have to declare it?" I asked.
"Yep, " wrote Okopnik. "Do a run-through of the 'wizard' at
http://creativecommons.org/license/
to make sure that you have exactly
the license you want - it only takes a few seconds - and it'll generate
the legal notice you need. Nice and neat."
I wrote, "I don't have to apply to anyone for approval?"
Okopnik wrote, "Nope; you're The Boss when it comes to licensing your
own stuff. Isn't that nice?"
Okopnik also gave me another perspective on the GNU revolution and its
major and minor aspects. For instance, the FSF's insistence on calling
"Linux" GNU/Linux though valid, violates the peoples' tendency to
abbreviate (hence the many abbreviations and acronyms in the Unix/GNU/Linux
operating systems themselves), and according to Software Developer and
Systems Designer Paul Ford (author of the popular Ftrain website), the 'GNU/' prefix came too late
in the game. All the "Linux" books and CDs had gone to print; the word
"Linux" came to mean both the kernel and operating system, though Linus
Torvalds and supporters developed the kernel and GNU the other essentials
of the OS.
"Looking through my 55 MB 'sent mail' archive (gads, but I do a lot of
writing! :), I find myself using the term 'GNU/Linux' exactly once,
back in 2000," wrote Okopnik.
Ford wrote, "Everyone I know calls it 'Linux.' Everyone appreciates
Richard
Stallman's extraordinary contributions. He's a genius, and has a
MacArthur genius grant to prove it. But the 'GNU/' prefix was added a
few years too late. I'll call it GNU/Linux in writing, sometimes,
though. Honestly, I don't care what people call it. That entire debate
seemed anathema to the open source ethos, too similar to someone
protecting their registered trademark."
Regardless of what it's called -- I'll call it GNU/Linux out of respect
for its GNU origins -- GNU/Linux is a political phenomenon, the
creation of user/developers for its own sake, or rather, for their own
sake. Rather than succumb to the Microsoft Monopoly, which places
Windows on virtually every PC sold in America, they created their own
free system and licensed it not for the benefit of an elite few, but
for anyone with the capability to alter the code, or learn how to alter
it (admittedly an elite group, but one based on merit and
intellectual, rather than corporate, "capital").
Okopnik describes the "typical" Linux user thus:
"I don't want to idealize Linuxers in my answers; none of us humans are
perfect saints, and the wild bunch who cleave to this OS and this
community are little different in that respect. However, just as
exaggeration in teaching is used to emphasize a point, isolating or
highlighting the common trends and traits in this case can provide an
interesting introduction. Keeping that in mind -
"The average Linux user, in my experience, is a product of a filtering
process - several of them, in fact. As in Darwin's description, the
selective pressure is not necessarily powerful, but is eventually
definitive; those who have, and propagate, the more effective traits are
better equipped to survive in a given environment.
"The first, and key, characteristic evident in Joe Linuxer is that he's
a maverick (Jane Linuxer is arguably even more so) - perhaps even somewhat
of a rebel; in any case, an individualist. The desire to not be one of the
herd, to not have to put up with chewing the same stale cud as everyone
else, often propels these people into experimentation with various facets
of their lives. Their computer-usage experience is just one of those
facets. Linux users tend strongly toward the non-usual - for all the good
and bad that this implies.
"Secondly, they tend to be capable -- not necessarily able to build a
spaceship out of a bicycle and cook a seven-course meal out of cupboard
scrapings, but willing to assume that they can do, build, repair,
"handle" things. (As a closely-coupled trait, they're often tinkerers,
always trying to unscrew the inscrutable. May the Great Ghu help the
Universe if one of us gets hold of the master key.) Larry Wall, the
creator of the Perl programming language, spoke of the three great
virtues of a programmer (Laziness, Impatience, and Hubris); impatience,
by Larry's definition, is the anger a programmer feels when the computer
is being lazy, which causes him to create long-term solutions rather
than short-term Band-Aids. Linuxers are often excellent examples of this
laudable kind of impatience - and Linux provides the opportunity to
exercise it effectively.
"As a result, Linuxers tend to feel a sense of ownership of
their system; they have not only gone out of the common path to install a
different OS but have tinkered with it, tweaked it, made it uniquely
"theirs" - and just as, e.g., long-distance sailors grant each other the
respect due to those who face the challenges of the ocean, Linuxers gather
in communities in which respect (specifically, respect for knowledge,
competence, and contribution to the community - but several other flavors
coexist as well) is the common thread as well as the coin of the realm."
KDE and GNOME: Linux for "The Rest of Us"
I first came to GNU/Linux in 1996, when it was already a world-wide
phenomenon, but still a 'techie/hacker' thing. I was on the command-line
for six months before I installed X-windows; nevertheless, I was surprised
at the number of alternatives a Unix-type system gave me, especially as a
writer in terms of text formatting, manipulation, etc.; Emacs, Vim and
though I didn't get into Tex or nroff, I used Applixware to do the
formatting one does in a WYSIWYG word-processor.
I wrote to Okopnik, "Because of various jobs I took as both a
free-lance and cubicle-bound ad-man and copywriter, I had to install
Windows merely to run 'Word' and 'Powerpoint' and gradually moved away from
Linux for about four years. When I came back to it with SuSE 9.0 in
December of 2003 I was astonished. KDE (KDE.org's free desktop environment), GNOME (GNU's
free desktop environment), Open Office (OpenOffice.org's free office suite), the
whole new GUI interface floored me.
Okopnik wrote, "This is not an uncommon reaction in this venue. The
rates of growth and development in Linux are simply phenomenal - and still
accelerating."
I wrote to Okopnik, "KDE and GNOME, especially KDE 3.1x, worked as
well as or better than the Win2000 I had installed four years ago, and
I've yet to experience a full crash -- that's par for the daily use of
Windows 2000. More significantly, I was turned onto the 'New Linux' by
someone who knew about as much about the GNU/Linux command-line as the
typical Windows user knows about Unix's retarded younger brother, DOS.
"Similarly, I turned someone else who was sick of Microsoft's shoddy
but expensive products to the mind-boggling array of free software programs
that run under GNU/Linux, though he had neither the time nor the
inclination to learn about the operating system. Like many users, all he
wanted and needed was a word-processor, a browser, a mail program, some
games, and as little trouble as possible. SuSE 9.0 provided him with all
of these things, and now, after a year on GNU/Linux, he knows slightly more
about the command-line than he did DOS. But no desire to go back to
Windows.
"Have you noticed more of an interest in Linux or an enhanced
readership for the Linux Gazette since Linux became both market and user
friendly? If so, are these new users less interested in the 'technical'
aspects than in having a stable GUI-based system to use for work and email
and net-surfing?"
Okopnik responded, "Actually, this is an issue that I brought up in an
involved discussion with the LG staff and the Answer Gangsters (The Answer Gang answers
Linux questions sent to us by our readers, and the discussions and the
answers become part of LG.) My viewpoint here is that it's actually a very
good thing - modulo the awareness that the command-line (CLI) exists. That
is, people are perfectly welcome to come to Linux and use only its GUI
capabilities as long as this serves their needs - but when the GUI proves
insufficient, the capabilities of the CLI are there, just underneath,
providing the perfect security blanket.
"In an article I wrote for Java Developers Journal, I related an example
of this. I had a client whose Web developer left them in the lurch with
several hundred HTML files without the '.html' extensions. This wouldn't
be too bad by itself - renaming a group of files isn't difficult - but
the thousands of HTML links within the files referred to those
extensionless names as well. With GUI-only tools, this is a
nearly-unsolvable disaster. From the CLI, it was a matter of a single
short line of code:
perl -i -wpe 's/(<a href="[^"]+)/$1.html/g' *
The readership of The Linux Gazette (LG) has certainly changed over
time. Where we used to get dozens of questions on fairly technical
topics in The Answer Gang, we now get only a few - and they tend to be
simpler, less technical. The email I get from our readers indicates that
there has indeed been a definite shift in the user base; the old Linuxer
who would bang on a problem for hours so that it could be reported (and
quickly fixed) is being... well, not replaced, but reduced,
percentage-wise, as the mainstay of the population. The new user is often
just that - a computer user who just wants that
email/web/document/spreadsheet processor and maybe a few games on the side.
There is, however, a cultural shift that occurs even in those users after a
while: you cannot live in a society based on a given moral premise and
ignore that premise, or even stop it from penetrating into your life (even
if you try to prevent it.) The original "hacker ethic" of Linux lives on,
strong as ever in those who use the full extent of this OS, and inherent
(and growing, however slowly) in those who use it even without that full
knowledge."
Paul Ford wrote, "I used to think there was too much emphasis in the
community on claiming the desktop, trying to compete with Windows, but the
latest GNOME is attractive and elegant, and works great, looks as good as
MacOS X, and doesn't feel like a thin skin over Unix at all. It's an
environment I could use every day. So I was wrong--the desktop was a good
aim, it just took a while to get things to a good point."
According to George Staikos, a KDE developer and spokesperson, "The KDE
project was formed by a small group of computer programmers and Linux and
UNIX users who were fed up with the lousy state of user interfaces
available for their operating systems. They wanted something that was
fast, powerful, featureful, and looked good. Notice that making money was
not one of the requirements. They set out to accomplish this task in the
most effective manner possible, which was to use the Qt toolkit (at the
time distributed free of charge for non-commercial use under a restrictive
license, but now distributed under the free GPL license for non-commercial
use). Because there were very many people around the world with similar
desires and compatible skills, because there was no risk of someone
hijacking the project and turning it into a business, and because there was
actually proof-of-concept working code already being produced, the project
quickly grew. After a few years, the core of the system was very solid and
new programmers could easily find a niche to work in, implementing that
feature they always wanted or fixing that bug that has bothered them for so
long. These individuals are what makes KDE work. They keep the project
evolving, bringing new ideas and new manpower. There is relatively no risk
involved in contributing, and the rewards are plenty. Developers
(including coders, translators, documenters, artists, and more) can
contribute whatever they have time for.
"Of course there are other requirements to keeping such a project
going. We need bandwidth, servers, funds for promotion and travel, and
more. This tends to come from corporate users who are kind enough to
contribute back to ensure the progress of the project, and from home
users who perhaps can't contribute in other ways. Some people also
contribute system administration time. This is all very vital to the
success of KDE.
"It's important to note, however, that KDE is indeed paid for, as much
as any other software is. KDE is paid for by individuals, and paid for
in a distributed manner. Our time, as KDE developers, is worth as much
money as any other software developer (More, if you ask me. KDE
developers tend to be one smart bunch!). KDE is indeed a very costly
project, and is paid for by society itself, as much as a result of the
lack of momentum of the commercial sector to create a useful solution
to existing problems.
"What is KDE 'worth'? The freely available SLOCCount tool gives me an
estimate of $22.6 million just for the KDE libraries alone, a small
fraction of what is KDE. Most of the code in the KDE libraries was
developed from 1999 through 2004, almost 6 years in total. Not
including the Qt toolkit, KDE must be worth well over $250 million.
This also doesn't include artwork, documentation and language
translations, which KDE is well known for.
"Total Physical Source Lines of Code (SLOC) = 608,228
Development Effort Estimate, Person-Years (Person-Months) = 167.61 (2,011.32)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months) = 3.75 (45.01)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule) = 44.69
Total Estimated Cost to Develop = $ 22,641,821
(average salary = $56,286/year, overhead = 2.40)
(this data was generated using David A. Wheeler's 'SLOCCount' tool)"
I wrote, "Was KDE originally supposed to be free-ware? I remember when
I first saw the specs in the late nineties, I thought it seemed too good to
believe. Yet it's here and working and continues to grow in functionality
and popularity. In fact, the SuSE 9.x package uses KDE as its Graphic
base. Can this go on without some serious cash flow?"
Staikos wrote, "Yes, KDE was definitely supposed to be free, both in
cost, and in terms of speech. KDE was to be available free of cost to all,
and available for modification and learning as desired as long as it was
not abused for commercial gain by others who weren't interested in
contributing back. That is to say, the licensing prohibits using KDE code
in a non-free project, though you may use KDE for that project. For
example, you cannot copy source code from KDE and embed it in your
commercial, closed-source application. The KDE license actually requires
you to release the source for your application as well. However, you may
make calls to the KDE libraries from your application. In short, free to
use, yes, free to steal from, no.
"Indeed KDE is growing rapidly in popularity. We do need to find new
ways to support the project in terms of getting the hardware we need, the
administration help we need, the legal work done, and paying for
conferences and developer meetings. It's an ongoing struggle.
"Making money is not a bad thing, and I think making money off of KDE
is a good thing for KDE. Stealing code from KDE is however not a good
thing, and that's what the GPL protects us from. Most of KDE is licensed
under the GPL, while the libraries tend to use the LGPL in order to permit
commercial KDE applications to be developed. Some portions of KDE are
under more liberal licenses such as the Berkely Software Development (BSD)
license because the author did not have concerns with others using that
code in non-free software. KDE as a project maintains that our software
must be compatible with the GPL, but it need not be specifically licensed
under the GPL," wrote Staikos.
I wrote, "I know several people who, finally fed up with Windows, and
not wanting to deal with getting a new Mac, switched over to GNU/Linux even
though they know only the rudiments of command-line arguments and don't
plan on learning much more. Like many users, all they use their computers
for is word-processing, presentations, a web browser, and email. Since the
advent of KDE and GNOME, people can use GNU/Linux the way they use Mac or
Windows without spending the time and effort necessary to learn a Unix-like
OS. This would have been unthinkable a few years back, even with relatively
user-friendly Window Managers like Windowmaker and IceWM. One of the
people I'm interviewing for this article, the editor of the Linux Gazette,
confirmed this 'trend.' More and more of his readers are concerned with
'typical' user issues rather than the more technical aspects of Linux. Do
you think that with the advent of GUI-based Desktop Environments such as
KDE that GNU/Linux will appeal to a wider audience who want a choice other
than Mac or Windows?"
Staikos wrote, "Most definitely. This was the original goal of KDE,
and still remains one. However KDE does not have the resources to provide a
real end-user system. We only 'ship' source code, and it is up to the
distributors to set their preferred configuration defaults, customize the
menus, and determine which applications to add or remove. These things are
absolutely vital to creating a valuable end-user experience, and in fact
are different for each target market. I think Linux with KDE is already a
perfectly suitable solution for the average desktop system. The obstacles
in place are more monopolistic in nature. Users are accustomed to the way
MS Windows works. They learned this over many years, and expect that all
computers work this way, even if it's inefficient or poorly designed.
They're impatient to learn a new system, and balk at the idea of using
Linux. Furthermore, most commercial applications are designed only for MS
Windows. It's hard to justify using Linux when your favorite video game or
other software only runs on Windows. Hopefully we will change this over
time, as KDE becomes more popular and software developers can justify
porting to Linux.
"I think KDE is one of the driving factors that is pushing Linux into
the mainstream, and I do hope that it will one day be a serious competitor
in terms of user base to MS Windows. The world needs an alternative, and
Linux is here to provide it," wrote Staikos.
While GNU/Linux is "inextricably political," both Ford and Okopnik
admit that most users are less into the politics than the practical
applications of the system itself.
Ford wrote, "Linux has a pretty amazing advantage in that you get
something for free, or a very low cost, out of the movement. It's hard for
any movement based on ideas to compete with that -- it's not like you can
say, 'if you buy into Noam Chomsky's theory of foreign policy, we'll give
you a free Chomsky hat.' Whereas Linux can say, 'if you're willing to
believe that Open Sourced software works, we'll give you a free operating
system with all the trimmings and cranberry sauce.' So the two 'movements'
don't really compare."
While for the Stallman and the FSF the difference between the "free"
and "open source" movements is crucial, for Ford they are pretty much the
same.
"I think they're usually interchangeable, " Ford wrote. "And in truth,
I don't really care that much. If a license is similar to the GPL, I'll go
with it. For things like OCR or image editing, I don't mind buying
commercial tools. They tend to be in much better shape than their
open-sourced counterparts. They're very task-based -- I'm scanning a page,
or creating an image. If good replacement software comes along, I'll use
that. But in the meantime, the work is done--I've got my output....But for
any programming project, where people need to work together, and thousands
of hours go into developing code, I'm terrified of commercial software.
Lock-in is terrifying.... Basically, when I'm looking for a tool, I go
"shopping" for the open-sourced version first. Open-sourced software lets
me try out a huge number of solutions to find the best one--if I don't
like one package, I can see if there's a better one," wrote Ford.
Adam Engel has published poetry, fiction and essays in such magazines and
periodicals as Counter Punch, Dissident Voice, Online Journal,
Strike-the-Root, LewRockwell.com, The New York Art Review, The Concord
Journal, The Middlesex News, Accent, The Littleton Review, Ark, Smart
Shoes, The Beacon, Literal Latte, Artemis, The Lummox Journal, Fearless,
POESY, The Half Moon Review, Art:Mag, Chronogram, Gnome and others.
Adam Engel's first book of poetry, Oil and Water, was
published by Maximum Capacity Press in 2001. His novel,
Topiary, will be published by Dandelion Books in the
Spring of 2005.
He has worked as a journalist, screenwriter, executive speechwriter,
systems administrator, and editorial consultant, and has taught writing at
New York University, Touro College and the Gotham Writer's Workshop in New
York City.
This article will serve as a primer to people who are new to the
world of Linux hacking, and are attempting to compile the Linux kernel
from source. The various steps from downloading the kernel source to
booting from the new kernel image are explained. Also given are tips
on cleaning up the source code, doing verbose compilation etc.
1. Downloading the kernel source code
In order to compile a new kernel we have to download the source code of
the Linux kernel. We can download the source from www.kernel.org. Here we
can find all versions of the Linux kernel source code. Let's take an
example. Suppose we want to compile the 2.6.9 version of the linux kernel.
We have to download the 2.6.9 source code from:
It's better to download the bzipped version, as that will be more
compressed than its gzipped counterpart; hence will take less
time to download. A wget from the command line will look like:
Once we download the required kernel version source, we need to bunzip
and untar it. We can do the following:
tar xvjf linux-2.6.9.tar.bz2
The 'x' option is to denote the untarring (e'x'traction), 'v' for
verbose, 'j' for specifying that we need to bunzip the file before
untarring and 'f' for stating the name of the input file.
The file will untar into the directory linux-2.6.9. Once it's
untarred 'cd' to linux-2.6.9.
2. Configuring the kernel
We have to configure the kernel before we start compiling it.
During the configuration phase, we will select the components which
we want to be part of the kernel. For example: suppose we are using
the ext3 filesystem. Then we need to select the ext3 filesystem support
while configuring the kernel. Typically we have to run a
make menuconfig
This will bring up the ncurses interface for configuring the kernel.
There are other options such as 'make xconfig' and 'make config'.
The former will bring up the configuration menu in graphical mode and
the latter in text mode.
Once we select the different components we want for our kernel, we
can exit the configuration interface. We should select the option
to save the configuration from the configuration menu, before exiting.
After we have configured the kernel as mentioned above, we can find
a file named '.config' in the top level directory of the source.
This file is the configuration file. It contains various options and
their states (whether they are selected or not). For example, if we choose
to have the PCI support in our kernel we can find an entry of the form:
CONFIG_PCI=y
in the .config file. Similarly, options which are selected as not
required will appear as not set. Suppose we have not selected
the XFS filesystem support in our kernel we will find the following
in the .config
# CONFIG_XFS_FS is not set
A great feature of 2.6 kernels is that if we are running
make menuconfig (or
xconfig or config) for the first time, then the configuration menu we are
presented with is based on our current kernel configuration. In my case,
I have a Fedora Core 1 system. The kernel which I run is '2.4.22-1.2115.nptl'.
Hence when I run a 'make menuconfig' for the first time on the source then the
configuration menu presented will contain the options as given in
'/boot/config-2.4.22-1.2115.nptl'.
3. Building Dependencies
This step is required in kernels prior to 2.6 series (here I am only
referring to the stable series kernels). For example if we are using a
2.4 kernel then we have to build the dependencies explicitly. We have
to run the following:
make dep
This will build the dependencies. But for a 2.6 kernel we can skip
this step. The dependencies are automatically created when making the
final image with a 2.6 kernel.
4. Creating the final image
We can build various types of kernel binary images.
We can build a plain kernel image, or a compressed version of it; the usual
choice is compressed, or the 'bzImage'. We can create the bzImage by running
make bzImage
In 2.6 kernels this step will also resolve the dependencies and proceed
to create a bzImage image.
After the compilation is over we can find the kernel image at the path
arch/i386/boot/bzImage in case of an image for a 386 based processor
(Pentium, AMD etc.).
5. Compiling and Installing the modules
In the configuring section if we have selected some components to be
built as kernel modules then we need to compile those modules.
To compile the modules we should run the command:
make modules
This command will compile the components (which are selected
for module compilation) to modules. In a 2.4 kernel the result
will be .o files of the corresponding components. But in a
2.6 kernel the output file will be a .ko module. For example
if we have given the option for the Network driver of Realtek
cards to be built as modules then after giving a 'make modules'
we can find in 'driver/net/' a file named 8139too.o
in the case of a 2.4 kernel and 8139too.ko in the case of a
2.6 kernel.
After we have compiled the modules, it's time now to install the modules.
To install the modules run:
make modules_install
as root. This will install the modules and other necessary files into the
/lib/modules/2.6.9 directory.
6. Booting from the new kernel
Once we are done with the installation of modules, we
can go for an automatic installation procedure for the
kernel binary. We just have to run
make install
This will update the kernel image on to the /boot area, update
the configuration file of the bootloader (lilo.conf or
grub.conf) and then do the necessary actions to make the new
kernel bootable.
After this we need to reboot the machine. When the machine boots
next time the boot menu will present us with the option to boot
from the new kernel we built. We choose that option and voila!!
boot into a kernel we built all by ourselves!
7. Manual installation of the kernel
In case 'make install' does not work, or if we cannot perform an
automatic installation due to some other reason, we can go for a manual
installation of the kernel. For example, if we are using the grub boot
loader then we have to copy the bzImage into the boot partition and then
change the '/etc/grub.conf' to reflect the presence of the new
image. If we are having lilo boot loader then we have to copy the bzImage
to the boot location and then modify the lilo.conf and then run
the 'lilo' command to make sure that next time we boot we will
have our new image as a choice to boot from. The following are the steps we
should perform as root user if we are using lilo boot loader:
cp -a arch/i386/boot/bzImage /boot/bzImage-2.6.9
After this we add the following entry to /etc/lilo.conf
We will reboot the machine after this. When we are prompted at
the lilo prompt enter '2.6.9-kernel' as the boot option and we
will be booting to the new custom built kernel.
8. Verbose compilation
We find that the compilation of the kernel is very quiet. Much less
information on what is getting compiled is shown on the screen while the
compilation proceeds.
After we have initiated compilation once on the source if we want
to clean the object files and other temporary files then
we have to run the following:
make clean
This will remove most generated files but will keep the configuration
file.
If we need an absolute cleaning, i.e. if we want to return the source to
the state in which it was before we started the compilation, then do a
make mrproper
This command will delete all generated files, the configuration file as
well as various backup files. This will in effect unwind all the changes
we made to the source. The source after this step will be as good as it
was just after the download and untar.
10. Conclusion
We have seen how to obtain the linux kernel source, how to configure it,
how to build the kernel image and modules, how to boot from the newly compiled
kernel and how to do a verbose compilation. Also we have seen how to clean up
the temporary files and configuration files which were created during the
compilation. The next step for a budding kernel hacker would be to modify the
kernel source and try experimenting with it.
Krishnakumar loves to hack the Linux kernel. He works
for Hewlett-Packard and is a BTech from Govt. Engg. College
Thrissur.
Originally published in Issue 52 of Linux Gazette, April
2000
Here's a hint. When you think your code to exec a shell function is
just not working, never, repeat NEVER send it "/etc/reboot" just to see
what happens.
-- Elliott Evans
Introduction
Shell scripting is a fascinating combination of art and science that
gives you access to the incredible flexibility and power of Linux with very
simple tools. Back in the early days of PCs, I was considered quite an
expert with DOS's "batch files", something I now realize was a weak and
gutless imitation of Unix's shell scripts. I'm not usually much given to
Microsoft-bashing - I believe that they have done some good things in their
time, although their attempts to create an operating system have been
pretty sad - but their BFL ("Batch File Language") was a joke by
comparison. It wasn't even particularly funny.
Since scripting is an inextricable part of understanding shell usage in
general, quite a bit of the material in here will deal with shell quirks,
methods, and specifics. Be patient; it's all a part of the knowledge that
is necessary for writing good scripts.
Philosophy of Scripting
Linux - Unix in general - is not a warm and fuzzy,
non-knowledgeable-user oriented system. Rather than specifying exact
motions and operations that you must perform (and thus imiting you
only to the operations described), it provides you with a myriad
of small tools which can be connected in a literally infinite number of
combinations to achieve almost any result (I find Perl's motto of
"TMTOWTDI" - There's More Than One Way To Do It - highly apropos for all of
Unix). That sort of power and flexibility, of course, carries a price -
increased complexity and a requirement for higher competence in the user.
Just as
![[BIO]](misc/cover/bear-mini.png)
bear has been The Editor Gal's teddy bear for some time now. He has
only gotten into Linux recently, as he has been looking up to
xteddy and how
much he enjoyed the desktop, bear enjoys Fvwm button bars best
(though large dock icons are nice too) and has a snug $HOME in
/usr/share/xteddy.
![[BIO]](../gx/2002/note.png)
Heather got started in computing before she quite got started learning
English. By 8 she was a happy programmer, by 15 the system administrator
for the home... Dad had finally broken down and gotten one of those personal
computers, only to find it needed regular care and feeding like any other
pet. Except it wasn't a Pet: it was one of those brands we find most
everywhere today...
Update vs Install, how best to manage /home?
Greetings from Heather Stern
[Thomas]
When you do a new install a distro (albeit a new one, or an upgrade to a
new point release of one you currently have) you can instruct it not to
touch certain partition (like /home). This means you don't have to
worry about loss of data. You mentioned UIDs. Backup
/etc/{passwd,shadow} beforehand.
Mostly, it's a REALLY good way to
screw yourself up; in general, it's not a good idea to use anything outside
of [a-zA-Z0-9_] as part of a filename.
Software Patents ![[Picture]](../gx/2002/tagbio/conry.jpg)
Originally hailing from Ireland, Michael is currently living in Baden,
Switzerland. There he works with ABB Corporate Research as a
Marie-Curie fellow, developing software for the simulation and design
of electrical power-systems equipment.
![[BIO]](../gx/authors/cherian.jpg)
![[BIO]](../gx/authors/engel.jpg)
Adam Engel has published poetry, fiction and essays in such magazines and
periodicals as Counter Punch, Dissident Voice, Online Journal,
Strike-the-Root, LewRockwell.com, The New York Art Review, The Concord
Journal, The Middlesex News, Accent, The Littleton Review, Ark, Smart
Shoes, The Beacon, Literal Latte, Artemis, The Lummox Journal, Fearless,
POESY, The Half Moon Review, Art:Mag, Chronogram, Gnome and others.
![[BIO]](../gx/authors/krishnakumar.jpg)
Krishnakumar loves to hack the Linux kernel. He works
for Hewlett-Packard and is a BTech from Govt. Engg. College
Thrissur.
