UNIX Tricks
See also Bootcamp and the PCFarm Info pages for detailed Unix information.
General
My program crashes but I don't get a core file - how can I get one ?
In a Bourne shell use ulimit -c unlimited Other values than unlimited are possible but it depends on the shell flavour whether one has to give a number in 512-byte or 1024-byte blocks
How can I rm files with funny file names, like ones that begin with "-" ?
rm "./<file to be deleted>"
Programs using the glibc “getopt()” and ”getopts()” functions (e.g. most programs under Linux) stop parsing commandline options if they encounter a token consisting of two single dashes (“--”). The remaining tokens will be treated as non-option arguments. See for example these examples:
grep -- -foo filename.txt ### look for “-foo” in filename.txt touch -- -rf ### create file -rf rm -- -rf ### delete file -rf
How can I search my command history ?
In bash, the command history lives in ~/.bash_history, but you can search it easily by doing Ctrl+r and then typing the beginning character of the command you want.
find files
find <path> -name '<filename>' -print
find name strings (e.g. in your code)
grep -in <string> source/*.c grep -i <string> include/*.h
option -i: switches off case sensitivity.
option -n: gives you line numbers.
How can I find out if a uDST run is corrupted?
Type:
cat /production/udst/05d1/smlinks/run19208.smdst.gz _ gunzip -t
If nothing is returned, the run is OK. If it returns something like
gunzip: stdin: invalid compressed data--format violated
the run is corrupted. Please inform data cops in this case.
Rename multiple files
- If the very useful command "mmv" (move multiple files) is not installed in your system and you want to rename many files with only one command line, you might find this www-page helpful.
gdb stuff
To get gdb to break under a condition, do something like this:
> break MyFile.c:345 ---> puts a breakpoint at line 345, if this is breakpoint 1 then you can do > condition 1 MyVar==10
See also gdb tutorial
emacs stuff
To configure your emacs properly you should create/modify your ".emacs" file. You can have spell checking on the fly etc. Please find an example in: [1]
To sort a file by a particular column, use
C-u # M-x sort-fields
Where # is the argument is indicate which column to sort by
To cut and paste marked columns or other rectangles. (Release "Ctrl" before typing the latter letters.)
C-x r k C-x r y
To query-replace something + line break with something else:
M-% "something" C-q C-j <enter> "something else" <enter>
The file Emacsreference.pdf summarizes nicely most useful short cuts. Alternatively please visit: [2].
pine stuff
To delete a bunch of mails with the same sender/subject/etc. try this:
; To get the the selection page T to use a text selector S to select on the subject Your subject [Enter] A to apply a command D to mark the selected mails for deletion
ssh tricks
These are many, and vastly useful, so expect this list to expand quickly :)
How to avoid typing password so often?
Insert into your ~/.ssh/config on the host of origin (e.g. local desktop/laptop) the following lines:
Host *.desy.de
ControlMaster auto
ControlPath ~/.ssh/.control-%r@%h:%p
This will allow you to login without password to the destination host, as long as the first connection stays open. (taken from faq on DESY bastion)
NOTE: This will work only with OpenSSH 4.4 or higher (unfortunately). For version 4.0-4.3 use the following commands:
ssh -M -S ~/.ssh/remote-host user@remotehost
for the first (master) connection, and
ssh -S ~/.ssh/remote-host user@remotehost
for every subsequent connections, which will then be passwordless.
Avoid typing the passwort, second possibility
Another possibility is to create two “virtual” hosts in your .ssh/config where you can configure a lot of useful options in one place. Then you login with “ssh worf-m” once and let this connection linger. Use “ssh worf-s” to open successive shells later on. Works great with the ”fish://worf-s/” filetransfer in KDE, or to copy files with rsync/sftp in scripts (rsync -avP file worf-s:/user/foobar/). I like that version better because it marks the first, important, ”master” connection as special:
Host worf-m
ControlMaster yes
ControlPath ~/.ssh/.control-%r@%h:%p
Hostname worf.desy.de
User useratdesy
## add even more useful options here... ##
Host worf-s
ControlMaster no
ControlPath ~/.ssh/.control-%r@%h:%p
How to use bastion as a ssh/scp/sftp proxy?
For example to reach the "pal" cluster over bastion: Insert into your ~/.ssh/config on the host of origin (not bastion) the following lines:
Host pal.desy.de ProxyCommand ssh bastion.desy.de netcat -w 3 pal.desy.de %p
If your userid is different from your AFS account on the target host, than use the following lines and replace "userid" with your AFS account:
Host non-afs-host.desy.de ProxyCommand ssh userid@bastion.desy.de netcat -w 3 non-afs.desy.de %p
You will be asked for password twice then, once for bastion and the second time for the destination host. Pay attention if your passwords are different. (taken from faq on DESY bastion)
Using a proxy (2nd possibility)
You can also forward ports using the forwarding feature of ssh that is enabled by using the “-L” commandline option or the “LocalForward” command in your .ssh/config. First the commandline variety, then the stanzas for .ssh/config:
shell1$ ssh -L 2222:slowroc.desy.de:22 worf.desy.de # open tunnel shell2$ ssh -p2222 onl@localhost # use tunnel
Host worf-m
# best to add this to the snippet with the ControlMaster, above...
LocalForward 2224 slowroc.desy.de:22
Host slowroc
Hostname 127.0.0.1
User onl
HostKeyAlias slowroc.desy.de
Port 2224
Note that this method can also be used to redirect access to CVS (port 2401) or any other TCP based protocol. If you want to conveniently access electronic journals from home, use the dynamic proxying feature (“ssh -D1080 worf.desy.de”, or “DynamicForward 1080” in .ssh/config) and set your web-browser to use a “SOCKS4” proxy on host “127.0.0.1“, port “1080”.
For your even more advanced tunneling needs, use Socat, the descendant of the venerable Netcat. It's manualpage is awe-inspiring.
Remote file access to DESY computers
Often one needs to comfortably work from outside DESY, still accessing the files on HERMES workgroups servers (worf/geordi) or his own desktop. SSH file transfer (scp) is the most common method for such an access, yet it's relatively uncomfortable, since one needs to type password for each copy operation, and desktops are not accessible from outside, so one needs to copy the files twice: desktop->geordi->local (see the [UNIX_Tricks#ssh_tricks_ssh tricks] section for solutions). In addition, it's often unfeasible to have an emacs running on geordi and have it tunneled through ssh X-forwarding due to slow network connection, but rather have the file mapped locally and use the local editor. Here are few workarounds to this problem:
unison
One of such possibilities is using the unison or rsyncfile synchronisation tools. They both allow directory-based syncronisations with diffs, using ssh as a transport mechanism. The drawbacks are that unison has a very strict version control, so you need to have the exactly same version installed at both ends (that's why it's not centrally maintained by DESY) and rsync needs a bit of getting used to
konqueror fish plugin
Most recent linux distributions now distribute KDE as the default window manager. It includes the Konqueror file manager/web browser, which allows remote filesystem access via ssh using the fish or sftp plugins. Just enter e.g. "fish://geordi.desy.de/group02/calogrp/pi0study/" in the location box, enter your password when asked for, and you'll navigate to the remote directory on geordi as if it was local (see the screenshot). You can then right-click on the file you'd like to edit and choose Kate as the editor. As part of KDE, it can handle remote files properly through fish, so when you make changes and Save your file it will automatically get transferred back to the remote host.
sshfs
A even more transparent method of accessing your files at DESY is to use the ssh-filesystem “sshfs” that makes use of the sftp subsystem. If you are lucky and your Linux distribution has taken the proper precautions, it should be as simple as running this single command (assuming that “/media/worf” is a directory owned by your userid and that you are using the “avoid typing your password” tricks above):
$ sshfs worf-s:/ /media/worf $ cd /media/worf $ ls acs@ data01/ data09/ data17/ data25/ data33/ data41/ data57/ hermes@ mcdata02/ net/ sbin/ user03/ afs/ data02/ data10/ data18/ data26/ data34/ data42/ data58/ home/ mcdata03/ opt/ scratch/ user04/ ...
Note that the symlinks are most likely pointing to the wrong locations, so remember which data-partition your group/userfiles are in. The convenient thing about ssh is that every application, kde, gnome, commandline or whatever else there might be will be able to directly read and write your files on worf. Unmount with “fusermount -u /media/worf”.
awk stuff
Summing a column of numbers
Yay for awk!
cat count.txt _ awk '{ sum+=$7} END {print sum}'
cat FileWithColumnOfNumbers _ awk 'BEGIN {FS="_"}; {sum += $1} END {printf("%d\n",sum)}'
Printing out certain precision
awk '{printf " %7.6f %6.5f %7.6f \n", $1, $2, $3}' file.txt
Linux meets Windows
If absolutely necessary :( ...:
Access to windows from linux
- If you want to access windows from linux at DESY, here the relevant webpage:
- Once you are registered, you can access windows via
rdesktop -f adterm.desy.de
How to extract the files embedded in a powerpoint presentation
- Open the *.ppt file with OpenOffice (soffice)
- Save it as OpenOffice document (*.sxi or similar)
- Type
unzip *.sxi
- The embedded files are extracted and dumped as *.png into the subdirectory 'Pictures'
- In order to convert a *.png file to *.ps, open it for example with xv and save it in a different format. Then convert to encapsulated postscript if you want to include it in a LaTex presentation.
xv *.png ps2epsi *.ps
Graphics and Document Creation
- Check the Bootcamp: Graphics and Document Creation
Manipulating postscript
Convert/edit images
There are a lot of possibilities to convert and modify pictures. Generally one should distinguish between two types of images:
- vector graphics (e.g. postscript or pdf)
- pixel graphics (e.g. png, jpg, gif, but can also be ps or pdf!)
In the first category vectors are saved while the second category some kind of sampling is applied which, in any case, limits the resolution. Vector graphics are most useful and smallest for simple drawings and histograms, the advantage being that you can zoom in as much as you wish and one can use ascii text. Photographs and images with colour gradients are, however, smaller and easier to handle if saved in some kind of pixel format, e.g. png, jpg, gif.
Depending on those differences one should also choose the format and best type of conversion.
Modify vector graphics (e.g. by using TGIF)
This offers the great advantage to easily change colours, line thickness, text and many more things in existing vector graphics. Please note that , if once converted to pixel graphics, there is no way back.
If the original file is pdf first convert to ps (e.g. pdf2ps but not programmes which pixelise!)
pstoedit -f tgif infile.ps outfile.obj tgif outfile.obj
You can also use XFIG. To create an eps file or alike in TGIF use print and set "File->Print/Export in colour" first. I find it often useful to deactivate "Layout->Snap to grid"
Modify pixel graphics
There are a lot of possibilities. One can use programmes like gimp or xv. A nice command-line method is using the ImageMagic package in linux:
convert infile outfile
Here you use the file endings to specify the format and one can use a lot of options to scale and modify pictures, e.g.:
convert -sample 20% infile outfile
to scale the image to 20% of its original size