Neither one nor Many

 
August 26 2016

Some stuff I keep here as a cheetsheat for myself

mass replace recursively

NOTE: do not run in case you have .git directory, it can corrupt it.

Source: https://stackoverflow.com/a/1583282/1958831

mv .git /tmp/git_tmp
find . -type f -print0 | xargs -0 sed -i 's/subdomainA\.example\.com/subdomainB.example.com/g'
mv /tmp/git_tmp .git 

return value

$? return last exit code

buffer control with stdbuf

time make -j24 "$@"  2>&1 | stdbuf -i0 -o0 -e0 tee ./mm.log

sets buffer to zero, everything is piped immediately, bad for throughput, good for latency

< <(command)

gdb on program that works on stream, with mkfifo to make this a combined example (you can issue the bash command instead of the tail -f in this case)

mkfifo /tmp/foo
bash tail_all.sh > /tmp/foo
gdb <application>
run <program args> < <(tail -f /tmp/foo)

This syntax is also useful for stuff like:

while read -d ':' line; do
    echo "$line"
done < <(hbase classpath)

A pipe | creates a new subshell, this is also a nice way to avoid this (subshells cannot modify variables of their parent)!

EDIT: This is called process substitution, see http://tldp.org/LDP/abs/html/process-sub.html

echoing multiple lines to a file

cat <<EOT >> greetings.txt
line 1
line 2
EOT

matching

if [[ $foo =~ foo ]]; then
    ;
done

Note that you DO NOT use asterisks like *foo*!

processing stdin

A pattern I use quite a bit is the following.

function func
{
    while read line; do
        echo $line
    done < /dev/stdin
}
cat foo.txt | func

subshells / pids

Sometimes subshells are useful.

echo mypid = $!

( sleep 10; ) &
typeset pid=$!
sleep 1
kill $pid # kill if still running..

Note that you can use exit in a subshell without destroying the parent.

trap CTRL+C

function myfunc
{
    echo Do some cleanup here..
    exit 0
}
trap "myfunc" 2
print "Press CTRL+C to abort 10 seconds of waiting.."
sleep 10

namerefs

First learned about this in kornshell, but seems to work for bash as well.

function func {
    local -n test_ref=$1
    test_ref="Hello world"
}

func test # note omitted $-sign

echo $test

If you want kornshell compatibility use typeset -n instead of local -n.

The same example, but with an array, you have to first explicitly declare the array before passing it to func.

declare -A array
function func {
    local -n array_ref=$1
    array_ref=([one]=aaa [two]=bbb [three]=ccc)
}
func array

(If you want kornshell compatibility again, use typeset -A instead of declare -A (and typeset -n for local -n)).

arrays

Continuing the previous paragraph, some useful examples regarding arrays.

echo ${array[@]}  # aaa bbb ccc
echo ${!array[@]} # one two three
echo ${#array[*]} # 3

for key in ${!array[@]}; do
    local value=${array[$key]}
    echo $key = $value # one = aaa, ... 
done

Instead of initializing with the array=([foo]=bar [baz]=ban) syntax, assignment syntax is array[foo]=bar.

bash substrings

typeset hostname=$(hostname -s)
hostname=${hostname:0:10}  # first 10 chars..

NFS caching

Trick to use cachefilesd on NFS mounts. Useful if you work remote on a mounted NFS share. btrfs is not supported, so I create an ext4 mount instead like this:

dd if=/dev/zero of=/tmp/fscache.txt bs=1M count=4000
# be careful with choosing /dev/loopN
losetup /dev/loop10 /tmp/fscache.txt
mkfs.ext4 /dev/loop10
mount /dev/loop10 /var/cache/fscache

Very useful other commands can be found here: http://www.cyberciti.biz/faq/centos-redhat-install-configure-cachefilesd-for-nfs/ (i.e.: cat /proc/fs/nfsfs/*)

Benchmark I/O

start iperf -s server, then iperf -c 127.0.0.1.

Show Network I/O per process

atop can do this, which comes provided by your package manager probably, but chances are you need to compile it yourself for a newer version. The new version makes it possible to compile a kernel module, which (after reboot) will make network counters available in atop. Quite awesome!

simulate slow network

export interface=enx9cebe8349fdc
sudo tc qdisc add dev $interface root netem delay 10000ms
sudo tc qdisc del dev $interface root netem 

source


Prints files count per directory for current directory level:

du -a | cut -d/ -f2 | sort | uniq -c | sort -nr

UDP traffic through SSH tunnel

http://superuser.com/questions/53103/udp-traffic-through-ssh-tunnel https://securesocketfunneling.github.io/ssf/#download

Using iptables to redirect ip address

iptables -t nat -A OUTPUT -d 10.141.0.1 -j DNAT --to-destination 8.8.8.8

http://superuser.com/questions/681705/using-iptables-to-redirect-ip-address

sslh

http://www.rutschle.net/tech/sslh.shtml i.e., http://www.rutschle.net/sslh#using-proxytunnel-with-sslh

WHAT IS IT? sslh accepts connections on specified ports, and forwards them further based on tests performed on the first data packet sent by the remote client.

sed

trigen@zenbook:~> echo "Hello world (y)" | sed 's/(y)/:-)/'
Hello world :-)
trigen@zenbook:~> echo "Hello world (y)" | sed 's/.*\((y)\)/smiley: \1/'
smiley: (y)

trigen@zenbook:~> sed -i.bak 's/H/J/g' test.txt
trigen@zenbook:~> cat test.txt
Jello world
trigen@zenbook:~> cat test.txt.bak
Hello world

check if port is open

nc -z -v -w1 cppse.nl 80

openssl check certificates

openssl x509 -in <INPUT> -text

openssl s_client -showcerts -connect cppse.nl:443 </dev/null
# in case of SNI (combined domains in certificate)
openssl s_client -showcerts -servername cppse.nl -connect www.cppse.nl:443 </dev/null

set boot target graphical or not

source: https://www.systutorials.com/239880/change-systemd-boot-target-linux/

systemctl enable multi-user
systemctl set-default multi-user
systemctl enable graphical
systemctl set-default graphical

commandline json prettify

echo '{"foo": "lorem", "bar": "ipsum"}' | python -m json.tool

sed extract piece of text

openstack server list | grep ...  | sed -n 's/.*\(10\.2[^;]*\);.*/\1/p'

kpartx dd img

kpartx -rav some.dd

readonly add partition devmappings verbose

mount /dev/mapper/loop0p5 /mnt/loop0p5/

umount /mnt/loop0p5

kpartx -d some.dd or kpartx -d /dev/loop0

see losetup -a for available loops

note losetup -D doesn't work for some reason when the partitions were created with kpartx

screen

source: https://unix.stackexchange.com/questions/26743/force-gnu-screen-to-reflow-to-current-terminal-width-on-reattach?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

"after you reattach a ctrl-a F runs the "fit" command to resize the current window. if you reattach using the -A option it should resize all windows when you reattach. "

git

Assuming the hash of the commit you want is c5f567 (source):

git checkout c5f567 -- file1/to/restore file2/to/restore

The git checkout man page gives more information. If you want to revert to the commit before c5f567, append ~1 (works with any number):

git checkout c5f567~1 -- file1/to/restore file2/to/restore

turn .a (thin) archive into "normal" one. (Source)

After some additional research, ar -t can be used to enumerate the object files in an archive, so after that it's just a matter of providing that list to ar as you usually would when creating an archive.

The following script handled this for all of the libraries at once:

for lib in `find -name '*.a'`;
    do ar -t $lib | xargs ar rvs $lib.new && mv -v $lib.new $lib;
done

clipboard fu

There is a difference in clipboards I've noticed, which confuses me bigtime.

Apparently one is the "X clipboard":

echo Hello world | xclip

Whenever pasting doesn't work try the middle mouse button.. Or use xsel instead:

echo Hello world | xsel --clipboard
echo Hello world | xsel -b # same

Some params for xsel:

Selection options
    -p, --primary         Operate on the PRIMARY selection (default)
    -s, --secondary       Operate on the SECONDARY selection
    -b, --clipboard       Operate on the CLIPBOARD selection

resize images in directory to smaller size (recursively)

find . -name '*.jpg' -execdir mogrify -resize 1024x {} \;

installing trusted certificates in system

  • chromium has some way to also import authorities you might also try it
  • get certs with openssl s_client -showcerts -connect 10.2.61.184:8081
  • I put the certs for example 3 different .crt files in /usr/local/share/ca-certificates/
  • do not use .pem or they won't be recognized.
  • you have to be root to put them there.
  • sudo update-ca-certificates should pick them up

git change author for bunch of commits

git rebase -i HEAD~2
mark commits as edit, then use this over and over:

git commit --amend --author="Ray Burgemeestre <ray.burgemeestre@brightcomputing.com>"
git rebase --continue

socat

socat TCP-LISTEN:8081,fork TCP:127.0.0.1:12345

git stuff

TODO: https://stackoverflow.com/a/10216050/1958831

find files less then a day old

find ./ -name '*.meta' -type f -mtime -1;

# then delete
find ./ -name '*.meta' -type f -mtime -1 -exec rm -f {} \;

source: https://sites.google.com/a/kossboss.com/main/linux---find-files-older-or-younger-then-certain-days---delete-them-too-if-you-want

get rid of rpm without dependencies example

source: https://ma.ttias.be/removing-a-package-without-its-dependencies-in-centos-or-rhel/

# rpm -qa | grep "php-sqlite2"
php-sqlite2-5.1.6-200705230937
# rpm -e --nodeps "php-sqlite2-5.1.6-200705230937"

determine octal representation of permissions

Sometimes it's annoying, to reverse engineering rwx -> 7, r-x -> 5 and r-x -> 5. For example, I misread the group part for -xr the first time.

shell# ls -althrst /var/lib|grep etcd
0 drwxr-xr-x  2 etcd          etcd             6 Nov  5 14:19 etcd

Thanks to google and stackoverflow (https://askubuntu.com/questions/152001/how-can-i-get-octal-file-permissions-from-command-line):

shell# stat -c "%a %n"  /var/lib/etcd/
755 /var/lib/etcd/

Much easier!

print dates for bunch of dirs

cat /tmp/dirs | xargs -n 1 stat -c '%y %n'

tmux

sometimes you have some disconnected session that still forces some pane to be non-full size. you can force a full resize again using: ctrl+b, :detach-client -a.

docker

Running an X application in docker I usually google for the right parameters, but sometimes they forget the X authority file. Now I use this:

#!/bin/bash
xhost +
docker run --name=foobar --privileged -it -v $PWD/../:/articlemanager --workdir /articlemanager \
       -e DISPLAY \
       -v /tmp/.X11-unix:/tmp/.X11-unix \
       -v $HOME/.Xauthority:/root/.Xauthority \
       --net=host \
    foobar:latest xeyes

If you want to do it in a more correct/secure way, I found this blog post: http://wiki.ros.org/docker/Tutorials/GUI

Also found some useful info here: https://stackoverflow.com/a/44434831/1958831 (In my above example the net=host was needed too, but it's annoying port 80 is already in use for example.)

bash if statements and defaults

There are some interesting snippets here that I need to extract:

https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash

add user to group

usermod -a -G docker ray

if you are added to a group, you can do newgrp docker or login again.

Cheatsheets Comments (0)


Leave a Reply

Comment may not be visible immediately, because I process everything manually.**

**) I plan to automate this.., but it's on my ToDo since for ever..


Author:
Ray Burgemeestre
february 23th, 1984

Topics:
C++, Linux, Webdev

Other interests:
Music, Art, Zen