Some stuff I keep here as a cheetsheat for myself

mass replace recursively

NOTE: do not run in case you have .git directory, it can corrupt it.

Source: https://stackoverflow.com/a/1583282/1958831

mv .git /tmp/git_tmp
find . -type f -print0 | xargs -0 sed -i 's/subdomainA\.example\.com/subdomainB.example.com/g'
mv /tmp/git_tmp .git 

return value

$? return last exit code

buffer control with stdbuf

time make -j24 "$@"  2>&1 | stdbuf -i0 -o0 -e0 tee ./mm.log

sets buffer to zero, everything is piped immediately, bad for throughput, good for latency

< <(command)

gdb on program that works on stream, with mkfifo to make this a combined example (you can issue the bash command instead of the tail -f in this case)

mkfifo /tmp/foo
bash tail_all.sh > /tmp/foo
gdb <application>
run <program args> < <(tail -f /tmp/foo)

This syntax is also useful for stuff like:

while read -d ':' line; do
    echo "$line"
done < <(hbase classpath)

A pipe | creates a new subshell, this is also a nice way to avoid this (subshells cannot modify variables of their parent)!

EDIT: This is called process substitution, see http://tldp.org/LDP/abs/html/process-sub.html

echoing multiple lines to a file

cat <<EOT >> greetings.txt
line 1
line 2
EOT

matching

if [[ $foo =~ foo ]]; then
    ;
done

Note that you DO NOT use asterisks like *foo*!

processing stdin

A pattern I use quite a bit is the following.

function func
{
    while read line; do
        echo $line
    done < /dev/stdin
}
cat foo.txt | func

subshells / pids

Sometimes subshells are useful.

echo mypid = $!

( sleep 10; ) &
typeset pid=$!
sleep 1
kill $pid # kill if still running..

Note that you can use exit in a subshell without destroying the parent.

trap CTRL+C

function myfunc
{
    echo Do some cleanup here..
    exit 0
}
trap "myfunc" 2
print "Press CTRL+C to abort 10 seconds of waiting.."
sleep 10

namerefs

First learned about this in kornshell, but seems to work for bash as well.

function func {
    local -n test_ref=$1
    test_ref="Hello world"
}

func test # note omitted $-sign

echo $test

If you want kornshell compatibility use typeset -n instead of local -n.

The same example, but with an array, you have to first explicitly declare the array before passing it to func.

declare -A array
function func {
    local -n array_ref=$1
    array_ref=([one]=aaa [two]=bbb [three]=ccc)
}
func array

(If you want kornshell compatibility again, use typeset -A instead of declare -A (and typeset -n for local -n)).

arrays

Continuing the previous paragraph, some useful examples regarding arrays.

echo ${array[@]}  # aaa bbb ccc
echo ${!array[@]} # one two three
echo ${#array[*]} # 3

for key in ${!array[@]}; do
    local value=${array[$key]}
    echo $key = $value # one = aaa, ... 
done

Instead of initializing with the array=([foo]=bar [baz]=ban) syntax, assignment syntax is array[foo]=bar.

bash substrings

typeset hostname=$(hostname -s)
hostname=${hostname:0:10}  # first 10 chars..

NFS caching

Trick to use cachefilesd on NFS mounts. Useful if you work remote on a mounted NFS share. btrfs is not supported, so I create an ext4 mount instead like this:

dd if=/dev/zero of=/tmp/fscache.txt bs=1M count=4000
# be careful with choosing /dev/loopN
losetup /dev/loop10 /tmp/fscache.txt
mkfs.ext4 /dev/loop10
mount /dev/loop10 /var/cache/fscache

Very useful other commands can be found here: http://www.cyberciti.biz/faq/centos-redhat-install-configure-cachefilesd-for-nfs/ (i.e.: cat /proc/fs/nfsfs/*)

Benchmark I/O

start iperf -s server, then iperf -c 127.0.0.1.

Show Network I/O per process

atop can do this, which comes provided by your package manager probably, but chances are you need to compile it yourself for a newer version. The new version makes it possible to compile a kernel module, which (after reboot) will make network counters available in atop. Quite awesome!

simulate slow network

export interface=enx9cebe8349fdc
sudo tc qdisc add dev $interface root netem delay 10000ms
sudo tc qdisc del dev $interface root netem 

source

---

Prints files count per directory for current directory level:

du -a | cut -d/ -f2 | sort | uniq -c | sort -nr

UDP traffic through SSH tunnel

http://superuser.com/questions/53103/udp-traffic-through-ssh-tunnel https://securesocketfunneling.github.io/ssf/#download

Using iptables to redirect ip address

iptables -t nat -A OUTPUT -d 10.141.0.1 -j DNAT --to-destination 8.8.8.8

http://superuser.com/questions/681705/using-iptables-to-redirect-ip-address

sslh

http://www.rutschle.net/tech/sslh.shtml i.e., http://www.rutschle.net/sslh#using-proxytunnel-with-sslh

WHAT IS IT? sslh accepts connections on specified ports, and forwards them further based on tests performed on the first data packet sent by the remote client.

sed

trigen@zenbook:~> echo "Hello world (y)" | sed 's/(y)/:-)/'
Hello world :-)
trigen@zenbook:~> echo "Hello world (y)" | sed 's/.*\((y)\)/smiley: \1/'
smiley: (y)

trigen@zenbook:~> sed -i.bak 's/H/J/g' test.txt
trigen@zenbook:~> cat test.txt
Jello world
trigen@zenbook:~> cat test.txt.bak
Hello world

Another nice example

sed -i '/upload_max_filesize/s/=.*/= 100M/' /etc/php/7.2/apache2/php.ini

(-n don't echo all (double check if this is true), /p to print, \1 to print between ()'s (escaped))

fdisk -l |& sed -n 's/^Disk \/\([^:]*\).*$/\/\1/p'
fdisk -l |& sed -n 's/^Disk \/\([^:]*\).*$/\/\1/p' | xargs -n 1 -I{} /bin/sh -c "echo {}; smartctl -H {}"

Nice resource for sed stuff: https://www.grymoire.com/Unix/Sed.html#uh-9

check if port is open

nc -z -v -w1 cppse.nl 80

openssl check certificates

openssl x509 -in <INPUT> -text

openssl s_client -showcerts -connect cppse.nl:443 </dev/null
# in case of SNI (combined domains in certificate)
openssl s_client -showcerts -servername cppse.nl -connect www.cppse.nl:443 </dev/null

# grep for alt subjects
openssl s_client -connect 10.141.0.1:10443 -servername 10.141.0.1 < /dev/null 2>/dev/null | openssl x509 -noout -text | grep -A1 'Subject Alternative Name'

# or..
openssl s_client -showcerts -connect 10.3.195.44:10443 </dev/null | openssl x509 -noout -ext subjectAltName

set boot target graphical or not

source: https://www.systutorials.com/239880/change-systemd-boot-target-linux/

systemctl enable multi-user
systemctl set-default multi-user
systemctl enable graphical
systemctl set-default graphical

systemctl

systemctl list-units --type target --state active

commandline json prettify

echo '{"foo": "lorem", "bar": "ipsum"}' | python -m json.tool

sed extract piece of text

openstack server list | grep ...  | sed -n 's/.*\(10\.2[^;]*\);.*/\1/p'

kpartx dd img

kpartx -rav some.dd

readonly add partition devmappings verbose

mount /dev/mapper/loop0p5 /mnt/loop0p5/

umount /mnt/loop0p5

kpartx -d some.dd or kpartx -d /dev/loop0

see losetup -a for available loops

note losetup -D doesn't work for some reason when the partitions were created with kpartx

screen

source: https://unix.stackexchange.com/questions/26743/force-gnu-screen-to-reflow-to-current-terminal-width-on-reattach?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

"after you reattach a ctrl-a F runs the "fit" command to resize the current window. if you reattach using the -A option it should resize all windows when you reattach. "

git

Assuming the hash of the commit you want is c5f567 (source):

git checkout c5f567 -- file1/to/restore file2/to/restore

The git checkout man page gives more information. If you want to revert to the commit before c5f567, append ~1 (works with any number):

git checkout c5f567~1 -- file1/to/restore file2/to/restore

git find when file was deleted

Taken from source:

git log --full-history -- [file path]

git log --full-history -1 -- [file path]

turn .a (thin) archive into "normal" one. (Source)

After some additional research, ar -t can be used to enumerate the object files in an archive, so after that it's just a matter of providing that list to ar as you usually would when creating an archive.

The following script handled this for all of the libraries at once:

for lib in `find -name '*.a'`;
    do ar -t $lib | xargs ar rvs $lib.new && mv -v $lib.new $lib;
done

clipboard fu

There is a difference in clipboards I've noticed, which confuses me bigtime.

Apparently one is the "X clipboard":

echo Hello world | xclip

Whenever pasting doesn't work try the middle mouse button.. Or use xsel instead:

echo Hello world | xsel --clipboard
echo Hello world | xsel -b # same

Some params for xsel:

Selection options
    -p, --primary         Operate on the PRIMARY selection (default)
    -s, --secondary       Operate on the SECONDARY selection
    -b, --clipboard       Operate on the CLIPBOARD selection

resize images in directory to smaller size (recursively)

find . -name '*.jpg' -execdir mogrify -resize 1024x {} \;

installing trusted certificates in system

• chromium has some way to also import authorities you might also try it
• get certs with openssl s_client -showcerts -connect 10.2.61.184:8081
• I put the certs for example 3 different .crt files in /usr/local/share/ca-certificates/
• do not use .pem or they won't be recognized.
• you have to be root to put them there.
• sudo update-ca-certificates should pick them up

git change author for bunch of commits

git rebase -i HEAD~2
mark commits as edit, then use this over and over:

git commit --amend --author="Ray Burgemeestre <ray.burgemeestre@brightcomputing.com>"
git rebase --continue

socat

socat TCP-LISTEN:8081,fork TCP:127.0.0.1:12345

git stuff

TODO: https://stackoverflow.com/a/10216050/1958831

find files less then a day old

find ./ -name '*.meta' -type f -mtime -1;

# then delete
find ./ -name '*.meta' -type f -mtime -1 -exec rm -f {} \;

find files for specific day

find ./var -printf "%p %TY-%Tm-%Td %TH:%TM:%TS %Tz\n" | grep 2022-01-01

source: https://sites.google.com/a/kossboss.com/main/linux---find-files-older-or-younger-then-certain-days---delete-them-too-if-you-want

get rid of rpm without dependencies example

source: https://ma.ttias.be/removing-a-package-without-its-dependencies-in-centos-or-rhel/

# rpm -qa | grep "php-sqlite2"
php-sqlite2-5.1.6-200705230937
# rpm -e --nodeps "php-sqlite2-5.1.6-200705230937"

determine octal representation of permissions

Sometimes it's annoying, to reverse engineering rwx -> 7, r-x -> 5 and r-x -> 5. For example, I misread the group part for -xr the first time.

shell# ls -althrst /var/lib|grep etcd
0 drwxr-xr-x  2 etcd          etcd             6 Nov  5 14:19 etcd

Thanks to google and stackoverflow (https://askubuntu.com/questions/152001/how-can-i-get-octal-file-permissions-from-command-line):

shell# stat -c "%a %n"  /var/lib/etcd/
755 /var/lib/etcd/

Much easier!

print dates for bunch of dirs

cat /tmp/dirs | xargs -n 1 stat -c '%y %n'

%x     Time of last access
%y     Time of last modification
%z     Time of last change

tmux

sometimes you have some disconnected session that still forces some pane to be non-full size. you can force a full resize again using: ctrl+b, :detach-client -a.

docker

Running an X application in docker I usually google for the right parameters, but sometimes they forget the X authority file. Now I use this:

#!/bin/bash
xhost +
docker run --name=foobar --privileged -it -v $PWD/../:/articlemanager --workdir /articlemanager \
       -e DISPLAY \
       -v /tmp/.X11-unix:/tmp/.X11-unix \
       -v $HOME/.Xauthority:/root/.Xauthority \
       --net=host \
    foobar:latest xeyes

If you want to do it in a more correct/secure way, I found this blog post: http://wiki.ros.org/docker/Tutorials/GUI

Also found some useful info here: https://stackoverflow.com/a/44434831/1958831 (In my above example the net=host was needed too, but it's annoying port 80 is already in use for example.)

bash if statements and defaults

There are some interesting snippets here that I need to extract:

https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash

add user to group

usermod -a -G docker ray

if you are added to a group, you can do newgrp docker or login again.

Assigning extra IP address to interface

sudo ip addr add 192.168.2.150/24 dev enp0s3
ip addr

avahi / mDNS<

Available on most Linux distro's, Raspberry PI's and Arduino devices.

It's a convenient way to allow devices to find each other on a network Now I let the raspberry pi's here publish some service named "_ray" so I can easily find them:

trigen@zenbook:~> avahi-browse -d local _ray._tcp --resolve -t
+ wlp3s0 IPv6 dashboardpi                                   _ray._tcp            local
+ wlp3s0 IPv4 domoticapi                                    _ray._tcp            local
+ wlp3s0 IPv4 dashboardpi                                   _ray._tcp            local
= wlp3s0 IPv6 dashboardpi                                   _ray._tcp            local
   hostname = [dashboardpi.local]
   address = [fe80::90c3:716c:7d24:ccc9]
   port = [80]
   txt = []
= wlp3s0 IPv4 domoticapi                                    _ray._tcp            local
   hostname = [domoticapi.local]
   address = [192.168.115.1]
   port = [8083]
   txt = []
= wlp3s0 IPv4 dashboardpi                                   _ray._tcp            local
   hostname = [dashboardpi.local]
   address = [192.168.2.126]
   port = [80]
   txt = []

Then:

pi@domoticapi:~ $ cat /etc/avahi/services/ray.service 
<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
    <type>_ray._tcp</type>
    <port>8083</port>
  </service>
</service-group>

and restart the avahi-daemon

NAS disk failure

Source: https://serverfault.com/questions/556667/how-do-i-figure-out-which-drive-is-failing

for x in /sys/block/sd*
do
    dev=$(basename $x)
    host=$(ls -l $x | egrep -o "host[0-9]+")
    target=$(ls -l $x | egrep -o "target[0-9:]*")
    a=$(cat /sys/class/scsi_host/$host/unique_id)
    a2=$(echo $target | egrep -o "[0-9]:[0-9]$" | sed 's/://')
    serial=$(hdparm -I /dev/$dev | grep "Serial Number" | sed 's/^[ \t]*//')
    echo -e "$dev \t ata$a.$a2 \t $serial"
done

in my case ata5 was throwing errors in dmesg -w,

output:

sda      ata1.00     Serial Number:      S12RNEAD500626J
sdb      ata2.00     Serial Number:      W4Z1GZLP
sdc      ata3.00     Serial Number:      R6GGJN0Y
sdd      ata4.00     Serial Number:      WD-WMC4M0F6RXR8
^Csde    ata5.00
sdf      ata6.00     Serial Number:      ZA19BLKY
sdg      ata7.00     Serial Number:      ZA19Z3D9
sdh      ata8.00     Serial Number:      45D0LWNAS

caveat: this command blocks if the hdd is broken, so in that case you want to make an async version of this command. but in my case I was lucky enough that control+c already told me sde was the bad apple.

vim insert non breaking space

 

In Vim, you can insert it with Ctrl-v x a 0

git stuff

git diff HASH^! diff against it's parent, sometimes reversed for me I then simply use

git diff HASH~1 HASH

git reset --soft HEAD~1 # also creates the reference ORIG_HEAD git commit --reuse-message=ORIG_HEAD # can be used to re-use the message

ldd tree

package pax-utils

lddtree $BINARY

test mic input real time in linux

Source:

https://askubuntu.com/questions/123798/how-to-hear-my-voice-in-speakers-with-a-mic

I'm using pavucontrol

pactl load-module module-loopback latency_msec=1

To unload loopback

pactl unload-module module-loopback

netgear r7000

nvram set wl_regdomain="EUROPE" nvram set wl_country_code="EU" nvram commit

to enable channel 13 and 14

image to pdf

https://unix.stackexchange.com/questions/42856/how-can-i-convert-a-png-to-a-pdf-in-high-quality-so-its-not-blurry-or-fuzzy

img2pdf -o sample.pdf sample.jp2

print queue view

lpstat -R

#

foo.py writing to fd 3, ./foo.py 3>&1

or beforehand:

exec 3>&2

lsof open deleted files

lsof +L1

diff rsync

Using rsync to merge two dirs with changes in both dirs. Check if rsync won't overwrite anything:

# dry run of rsync, note no --delete
rsync --dry-run -raPv owncloud_deskmini/ owncloud/ |& tee out.log
# dry run of rsync with "u" flag that ignores updates
rsync --dry-run -rauPv owncloud_deskmini/ owncloud/ |& tee out2.log
diff out.log out2.log 

If files show up in the diff those might be overwritten

gdb

coredumpctl -1 debug
/var/lib/systemd/coredump

bash

foo="12345"

# keep foo only if it contains a valid integer, otherwise truncate
[[ $foo =~ ^-?[0-9]+$ ]] || foo=""

heredoc

cat << EOF > file.yaml
Contents
EOF

The above has a flaw, it will replace for example $1 stuff, but if you put the EOF in single quotes, it will preserve it:

cat << 'EOF' > script.sh
Contents $1
EOF

This is also possible:

if [[ -z $1 ]]; then
    cat <<- EOF > file.yaml
    Contents
    EOF   # as long as EOF is indented with TAB character(s)
fi

https://www.gnu.org/software/bash/manual/bash.html#Here-Documents

dates

trigen@xps:~/projects/jirahours[master]> date +"%Y-%m-%d %T +0200 CEST"
2021-06-01 14:04:56 +0200 CEST
root@rb-b91-s15sp2-03-21-a:~# date +"%Y%m%d%H%M%S"
20220321135439

sudo dpkg-reconfigure tzdata to fix the timezone.

source: https://help.ubuntu.com/community/UbuntuTime

gdb

gdb -p $(pidof myprogram) < <(printf "yes\nset pagination off\nset logging on\nbt\nthread apply all bt\ninfo threads\nquit\n")

prevent X from turning off screen/sleeping

xset dpms 0 0 0 && xset s noblank  && xset s off

mysql

mysqldump --skip-extended-insert --default-character-set=utf8mb4 -u root -pPASSWORD wordpress -r ./wordpress.sql
mysql --default-character-set=utf8mb4 --protocol=TCP --host 127.0.0.1 -uroot -pPASSWORD --database=wordpress  < wordpress.sql

awk

comma-separate multiple lines

$ awk -v ORS=, '{print $2}' data.txt | sed 's/,$//'
+12.0,+15.5,+9.0,+13.5