Neither one nor Many

EBPF Flamegraphs C++ Ubuntu 20.04

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 10 Feb 2022 00:00:00 +0000

February 10 2022

EBPF Flamegraphs C++ Ubuntu 20.04

Some of the stuff I'm posting on my 'blog' might be better categorized as a knowledge base articles, or more simply 'notes'.. In any case, this might be one more such posts , just some caveats I ran into setting stuff up on my Ubuntu Server 20.04 LTS.

Flame Graphs

Invented by Brendan Gregg, who is someone that I honestly greatly admire. He is behind a lot of amazing presentations/talks, online content and tools w/r/t performance. Recently, when I got very interested in EBPF, stumbled upon his work again.

I've used Flame Graphs in the past with perf, and so I was curious to try it with ebpf this time. Flame Graphs can give very useful insights into potential bottlenecks. Below is one that I've created as an example, but will refer to Brendan's website if you want to dive into them more, and see more interesting examples.

Click here to open the SVG in interactive mode (warning: 12 MiB, loading takes a while!)

As I tried to make all the tools work, I discovered the Ubuntu packages are a bit old, and I ran into a few issues. The steps that worked for me are based on this link, section 4.2.:

1) Install prerequisites

sudo apt install -y bison build-essential cmake flex git libedit-dev \
  libllvm7 llvm-7-dev libclang-7-dev python zlib1g-dev libelf-dev libfl-dev python3-distutils

Remove the old bpfcc-tools package, if you've installed them before (gave me nothing but issues, such as python scripts raising errors, that have already been fixed upstream). We will fetch the latest version from github instead.

apt remove -y bpfcc-tools

2) Install bcc tools (bpfcc-tools)

git clone https://github.com/iovisor/bcc
cd bcc

Then execute the following:

mkdir -p build; cd build
export LLVM_ROOT=/usr/lib/llvm-7
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd

Note the export LLVM_ROOT=/usr/lib/llvm-7, this was critical in my case, since I had newer versions:

trigen@ideapad:~> find /usr/lib -name 'llvm-*' -type d
/usr/lib/llvm-7
/usr/lib/llvm-10
/usr/lib/llvm-12

CMake would pick up the latest llvm-12, and that would cause compilation errors. See: https://github.com/iovisor/bcc/issues/3601

3) Do the profiling with EBPF

Step 1: I would start my executable program starcry in a terminal, have it render a bunch of stuff.
Step 2: Then sample for 60 seconds on the specific pid as root, see below.

export PYTHONPATH=$(dirname `find /usr/lib -name bcc | grep dist-packages`):$PYTHONPATH
/usr/share/bcc/tools/profile  # see if it produces meaningful output

The PYTHONPATH had to be exported correctly first (on my system in any case) or the profile tool would raise a Python error. THen do the actual sampling with:

sudo python3 /usr/share/bcc/tools/profile  -F 99 -adf 10 -p $(pidof starcry) > /path/to/out.profile-folded

4) Generate the Flame Graph

git clone https://github.com/brendangregg/FlameGraph
cd FlameGraph
./flamegraph.pl --colors=java /path/to/out.profile-folded > profile.svg

That should be it!

Example

From the example PNG included in this blog post:

trigen@ideapad:~/projects/FlameGraph[master]> export PYTHONPATH=$(dirname `find /usr/lib -name bcc | grep dist-packages`):$PYTHONPATH
trigen@ideapad:~/projects/FlameGraph[master]> sudo python3 /usr/share/bcc/tools/profile -U -F 99 -adf 10 -p $(pidof starcry) > out.profile-folded
WARNING: 17 stack traces could not be displayed. Consider increasing --stack-storage-size.
trigen@ideapad:~/projects/FlameGraph[master]> ./flamegraph.pl --colors=java ./out.profile-folded > profile.svg

Settings for Tide graph Casio G-Shock GLX 150

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 04 Apr 2020 00:00:00 +0000

April 4 2020

Settings for Tide graph Casio G-Shock GLX 150

lunitidal interval

In ADJUST Settings set the right Longitude. I.e., if Lat/Lon for my city is: 52.601234, 4.700493 The only relevant part is the longitude, +4 or +5 in this case. I configured 4 for longitude, and E for EAST, negative numbers should be WEST.

Then lookup whatever beach you are interested in, the closest to me is Egmond Aan Zee. The manual contains a bunch of them, but this list is not very complete, or useful at all IMO. And I learned in The Netherlands these times differ greatly per beach.

So, better find the High and Low tide times yourself, in my case for Egmond Aan Zee:

https://egmondaanzee.org/nl/getijden-eb-vloed.php

This lunitidal interval (HH:MM) is called a "havengetal" or "haventijd" in my language. And it's dutch definition is:

"De haventijd is het tijdsverschil tussen de hoogste waterstand en de doorgang van zon of maan door de meridiaan, voor een gegeven plaats."

Translated: the difference between the highest tide and the passing of the sun or moon through the meridian, for a given place.

Today is the 4th of april 2020 with the highest tide at 13:44 (84 cm).

According to this site the moon passes the meridian at exactly 23:00 for my location today.

https://www.timeanddate.com/moon/@2754516

We need the difference:

From 23:00 to 13:44 is: 14:44.

This results in settings for my casio: 4 LONG E + INT 14:44.

testing

Then test in the tide mode for different dates, and it should work! Personally I noticed that low tide is not completely synchronized, it's one "bar" later with the wave on the watch. I suspect that is because the actual graph is not a perfect sine wave, but looks a little skewed. I guess this may vary per beach.

Deploying owncloud via Helm on Kubernetes

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 19 Oct 2018 00:00:00 +0000

October 19 2018

Deploying owncloud via Helm on Kubernetes

For this tutorial I'm assuming Kubernetes with Helm + Ingress is already deployed. If not, I still included the commands I used near the end of this article.

OwnCloud

My NAS is running at home with Rockstor, and I'm using RAID10 with btrfs. Rockstor has (docker) apps support with the feature called Rock-on's, they also include OwnCloud, but after updating and some other issues with Rockstor at some point my deployment broke. This frustrated me so I've decided to switch to Kubernetes instead.

I use my own cloud (no pun intended) as an alternative over using services owned by Google/Amazon/Apple. When you plan to do the same, just make sure to also make proper backups.

Deploy OwnCloud with Helm

Following the instructions; copy their default values.yaml (from here). Tweak all the values. It seems important to define a hostname! (If you try accessing the service later via IP address, the webinterface will not accept this.)

helm install --name my-owncloud -f owncloud.yaml stable/owncloud --set rbac.create=true

Notes: owncloud.yaml is my values.yaml, and I expect the rbac.create=true not to be needed but I used it anyway it was left over when copy & pasting another command.. For convenience you can download my owncloud.yaml.

Owncloud will require some storage.

In my case I made a btrfs share named /mnt2/NAS/kubeowncloudstorage.

Then created three folders inside it:

mkdir -p /mnt2/NAS/kubeowncloudstorage/data
mkdir -p /mnt2/NAS/kubeowncloudstorage/mariadb
mkdir -p /mnt2/NAS/kubeowncloudstorage/apache

Set the right permissions for these folders, owncloud will write as user id(1).

chown 1:1 /mnt2/NAS/kubeowncloudstorage -R

Then apply the following yaml (kubectl apply -f kube_owncloud_storage.yaml):

nas:/root # cat kube_owncloud_storage.yaml 
kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-owncloud-storage-data
  labels:
    type: local
spec:
  capacity:
    storage: 3072Gi
  storageClassName: owncloud-storage-data
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt2/NAS/kubeowncloudstorage/data
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-owncloud-storage-mariadb
  labels:
    type: local
spec:
  capacity:
    storage: 8Gi
  storageClassName: owncloud-storage-mariadb
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt2/NAS/kubeowncloudstorage/mariadb
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-owncloud-storage-apache
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  storageClassName: owncloud-storage-apache
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt2/NAS/kubeowncloudstorage/apache

If you redeploy Kubernetes and/or the system in general, I forgot when exactly but a PersistentVolume may end up in a state that prevents PersistentVolumeClaim's to not bind to the Volumes. There was a trick to force it to bind, IIRC kubectl edit pv kube-owncloud-storage-data and you can remove the reference it has to an existing PVC. But it was a few weeks ago I experimented with this so sorry I don't remember the details. Only now I stumbled upon my notes and decided to wrap it up in a blog post.

Setup ingress (beware of caveat!)

nas:/root # cat owncloud_ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/proxy-body-size: 500m
    nginx.ingress.kubernetes.io/proxy-body-size: 500m
  name: owncloud
  namespace: default
spec:
  rules:
    - host: ******DOMAIN NAME*******
      http:
    paths:
      - backend:
          serviceName: my-owncloud-owncloud
          servicePort: 80
        path: /

Take a careful look at these two options in the annotations:

ingress.kubernetes.io/proxy-body-size: 500m
nginx.ingress.kubernetes.io/proxy-body-size: 500m

They took me two hours of debugging, owncloud was throwing errors 413 Request Entity Too Large when syncing some larger video files from my phone to owncloud. Thinking this must be an issue inside owncloud I experimented with lots of parameters, fixes for php, apache, etc. Then realized it could be the Ingress in Kubernetes. The above example makes sure it doesn't block uploads up to half a gigabyte.

DONE!

The end result should look something like this in Kubernetes:

nas:/root # kubectl get all
NAME                                                          READY   STATUS    RESTARTS   AGE
pod/my-nginx-nginx-ingress-controller-664f4547d8-vjgkt        1/1     Running   0          16d
pod/my-nginx-nginx-ingress-default-backend-5bcb65f5f4-qrwcd   1/1     Running   0          16d
pod/my-owncloud-mariadb-0                                     1/1     Running   0          16d
pod/my-owncloud-owncloud-6cddfdc8f4-hmrh5                     1/1     Running   2          16d

NAME                                             TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
service/kubernetes                               ClusterIP      10.96.0.1       <none>          443/TCP                      16d
service/my-nginx-nginx-ingress-controller        LoadBalancer   10.103.57.37    192.168.2.122   80:32030/TCP,443:30453/TCP   16d
service/my-nginx-nginx-ingress-default-backend   ClusterIP      10.101.16.224   <none>          80/TCP                       16d
service/my-owncloud-mariadb                      ClusterIP      10.104.48.71    <none>          3306/TCP                     16d
service/my-owncloud-owncloud                     LoadBalancer   10.102.95.4     <pending>       80:32287/TCP                 16d

NAME                                                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-nginx-nginx-ingress-controller        1         1         1            1           16d
deployment.apps/my-nginx-nginx-ingress-default-backend   1         1         1            1           16d
deployment.apps/my-owncloud-owncloud                     1         1         1            1           16d

NAME                                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/my-nginx-nginx-ingress-controller-664f4547d8        1         1         1       16d
replicaset.apps/my-nginx-nginx-ingress-default-backend-5bcb65f5f4   1         1         1       16d
replicaset.apps/my-owncloud-owncloud-6cddfdc8f4                     1         1         1       16d

NAME                                   DESIRED   CURRENT   AGE
statefulset.apps/my-owncloud-mariadb   1         1         16d
nas:/root # kubectl get ingress
NAME       HOSTS               ADDRESS   PORTS   AGE
owncloud   *****************             80      16d
nas:/root # kubectl get pv
NAME                            CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                   STORAGECLASS               REASON   AGE
kube-owncloud-storage-apache    1Gi        RWO            Retain           Bound    default/my-owncloud-owncloud-apache     owncloud-storage-apache             16d
kube-owncloud-storage-data      3Ti        RWO            Retain           Bound    default/my-owncloud-owncloud-owncloud   owncloud-storage-data               16d
kube-owncloud-storage-mariadb   8Gi        RWO            Retain           Bound    default/data-my-owncloud-mariadb-0      owncloud-storage-mariadb            16d
nas:/root # kubectl get pvc
NAME                            STATUS   VOLUME                          CAPACITY   ACCESS MODES   STORAGECLASS               AGE
data-my-owncloud-mariadb-0      Bound    kube-owncloud-storage-mariadb   8Gi        RWO            owncloud-storage-mariadb   16d
my-owncloud-owncloud-apache     Bound    kube-owncloud-storage-apache    1Gi        RWO            owncloud-storage-apache    16d
my-owncloud-owncloud-owncloud   Bound    kube-owncloud-storage-data      3Ti        RWO            owncloud-storage-data      16d

Deploying Kube on a single node machine notes

Just in case you are also attempting to install Kubernetes for the first time, a reference of the commands used in my setup. First I followed the official docs to deploy kubeadm,kubelet etc. See here.

My init looked like this:

kubeadm init --pod-network-cidr=192.168.0.0/16

At this point you may get some errors, and you have to fix the errors, maybe even kubeadm reset and then retry. Until I was okay with the remaining errors, I proceeded with:

kubeadm init --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors=all

# these steps will be recommended from above command:
mkdir -p $HOME/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# I chose calico for networking
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

# Then after a while (maybe check if kubelet etc. come up correctly, try "kubectl get no")
# Make sure the master node is not excluded for running pods.
kubectl taint nodes --all node-role.kubernetes.io/master-

# I also executed this patch, but I think it's not needed anymore, it was still in my helper script
kubectl -n kube-system get deployment coredns -o yaml |   sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' |   kubectl apply -f -

# Then I looked up the kubelet service file with `systemctl cat kubelet` and edited:
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
# added this to above file, the --resolv-conf: 
#
#ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --resolv-conf=/etc/resolv.conf
#
#ALSO: I edited /etc/resolv.conf, I removed the ipv6 nameserver entry, and added 8.8.8.8 as per https://hk.saowen.com/a/e6cffc1e02c2b4643bdd525ff9e8e4cfb49a4790062508dca478c0c8a0361b5a
systemctl daemon-reload
systemctl restart kubelet

kubectl get pod -n kube-system
kubectl delete pod coredns-68fb79bcf6-9zdtz -n kube-system
kubectl delete pod coredns-68fb79bcf6-t7vsm -n kube-system
kubectl get pod -n kube-system -o wide

Solution for the last bit I got from here. However this may have been a random issue that I just ran into, because on different servers I don't recall I had to the steps regarding coredns.

Possible commands

helm reset --force
helm init --upgrade --service-account tiller

# don't remember if these two commands were still necessary
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

Unrelated notes about Owncloud itself

Links for solutions for problems that I ran into at some point in time:

https://central.owncloud.org/t/file-is-locked-how-to-unlock/985

Links that eventually pointed me in the right direction for the 413 Request Entity Too Large error.

https://forum.owncloud.org/viewtopic.php?t=23199
https://www.keycdn.com/support/413-request-entity-too-large
https://stackoverflow.com/questions/18740419/how-to-set-allowoverride-all
https://github.com/nginxinc/kubernetes-ingress/issues/21

How to print an endlessly folding card

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 21 Jun 2018 00:00:00 +0000

June 21 2018

How to print an endlessly folding card

I remember having something like this as a child: http://www.endlessfoldingcard.com/endless-folding-card-p-768-918.html. It was not that one but something similar, a flyer for some stupid product. I was fascinated by it and could keep folding it for hours. It was nice made out of some very strong paper/cardboard, unfortunately I probably got rid of it at some point.

It took me a long time to find it again, every now and then I would try to look it up on Google (with years in between), unable to find it. Until I had a moment of clarity and tried the additional keyword "Endless", and finally found something that I remembered.

Figuring out how it works

All the YouTube videos I found would basically fold the thing together, and then decorate the card afterward, however I wanted to print it beforehand in such a way that it would turn out nicely when folded. To be honest this one does attempt to explain some of the layouting, but it wasn't clear enough for me. This is another video that shows how to fold it clearly.

There are a few things that you will notice when you fold one, some parts of the card stay constant for example, and not all views have corners. Anyway I decided to treat it as a grid of tiles. I just printed 2 two-sided pieces of paper with unique numbers for each tile. Then I deciphered which tiles end up where after folding, and which ones are rotated 180 degrees. See madness below.

See the cardboard in action here: VID_20180531_011204.mp4

Design and print the card!

Designing the card in something like gimp is a lot of work of course, and it would be hell if you need to prepare it manually for printing. Luckily I wrote a C++ program that uses the very cool Selene Image library, which I learned about via the Dutch C++ User Group , Michael (the author) gave an awesome lightning talk about it. It was good timing because a few days after that I needed to write this program.

I didn't put this code on github, but here is the source code:

// This file is using the `Selene` library.
// Copyright 2017-2018 Michael Hofmann (https://github.com/kmhofmann).
// Distributed under MIT license. See accompanying LICENSE file in the top-level directory.

#include <selene/img/ImageAccess.hpp>
#include <selene/img/ImageDataToImage.hpp>
#include <selene/img/ImageToImageData.hpp>
#include <selene/img_io/IO.hpp>
#include <selene/img_ops/Algorithms.hpp>
#include <selene/img_ops/Transformations.hpp>
#include <selene/io/FileReader.hpp>
#include <selene/io/FileWriter.hpp>

#include <cassert>
#include <iostream>
#include <string>

#include <boost/filesystem.hpp>

using namespace sln::literals;

struct tile
{
  size_t index;
  size_t folded_index;
  bool flipped;
};

struct tile_index
{
  sln::PixelIndex x;
  sln::PixelIndex y;
};

// clang-format off
std::array<tile, 64> tiles {{
  /**
   * There are basically four sides (a,b,c,d), which I've divided into 16 tiles per side.
   * Side a & b will be printed on one piece of paper, double sided, as will sides c & d.
   * In total there are 64 tiles, and this table basically combines two states for each tile.
   *
   * When designing you create four "views" for each side, which is the desired outcome,
   * or the first state of each tile. Like:
   * {1, 2, 3, ..., 17, 18, 19, ..., 62, 53, 64.} (nothing will be rotated)
   *
   * But once you print them and glue them together, and you fold your way through the card,
   * the tiles appear in different order, and sometimes even rotated 180 degrees.
   * This is the second state of the tiles. Something like:
   * {1, 2, 3, ..., 19, 50, 51, ..., 43+rotate, 42+rotate, 43+rotate}
   *
   * Both states are combined in this table, for each tile the following fields:
   * - index: the tile number (for the first state)
   * - folded index: the tile number (for the second state)
   * - flipped: whether the tile is rotated 180 degrees in the 2nd state.
   *
   * So basically what needs to happen is move each tile from index -> folded_index, rotate
   * if needed, and you can just print the resulting images and when you assemble the card
   * all your designs appear the way you intended them to. Doing this in something like
   * Photoshop or Gimp would be a huge pain, so that's why I made this program.
   */
  // a
  {1, 1, false},   {2, 2, false},   {3, 3, false},   {4, 4, false},
  {5, 5, false},   {6, 6, false},   {7, 7, false},   {8, 8, false},
  {9, 9, false},   {10, 10, false}, {11, 11, false}, {12, 12, false},
  {13, 13, false}, {14, 14, false}, {15, 15, false}, {16, 16, false},

  // b (corner tiles are glued)
  {17, 19, false}, {18, 50, false}, {19, 51, false}, {20, 18, false},
  {21, 23, false}, {22, 54, false}, {23, 55, false}, {24, 22, false},
  {25, 27, false}, {26, 58, false}, {27, 59, false}, {28, 26, false},
  {29, 31, false}, {30, 62, false}, {31, 63, false}, {32, 30, false},

  // c (corner tiles are unset, tiles on left and right are similar as previous (side b))
  {33, 0, false},  {34, 53, true},  {35, 56, true},  {36, 0, false},
  {37, 0, false},  {38, 24, false}, {39, 21, false}, {40, 0, false},
  {41, 0, false},  {42, 28, false}, {43, 25, false}, {44, 0, false},
  {45, 0, false},  {46, 57, true},  {47, 60, true},  {48, 0, false},

  // d (corner tiles are glued)
  {49, 40, true},  {50, 39, true},  {51, 38, true},  {52, 37, true},
  {53, 36, true},  {54, 35, true},  {55, 34, true},  {56, 33, true},
  {57, 48, true},  {58, 47, true},  {59, 46, true},  {60, 45, true},
  {61, 44, true},  {62, 43, true},  {63, 42, true},  {64, 41, true},
}};
// clang-format on

template <typename PixelType>
sln::Image<PixelType> load_image(const char* filename)
{
  auto img_data = sln::read_image(sln::FileReader(filename));
  if (!img_data.is_valid())
  {
    std::cerr << "Image data could not be decoded." << std::endl;
  }
  const auto img = sln::to_image<PixelType>(std::move(img_data));
  assert(img.is_valid());
  return img;
}

template <typename T>
void copy(T src, T dst)
{
  for (auto y = 0_idx; y < src.height(); ++y)
  {
    for (auto ptr = src.data(y), end = src.data_row_end(y), ptr2 = dst.data(y); ptr != end; ++ptr, ++ptr2)
    {
      *ptr2 = *ptr;
    }
  }
}

int main()
{
  // load the four all-equal in size views for the endless card
  auto img1 = load_image<sln::Pixel_8u4>("card_a.png");
  auto img2 = load_image<sln::Pixel_8u4>("card_b.png");
  auto img3 = load_image<sln::Pixel_8u4>("card_c.png");
  auto img4 = load_image<sln::Pixel_8u4>("card_d.png");

  assert(img1.width() == img2.width() && img1.height() == img2.height());
  assert(img2.width() == img3.width() && img2.height() == img3.height());
  assert(img3.width() == img4.width() && img3.height() == img4.height());

  // concat all images into one long image, making it easier to calculate coordinates for each tile
  sln::Image<sln::Pixel_8u4> input(img1.width(), sln::to_pixel_length(img1.height() * 4));
  sln::Image<sln::Pixel_8u4> output = clone(input);

  copy(img1, sln::view(input, 0_idx, 0_idx, img1.width(), img1.height()));
  copy(img2, sln::view(input, 0_idx, sln::to_pixel_index(img1.height() * 1), img1.width(), img1.height()));
  copy(img3, sln::view(input, 0_idx, sln::to_pixel_index(img1.height() * 2), img1.width(), img1.height()));
  copy(img4, sln::view(input, 0_idx, sln::to_pixel_index(img1.height() * 3), img1.width(), img1.height()));

  // helper lambda to get coordinate for a given tile number
  const auto tile_width = sln::to_pixel_length(img1.width() / 4);
  const auto tile_height = sln::to_pixel_length(img1.height() / 4);
  auto index_to_x_and_y = [=](size_t index) -> tile_index {
    const size_t x = ((index - 1) % 4) * static_cast<sln::PixelLength::value_type>(tile_width);
    const size_t y = ((index - 1) / 4) * static_cast<sln::PixelLength::value_type>(tile_height);
    return {sln::to_pixel_index(x), sln::to_pixel_index(y)};
  };

  // copy each tile to the correct location for printing the endless card
  for (const auto& tile : tiles)
  {
    auto src_index = index_to_x_and_y(tile.index);
    auto dst_index = index_to_x_and_y(tile.folded_index);
    if (tile.folded_index == 0)
    {
      continue;
    }
    auto src = sln::view(input, src_index.x, src_index.y, tile_width, tile_height);
    auto dst = sln::view(output, dst_index.x, dst_index.y, tile_width, tile_height);
    if (tile.flipped)
    {
      copy(sln::rotate<sln::RotationDirection::Clockwise180>(src), dst);
    }
    else
    {
      copy(src, dst);
    }
  }

  // debug
  sln::write_image(to_image_data_view(input, sln::PixelFormat::RGBA), sln::ImageFormat::PNG,
                   sln::FileWriter("result.png"));
  sln::write_image(to_image_data_view(output, sln::PixelFormat::RGBA), sln::ImageFormat::PNG,
                   sln::FileWriter("result2.png"));

  // write the resulting cards ready to print, result a + b double sided, and c + d double sided.
  sln::write_image(
      to_image_data_view(sln::view(output, 0_idx, 0_idx, img1.width(), img1.height()), sln::PixelFormat::RGBA),
      sln::ImageFormat::PNG, sln::FileWriter("result_a.png"));
  sln::write_image(
      to_image_data_view(sln::view(output, 0_idx, sln::to_pixel_index(img1.height() * 1), img1.width(), img1.height()),
                         sln::PixelFormat::RGBA),
      sln::ImageFormat::PNG, sln::FileWriter("result_b.png"));
  sln::write_image(
      to_image_data_view(sln::view(output, 0_idx, sln::to_pixel_index(img1.height() * 2), img1.width(), img1.height()),
                         sln::PixelFormat::RGBA),
      sln::ImageFormat::PNG, sln::FileWriter("result_c.png"));
  sln::write_image(
      to_image_data_view(sln::view(output, 0_idx, sln::to_pixel_index(img1.height() * 3), img1.width(), img1.height()),
                         sln::PixelFormat::RGBA),
      sln::ImageFormat::PNG, sln::FileWriter("result_d.png"));

  return 0;
}

Input & Output

The program reads card_a.png, card_b.png, card_c.png, card_d.png. Does it's magic and procudes: result_a.png, result_b.png, result_c.png, result_d.png.

<-- the *input* images <-- the *output* images

As you can see the resulting images look a bit weird, but when printed double sidedly, card A and B on one side, and card C and D on the other side of the paper, you can cut them out and fold them.

How to fold?

This is how I did it, I didn't plan to make a blog post so I didn't completely document every step. But at least on this picture should give you an idea:

For folding I bought a "Scor-Pal", I can recommend this tool it makes nice folds, doing without would give crappy results on thick paper IMO.

The polar bear piece of paper is side A & C, and cut in the middle already, and two horizontal pre-folds are made (sloppy ones though ). The other two sides C & D are pre-folded as well, but I didn't cut it horizontally yet. After cutting, glue the white edges on sides B & C together, and have it dry, the card should be done.

Conclusion

After having folded about 60 of these I finally got the hang of it and I could produce pretty slick cards. One thing I did was print myself with a color laser printer on very thick paper, this gave "meh" results, the toner had trouble sticking to the paper especially after folding. I recommend doing the printing at a specialized shop, maybe to avoid the toner letting loose, but also because aligning both sides is important. Not all printers are great at this I've discovered, especially if you have to use the manual feed for the thick paper.

What worked best is this order:

Printing both sides
Cut out the two sides with something like
Do the (in total four) pre-folds on both pieces of paper (do them in the right direction)
Cut the first one vertically, perfectly in the middle.
Cut the second one horizontally, perfectly in the middle.
Put glue on the corners for the second one now (the one cut horizontally)
Then align those two pieces perfectly so it appears whole again.
One by one add the other (the one cut vertically) two pieces, you'll find with the glue still wet it's easy to adjust.
When done, let it dry, and later on pre-fold some more for the definitive card.

i3 floating point window resize percentage wise and centered

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 30 May 2018 00:00:00 +0000

May 30 2018

i3 floating point window resize percentage wise and centered

I'm a heavy user of scratch pads with i3, I often don't like the dimensions of a window after you make them floating. As do other people, see here and here2.

I've used a customized version of the solution proposed in one of the comments by the creator of i3-gaps (Airblader) here3. This has served me well, but one thing bugged me when using multiple monitors it wouldn't center the window correctly, so I made a Python script that first uses Qt to get all screen dimensions and determine the correct offset based on the Mouse position. It's a bit overkill probably, but it works, so I'm happy with it.

Step 1: download, make, and make install wmutils.

Note that if you update your system in the meantime, it may have to be recompiled at some point, I've experienced this with the lsw command which is using some X calls that changed after I updated from Ubuntu 17.04 -> 17.10.

Step 2: put this python script somewhere

#!/usr/bin/env python3
import os
import os.path
import psutil
import subprocess
import sys
from PyQt5.QtCore import *
from PyQt5.QtGui import *
from PyQt5.QtWidgets import *
from pymouse import PyMouse
from sys import platform


def clamp(n, smallest, largest):
    return max(smallest, min(n, largest))


def current_screen_size(mouse_position):
    for screen in app.screens():
        (x, y) = mouse_position
        screen_geom = screen.availableGeometry();
        if (x >= screen_geom.left() and y >= screen_geom.top() and
            x <= screen_geom.left() + screen_geom.width() and
            y <= screen_geom.top() + screen_geom.height()):
            return screen_geom
    return app.desktop().availableGeometry(-1)


# read ratio from config file
ratio_file = "/tmp/resize_ratio.txt"
ratio = 70

if os.path.isfile(ratio_file):
    file = open(ratio_file, "r") 
    ratio = int(file.read())
    file.close()

if len(sys.argv) > 1:
    if sys.argv[1] == 'inc':
        ratio += 10
    elif sys.argv[1] == 'dec':
        ratio -= 10

ratio = clamp(ratio, 10, 100)

# get mouse and screen specs
app = QApplication(sys.argv)
mouse = PyMouse()
screen = current_screen_size(mouse.position())

# call wmutils::core utilities
currentWindow = subprocess.check_output(["pfw"])

# resize the window
new_width = (screen.width() / 100) * ratio
new_height = (screen.height() / 100) * ratio
# subprocess.call([
#     "wrs",
#     "-a",
#     str(new_width),
#     str(new_height),
#     currentWindow
# ])

# position the window centered (+ resize)
new_x = screen.left() + ((screen.width() - new_width) / 2)
new_y = screen.top() + ((screen.height() - new_height) / 2)
subprocess.call([
    "wtp",
    str(new_x),
    str(new_y),
    str(new_width),
    str(new_height),
    currentWindow
])

# persist current ratio
file = open("/tmp/resize_ratio.txt", "w") 
file.write(str(ratio)) 
file.close()

Don't forget chmod +x /path/to/resize.py

You may need to install some python3 -m pip install ... when you try to run it you'll probably discover what exactly, I forgot to keep a requirements.txt. From what I remember you need at least: python -m pip install pyuserinput pyqt5 python-xlib

Step 3: modify resize mode in i3

Probably you already have a "resize" mode, just add something like SHIFT+J and SHIFT+K to that mode to call the python script:

mode "resize" {
    bindsym Shift+k exec $HOME/.bin/resize.py inc
    bindsym Shift+j exec $HOME/.bin/resize.py dec
    ...
}
bindsym $mod+r mode "resize"

My visit to Meeting C++ 2016!

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 21 Nov 2016 00:00:00 +0000

November 21 2016

My visit to Meeting C++ 2016!

Finally, I was able to attend this conference, missing out two years in a row, and it was great. So far it has been the largest yet with 600 attendees, and AFAIK Bjarne Stroustrup was present for the first time this year.

I went to Berlin with my girlfriend two days before the event so we had a chance to see Berlin. Even though the weather was very much what you would expect around this time of year, cloudy, rainy, etc. we had a great time. Especially renting bikes and sightseeing.

[Image caption] Brief moment of no-rain..

Talks I attended... DAY 1

Opening Keynote - Bjarne Stroustrup

What is C++ and what will it become? It was a nice presentation showing the strength of C++ and providing a little history here and there (like code below). Funny quote from the presentation "Only a computer scientist makes a copy then destroys the original"; The committee has a difficult task, making the C++ language less complex, but the only thing the committee can do is add more to it , but they still succeed (i.e., with auto, constexpr, ..).

int i; // 70's? 
for (i=0; i<10; i++) a[i] = 0;
----------
for (int i=0; i<10; i++) a[i] = 0; // 80's? no declaration outside the for
----------
for (auto &el : a) el = 0; // mistakes like reading out of bounds no longer possible
                           // ... also mistakes like; for (int i=0; i<10; j++) {}

Boris Schäling asked "Scott Meyers retired from C++ a year ago; do we need to be worried about you?", luckily we don't have to worry ;-). Bjarne answered that he tried a few times to quit C++ in the past, but apparently he is not very good at it .

Learning and teaching Modern C++ - Arne Mertz

The speaker made an interesting point regarding some pitfalls, i.e. that many C++ developers learned C first, pointers, pointer arithmetic, C++03, C++11, .., basically a "layered evolution". However Modern C++ isn't a layered evolution, rather it is a "moving target". Nowadays we prefer make_unique, unique_ptr and therefor why not postpone teaching new, delete, new[], delete[], pointer arithmetic etc. when teaching Modern C++? The same goes for C-style arrays, more complex to teach as opposed to std::array.

Actually kind of sad news; there are still schools in some Countries where C++ is taught with Turbo C++ (see this SO question from a few days ago) compiler (which is extremely outdated). Other notes I scribbled down were for me to check "clang tidy" and adding "isocpp.org" to my RSS feeds.

Wouter van OOijen--a professor teaching C++ in the context of embedded devices--made a good point: the order in which material is presented to students is the most difficult thing to get right. In most books on C++ the order doesn't make sense for embedded, that's why he creates his own material.

Implementation of a multithreaded compile-time ECS in C++14 - Vittorio Romeo

This was quite interesting, maybe it was just me but in the beginning of the presentation it wasn't clear to me what an Entity Component System was, it became clear to me during the talk though. He walked us through the implementation, advanced templating, lambdas, bit fiddling, all quite interesting, maybe a bit too much content for one presentation but very impressive stuff. The room temperature during the presentation was extremely hot, making it sometimes difficult to concentrate and the talk went a bit over the scheduled time.

Some stuff I found interesting: the usage of sparse sets, the use of proxy objects to make sure that certain methods of the library cannot be called at the wrong time.

ctx->step([&](auto& proxy)
    {
        // do something with proxy
    });

He went through a large list of features and how they are implemented

Ranges v3 and microcontrollers, a revolution -- Odin Holmes

Quite an awesome talk this one, the speaker is extremely knowledgeable on meta programming and embedded programming. His company works with devices with very little memory (just a few kilobyte) and this talk was very forward looking. There was a crash course regarding limitations for such devices, there is limited stack space, how do exceptions and interrupts play along with it.

He then started with real demo/hello world for such a device and demonstrated how even that small code contained bugs and a lot of boilerplate. The rest of the talk he showed how to improve it, like instead of parsing (dangerously) with scanf (you can overflow the buffer, so you need a "large enough" buffer up-front... "And we all know that coming up with a size for a large enough buffer is easy, right?" ) can be replaced with a statemachine known at compile time. Ranges can be applied to lazy evaluate input, and as a result it would consume only the minimal memory.

C++ Today - The Beast is back - Jon Kalb

Why was C/C++ successful? It was based on proven track record, and not a "pure theoretical language". High-level abstractions at low cost, with a goal of zero-abstraction principle. In other words; not slower than you could do by coding the same feature by hand (i.e., vtables).

If you like a good story and are curious about why there was a big red button on the IBM 360, the reason behind the C++ "Dark ages" (2000 - 2010), where very little seem to happen, then this is the presentation to watch. Spoiler alert: cough Java cough, OOP was the buzzword at the time, it was "almost as fast", computers got faster and faster, we "solved the performance issue"!

Interesting statements I jotted down "Managed code optimizes the wrong thing (ease of programming)", and regarding Java's finally (try {} catch {} finally {}): "finally violates DRY". He then asked the audience a few times what DRY stands for, which is quite funny as some people realize they were indeed repeating themselves, not all as someone else yelled "the opposite of WET" . He also "pulled the age card" when discussing Alexander Stephanov (the author of the STL) "You kids think std::vector grew on trees!".

DAY 2

Functional reactive programming in C++ - Ivan Cukic

Talk of two parts, first functional programming: higher order functions, purity, immutable state. Functional thinking = data transformation. He discussed referential transparency, f.i. replacing any function with its value should produce the same outcome. This can depend on your definition.

int foobar()
{
    std::cout << "Returning 42..." << '\n';
    return 42;
}

Above function when used in int n = foobar(); can be replaced by 42, and the line of code would result in exactly the same thing (n containing 42), however the console output won't be printed. Whether you consider std::cout to count as part of the referential transparency is up to you.

He continued with Object thinking = no getters, ask the object to do it. "Objects tend to become immutable.". I will have to review the presentation to get exactly what was meant by this.

Next: reactive programming, if I am correct this was his definition:

responds quickly
resilient to failure
responsive under workload
based on message-passing

Note: reacting not replying, i.e., piping Linux shell commands there is only one-way data flow. To conclude, some random notes I made during his talk below.

He's writing a book on Functional programming in C++
flatmap from functional programming does [x, a], [y, b, c] -> x, a, y, b, c.
His talk reminded me to lookup the meaning of placing && behind a member function declaration.

See below for an example from cppreference.com.

#include <iostream>

struct S {
    void f() & { std::cout << "lvalue\n"; }
    void f() &&{ std::cout << "rvalue\n"; }
};

int main(){
    S s;
    s.f();            // prints "lvalue"
    std::move(s).f(); // prints "rvalue"
    S().f();          // prints "rvalue"
}

The Speed Game: Automated Trading Systems in C++ - Carl Cook

This talk was probably one of the most well attended talks at the conference. The room was packed. Coming in slightly late I had to sit down on my knees for the entire talk. Which was worth it, I think I liked this talk most of all I attended. It was just the right mix of super interesting material and practical advice.

Coming from Amsterdam where Automated Trading companies seem to kind of dominate C++, it has always been very mysterious what exactly it is they do. It felt to me like it was basically the first time the veil was lifted a little bit. It's just amazing to hear how far they go in order to get the lowest latency possible. Within the time it takes for light to travel from the ground to the top of the Eiffel tower they can take an order, assess whether it's interesting or not, and place the order... times ten!

// Some practical advice, instead of the following..
if (checkForErrorA)
    handleErrorA();
elseif (checkForErrorB)
    handleErrorB();
elseif (checkForErrorC)
    handleErrorC();
else
    executeHotPath();

// Aim for this..
uint32_t errorFlags;
if (errorFlags)
    handleError(errorFlags);
else
{
    ... hotpath
}

Really interesting talk to watch whenever it comes online, it shows the importance of optimizing hardware, bypassing the kernel completely in the hot path, staying in user space for 100%, this includes network I/O (f.i., OpenOnload), cache warming, beware of signed/unsigned conversions, check the assembly, inplace_function (the speakers proposals, stdext::inplace_function<void(), 32>), benchmarking without the 'observable effect' by observing network packets, and more.

One note regarding Network I/O for example; if you read a lot but very little is interesting to the hot path, you may negatively affect your cache. A solution would be to offload all the reads to a different CPU and cherry-pick only the interesting reads and send them to the "hot" CPU.

Lock-free concurrent toolkit for hazard pointers and RCU - Michael Wong

Well, I was a bit tired at this point, so I cannot do the talk justice with a very thorough summary. Even if I could it's better to watch it from Michael Wong himself, because the slides help a lot in understanding the story.

I did learn a few things, maybe the first lesson for me is to try stay away from all of this.. Still, aside from being super complicated, it's also an interesting topic, and good to know more about. The ABA problem: he had good slides that visualized actually step-by-step the challenge of updating data in a multi-threading situation, having readers while writing to it, all wrapped in a fun story of Schrödingers Cat (and Zoo). Solutions discussed were hazard pointers and RCU (Read Copy Update).

The gains you can get by starting late, having a grace period so you can do multiple updates at the same time are interesting to learn about. Situations where "being lazy" actually pays off!

Lightning talks!

Surprise! They had secret lightning talks planned. To be honest at first I thought it was a bit long to have 1 hour and 40 minutes planned for a Meeting C++ update/review, so this was a nice surprise. My favorite lightning talk was from Michael Caisse reading from the standard as if it were a very exiting story, hilarious. Second James McNellis' "function pointers all the way down" (like "Turtles all the way down", actually Bjarne also had a reference to this in his keynote). The remaining lightning talks were also very good: Michael Wong, Jens Weller, Chandler Carruth, and Bjarne's. The latter on Concepts was quite interesting; "what makes a good concept?" It has to have semantics specifying it, which in practice seems to be an efficient design technique. Quite funny was his "Onion principle" on abstractions (IIRC?), "you peel away layer by layer, and you cry more and more as you go along" . Also Jens talk was really fun, it started with end of the world scenarios, working towards the future C++ standards.

C++ metaprogramming: evolution and future directions - Louis Dionne

The closing keynote was a really clear and relaxed presentation of how meta programming evolved, and in particular how boost::hana did. Again a nice lesson of history where Alexandrescu's Modern C++, boost::mpl, boost::fusion and the like all passed the revue. He showed what you can do with boost::hana at compile-time and runtime. His talk really opened my eyes on using constexpr, integral_constant, differences in meta programming with types and objects, and a lot more. It's amazing what his library can do. He argued the world needs more meta programming, but less template meta programming and concluded by sharing his view for the future.

The conference

There was a fun quiz, with really difficult puzzles (C++ programs) that had to be solved in < 3 minutes each. This was basically similar to peeling Bjarne's Onion.. but in a good way.

Between talks there were lunch-break Meetups planned (each 20 minutes, each had a specific topic). I attended two and my view is that it's a great idea, but the fact people have to come from talks, and leave on time to catch the next one, sometimes caused the time to be way too short (or yourself missing out on a talk because the room is now full).

The organization was super, the drinks and food, especially the second day. The Andel's Hotel is a really good location, the Hotel as well (if you are lucky enough to get a room there). For me it was all really worth the money.

Personally I like to write down a summary for myself, but I hope this blog post was also a fun to read to someone else!

Kerberizing Cloudera Manager

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Tue, 06 Sep 2016 00:00:00 +0000

September 6 2016

Kerberizing Cloudera Manager

The following steps are to quickly test how this stuff works.

Using my docker images (master, slave) and helper scripts on github, it's easy to get Cloudera Manager running inside a few docker containers. Steps: get most recent docker, install (GNU) screen, checkout the repo, in there do cd cloudera, bash start_all.sh. This should do it. Note that the image(s) require being able to invoke --privileged and the scripts currently invoke sudo. After running the script you get something like (full example output here).

CONTAINER ID        IMAGE                               COMMAND             CREATED             STATUS              PORTS                    NAMES
31e5ee6b7e65        rayburgemeestre/cloudera-slave:3    "/usr/sbin/init"    20 seconds ago      Up 17 seconds                                node003
f052c52b02bf        rayburgemeestre/cloudera-slave:3    "/usr/sbin/init"    25 seconds ago      Up 23 seconds                                node002
1a50df894f28        rayburgemeestre/cloudera-slave:3    "/usr/sbin/init"    30 seconds ago      Up 29 seconds       0.0.0.0:8888->8888/tcp   node001
54fd3c1cf93b        rayburgemeestre/cloudera-master:3   "/usr/sbin/init"    50 seconds ago      Up 48 seconds       0.0.0.0:7180->7180/tcp   cloudera

Not really in the way docker was designed perhaps, it's running systemd inside, but for simple experimentation this is fine. These images have not been designed to run in production, but perhaps with some more orchestration it's possible .

Step 1: install Cloudera Manager

One caveat because of the way docker controls /etc/resolv.conf, /etc/hostname, /etc/hosts, these guys show up in the output for the mount command. The Cloudera Manager Wizard does some parsing of this (I guess) and pre-fills some directories with values like:

/etc/hostname/<path dn>
/etc/resolv.conf/<path dn>
/etc/hosts/<path dn>

Just remove the additional two paths, and change one to <path dn> only. There is a few of these configuration parameters that get screwed up. (Checked until <= CDH 5.8)

Step 2: install kerberos packages on the headnode

docker exec -i -t cloudera /bin/bash # go into the docker image for headnode

yum install krb5-server krb5-workstation krb5-libs

# ntp is already working

systemctl enable krb5kdc
systemctl enable kadmin

Configuration files need to be fixed, so starting will not work yet.

Step 3: modify /etc/krb5.conf

Into something like:

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = MYNET
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 MYNET = {
  kdc = cloudera.mynet
  admin_server = cloudera.mynet
 }

[domain_realm]
 .mynet = MYNET
 mynet = MYNET

In this example cloudera.mynet is just hostname --fqdn of the headnode which will be running kerberos. (Note that mynet / MYNET could also be something like foo.bar / FOO.BAR.)

Step 4: modify /var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 MYNET = {
  #master_key_type = aes256-cts

  master_key_type = aes256-cts-hmac-sha1-96
  max_life = 24h 10m 0s
  max_renewable_life = 30d 0h 0m 0s

  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal aes256-cts-hmac-sha1-96
 }

I specifically added aes256-cts-hmac-sha1-96 as master key and supported encryption types, and the max_life plus max_renewable_life properties.

But there is a chance Cloudera Manager might add this stuff as well.

Step 5: modify /var/kerberos/krb5kdc/kadm5.acl

*/admin@MYNET      *

Step 6: initialize the database

# kdb5_util create -r MYNET -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'MYNET',
master key name 'K/M@MYNET'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key: ******
Re-enter KDC database master key to verify: ******

Step 7: add master `root/admin` user

[root@rb-clouderahadoop2 krb5kdc]# kadmin.local
Authenticating as principal root/admin@MYNET with password.
kadmin.local:  addprinc root/admin
WARNING: no policy specified for root/admin@MYNET; defaulting to no policy
Enter password for principal "root/admin@MYNET": ******
Re-enter password for principal "root/admin@MYNET": ******
Principal "root/admin@MYNET" created.
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/admin
Entry for principal kadmin/admin with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/admin with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/admin with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/admin with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/admin with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/admin with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/admin with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/admin with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
kadmin.local:  ktadd -kt /var/kerberos/krb5kdc/kadm5.keytab kadmin/changepw
Entry for principal kadmin/changepw with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/changepw with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/changepw with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/changepw with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/changepw with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/changepw with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/changepw with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
Entry for principal kadmin/changepw with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/var/kerberos/krb5kdc/kadm5.keytab.
kadmin.local:  exit

This will be the user we will give Cloudera to take over managing kerberos.

Step 8: start services

systemctl start krb5kdc
systemctl start kadmin

Step 9: do the `Enable security` wizard in Cloudera Manager

This should be self explanatory, but in summary:

Enable the four checkboxes on the first page of the wizard.
Next page, kdc = hostname --fqdn headnode, realm = MYNET (in our example). Leave other defaults.
Next page, select Manage krb5.conf through Cloudera Manager. Leave all defaults.
Next page, Username root/admin and password you typed in step 7.

The wizard will do it's magic and hopefully succeed without problems.

How to screen capture in Windows 10 with HiDPI support

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 07 May 2016 00:00:00 +0000

May 7 2016

How to screen capture in Windows 10 with HiDPI support

In case you are looking for a free alternative to Camtasia Studio or many other alternatives... One of my favorite tools of all time, ffmpeg can do it for free!

The simplest thing that will work is ffmpeg -f gdigrab -framerate 10 -i desktop output.mkv (source) This gives pretty good results already (if you use an MKV container, FLV will give worse results for example).

HiDPI: Fix mouse pointer

gdigrab adds a mouse pointer to the video but does not scale it according to HiDPI settings, so it will be extremely small. You can configure the mouse pointer to extra large to fix that. That mouse pointer won't scale either, but at least you end up with a regular size pointer in the video

Optional: Use H264 codec

More options you can find here, I've settled with single pass encoding using -c:v libx264 -preset ultrafast -crf 22.

ffmpeg -f gdigrab -framerate 30 -i desktop ^
       -c:v libx264 -preset ultrafast -crf 22 output.mkv

Optional: Include sound in the video

First execute ffmpeg -list_devices true -f dshow -i dummy this will give you directshow devices. (source) On my laptop this command outputs:

[dshow @ 00000000023224a0] DirectShow video devices (some may be both video and audio devices)
[dshow @ 00000000023224a0]  "USB2.0 HD UVC WebCam"
[dshow @ 00000000023224a0]     Alternative name "@device_pnp_\\?\usb#vid_04f2&pid_b3fd&mi_00#6&11eacec2&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global"
[dshow @ 00000000023224a0]  "UScreenCapture"
[dshow @ 00000000023224a0]     Alternative name "@device_sw_{860BB310-5D01-11D0-BD3B-00A0C911CE86}\UScreenCapture"
[dshow @ 00000000023224a0] DirectShow audio devices
[dshow @ 00000000023224a0]  "Microphone (Realtek High Definition Audio)"
[dshow @ 00000000023224a0]     Alternative name "@device_cm_{33D9A762-90C8-11D0-BD43-00A0C911CE86}\wave_{1DDF1986-9476-451F-A6A4-7EBB5FB1D2AB}"

Now I know the device name I can use for audio is "Microphone (Realtek High Definition Audio)". Use it for the following parameters in ffmpeg -f dshow -i audio="Microphone (Realtek High Definition Audio)".

The end result

I ended up with capture-video.bat like this:

ffmpeg -f dshow -i audio="Microphone (Realtek High Definition Audio)" ^
       -f gdigrab -framerate 30 -i desktop ^
       -c:v libx264 -preset ultrafast -crf 22 output.mkv

This is a resulting video where I used this command, resolution of the video is 3840x2160 and the HiDPI scale is set to 2.5.

Update 1> Add more keyframes for better editing

For this I use the following command, to insert a keyframe every 25 frames (the closer to one, the larger the output file will be):

ffmpeg.exe -i %1 -qscale 0 -g 25 %2

The option -qscale 0 is for preserving the quality of the video.

(Changing the container to .mov was probably not necessary, I tried this hoping that Adobe Premiere would support it, but it didn't!)

Update 2> Editing 4K on Windows 10...

Found the following tool for editing: Filmora and (on my laptop) it was able to smoothly edit the footage. They support GPU acceleration, but the additional keyrames really help with a smooth experience.

Once you get the hang of it (shortcut keys are your friend) it's pretty easy to cut & paste your videos.

Update 3> Support Adobe Premiere

As I discovered Adobe Premiere earlier, doesn't like MKV, but it also doesn't like 4:4:4 (yuv444p), the pixel format used by default (it seems). You can view such information using ffprobe <VIDEO FILE>. Anyway, it seems to like yuv420p, so add -pix_fmt yuv420p to make it work for Premiere:

ffmpeg.exe -i input.mkv -qscale 0 -g 25 -pix_fmt yuv420p output.mov

Profiling and visualizing with GNU strace

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 02 Apr 2016 00:00:00 +0000

April 2 2016

Profiling and visualizing with GNU strace

A crazy idea, building a profiler/visualizer based on strace output. Just for fun. But, who knows there may even be something useful we can do with this..

The following image shows exactly such a visualization for a specific HTTP GET request (f.i., to http://default-wordpress.cppse.nl/wp-admin/index.php (URL not accessible online)). The analysis from the image is based on the strace log output from the Apache HTTP server thread handling the request. Parameters for the strace call include -f and -F so it includes basically everything the Apache worker thread does for itself. (If it were to start a child process, it would be included.)

This request took 1700 milliseconds, which seems exceptionally slow, even for a very cheap micro compute instance. It is, I had to cheat a little by restarting Apache and MySQL in advance, to introduce some delays that make the graph more interesting. It's still still normal though that strace will slow down the program execution speed.

I grouped all strace lines by process ID and their activity on a specific FD (file descriptor). Pairs like open()/close() or socket()/close() introduce a specific FD and in between are likely functions operating on that FD (like read()/write()). I group these related strace lines together and called them "stream"s in the above image.

In the image you can see that the longest and slowest "stream" is 1241 milliseconds, this one is used for querying MySQL and probably intentionally closed last to allow re-use of the DB connection during processing of the request. The three streams lower in the visualization follow each other sequentially and appear to be performing a lookup in /etc/hosts, follewed by two DNS lookups directed to 8.8.4.4.

Why are we doing this? (Other than because it's Awesome!)

This works for any strace output, but my idea originated while doing web development. This was for a relatively complicated web application, that was divided in many sub-systems that communicate mostly via REST calls with each other. All these systems had lots of external calls to other systems, and I wanted a view where I could see regardless of which sub-system or actual PHP code being executed, how the performance was for specifically: I/O with (i.e. for i18n/locale) files, scripts, SQL queries to MySQL, Oracle, the REST API calls to system X, Y & Z, Redis, Memcached, Solr, Shared memory even and Disk caching.

If only there was a tool really good at capturing that kind of I/O... ahh yeah there is, strace! I switched jobs 7 months ago, before applying my strace tool to this code-base, but I've applied it to similar complex applications with success.

We already had tools for (more traditional) profiling of PHP requests. Quite often the interpretation was difficult, probably because of a lot of nasty runtime reflection being used. Also when you needed to follow a slow function (doing a REST call) it was a lot of effort to move profiling efforts to the other system (because of OAuth 1.0b(omg..), expired tokens, ..). Nothing unsolveable of course, but with strace you can just trace everything at once on a development environment (especially in Vagrant which we used), spanning multiple vhosts. If it's just you on the VM, perhaps you can strace the main Apache PID recursively, I didn't try that however, but I think that would work.

Products like NewRelic provide dashboards for requests where you can gain such deep insights, "off the shelve", basically, but the downside is that it's not cheap. NewRelic f.i. hooks into Apache & PHP and has access to actual PHP function calls, SQL queries, etc. strace cant do that, because it only sits between the process(es) and the Linux kernel.

First, let's take one step back & properly parse the strace output..

It quickly became apparent that I couldn't get away with some trivial regex for parsing it, so I turned to bnfc and created the following BNF grammer to generate the parser. I was quite suprised that this was so easy that it took me less than a working day to find a tool for the job, learn it and get the grammer right for some strace output.

With this tool you are provided with an autogenerated base class "Skeleton" which you can extend to create your own Visitor implementation. With this pattern it becomes quite easy to extract some meta-data you are interested in. I will show a simply example.

The grammer

I came up with the following grammer that bnfc uses to generate the Parser. Reading it from top to bottom is more or less the way you can incrementally construct this kind of stuff. You start really small; first chunking multiple strace-lines into single strace-lines, then chunk strace-lines into Pid, Timestamp and (remaining) Line. Then further specify a Pid, the Timestamp, Line, etc., slowly making the grammer more coarse-grained.

EStraceLines.          StraceLines         ::= [StraceLine];
EStraceLine.           StraceLine          ::= [Pid] [Timestamp] Line;

EPidStdOut.            Pid                 ::= "[pid " PidNumber "] ";
EPidOutput.            Pid                 ::= PidNumber [Whitespace] ;
EPidNumber.            PidNumber           ::= Integer;

ETimestamp.            Timestamp           ::= EpochElapsedTime;

ELine.                 Line                ::= Function "(" Params ")" [Whitespace] "=" [Whitespace] ReturnValue [TrailingData];
ELineUnfinished.       Line                ::= Function "(" Params "<unfinished ...>";
ELineContinued.        Line                ::= "<... " Function " resumed> )" [Whitespace] "=" [Whitespace] ReturnValue [TrailingData];
ELineExited.           Line                ::= "+++ exited with" [Whitespace] Integer [Whitespace] "+++" ;

EFunction.             Function            ::= Ident ;
EFunctionPrivate.      Function            ::= "_" Ident ;

EParams.               Params              ::= [Param];

EParamArray.           Param               ::= "[" [Param] "]" ;
EParamObject.          Param               ::= "{" [Param] "}" ;
EParamComment.         Param               ::= "/* " [CommentString] " */";
EParamInteger.         Param               ::= Number ;
EParamFlags.           Param               ::= [Flag] ;
EParamIdent.           Param               ::= Ident ;
EParamString.          Param               ::= String ;
EParamWhitespace.      Param               ::= Whitespace ;
EParamAddress.         Param               ::= Address ;
EParamDateTime.        Param               ::= DateYear "/" DateMonth "/" DateDay "-" TimeHour ":" TimeMinute ":" TimeSecond ;
EParamKeyValue.        Param               ::= Param "=" Param ;
EParamKeyValueCont.    Param               ::= "...";
EParamExpression.      Param               ::= Integer Operator Integer;
EParamFunction.        Param               ::= Function "(" [Param] ")" ;

EDateYear.             DateYear            ::= Integer ;
EDateMonth.            DateMonth           ::= Integer ;
EDateDay.              DateDay             ::= Integer ;
ETimeHour.             TimeHour            ::= Integer ;
ETimeMinute.           TimeMinute          ::= Integer ;
ETimeSecond.           TimeSecond          ::= Integer ;

EOperatorMul.          Operator            ::= "*";
EOperatorAdd.          Operator            ::= "+";

EEpochElapsedTime.     EpochElapsedTime    ::= Seconds "." Microseconds ;
ESeconds.              Seconds             ::= Integer ;
EMicroseconds.         Microseconds        ::= Integer ;

ECSString.             CommentString       ::= String ;
ECSIdent.              CommentString       ::= Ident ;
ECSInteger.            CommentString       ::= Integer ;

ENegativeNumber.       Number              ::= "-" Integer;
EPositiveNumber.       Number              ::= Integer;

EFlag.                 Flag                ::= Ident;
EFlagUmask.            Flag                ::= Integer;

ERetvalAddress.        ReturnValue         ::= Address ;
ERetvalNumber.         ReturnValue         ::= Number ;
ERetvalUnknown.        ReturnValue         ::= "?";

EAddress.              Address             ::= HexChar;

ETrailingDataConst.    TrailingData        ::= " " [Param] " (" [CommentString] ")";
ETrailingDataParams.   TrailingData        ::= " (" [Param] ")" ;

ESpace.                Whitespace          ::= " ";
ESpace4x.              Whitespace          ::= "    ";
ETab.                  Whitespace          ::= "    ";

terminator             CommentString       "" ;
terminator             Param               "" ;
terminator             Pid                 " " ;
terminator             Timestamp           " " ;
terminator             TrailingData        "" ;
terminator             Whitespace          "" ;

separator              CommentString       " " ;
separator              Flag                "|" ;
separator              Param               ", " ;
separator              Pid                 " " ;
separator              StraceLine          "";

token HexChar ('0' 'x' (digit | letter)*);

Given the above grammer bnfc can parse this strace line 15757 1429444463.750111 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}]) into an Abstract Syntax Tree.

[Abstract Syntax]

(EStraceLines [
    (EStraceLine 
        [(EPidOutput [(EPidNumber 15757)])] 
        [(ETimestamp [(EEpochElapsedTime
                         [(ESeconds 1429444463)]
                         [(EMicroseconds 750111)])])] 
        [(ELine 
            [(EFunction "poll")] 
            [(EParams [
                (EParamArray [
                    (EParamObject [
                        (EParamKeyValue (EParamIdent "fd") 
                                        (EParamInteger [(EPositiveNumber 3)])),
                        (EParamKeyValue (EParamIdent "events")
                                        (EParamFlags [
                                            (EFlag "POLLIN"),
                                            (EFlag "POLLPRI"),
                                            (EFlag "POLLRDNORM"),
                                            (EFlag "POLLRDBAND")]))])]), 
                (EParamInteger [(EPositiveNumber 1)]),
                (EParamInteger [(EPositiveNumber 0)])])]

            ESpace ESpace

            [(ERetvalNumber [(EPositiveNumber 1)])]

            [(ETrailingDataParams 
                [(EParamArray 
                    [(EParamObject [
                        (EParamKeyValue (EParamIdent "fd")
                                        (EParamInteger [(EPositiveNumber 3)])),
                        (EParamKeyValue (EParamIdent "revents")
                                        (EParamFlags [
                                            (EFlag "POLLIN"),
                                            (EFlag "POLLRDNORM")]))])])])
            ]
          )
        ]
     )
])

No matter how nested these lines get, it will parse them as long as I didn't forget anything in the grammer. (So far it seems to be complete to parse everything.)

Visitor example

Using the BNF grammer, the above structure and occasional peeking at the generated Skeleton base class, you can simply override methods in your own visitor to do something "useful". The following visitor is a less "useful" but simple example that outputs all the strings captured for strace lines containing the open() function. Just to illustrate how you use this Visitor.

class OutputOpenVisitor : public Skeleton
{   
    string timestamp;
    string function;
    string strings;
public:
    void visitEStraceLine(EStraceLine* p)
    {   
        timestamp = "";
        function  = "";
        strings   = "";
        Skeleton::visitEStraceLine(p);
        if (function == "open") {
            cout << timestamp << " " << function << " " << strings << endl;
        }
    }
    void visitEFunction(EFunction* p)
    {   
        function = p->ident_;
        Skeleton::visitEFunction(p);
    }
    void visitEEpochElapsedTime(EEpochElapsedTime *p)
    {   
        auto secs      = static_cast<ESeconds *>(p->seconds_);
        auto microsecs = static_cast<EMicroseconds *>(p->microseconds_);
        timestamp = to_elasticsearch_timestamp(secs, microsecs);
        Skeleton::visitEEpochElapsedTime(p);
    }
    void visitString(String x)
    {   
        strings.append(x);
        Skeleton::visitString(x);
    }
};

You can find this example in the examples forder in the git repository here.

After compiling this example into strace-output-visualizer:

# capture a strace log
trigen@firefly:/projects/strace-output-parser[master]> strace -f -F -ttt -s 512 -o test.log uptime
17:53:02 up 32 days, 22:44, 23 users,  load average: 2.39, 2.20, 2.12

# strace log contains stuff like
trigen@firefly:/projects/strace-output-parser[master]> head -n 10 test.log 
19151 1458147182.196711 execve("/usr/bin/uptime", ["uptime"], [/* 47 vars */]) = 0
19151 1458147182.197415 brk(0)          = 0x7c1000
19151 1458147182.197484 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
19151 1458147182.197555 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f45cd85e000
19151 1458147182.197618 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
19151 1458147182.197679 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
19151 1458147182.197740 fstat(3, {st_mode=S_IFREG|0644, st_size=156161, ...}) = 0
19151 1458147182.197813 mmap(NULL, 156161, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f45cd830000
19151 1458147182.197888 close(3)        = 0
19151 1458147182.197969 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)

# pipe the log through the example program
trigen@firefly:/projects/strace-output-parser[master]> cat test.log | ./strace-output-parser 
2016-03-16T16:53:02.198Z open /etc/ld.so.cache
2016-03-16T16:53:02.198Z open /lib/x86_64-linux-gnu/libprocps.so.3
2016-03-16T16:53:02.199Z open /lib/x86_64-linux-gnu/libc.so.6
2016-03-16T16:53:02.200Z open /sys/devices/system/cpu/online
2016-03-16T16:53:02.200Z open /usr/lib/locale/locale-archive
2016-03-16T16:53:02.200Z open /etc/localtime
2016-03-16T16:53:02.201Z open /proc/uptime
2016-03-16T16:53:02.202Z open /var/run/utmp
2016-03-16T16:53:02.273Z open /proc/loadavg

Opposed to a simple Visitor like this example, I parse all the lines, prepare a JSON representation for each line and store that in ElasticSearch. This way selecting and filtering can be done afterwards. And also ElasticSearch is really a fast solution in case you want to do more complex queries on your log.

A Proof of concept for Web

This time at the beginning of each request I have PHP instruct some script to run a strace on the process id for the current PHP script's pid (or rather the Apache worker's) and all it's (virtual) threads and sub processes. (If I would track the Request accross the stack with "Cross application tracing" you can even combine all the relevant straces for a given request. I didn't implement this (again) because of I switched jobs. (Info on Cross application tracing in newrelic). This is even relatively easy to implement if you have a codebase where you can just make the change (like inject a unique id for the current request in curl call for example).)

The following image and code shows how I capture straces from specific PHP requests, like the wordpress example I started this blog with. You can skip this part. Eventually these straces are linked to a specific request, ran through a slightly more elaborate Visitor class and fed into ElasticSearch for later processing.

(This omits also some other details w/respect to generating a UUID for each request, and keeping track of what strace outputs are related to each request.)

Inject in your application 'header', i.e., top index.php:

register_shutdown_function(function () { touch("/tmp/strace-visualizer-test/done/" . getmypid()); });
$file = "/tmp/strace-visualizer-test/todo/" . getmypid();
touch($file);
while (file_exists($file)) { sleep(1); } // continue with the request when removed from todo folder

A separate long running process runs the following:

trigen@CppSe:~/strace-visualizer-test> cat run.ksh 
#!/bin/ksh93
mkdir -p /tmp/strace-visualizer-test/todo
mkdir -p /tmp/strace-visualizer-test/done
while true; do
    find /tmp/strace-visualizer-test/todo/ -type f | \
        xargs -I{} -n 1 sh -c "strace -f -F -ttt -s 4096 -o \$(basename {}).strace -p \$(basename {}) & rm -rf {};"
    find /tmp/strace-visualizer-test/done/ -type f | \
        xargs -I{} -n 1 sh -c "(ps axufw | grep [s]trace.*\$(basename {}) | grep -v grep | awk -F ' ' '{print \$2}' | xargs -n 1 kill -1 ) & (sleep 1; rm -rf {};)"
    printf ".";
done

This way you end up with .strace files per process ID (it should probably include a timestamp too). The long running process removes the file the client checks from the todo folder as soon as it started strace. That way the client will no longer block and the interesting stuff will be captured. It uses a shutdown handler to instruct the long running process to stop the capture (the Apache thread won't exit, it will wait for a next request).

Final step, To ElasticSearch!

I use a Visitor and my strace parser to create JSON representations for the strace log lines. Containing the meta-data I need: file descriptors, an array with all strings, a timestamp that ElasticSearch can understand out of the box, etc.

To get to my previous example, I can use cat test.log | ./strace-output-parser elasticsearch localhost 9200 strace_index to import the parsed lines to ElasticSearch.

In above example I use filtering with a plugin called "head" to basically make the same selection as I did with the simple visitor example. I also highlighted one specific line to show the JSON representation.

I used PHP for processing the wordpress strace output from ElasticSearch and generated the visualization from the very first image in this blog post. You can view the HTML output here.

Hopefully this blog post was interesting to read, and maybe you find some use for the strace parser yourself. If you do, please let me know, that would be fun to know .

Remote debugging with NetBeans 8.1 for C/C++

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 17 Dec 2015 00:00:00 +0000

December 17 2015

Remote debugging with NetBeans 8.1 for C/C++

Most people are probably familiar with gdb, and Ribamar pointed out to me there is also a ncurses frontend inside gdb. But in case anyone is interested I learned that NetBeans also supports remote debugging. Even though it's not the most modern IDE in the world, and it's vi emulation is cumbersome , it seems to have pretty good support for remote-debugging. It will just login to some machine via ssh (i.e., dev11 or a real cluster), and issue gdb <something> and wrap around it. If you make sure it knows where the sources files are on your development machine, you can use all the step-debugging features.

The only downside is that loading up cmd in gdb takes a while probably ~ 30 seconds. Still it's a lot faster than debugging with print-statements and recompiling. For cmsh it's already a lot faster and on top of that you can issue a command multiple times via the REPL, so you can step debug it multiple times within the same gdb session. (Beware though that you probably need to connect again as your connection may be lost)

Example workflow

To show off how it works first with CMDaemon. My workflow is to create a unit-test that fails, set a breakpoint in the unit-test and start the debug.

-
break point set followed by the debugger stopping execution at that point.

-
step-into example, select the function to step into -and click the button highlighted with -.

There is also the F7 key to "step into", but be prepared to step into assembly a lot of times (use CTRL+F7 to step out, and try again). You will jump into the -> operator, shared pointer dereferences, std::string constructors, before getting into the function you want. (Also note that the first time you step into assembly it will be very slow, but it will get faster the next few times).

Wizard example to debug cmd unit test

Download from https://netbeans.org/downloads/
chmod +x netbeans-8.1-cpp-linux-x64.sh
./netbeans-8.1-cpp-linux-x64.sh

Note that you want to set some bogus command like whoami.
Netbeans will try to be smart and clean your project directory for you
(and rebuild without using multiple cores, ..)

Note the working directory should be including src.
This is to help gdb later with finding source code.

There is one fix needed that the Wizard didn't set properly for us.
Go to project properties, Build / Make, and set Build Result to the executable.
The remote debugger will use this value for issuing with gdb, and it's somehow empty by default.

Use ALT+SHIFT+o to Jump to the file containing the test.
Set a breakpoint there using CTRL+F8

The final thing we want to pass to gdb is the parameters for running our specific unittest.
In my example "${OUTPUT_PATH}" --unittests --gtest_filter=LineParserTest.empty.

You can use these settings to double check if everything is correct

How to "inspect element" XUL applications

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Tue, 01 Dec 2015 00:00:00 +0000

December 1 2015

How to "inspect element" XUL applications

In addition to my previous blog post How to debug XUL applications.

Last friday I learned that you can use the DOM inspector on XUL applications as well. This is quite useful if you want to see what events are hidden behind a button, try out layout changes, etc., etc. It is also quite fast, I don't notice any performance difference.

These instructions are taken from a very useful stackoverflow answer. Summarizing:

Add [XRE] EnableExtensionManager=1 to your application.ini if it isn't already.
If you are using the xulrunner app you already have the Error Console available (for info see my previous blog post for this). Type in it the following: window.openDialog("chrome://mozapps/content/extensions/extensions.xul", "", "chrome,dialog=no,resizable=yes");.
You will be presented the Add-ons Manager, in there choose "Install Add-on From File..." and download the "DOM Inspector". (I have a local copy here: addon-6622-latest.xpi (downloaded from: here)).
You need to restart and start xulrunner with an additional -inspector flag.

One tip with the DOM inspector, if you use "File >> Inspect Chrome Document" and the list is huge, highlight an item with your mouse and press the End key on your keyboard. You likely need one at the bottom of the list because those are the XUL files loaded most recently.

How to debug XUL applications

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 25 Nov 2015 00:00:00 +0000

November 25 2015

How to debug XUL applications

You can use Mozilla Firefox (Javascript) debugging on your XUL application using the Remote Debugging facility. This blog post could be useful as a HOWTO, because I was lucky enough to attempt this 3rd of July 2015. You see had I tried this today I would have failed, because stuff seems broken in newer versions of xulrunner (and Firefox). This is true for the project I work on at least. The very fact that I struggled with setting this up today was my motivation to dig into why it wasn't working and made me think this might be useful to others.

I know everything in this blog post to work for both CentOS 6.6 and Ubuntu 15.04. These steps (except for the xulrunner download) should be platform independent.

First get a slightly older xulrunner

You need a reasonably new xulrunner in order for Remote Debugging to work. I downloaded xulrunner version 38 at the time from The Mozilla Project Page (xulrunner-38.0.5.en-US.linux-x86_64.tar should be on their FTP somewhere, but you can also use this local copy hosted with this blog). I think we should cherish that version, because that one works.

The newest and version is version 41, but also the last because they started integrating it in Mozilla Firefox since then. I tried version 41, and grabbing a recent ~~Thunderbird~~ Firefox, but all steps work, except when you arrive in the "Connect Dialog", the clickable Main Process hyperlink (as shown in the image) is simply not there for you to click on.

Enable a debug listener in the code

In your application you need to start the debug listener. Probably in the top of your main.js include the following lines.

Components.utils.import('resource://gre/modules/devtools/dbg-server.jsm');
if (!DebuggerServer.initialized) {
  DebuggerServer.init();
  // Don't specify a window type parameter below if "navigator:browser"
  // is suitable for your app.
  DebuggerServer.addBrowserActors("myXULRunnerAppWindowType");
}
var listener = DebuggerServer.createListener();
listener.portOrPath = '6000';
listener.open();

Also enable in the preferences (probably defaults/preferences/prefs.js).

pref("devtools.debugger.remote-enabled", true);

If you forget to change this last preference you will get the following error.

JavaScript error: resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/devtools/server/main.js, line 584: Error: Can't create listener, remote debugging disabled

Start the application with this xulrunner

Extract the xulrunner runtime to somewhere, i.e. /projects/xulrunner, and issue from the your program's directory like this:

shell$> /projects/xulrunner/xulrunner application.ini

Attach debugger from Mozilla Firefox

Open a fairly recent Firefox browser and open the remote debugger which is available via "Tools -> Web Developer -> Connect...".

If the above "Connect.." option is not available, you have to enable the same preference inside Firefox in the "about:config" page. Search for remote-enabled.

Then connect to localhost port 6000.

Your program will present you a dialog to accept the incoming connection from the debugger.

After accepting you can click to attach to "Main Process" (your program).

You should be presented with a debugger that will automatically break when it encounters the debugger keyword. You can also set breakpoints inside.

This can look similar to the following image where a call stack is shown, and you have your usual ways to inspect variables and perform step debugging with F10, F11, Shift+F11

I am convinced it should also be possible to make it so that the javascript in can handle inspection from the debuggers console. In order to get a REPL working there (for inspecting variables), but I didn't find out how this can be achieved. Using the Watch (and Auto) expressions you can already inspect everything.

Just beware that once you attach to the process your program can freeze up for a while as the debugger is loading all the javascript files.

Circlix Clock

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 13 Sep 2015 00:00:00 +0000

September 13 2015

Circlix Clock

Today I published my first Android (Wear) App! . The idea behind this clock is that it uses concentric circles to show time, and doesn't use analog clock hands or numeric time notation. This is something I have on a bigger LCD screen at home for a while now, and now that there is Android Wear for a while, I wanted to implement this for Android.

Some example visualizations

There is more theory behind the visualization, more on that on the website: http://circlix.click.

Android Watch Face

WebGL from the Website

You need to have WebGL support in your browser in order to see the following live-clock.

Some comments on Android Wear development

Android Wear is relatively new, and I never read any book on the Android Framework. Luckily I had some Java experience. Overall I am impressed by the design of the Framework, although it also confused the hell out of me on various occasions @:|@.

Some stuff I needed to realize or discover during development:

(Very basic:) an Activity only runs when it's the current activity.
If you need stuff running for longer than an Activity, you need Services.
In Java you don't have RAII like in C++/PHP. If you have handlers for threads etc. you should stop them in some onDestroy() method.
Packaging, creating the APK for use in f.i. the Play Store was counter intuitive, at least for me. Follow the example project provided by Google closely in your project w/respect to Gradle files. I had a perfectly good working APK that came out of Android Studio, it worked whenever I sent it to others, but it was not accepted by the Play store.
There is already OpenGL support for Watch Faces. You need to extend Gles2WatchFaceService.

Improve performance Jetbrains IDE when working on remote machine

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 02 Sep 2015 00:00:00 +0000

September 2 2015

Improve performance Jetbrains IDE when working on remote machine

I use CLion in this blog post, but it should be the same for any of the other editors. (PyCharm, PhpStorm, Intellij, etc.).

It took me a while to get a setup that works reasonably well for me at work, for what I expect not a very uncommon setup. That's why I'm sharing this in a blog post.

The project I'm working on is quite big, 10yr under development; large codebase and a complex build process. The debug build results in a 1.2 GiB executable, all intermediate files generated by the compiler/linker are many, and big. During build a lot of files are removed/(re)created/generated, so in general a lot of I/O happens.

Our build machines are extremely powerful, so it doesn't make sense to work on a local machine because of the build times. That's why compiling happens on remote machines. I have worked remotely at a lot of companies, and usually I would simply use vim + a lot of plugins. However, nowadays I'm accustomed to the power IDE's can provide, primarily navigation-wise (jumping to classes, files, finding usages, etc.) and simply don't want to work without a proper IDE.

This is my setup

I use an NFS mount (sshfs would suffice as well) where I mount from the remote to local, not the other way around, or compiling will be extremely slow. In my opinion using file synchronization in these kinds of setups is too error prone and difficult to get right.

As a side-note; I've seen synchronization work moderately okay within a PHP project. But so far not in a C++ project where intermediate/build-files/libraries are first of all large and scattered throughout the project folder.

In my previous blog post we fixed fsnotifier such as in the previous image, but this also causes a new problem.

Lot's of I/O is slow over network mount

During compiling I noticed my IDE would hang, the only cause could be that it's somehow flooded by the enourmous lines of input it now receives from fsnotifier. Perhaps when we're working with the project files on a local disk the IDE wouldn't hang, because simple I/O (even just checking file stats) doesn't have network overhead.

Solution, ignore as much (irrelevant) I/O as possible

Here I made the fsnotifier script--that was at first just a simple proxy (calling the real fsnotifier via ssh)--more intelligent. It now filters out intermediate files generated by the compiler (.o, .d, and some other patterns).

function custom_filter
{
    typeset -n return_val=$1
    typeset cmd=$2  # i.e., DELETE/CREATE/CHANGE/...
    typeset file=$3 # i.e., /full/path/to/file

    # Ignore some files that are not interesting to my IDE
    if [[ $file =~ (cmd|mm)\.log$ ]] || \
       [[ $file =~ deps.*\.d$ ]]         || \
       [[ $file =~ \.o$ ]]            || \
       [[ $file =~ \.o\. ]]            || \
       [[ $file =~ \.html$ ]]         || \
       [[ $file =~ core\.*([0-9])$ ]];
    then
        return_val=false
        return
    fi

    return_val=true
    return
}

Download all source code from GitHub: https://github.com/rayburgemeestre/fsnotifier-remote/.

Alternative solutions

The fsnotifier script outputs it's process id to /tmp/fsnotifier.pid and hooks two signals, so you can enable/disable it with a signal. Disabling will simply pause outputting all updates from the real fsnotifier (that is invoked via ssh).

kill -SIGINT $(cat /tmp/fsnotifier.pid) - pause all activity
kill -SIGHUP $(cat /tmp/fsnotifier.pid) - continue all activity

Another extension you may find useful would be to make the buildscript touch a file like, i.e. /path/to/project/DISABLE_FSNOTIFIER and make the fsnotifier script pause itself (or behave differently) during the build until it sees for example the ENABLE_FSNOTIFIER file.

Simply disabling fsnotifier again doesn't fix the problem, CLion would keep nagging occasionally about conflicts with files that have changed both on disk and in memory. And when auto-generated files are being re-generated by the build, I want my IDE to reflect them immediately.

Fine-tuning your filter

The filter is just a bash/ksh function, so you can easily extend it with patterns appropriate to your project. The fun thing is you can "killall -9 fsnotifier", and Jetbrains will simply restart it. So no need to restart Jetbrains (and with that having it re-index your project). Debug the filters by tailing: /tmp/fsnotifier-included.log and /tmp/fsnotifier-filtered.log.

Update: 13th October 2016

No longer do I need to filter out *.o files etc. to get a better responsive IDE nowadays. The network improved (and perhaps it's something that improved in newer CLion versions). Another change I did make to the script is based on the ROOTS that get registered (for monitoring the project path) use fsnotifier over ssh or not. (for local projects it would try to login via ssh otherwise, finding nothing and the IDE would hang at that point).

https://github.com/rayburgemeestre/fsnotifier-remote/commit/414e2e1f937a59a9ab11eede6b999c8170e30af0

Enable fsnotifier for Jetbrains IDE's like PyCharm over NFS/SSHFS network share

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 14 Aug 2015 00:00:00 +0000

August 14 2015

Enable fsnotifier for Jetbrains IDE's like PyCharm over NFS/SSHFS network share

This should work for all their editors, PyCharm, Intellij, CLion, PhpStorm, Webstorm, etc.

The editor(s) use this tool to "subscribe" to changes on the filesystem. So if you change a file that's also in a buffer in for example CLion, it will know it needs to reload that file from disk in order to show the latest changes.

Without this tool it will fallback to periodically checking for changes or when a specific file is activated, I don't know exactly, but it's slower anyway.

You probably started searching for a solution because you saw this error in the console or in a popup in the IDE:

trigen@baymax:/home/trigen/Downloads/clion-1.0.4> FSNOTIFIER_LOG_LEVEL=info ./bin/clion.sh  
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=350m; support was removed in 8.0
[   3243]   WARN - om.intellij.util.ProfilingUtil - Profiling agent is not enabled. Add -agentlib:yjpagent to idea.vmoptions if necessary to profile IDEA. 
[  14166]   WARN - api.vfs.impl.local.FileWatcher - Project files cannot be watched (are they under network mount?)  <<<<<<<<<<<

Let's fix it by having the IDE run fsnotifier over SSH on the actually server.

I will use as an example a project named MyFirstProject mounted via NFS from a server named DevelopmentMachine:

sudo mount -t nfs DevelopmentMachine:/home/ray/projects/MyFirstProject /projects/MyFirstProject -o rw,user,hard,intr,tcp,vers=3,timeo=600,_netdev,nolock,exec

First you need fsnotifier on DevelopmentMachine, because that machine should be able to subscribe to the filesystem events. I downloaded and build the one from ThiefMaster/fsnotifier-remote.

Test it by starting it and adding the project like this (>>> is your input, <<< the output you get):

[ray@DevelopmentMachine linux]$ ./fsnotifier
>>> ROOTS
>>> /home/ray/projects/MyFirstProject
>>> #
<<< UNWATCHEABLE
<<< #

Now it's watching, trigger some changes on something in that root (i.e. open a vim hi.txt):

<<< CREATE
<<< /home/ray/projects/MyFirstProject/.hi.txt.swp
<<< CHANGE
<<< /home/ray/projects/MyFirstProject/.hi.txt.swp
<<< STATS
<<< /home/ray/projects/MyFirstProject/.hi.txt.swp
...

In this example I work locally on /projects/MyFirstProject, where it's /home/ray/projects/MyFirstProject on the server. The super easy solution is to make sure your local path is exactly the same. In my case I made a symlink so I have /home/ray/projects/MyFirstProject both on my local- and remote machine.

On the local machine I can run the above ./fsnotifier example through ssh, lets test that (make sure you have ssh keys configured correctly for this, otherwise you will get an authentication prompt):

trigen@baymax:/projects/fsnotifier-remote[master]> ssh -l ray DevelopmentMachine /home/ray/projects/fsnotifier-remote/linux/fsnotifier64
>>> ROOTS
>>> /home/ray/projects/MyFirstProject
...

The fun thing is that the displayed files are actually already correct, so you don't need to do some any mapping. Just make sure you launch your IDE on the /home/ray/projects/MyFirstProject folder. (Which the beforementioned fsnotifier-remote script should be able to do, but I encountered multiple issues executing it under Linux and I didn't like to dive into it's Python code).

I created a local fsnotifier script with the following contents:

#!/bin/ksh93
ssh -l ray DevelopmentMachine /home/ray/projects/fsnotifier-remote/linux/fsnotifier64

Then told my IDE to use this wrapper (make sure it's executable with chmod +x)

trigen@baymax:/home/trigen/Downloads/clion-1.0.4/bin> vim idea.properties
idea.filewatcher.executable.path=/projects/fsnotifier-remote/fsnotifier <<< add this line!

You can log the communication between the IDE and fsnotifier over ssh by inserting this in the fsnotifier wrapper script: strace -f -F -ttt -s 512 -o /tmp/fsnotifier-debug.log (put it before the ssh command). Then you can find stuff like this in the /tmp/fsnotifier-debug.log:

823 3722  1439468464.159229 read(4, "ROOTS\n", 16384) = 6
833 3722  1439468464.159644 read(4, "/home/ray/projects/MyFirstProject\n", 16384) = 28
843 3722  1439468464.160011 read(4, "/home/trigen/.clion10/system/extResources\n", 16384) = 42
853 3722  1439468464.160489 read(4, "#\n", 16384) = 2

Hope this approach will help others!

Update: 13th October 2016

Having this script in place allows for some fun "performance optimizations" in case you have a slow network connection.

Nagios 4 + Nagvis + Nagiosgraph + Nagios plugins Dockerfile / Docker image

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 17 May 2015 00:00:00 +0000

May 17 2015

Nagios 4 + Nagvis + Nagiosgraph + Nagios plugins Dockerfile / Docker image

Setting up Nagios + Nagvis + Nagiosgraph on Ubuntu (14.04) can be a pain in the neck.

Default Ubuntu (14.04) ships with Nagios3, which is plain ugly and old, also the Nagvis is pretty old and less user friendly. So I created a Docker image with that install the—at the time of writing—newest versions of Nagios, Nagvis, Nagios plugins and Nagios graph. (Along with Apache2.4 + PHP 5.5 for the Web interfaces.)

I'm new to Docker, so leaving comments/rants/improvements is appreciated

TL;DR

docker run -P -t -i -v /your/path/to/rrd-data:/usr/local/nagiosgraph/var/rrd rayburgemeestre/nagiosnagvis
docker ps              # to discover port
boot2docker ip         # to discover host other than localhost (if you are using boot2docker on OSX)
open http://host:port  # you will get a webinterface pointing to nagios/nagvis or nagiosgraph

Caveats with the install

For Nagvis you need a different broker called livestatus, where both Nagios and Nagvis need to change their configs for, and you must specifically configure it to support Nagios Version 4, otherwise you will get an error starting Nagios. Specifically this one:

Error: Could not load module '/usr/local/lib/mk-livestatus/livestatus.o' -> /usr/local/lib/mk-livestatus/livestatus.o: cannot open shared object file: No such file or directory
Error: Failed to load module '/usr/local/lib/mk-livestatus/livestatus.o'.

Which is fixed by this instruction from the Dockerfile:

WORKDIR /usr/local/src/check-mk-raw-${livestatusversion}.cre.demo
RUN ./configure --with-nagios4 && \
    make && \
    ### specifically make mk-livestatus package /again/ with the --with-nagios4 flag, by default it's build for nagios3 which doesn't work.. \
    cd ./packages/mk-livestatus/mk-livestatus-${livestatusversion} && \
    make clean && \
    ./configure --with-nagios4 && \
    make && \
    make install

In the source root the --with-nagios4 flag is not propagated to it's sub-packages. So I just make everything and then specifically clean the mk-livestatus-xx package and re-configure with --with-nagios4, make, make install.

If I had to guess the livestatus configure script probably by default tries to detect the Linux distrubition, and as Ubuntu 14.04 ships with Nagios 3 by default it probably assumes to use version 3.

After the (re)configure, this is the normal output:

Event broker module '/usr/local/lib/mk-livestatus/livestatus.o' initialized successfully.

Build docker image yourself

You can use the Dockerfile on Github to build the image yourself. It contains in one file all the commands you would need to execute to do everything manually.

Build with: docker build -t rayburgemeestre/nagiosnagvis ..

Get docker image from Docker Hub

You can also use the image rayburgemeestre/nagiosnagvis as a base for your own projects using Docker Hub.

You can also run the base image with: docker run -P -t -i rayburgemeestre/nagiosnagvis.

The -P auto-portforwards the port for Apache that runs inside (use docker ps to detect the port).

[Snoop around on the container with: docker run -P -t -i --entrypoint /bin/bash rayburgemeestre/nagiosnagvis.]

Directories in the container

Nagios, Nagvis and Nagiosgraph are all installed in subdirectories of /usr/local.

You are likely to want /your/own/rrd-data directory mounted as /usr/local/nagiosgraph/var/rrd inside the container, so the RRD databases are not stored inside the container and retained after rebuilding/upgrading the container. This is possible with the -v flag: docker run -P -t -i -v /your/own/rrd-data:/usr/local/nagiosgraph/var/rrd rayburgemeestre/nagiosnagvis

Don't forget that the docker user (uid 1000) has the appropriate read-write permissions on that rrd directory.

Qt Applications in Browser

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 21 Feb 2015 00:00:00 +0000

February 21 2015

Qt Applications in Browser

While thinking about a Qt application that I also wanted to create a Web-version for, I experimented a bit with the “lazy man's approach”. I was wondering if I could simply “bridge” from Qt app to Browser in an easy way.

So I created the following proof of concept (Note that this is not just an image, use your mouse to interact):

You can control above window from your browser, it doesn’t always feel like a real-time experience yet though (mind that it's hosted on a very weak server). I took one of the Qt Examples (gradients), stripped it down a bit and wrote a simple plugin. This plugin once loaded, exposes the graphics through a Websocket to the browser and provides an API for handling mouse behaviours. For the graphics currently PNG images are transferred over the Websocket.

The required patch in main.cpp (for gradients) and an example running locally.

Streaming

Then thinking about steps on how to improve this, I encountered some really cool Javascript stuff! My next idea was to find a way to stream the window instead of working with PNG images. I didn’t pursue this through.

One of my frustrations with the web is that there is still not a real standard for streaming with HTML5. Then I found someone who implemented an MPEG1 decoder in javascript and uses that for streaming: http://phoboslab.org/log/2013/09/html5-live-video-streaming-via-websockets

I experimented a bit with hooking up my Raspberry Pi Camera on it and I got it working with < 100ms delay, so it’s a nice solution. Only perk is that it doesn’t support sound.

Compiled to Javascript (emscripten-qt)

jsmpeg is a MPEG1 Decoder, written in JavaScript. It's "hand ported", i.e. not compiled with emscripten or similar. This will probably make it obsolete with the advent of asmjs.

Reading that I further investigated emscripten, demos here. And also encountered emscripten-qt(!), view some demos here. I didn't get it to work on my own system(s) unfortunately (OSX 10.9.4 and an Ubuntu server). I might attempt on it again later, see if I can include the gradients example compiled to the browser .

How emscripten-qt basically works is: it uses a slightly older version of Qt and patches it (I think heavily), it also requires a patched main() in the Qt app. It uses LLVM to compile your application from C++ not to a binary like you'd normally do, but to an assembly for use with asm.js. The end result is quite awesome, with some Javascript your Qt program running in the browser, completely client-side. (Some stuff won't be supported like Network).

asm.js was originally a Research project at Mozilla and modern browsers will support it. There is a chance that asm.js will be supported natively in the future, but it's already very fast.

Download + sourcecode

See https://bitbucket.org/rayburgemeestre/qwebcontrollerserverplugin.

git clone git@bitbucket.org:rayburgemeestre/qwebcontrollerserverplugin.git

cd qwebcontrollerserverplugin
cmake .
make
make install

cd example/gradients
cmake .
make

./gradients

Or use an IDE, QCreator should work out of the box, and one that supports CMake will probably too (I tested with CLion).

Tweak Battle....

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 19 Jan 2015 00:00:00 +0000

January 19 2015

Tweak Battle....

.... evolved from Smash Battle and was launched by Tweakers on April fools with the title (translated): “Tweakers releases Tweak Battle - Tech-site starts Game Studio”. It was pretty cool, the day before all Tweakers staff changed their avatars to their “8-bit” style character. Why blog about this now? Well, now some Tweakers created an Arcade machine a few days ago for it, it turned out to be quite awesome and I also accidentally stumbled upon some stats from April-fools day.

A while ago I added network multiplayer to Smash Battle (see old blog post) and then after that we/Jeroen got the idea to change all the characters into Tweakers editors/developers and launch the game as an April fools joke. The deadline was pretty tight, we had a lot of ideas for improvements and there were many glitches but we fixed all of them. We had to work many long nights and the night- and morning before the publication of the News at 8 o'clock.

Play stats

22:00 we fixed a problem that occasionally made a server crash, also you may notice the "active" & "joined" players lines to swap at that point, before that they were mixed up. The difference between the two is simply the number of spectators (you are "joined" if you connect and "active" when you play). Spectators were necessary because the game can hold a maximum of 4 players.

The only statistics we have, at the time gathered using a simple Munin plugin.

Ten blade servers

Ten seriously over-the-top servers were sponsored by True, and I'm sorry but I forgot the exact specs. We provisioned one of them as the main API server and the other we started nine games per (nine) server(s) on with all the different levels evenly distributed.

We did quite some last-minute optimizations, like removing the GUI for Servers, so running servers became a lot less CPU intensive. Previously we had to start them with xvfb (we basically used it as a /dev/null for the graphics). Even though I discovered by accident that SDL is so Awesome that it falls back to ncurses (see following image).

But in retrospect, we could have ran all the servers from my laptop . It was surely overkill for such a simple game with not that much Network traffic. But just in case it would go world-wide-viral, we could have handled it .

ncurses ASCII art rendering of a Tweak Battle server with no joined players.

Further improvements

Jeroen & Bert pushed me to replace the TCP/IP implementation with a UDP one, and this was a good idea. It made a big difference, even more than I expected. We also had to fix some glitches/artifacts that were caused by previous refactorings, i.e. that the game now runs on elapsed time, this created some problems with powerups to disappear too fast, and animations to go to fast here and there. Jeroen also designed and implemented all the Tweakers characters, which was a lot of work, Bert helped all-round and improved server provisioning a lot.

The “main” server

The main server is written in Symfony2 as a REST API (inspired by this implementation and Scenario Driven API Design). For documentation and source code check the bitbucket repo.

Maximum number of requests on /server/listing per second with- and without Varnish:

Requests per second:    43.34 [#/sec] (mean)
Requests per second:    12209.64 [#/sec] (mean) # with Varnish

We let Varnish cache this listing every 1 second(s). We talked to it asynchronously with C++ using libcurl. Storage used by the API was Redis.

The future

git network_multiplayer branch merged into master. But I would like to preserve both character sets, Smash Battle and Tweak Battle.
refactoring^2
nicknames support, online rankings.
(for Tweak Battle:) actual Nerf Guns + Bullets.
more game types like capture the flag, instagib, last man standing.
intelligent bots.

Example rendered video

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 18 Jan 2015 00:00:00 +0000

January 18 2015

Example rendered video

wxhttpproxy new version 1.2 supports CLI

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 14 Jan 2015 00:00:00 +0000

January 14 2015

wxhttpproxy new version 1.2 supports CLI

Just to announce I created a new version that supports a non-GUI interface. This makes installing and running it commandline on a linux server easy.

CLI interface

ksh$ wxhttpproxy --help
Usage: wxhttpproxy [-h] [-f <str>] [-n <num>] [-p <num>] [-q]
  -h, --help            displays help on the command line parameters.
  -f, --file=<str>      output everything to <file>.
  -n, --number=<num>    truncate file each <num> lines (default 500000).
  -p, --port=<num>      bind to port <port> (default 8888).
  -q, --quiet           silent output (nothing to stdout).

Building

trigen@Rays-MacBook-Pro.local:/tmp> git clone git@bitbucket.org:rayburgemeestre/wxhttpproxy.git
Cloning into 'wxhttpproxy'...
remote: Counting objects: 394, done.
remote: Compressing objects: 100% (391/391), done.
remote: Total 394 (delta 269), reused 0 (delta 0)
Receiving objects: 100% (394/394), 1.04 MiB | 225.00 KiB/s, done.
Resolving deltas: 100% (269/269), done.
Checking connectivity... done.
trigen@Rays-MacBook-Pro.local:/tmp> cd wxhttpproxy/
/tmp/wxhttpproxy
trigen@Rays-MacBook-Pro.local:/tmp/wxhttpproxy[master]> cmake .
-- The C compiler identification is AppleClang 6.0.0.6000056
-- The CXX compiler identification is AppleClang 6.0.0.6000056
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found wxWidgets: TRUE
/usr/local/Cellar/cmake/3.0.2/share/cmake/Modules/UsewxWidgets.cmake
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/wxhttpproxy
trigen@Rays-MacBook-Pro.local:/tmp/wxhttpproxy[master]> make
Scanning dependencies of target wxhttpproxy
[ 10%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/application.cpp.o
[ 20%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/cache.cpp.o
[ 30%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/clientdata.cpp.o
[ 40%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/consoleoutputhandler.cpp.o
[ 50%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/guioutputhandler.cpp.o
[ 60%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/httpbuffer.cpp.o
[ 70%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/httpproxywindow.cpp.o
[ 80%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/logger.cpp.o
[ 90%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/socketbuffer.cpp.o
[100%] Building CXX object CMakeFiles/wxhttpproxy.dir/src/socketserver.cpp.o
Linking CXX executable wxhttpproxy
[100%] Built target wxhttpproxy
trigen@Rays-MacBook-Pro.local:/tmp/wxhttpproxy[master]>

Changelog

Version 1.2:

Proper Connection: keep-alive support (with reconnects when necessary)
Support switching to different hosts/backends within a connection.
Commandline (Non-GUI) version, with logging+truncate feature.
An actual 1337 icon for the program!

Version 1.1:

HTTPS support
OSX support
Stability fixes
Support for caching content and serving from cache.

Version 1.0:

HTTP proxy that supports CONNECT and GET syntax.

CLI example usage (`http_proxy=http://localhost:8888 curl http://cppse.nl/test.txt`)

ksh$ wxhttpproxy
Info - Server listening on port 8888.
Thread #1 - New client connection accepted
Thread #1 - Connecting to host cppse.nl:80 for client.
Thread #1 - Request from client: GET /test.txt HTTP/1.1
Thread #1 - Host socket connected.
Thread #1 - >>>: GET /test.txt HTTP/1.1
Thread #1 - >>>: User-Agent: curl/7.30.0
Thread #1 - >>>: Host: cppse.nl
Thread #1 - >>>: Accept: */*
Thread #1 - >>>: Proxy-Connection: Keep-Alive
Thread #1 - >>>:
Thread #1 - Response from host.
Thread #1 - <<<: HTTP/1.1 200 OK
Thread #1 - <<<: Date: Sun, 28 Dec 2014 21:05:09 GMT
Thread #1 - <<<: Server: Apache/2.4.6 (Ubuntu)
Thread #1 - <<<: Last-Modified: Sun, 09 Feb 2014 00:48:33 GMT
Thread #1 - <<<: ETag: "c-4f1ee951f920c"
Thread #1 - <<<: Accept-Ranges: bytes
Thread #1 - <<<: Content-Length: 12
Thread #1 - <<<: Connection: close
Thread #1 - <<<: Content-Type: text/plain
Thread #1 - <<<:
Thread #1 - <<<: Hello world
Thread #1 - Host socket disconnected.

Zend server And the Return of the Oracle Instant Client

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 28 Nov 2014 00:00:00 +0000

November 28 2014

Zend server And the Return of the Oracle Instant Client

The problem

The title being a reference to this article from 2011, a blog post from someone who encountered a similar issue once . Hopefully my blog post will prevent someone else from spending a day on this issue. We are in the middle of a migration from Oracle 11.2 to 12.1, and from PHP, Zend server more specifically, we had some connectivity problems to Oracle, the PHP function oci_connect() returned:

PHP Warning:  oci_connect(): ORA-28547: connection to server failed, probable Oracle Net admin error in /home/webro/test.php on line 2

Good luck googleing that Oracle error code, nothing hints in the right direction, only that it's an error that occurs after the connection is established. Quote from http://ora-28547.ora-code.com/:

A failure occurred during initialization of a network connection from a client process to the Oracle server: The connection was completed but a disconnect occurred while trying to perform protocol-specific initialization, usually due to use of different network protocols by opposite sides of the connection.

The problem in our case was with the characterset

The "tl;dr" is: you may be using an Oracle "Light" client instead of the "Basic" client. In Zend Server this means that in the Zend Server lib path some libraries are missing. The Light client only supports a few charactersets. If you have some other Characterset that isn't default, that may be the problem. You need to make sure the Oracle Instant client Zend Server is using is the Basic client.

Unfortunately you cannot tell this from the phpinfo() output. Both Light and Basic return exactly the same version information.

Oracle Run-time Client Library Version => 11.2.0.2.0
Oracle Instant Client Version => 11.2

How we found out..

Luckily I was able to succesfully connect from another virtual machine to the new database server. This was an older Zend server instance, where the Oracle instant client was patched from 11.1 to 11.2. The Zend server that failed had 11.2, so we assumed patching wasn't necessary. I compared the strace outputs.

The first observation was that during the communication the server--on the right in the following image--stopped and concludes there is a communication error.

Working VM on the left, failing server on the right.

The second observation in the diff was that there was also a difference between libraries loaded.

- 9166  open("/usr/local/zend/lib/libociei.so", O_RDONLY) = -1 ENOENT (No such file or directory)
- 9166  open("/usr/local/zend/lib/libociicus.so", O_RDONLY) = 4
+ 17606 open("/usr/local/zend/lib/libociei.so", O_RDONLY) = 4

The fix for Zend server

Replace the Light client with the Basic client, in our case this meant adding a library to Zend Server's libs:

# After installing the oracle instant client BASIC
ln -s /usr/lib/oracle/11.2/client64/lib/libociei.so /usr/local/zend/lib/libociei.so

Apparently the difference between Light & Basic is just this one library. The package that provides the Basic client may differ per Linux distribution, you can also download it from oracle.com.

Visualize Issue Ranks in Atlassian Jira Agile board

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 01 Nov 2014 00:00:00 +0000

November 1 2014

Visualize Issue Ranks in Atlassian Jira Agile board

In Jira's Agile Board the Ranks of the Issues are visualized underneath the Story point estimates. The highest rank is colored green, the lower the priority becomes, the more the color changes to red. This way it also becomes visible who is not working according to priorities. See the following screenshot.

King - Servant pattern

At work we use the King-Servant pattern (see screenshot I made of slide 47 from this pdf) to avoid too much concurrent work. This pattern tries to solve the problem that you could end up with four unfinished tickets rather than two or even one complete ticket(s).

We work with remote programmers and therefore don't use a physical Scrum board. During the day and standups we view this board often and found it annoying that in order to determine the "King" ticket, you would need to switch back and forth the "Plan mode" a lot to see the Ranks. The different swimlanes often obfuscate the ranking information. With this script the king ticket is simply the one with rank "1".

The tampermonkey / greasemonkey script to give you this precious feature

Editing configuration may be required to adjust it to your board.

You can view/copy/download it from my gist at github and click the Raw button and copy & paste.
Chrome: You can use the Tampermonkey extension, "add new script", copy & paste the script and save.
FireFox: You can use the Greasemonkey add-on, add user script, and you need to explicitly add http*://*.atlassian.net/secure/RapidBoard.jspa?* as an include url, then copy & paste the script code from clipboard and save. Sorry Greasemonkey is not very user friendly.

[Edit 2-Nov-2014, I've changed my mind with the coloring, and some other things. The original version 1 is still available here]

[Edit 2-Jun-2015, I actually use the following script URL: //cppse.nl/public/tampermonkey_agile_board_prios.js and keep it up-to-date with my gist]

PhpStorm or IntelliJ suddenly hangs / freezes / keyboard not responsive/ unresponsive while editing in Linux / Ubuntu

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 31 Oct 2014 00:00:00 +0000

October 31 2014

PhpStorm or IntelliJ suddenly hangs / freezes / keyboard not responsive/ unresponsive while editing in Linux / Ubuntu

Problem

The symptom is that while editing the IDE freezes, somehow the keyboard no longer responds. Two years ago at Tweakers there was luckily someone using Ubuntu who could tell me right away how to fix it ("Just killall -9 ibus-x11", and it would magically continue to work). Now more recently at AutoTrack--where I now work--a collegue encountered the same issue. Luckily I knew the fix this time.

The fact that I personally know at least five different individuals who spent time fixing this makes me guess there are a lot more people still. Hence this blogpost with some keywords that will hopefully lure others into this fix...

Fix

According to this bug report:

killall -HUP ibus-daemon

Or use this one:

killall -9 ibus-x11

I have no idea what ibus exactly is, and I don't really care Also it may be possible to just uninstall ibus, I didn't try this, but I can imagine the following to work..

sudo apt-get remove ibus

Deflate and Gzip compress and decompress functions

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 01 Jun 2014 00:00:00 +0000

June 1 2014

Deflate and Gzip compress and decompress functions

When implementing a gzip compression & decompression for my blog. I stumbled upon two C++ functions by Timo Bingmann:

string : compress_string(string)
string : decompress_string(string)

Renamed these and created two gzip versions from them:

string : compress_deflate(string)
string : decompress_deflate(string)
string : compress_gzip(string)
string : decompress_gzip(string)

These are the differences:

Complete source

#include <string>
#include <sstream>
#include <stdexcept>

#include <string.h>
#include "zlib.h"

using std::string;
using std::stringstream;

// Found these here http://mail-archives.apache.org/mod_mbox/trafficserver-dev/201110.mbox/%3CCACJPjhYf=+br1W39vyazP=ix
//eQZ-4Gh9-U6TtiEdReG3S4ZZng@mail.gmail.com%3E
#define MOD_GZIP_ZLIB_WINDOWSIZE 15
#define MOD_GZIP_ZLIB_CFACTOR    9
#define MOD_GZIP_ZLIB_BSIZE      8096

// Found this one here: http://panthema.net/2007/0328-ZLibString.html, author is Timo Bingmann
// edited version
/** Compress a STL string using zlib with given compression level and return
  * the binary data. */
std::string compress_gzip(const std::string& str,
                             int compressionlevel = Z_BEST_COMPRESSION)
{
    z_stream zs;                        // z_stream is zlib's control structure
    memset(&zs, 0, sizeof(zs));

    if (deflateInit2(&zs, 
                     compressionlevel,
                     Z_DEFLATED,
                     MOD_GZIP_ZLIB_WINDOWSIZE + 16, 
                     MOD_GZIP_ZLIB_CFACTOR,
                     Z_DEFAULT_STRATEGY) != Z_OK
    ) {
        throw(std::runtime_error("deflateInit2 failed while compressing."));
    }

    zs.next_in = (Bytef*)str.data();
    zs.avail_in = str.size();           // set the z_stream's input

    int ret;
    char outbuffer[32768];
    std::string outstring;

    // retrieve the compressed bytes blockwise
    do {
        zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
        zs.avail_out = sizeof(outbuffer);

        ret = deflate(&zs, Z_FINISH);

        if (outstring.size() < zs.total_out) {
            // append the block to the output string
            outstring.append(outbuffer,
                             zs.total_out - outstring.size());
        }
    } while (ret == Z_OK);

    deflateEnd(&zs);

    if (ret != Z_STREAM_END) {          // an error occurred that was not EOF
        std::ostringstream oss;
        oss << "Exception during zlib compression: (" << ret << ") " << zs.msg;
        throw(std::runtime_error(oss.str()));
    }

    return outstring;
}

// Found this one here: http://panthema.net/2007/0328-ZLibString.html, author is Timo Bingmann
/** Compress a STL string using zlib with given compression level and return
  * the binary data. */
std::string compress_deflate(const std::string& str,
                            int compressionlevel = Z_BEST_COMPRESSION)
{
    z_stream zs;                        // z_stream is zlib's control structure
    memset(&zs, 0, sizeof(zs));

    if (deflateInit(&zs, compressionlevel) != Z_OK)
        throw(std::runtime_error("deflateInit failed while compressing."));

    zs.next_in = (Bytef*)str.data();
    zs.avail_in = str.size();           // set the z_stream's input

    int ret;
    char outbuffer[32768];
    std::string outstring;

    // retrieve the compressed bytes blockwise
    do {
        zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
        zs.avail_out = sizeof(outbuffer);

        ret = deflate(&zs, Z_FINISH);

        if (outstring.size() < zs.total_out) {
            // append the block to the output string
            outstring.append(outbuffer,
                             zs.total_out - outstring.size());
        }
    } while (ret == Z_OK);

    deflateEnd(&zs);

    if (ret != Z_STREAM_END) {          // an error occurred that was not EOF
        std::ostringstream oss;
        oss << "Exception during zlib compression: (" << ret << ") " << zs.msg;
        throw(std::runtime_error(oss.str()));
    }

    return outstring;
}

/** Decompress an STL string using zlib and return the original data. */
std::string decompress_deflate(const std::string& str)
{
    z_stream zs;                        // z_stream is zlib's control structure
    memset(&zs, 0, sizeof(zs));

    if (inflateInit(&zs) != Z_OK)
        throw(std::runtime_error("inflateInit failed while decompressing."));

    zs.next_in = (Bytef*)str.data();
    zs.avail_in = str.size();

    int ret;
    char outbuffer[32768];
    std::string outstring;

    // get the decompressed bytes blockwise using repeated calls to inflate
    do {
        zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
        zs.avail_out = sizeof(outbuffer);

        ret = inflate(&zs, 0);

        if (outstring.size() < zs.total_out) {
            outstring.append(outbuffer,
                             zs.total_out - outstring.size());
        }

    } while (ret == Z_OK);

    inflateEnd(&zs);

    if (ret != Z_STREAM_END) {          // an error occurred that was not EOF
        std::ostringstream oss;
        oss << "Exception during zlib decompression: (" << ret << ") "
            << zs.msg;
        throw(std::runtime_error(oss.str()));
    }

    return outstring;
}

std::string decompress_gzip(const std::string& str)
{
    z_stream zs;                        // z_stream is zlib's control structure
    memset(&zs, 0, sizeof(zs));

    if (inflateInit2(&zs, MOD_GZIP_ZLIB_WINDOWSIZE + 16) != Z_OK)
        throw(std::runtime_error("inflateInit failed while decompressing."));

    zs.next_in = (Bytef*)str.data();
    zs.avail_in = str.size();

    int ret;
    char outbuffer[32768];
    std::string outstring;

    // get the decompressed bytes blockwise using repeated calls to inflate
    do {
        zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
        zs.avail_out = sizeof(outbuffer);

        ret = inflate(&zs, 0);

        if (outstring.size() < zs.total_out) {
            outstring.append(outbuffer,
                             zs.total_out - outstring.size());
        }

    } while (ret == Z_OK);

    inflateEnd(&zs);

    if (ret != Z_STREAM_END) {          // an error occurred that was not EOF
        std::ostringstream oss;
        oss << "Exception during zlib decompression: (" << ret << ") "
            << zs.msg;
        throw(std::runtime_error(oss.str()));
    }

    return outstring;
}


// Compile: g++ -std=c++11 % -lz
// Run: ./a.out

#include <iostream>

int main()
{
    std::string s = "Hello";

    std::cout << "input: " << s << std::endl;
    std::cout << "gzip compressed: " << compress_gzip(s) << std::endl;;
    std::cout << "gzip decompressed: " << decompress_gzip(compress_gzip(s)) << std::endl;;
    std::cout << "deflate compressed: " << compress_deflate(s) << std::endl;;
    std::cout << "deflate decompressed: " << decompress_deflate(compress_deflate(s)) << std::endl;;
}

Compile

g++ -std=c++11 % -lz

Run

./a.out

Output

Optimizing your website for performance

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 29 May 2014 00:00:00 +0000

May 29 2014

Optimizing your website for performance

Sometimes as a webdeveloper I have to work with websites where performance is not optimal and sacrifised in exchange for some other quality attribute. Perhaps that's why I "over optimize" some sites--like this blog--which probably is not even worth the effort considering the traffic it gets. However, it's something I spend my time on and it could be useful to others. And this being a blog, I blog about it.

Statically compile

In this blog I statically "compile" this websit with a minimalist tool I created. It was a project with a similar approach as octopress (based on Jekyll). I've never used octopress, I'm not sure if it even existed back when starting this blog.

A webserver likes plain file serving more than CPU intensive PHP scripts (that generate pages per request). You may not need to statically compile everything like I do, there are alternatives like using Varnish cache to optimize your performance perhaps. Tweakers.net (and many more high performance websites) use Varnish.

YSlow & Page Speed diagnostiscs

These are browser plugins, but you can do the checks online via services like: GTmetrix or WebPageTest. In the following screenshot are as an example some quick wins I made using GTmetrix in two ours of optimizing my blog.

In this blog I won't go over all the Tips given on those websites, but there is a lot of useful advice in there I won't repeat. Except for maybe one more: Caching headers for clients (and (caching) proxies), you might want to make sure you get those right.

Cloudfront as a Content Delivery Network (CDN)

In google analytics I noticed that in some countries my blog was a lot slower. Especially the loading of media. I verified this with WebPageTest, apparently my AWS server that was located in Ireland was too far away for some countries. Maybe not especially slow for the HTML, but especially for media, like images, a.k.a. "Content". Hence, CDN

First how to setup a CDN with Cloudfront...

You start with creating an "S3" bucket, which is basically a "dumb" but fast hosting for files. For every file you put there you have to set permissions, headers (if you want those headers to be returned to the client (i.e. Content-Type, etc.)). Normally a webserver like apache would do most of that stuff for you, here you have to do it yourself. All your static content should be uploaded to such a bucket.

Your bucket is something named like <BUCKETNAME>.s3-website-eu-west-1.amazonaws.com. As you can guess from the domain name, the bucket is hosted in one location. With "one location" I mean the unique URL, but also "EU West 1", which is one region (or location).

Files in the bucket are accessible via HTTP: //cdn-cppse-nl.s3-website-eu-west-1.amazonaws.com/some/path/to/a/file.txt.

If you put Cloudfront in front of your bucket, you can make if fetch from your bucket, and "cache" it in X locations all over the world. Your cloudfront endpoint is something like <UNIQUEID>.cloudfront.net.

When you want to use another domain in your HTML, you can define a CNAME (canonical name, a hostname that resolves to another hostname basically), which is a dns record that points to the cloudfront.net domain. In my case cdn.cppse.nl points to d15zg7znhuffnz.cloudfront.net (which reads from cdn-cppse-nl.s3-website-eu-west-1.amazonaws.com).

Creating a CNAME has the advantage that you can at DNS level switch to another CDN. If I were to use akamai, I can make the CNAME point to something from akamai. I could also prepare a totally new bucket and/or cloudfront endpoint, and then switch to the new endpoint by changing the CNAME to another <UNIQUEID2>.cloudfront.net.

Second how Cloudfront / CDNs work...

Cloudfront hostname resolves to multiple IP addresses. At DNS level the actual "edge" (a.k.a. server, in terms of it's ip address) is chosen where the actual files are to be fetched from. You can interpret all the cloudfront edges as "mirrors" for your S3 bucket. On my home PC (which is currently near Amsterdam) when I resolve cdn.cppse.nl (d15zg7znhuffnz.cloudfront.net) it is resolved to a list of IP's:

> cdn.cppse.nl
Server:     127.0.1.1
Address:    127.0.1.1#53

Non-authoritative answer:
cdn.cppse.nl    canonical name = d15zg7znhuffnz.cloudfront.net.
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.13.72
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.12.206
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.12.246
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.13.12
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.13.65
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.12.249
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.13.2
Name:   d15zg7znhuffnz.cloudfront.net
Address: 54.230.15.34

The fastest "edge" for my location is returned first and has the address: 54.230.13.72. If you reverse DNS lookup that IP you can see "ams1" in the hostname the ip resolved too.

> 54.230.13.72
Server:     127.0.1.1
Address:    127.0.1.1#53

Non-authoritative answer:
72.13.230.54.in-addr.arpa   name = server-54-230-13-72.ams1.r.cloudfront.net.

Authoritative answers can be found from:

This is specific to how Amazon structures their hostnames. ams1 in server-54-230-13-72.ams1.r.cloudfront.net stands for Amsterdam Airport Schiphol. Their coding is based on the closest International Airport IATA code. You can check how the domain resolves and what latency/packetloss is from a lot of different "checkpoints" (locations) at once, with tools like APM Cloud Monitor. In China/Hong Kong the address it resolves to is: 54.230.157.116. The reverse dns resolution for that ip is server-54-230-157-116.sin3.r.cloudfront.net, where SIN is the code for Republic of Singapore. So they won't have to download my javascript/css/media there all the way from Amsterdam.

If your website is entirely static, you could host everything this way. And hopefully people from all over the world can benefit from fast downloads from their nearest edges.

A few challenges I had with Cloudfront

After switching to Cloudfront I first noticed that loadtimes increased! I forgot that my apache used mod_gzip (and mod_deflate) to send text-based content compressed to the http client/webbrowser. But I uploaded my stuff to S3 "as is", which is plain/text and not gzipped.

A webbrowser normally sends in it's request whether it supports gzip or deflate encoding. If it does, apache will compress the content in a way to the client's preference, otherwise it will serve the content "as is". S3 is simply a key-value store in a way, so this conditional behaviour based on a client's headers like Accept-Encoding:gzip,deflate,sdch isn't possible. In the documentation you see that you have to create separate files.

Unfortunately Javascript doesn't have access to the browsers Accept-Encoding header (that one my chromium sends). So you cannot document.write the correct includes based on this client-side. That was my first idea.

How I now resolved it: For the CSS and Javascript files served from Cloudfront, I upload the plain version file.css and a compressed version file.gz.css. With correct headers etc., like this:

# Create expiry date
expir="$(export LC_TIME="en_US.UTF-8"; date -u +"%a, %d %b %Y %H:%M:%S GMT" --date "next Year")"

# Copy global.js to global.gz.js and gzip compress the copy
cp -prv global.js global.gz.js
gzip -9 global.gz.js

# Upload the files to bucket
s3cmd --mime-type=text/css \
      --add-header="Expires:$expir" \
      --add-header=Cache-Control:max-age=31536000 \
      -P put global.js s3://cdn-cppse-nl/global.js

s3cmd --mime-type=text/css \
      --add-header=Content-Encoding:gzip \
      --add-header="Expires:$expir" \
      --add-header=Cache-Control:max-age=31536000 \
      -P put global.gz.js s3://cdn-cppse-nl/global.gz.js

s3cmd is awesome for uploading stuff to s3 buckets. It also has a very useful sync command.

Perfect routing

Because I now have separate files for compressed and uncompressed javascript/css files, I cannot serve my static HTML files "blindly" from my CDN anymore. I now have to make sure I send the HTML either with references to file.gz.css or file.css based on the client's browser request headers.

So I introduced "Perfect routing", okay, I'm kind of trolling with the "Perfect" here, but I use "perfect hash generation" with gperf. At compiletime I output an input file for gperf and have gperf generate a hash function that can convert an article name string (the Request URI) to an Index (or to nothing in case the request is invalid). That Index points directly to the position in a map that it also generates containing the filename that corresponds to the article name. In that way it can fetch the file in a single lookup, and the filesystem is never hit for invalid requests.

My routing.cgi program does the following:

Read the Accept-Encoding header from the client.
If the generated hash function returns the filename: read the HTML into memory.
If client requested gzip encoding: replace in the HTML javascript and CSS includes with a regex to their .gz.js versions. Compress the HTML itself too in the same encoding.
If client requested deflate encoding: do the same with deflate encoding on the HTML, but I currently didn't implement .defl.js versions for the Javascript and CSS.
Also after fixing the javascript and CSS includes, compress the HTML now in the same way (gzip or deflate).
For fun it will add a X-Compression header with compression ratio info.
If the compressed length exceeds the plain version, it will use the plain version.

routing.cgi apache 2.4 config

For now routing.cgi is a simple cgi program, it could be further optimized by making it an apache module, or perhaps using fastcgi.

AddHandler cgi-script .cgi
ScriptAlias /cgi-bin/ "/srv/www/vhosts/blog.cppse.nl/"

<Directory "/srv/www/vhosts/blog.cppse.nl/">
AllowOverride None
Options +ExecCGI -Includes
Order allow,deny
Allow from all
Require all granted

DirectoryIndex /cgi-bin/routing.cgi
FallbackResource /cgi-bin/routing.cgi
</Directory>

routing.cpp a few code snippets

1) Determine encoding:

char *acceptEncoding = getenv("HTTP_ACCEPT_ENCODING");
enum encodings {none, gzip, deflate};
encodings encoding = none;

if (acceptEncoding) {
    if (strstr(acceptEncoding, "gzip"))
        encoding = gzip;
    else if (strstr(acceptEncoding, "deflate"))
        encoding = deflate;
}

2) The hash lookup:

static struct knownArticles *article = Perfect_Hash::in_word_set(requestUri, strlen(requestUri));
if (article) {
    printf("X-ArticleId: %d\n", article->articleid);
    printf("X-ArticleName: %s\n", article->filename);
    printf("Content-Type: text/html; charset=utf-8\n");
    ....
    ss << in.rdbuf();
} else {
    printf("Status: 404 Not Found\n");
    printf("Content-Type: text/html; charset=utf-8\n");
    ss << "404 File not found";
}

3) The regexes:

if (encoding == gzip) {
    //std::regex regx(".css");
    //str = std::regex_replace(str, regx, string(".gz.css"));
    const boost::regex scriptregex("(<script[^>]*cdn.cppse.nl[^ ]*)(.js)");
    const boost::regex cssregex("(<link[^>]*cdn.cppse.nl[^ ]*)(.css)");
    const std::string fmt("\\1.gz\\2");
    str = boost::regex_replace(str, scriptregex, fmt, boost::match_default | boost::format_sed);
    str = boost::regex_replace(str, cssregex, fmt, boost::match_default | boost::format_sed);

    cstr.assign(compress_string2(str));
}

4) Compressing to gzip or deflate:

I simply took of Timo Bingmann his {de}compress_string functions, whom use deflate, and created gzip versions of these. They took me a while to get right, to find the correct parameters etc., so you may find them useful.

You can find them here: Deflate and Gzip Compress- and Decompress functions.

More "performance wins": deferred javascript loading

As a base for the html I use adaptiv.js which provides a grid layout for a responsive design, with the following config:

// Find global javascript include file
var scripts = document.getElementsByTagName('script'),
    i = -1;
while (scripts[++i].src.indexOf('global') == -1);

// See if we're using gzip compression, and define config for adaptiv.js
var gzip = scripts[i].src.indexOf('.gz.js') != -1,
    rangeArray = gzip
        ? [
             '0px    to 760px  = mobile.gz.css',
             '760px  to 980px  = 720.gz.css',
             '980px  to 1280px = 960.gz.css',
             '1280px           = 1200.gz.css',
          ]
        : [
             '0px    to 760px  = mobile.css',
             '760px  to 980px  = 720.css',
             '980px  to 1280px = 960.css',
             '1280px           = 1200.css'
         ];

Only after loading the javascript will it correctly "fix" the right css include, introducing an annoying "flicker" effect. This makes it necessary to require the adaptiv.js javascript asap (a.k.a. in the header of the page).

To fix this I simply added the same css includes with media queries:

<link href='//cdn.cppse.nl/global.css' rel='stylesheet'>
<link href='//cdn.cppse.nl/assets/css/960.css' media='only screen and (min-width: 980px) and (max-width: 1280px)' rel='stylesheet'>
<link href='//cdn.cppse.nl/assets/css/720.css' media='only screen and (min-width: 760px) and (max-width: 980px)' rel='stylesheet'>
<link href='//cdn.cppse.nl/assets/css/1200.css' media='only screen and (min-width: 1280px)' rel='stylesheet'>
<link href='//cdn.cppse.nl/assets/css/mobile.css' media='only screen and (min-width: 0px) and (max-width: 760px)' rel='stylesheet'>

Now the javascript in adaptiv.js is only a fallback for browsers that don't support these queries. All javascript can now be included after DOM elements are loaded/rendered. Nowadays there may be different libraries that don't have this problem. But I'm not up-to-date on that .

As long as you make sure the javascript is loaded after all elements are drawn. Personally I don't put do that in the <body>'s onload="" attribute, as that is executed after everything on your page is loaded. I prefer to put it right before the body tag closes (</body>), as only the static DOM should have been outputted for the browser. (You may want your javascript photo-viewer loaded before the very last thumbnail is done loading for example.)

// Now start loading the javascript...
var j = document.createElement('script');
j.type = 'text/javascript';
j.src = '//cdn.cppse.nl/global.' + (gzip ? 'gz.' : '') + 'js?' + version;
document.body.appendChild(j);

You can also do this for certain stylesheets, like the print.css perhaps.

Image compression

Another huge gain is of course compressing images. I have PNG, JPG and GIF images. Before I choose a format I already try to choose the most appropriate encoding. Typically PNG or GIF for screenshots, JPG for everything else. In my static site generation, I also optimize all images.

TruePNG + PNGZopfli are the best for lossless compression, see this awesome comparison. They are windows tools, but they run perfectly with wine, albeit a little slower that's how I use them. For gifs I use gifsicle (apt-get install gifsicle), and for JPG jpegoptim (apt-get install jpegoptim).

minimize_png:
    wine /usr/local/bin/TruePNG.exe /o4 "file.png"
    wine /usr/local/bin/PNGZopfli.exe "file.png" 15 "file.png.out"
    mv "file.png.out" "file.png"

minimize_jpg:
    jpegoptim --strip-all -m90 "file.jpg"

minimize_gif:
    gifsicle -b -O3 "file.gif"

Excerpt from minimize.log:

compression of file 86-large-thumb-edges.png                                               from 96K        to 78K
compression of file 86-edges.png                                                           from 45K        to 35K
compression of file 86-thumb-edges.png                                                     from 38K        to 31K
compression of file 600width-result.png                                                    from 1,3M       to 121K
compression of file 72-large-thumb-userjourney.jpg                                         from 100K       to 29K
compression of file 57-large-thumb-diff_sqrt_carmack_d3rsqrt.jpg                           from 557K       to 95K
compression of file 63-result.jpg                                                          from 270K       to 137K
compression of file 55-large-thumb-vim.jpg                                                 from 89K        to 27K

Replace google "Custom search" with something "faster"

Google's custom search is pretty useful and easy to setup. I did have to do a few nasty CSS hacks to integrate it in my site the way I wanted.

The advantage is that is does the crawling automatically.
The disadvantage is that it does the crawling automatically.
Another disadvantage is that it adds a lot of javascript to your site.

In my case when I search for a keyword "foobar", google search would yield multiple hits:

/index - main page
/blog - category
/1 - paging
/foobar - individual article

Wanting more control I switched to Tokyo Dystopia . You can see it in action when searching via the searchbox on top of the page. For this to work I now also generate a "searchdata" outputfile alongside the HTML, which is simply the "elaborated" Markdown text. With elaborated I mean code snippets and user comments included.

CSS and Javascript minification

Not going into detail on this one. As a tool I use mince, which by default provides css and js minification with with csstidy and jsmin. For javascript I made a small adjustment to make it use crisp (who is an old collegue of mine)'s JSMin+. Because it's a more advanced minifier that yields even smaller javascript files.

Deferred loading of embedded objects like YouTube videos

I found YouTube/Vimeo & Other flash embeds in my blog posts annoyingly slow. So what I do is put the actual embed code in a container as an HTML comment. For example: I have a custom video player for an mp4 stream like this:

<p><center>
<a class="object_holder" href="#" onclick="videoInLink(this)" style="background:url(//cdn.cppse.nl/83-videothumb.png); width: 750px; height:487px; display:block;">
<!--
<object id="player" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" name="player" width="750" height="487">
    <param name="movie" value="//cdn.cppse.nl/player.swf" />
    <param name="allowfullscreen" value="true" />
    <param name="allowscriptaccess" value="always" />
    <param name="flashvars" value="file=//blog.cppse.nl/videos/wxhttpproxy.mp4&repeat=never&shuffle=true&autostart=true&streamer=lighttpd&backcolor=000000&frontcolor=ffffff" />
    <embed
        type="application/x-shockwave-flash"
        id="player2"
        name="player2"
        src="//cdn.cppse.nl/player.swf"
        width="750"
        height="487"
        allowscriptaccess="always"
        allowfullscreen="true"
        flashvars="file=//blog.cppse.nl/videos/wxhttpproxy.mp4&repeat=never&shuffle=true&autostart=true&streamer=lighttpd&backcolor=000000&frontcolor=ffffff"
    />
</object>
-->
</a>
</center></p>

The object_holder displays a screenshot of the player, giving the illusion it's already loaded. Only when the user clicks the screenshot the commented <object> is inserted using the videoInLink javascript function.

function videoInLink(anchor)
{
    anchor.innerHTML = anchor.innerHTML.replace('<!--','').replace('-->','');
    anchor.removeAttribute('href');
    anchor.onclick = null;
    return false;
}

The end

Concluding my ramblings, I hope you may have find something of use in this post.

Lame PHP Parser, visualization and how cool hhvm is!

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 12 May 2014 00:00:00 +0000

May 12 2014

Lame PHP Parser, visualization and how cool hhvm is!

Created a "Lame PHP Parser" that in this picture visualizes it's own sourcecode!

I thought the output was pretty cool.

It's not the most efficient code, which is fine for my current use case. I was wondering if I could speed it up with Facebook's HHVM (HipHop VM), a PHP virtual machine.

HipHop VM executes the code 5,82 times faster!(*)

Parsing the Symfony code in my 'vendor' directory, takes ~57 seconds with the PHP executable provided by Ubuntu.

PHP 5.5.3-1ubuntu2.1 (cli) (built: Dec 12 2013 04:24:35) 
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2013 Zend Technologies
    with Zend OPcache v7.0.3-dev, Copyright (c) 1999-2013, by Zend Technologies

Benchmarks from three successive runs:

real 0m57,34s, user 0m56,75s, sys 0m0,29s
real 0m57,18s, user 0m56,58s, sys 0m0,28s
real 0m58,38s, user 0m57,72s, sys 0m0,34s

Parsing the same directory three times with hhvm.

HipHop VM 3.0.1 (rel)
Compiler: tags/HHVM-3.0.1-0-g97c0ac06000e060376fdac4a7970e954e77900d6
Repo schema: a1146d49c5ba0d6db903beb3a4ed8a3766fef182

Benchmarks from three successive runs:

real 0m9,86s, user 0m9,04s, sys 0m0,54s
real 0m9,85s, user 0m9,15s, sys 0m0,45s
real 0m9,74s, user 0m9,06s, sys 0m0,44s

(*) 57.34 / 9.86 = 5,815415822

PHP Benelux 2014

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Tue, 22 Apr 2014 00:00:00 +0000

April 22 2014

PHP Benelux 2014

Solving the puzzle at PHPBenelux 2014

Last conference at the Dutch PHP Conference (DPC) I wrote a summary (in Dutch). This time I couldn't find the time, I took notes though, but I was/am too lazy. For PHP Benelux 2014 experiences you can read on other people their blogs, for example the one David wrote at his blog. And if you're looking for sheets, chances are you can find them at the event's joind.in page.

I made one big mistake at PHPBenelux, that was missing (by accident (don't ask why :p)) Ross Tuck's talk. Was really curious about that one. Of the other talks I especially liked Bastian Hofmann's talk "Marrying front with back end". His talk at DPC was also very good IIRC.

Dutch web alliance puzzle

An old collegue of mine (Marijn) and I decided to programmatically solve the puzzle created for the event there DWA. You can find the puzzle instructions in the photo I took (see fig.2.) We heard a rumour that you could win an ipad if you solved it. Unfortunately that wasn't true . In short you have to place all cubes side-by-side in such a way that rotating them combined should yield four unique labels each rotation step. There is only one possible solution.

We decided both on different approaches and see who would solve it first. Challenge accepted! I lost unfortunately, I even had to abandon my first approach and made the switch to semi-brute-force as soon as I gained more insight in how it could be solved more easily this way. Marijn approached it brute force from the start.

I gave each cube a number {1, 2, 3, 4}, and each sidename a letter {a, b, c, d, x, y}. Next I defined for each cube three directions {a, b, c, d}, {x, b, y, d} and {x, c, y, a} (see fig.1.). All these directions can also be in reverse ({d, c, b, a}, ...).

fig.1. possible directionsfig.2. puzzle instructions

My first idea was to place a cube on a side randomly, choosing for each cube a random direction to rotate in, and try four rotation steps. If in all steps the pictures were unique that would be a solution. I made a mistake in the implementation, totally forgot that {x, c, y, a} is also a possible sequence. What also didn't help is that I misread the instructions at first, thinking all sides had to be equal (instead of not-equal), which isn't possible if you take a good look at the actual cubes. I guess beer + coding isn't always a good combination.

My second approach was simply, generate "layouts" for each cube's three directions, i.e. for {a, b, c, d}:

{a, b, c, d}
{b, c, d, a}
{c, d, a, b}
{d, a, b, c}
{d, c, b, a}
{c, b, a, d}
{b, a, d, c}
{a, d, c, d}

For each cube a, b, c, d mean different side types (the different logos, php, phpbenelux, dwa, phpelephant). I simply compare every combination, and filter out the duplicate results. You initially get eight solutions, because every solution you can start at the first, second, third or fourth "step" of the solution, and eight because of the extra "times two" multiplier you get for forward versus backward rotation.

Solver in C++

Code also available on http://ideone.com/ttFYMD.

#include <iostream>
#include <algorithm>
#include <stdexcept>
#include <string>
#include <sstream>
#include <vector>
#include <array>
#include <iomanip>

enum class sidename
{
    a, b, c, d, x, y
};

enum class sidetype
{
    phpbenelux, php, phpelephant, dwa
};

std::string sidename_str(sidename sn) {
    switch (sn) {
        case sidename::a: return "a";
        case sidename::b: return "b";
        case sidename::c: return "c";
        case sidename::d: return "d";
        case sidename::x: return "x";
        case sidename::y: return "y";
    }
    throw std::runtime_error{"invalid sidename"};
}

std::string sidetype_str(sidetype st) {
    switch (st) {
        case sidetype::phpbenelux: return "phpbenelux";
        case sidetype::php: return "php";
        case sidetype::phpelephant: return "phpelephant";
        case sidetype::dwa: return "dwa";
    }
    throw std::runtime_error{"invalid sidetype"};
}

/**
 * cubes define 6 side types (or pictures)
 *
 * cubes calculate for themselves all possible layouts, meaning
 * if you rotate them into some direction, you get a side, followed by side+1, side+2, side+3.
 * there are a few possibilities: reverse order, and in each layout (or "path") you can start
 * rotating the cube on side, side+1, side+2 or side+4 (starting point "shifts").
 */
class cube
{
public:

    cube(sidetype a, sidetype b, sidetype c, sidetype d, sidetype x, sidetype y)
        : a_{a}, b_{b}, c_{c}, d_{d}, x_{x}, y_{y},
          directionnames_({{ 
             {sidename::a, sidename::b, sidename::c, sidename::d},
             {sidename::x, sidename::b, sidename::y, sidename::d},
             {sidename::a, sidename::y, sidename::c, sidename::x} }}),
          directions_({{ {a, b, c, d}, {x, b, y, d}, {a, y, c, x} }})
    {
        for (int i=0; i<4; i++) {
            for (auto &sides : directions_) {
                // normal insert
                layouts_.push_back(sides);

                // reverse insert
                auto sidesrev = sides;
                std::reverse(std::begin(sidesrev), std::end(sidesrev));
                layouts_.push_back(sidesrev);

                // shift all
                sidetype temp = sides[0];
                for (int i=1; i<=3; i++)
                    sides[i - 1] = sides[i];
                sides[3] = temp;
            }
        }
    }

    const std::vector<std::array<sidetype, 4>> & layouts() { return layouts_; }

private:

/**
 * This is how I labeled each sidetype:
 * 
 *    X  =   a
 *  X X X   = x b y 
 *    X  =   c
 *    X  =   d
*/

    sidetype a_;
    sidetype b_;
    sidetype c_;
    sidetype d_;
    sidetype x_;
    sidetype y_;

    std::array<std::array<sidename, 4>, 3> directionnames_;
    std::array<std::array<sidetype, 4>, 3> directions_;
    std::vector<std::array<sidetype, 4>> layouts_;
};

/**
 * helper class that can see if a given solution is a duplicate from a previous solution
 * 
 * if you have a solution that is simply the same one, but rotating in a different direction
 * is not really a new solution. also note the four possible starting point in each layout/path.
 * so it will check if duplicates exist in both forward and backward directions, and for each
 * possible four shifts
 */
class solutions
{
public:
    solutions()
    {}

    bool is_dupe(std::array<std::array<sidetype, 4>, 4> temp2)
    {
        // Check if found solution isn't a duplicate
        bool duplicate = false;
        for (auto &solution : solutions_) {
            for (int j=0; j<8; j++) {
                duplicate = true;
                int sidenum = 0;
                for (auto &side : solution) {
                    auto &temp = temp2[sidenum++];

                    int count = 0;
                    int offset = j % 4;
                    if (j < 4) {
                        // Check if they are duplicates, as we use offsets of +0, +1, +2, +3 we can 
                        //  detect shifted duplicate results.
                        for (auto i = side.begin(); i != side.end(); i++) {
                            duplicate = duplicate && temp[(count + offset) % 4] == *i;
                            count++;
                        }
                    }
                    else {
                        // Check if they are duplicates simply in reverse order, also with the 
                        //  detect for shifted duplicates.
                        for (auto i = side.rbegin(); i != side.rend(); i++) {
                            duplicate = duplicate && temp[(count + offset) % 4] == *i;
                            count++;
                        }
                    }
                }
                if (duplicate)
                       return true;
            }
        }

        // Remember found solution, for duplicates checking
        solutions_.push_back(temp2);

        return false;
    }

private:
    std::vector<std::array<std::array<sidetype, 4>, 4>> solutions_;
};

int main (int argc, char *argv[]) 
{
/*
 * on the sheet:
 * 
 *    cube 4 (sideways)
 * 
 *   cube 1, 2, 3
*/

    cube one{
        sidetype::dwa,
        sidetype::phpelephant,
        sidetype::phpbenelux,
        sidetype::dwa,
        sidetype::php,
        sidetype::phpbenelux};

    cube two{
        sidetype::phpelephant,
        sidetype::phpbenelux,
        sidetype::phpbenelux,
        sidetype::phpbenelux,
        sidetype::php,
        sidetype::dwa};

    cube three{
        sidetype::phpbenelux,
        sidetype::dwa,
        sidetype::phpelephant,
        sidetype::php,
        sidetype::dwa,
        sidetype::phpelephant};

    cube four{
        sidetype::php,
        sidetype::phpelephant,
        sidetype::phpbenelux,
        sidetype::phpelephant,
        sidetype::dwa,
        sidetype::php};

    solutions solution;

    for (auto &cube1sides : one.layouts()) {
    for (auto &cube2sides : two.layouts()) {
    for (auto &cube3sides : three.layouts()) {
    for (auto &cube4sides : four.layouts()) {

        // Pictures have to be unique on each four cubes to be considered a unique solution..
        bool flag = false;
        for (int i=0; i<4; i++) {
            // .. Also on each four rotations of course
            flag = flag || (!(cube1sides[i] != cube2sides[i] &&
                              cube1sides[i] != cube3sides[i] &&
                              cube1sides[i] != cube4sides[i] &&
                              cube2sides[i] != cube3sides[i] &&
                              cube2sides[i] != cube4sides[i] &&
                              cube3sides[i] != cube4sides[i]));
        }
        if (!flag){
            // Skip duplicate solutions
            if (solution.is_dupe({cube1sides, cube2sides, cube3sides, cube4sides})) {
                continue;
            }

            // Print the result
            std::cout << "The cube-layout for the solution:" << std::endl << std::endl;

            static auto print = [](const std::string &cube, decltype(cube1sides) &sides) {
                std::cout << cube << ": " 
                    << " front: " << std::setw(15) << sidetype_str(sides[0]) << ", "
                    << " up: " << std::setw(15) << sidetype_str(sides[1]) << ", "
                    << " top:   " << std::setw(15) << sidetype_str(sides[2]) << ", "
                    << " back:  " << std::setw(15) << sidetype_str(sides[3])
                    << std::endl;
            };

            print("cube #1", cube1sides);
            print("cube #2", cube2sides);
            print("cube #3", cube3sides);
            print("cube #4", cube4sides);
        }
    }}}}
}

Output:

cube #1:    front:             php,     up:     phpelephant,    top:        phpbenelux,     back:              dwa
cube #2:    front:      phpbenelux,     up:             dwa,    top:       phpelephant,     back:              php
cube #3:    front:     phpelephant,     up:      phpbenelux,    top:               dwa,     back:      phpelephant
cube #4:    front:             dwa,     up:             php,    top:               php,     back:       phpbenelux

Performance is on my pc 0,00202 seconds on average per run (see perf.txt). The average is 0,00181 seconds, 11.60% faster, if we stop processing when we find the 'first' solution.

I still think my first idea should also work and I'm curious if that "random brute-force" algorithm would on average find a solution faster compared to finding the first solution in my current solver (in terms of required compares). .

Use cache in http proxy for debugging in webdevelopment

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 05 Feb 2014 00:00:00 +0000

February 5 2014

Use cache in http proxy for debugging in webdevelopment

Serving specific requests from a mutable cache

You can record requests now in a new tab "Cache", and you can choose to use this cache per request. Any such request will be immediately returned from the cache (instead of being proxied).

The cool thing is that you can modify the cache, so modifying CSS / Javascript is easy for example. I wanted this feature so I can debug some stuff on environments other than development. There are other ways to do this, but I think this is very convenient.

Some use cases

Change <script src=... includes to URL's from production to your dev environment, to simply test javascript on production pages.
Change javascript includes to their unminified / deobfuscated versions..
Add 'debugger' / console.log's to the source.
I've used it to more conveniently debug www.google-analytics.com/ga.js (an obfuscated javascript), by de-obfuscating it and using the step debugger in chrome.

By the way, a cool way to set a breakpoint in Javascript on specific function calls could be this. In ga.js there is a _gaq array, with a push function _gaq.push([something]);, I simply added this with wxhttpproxy in ga.js:

var foo = _gaq.push;
_gaq.push = function () {
    debugger;         <<<<<  and the debugger will intercept, you can view the stacktrace
    return foo.apply(_gaq, arguments);
}

Features / Roadmap

Note that the wxhttpproxy will intelligently modify the Content-Length header in case you modify the content. Content-Encodings gzip/deflate are not yet supported, for the time-being the proxy will garble the Accept header (by replacing gzip -> nogz, and deflate -> noflate). This is a workaround, that makes sure gzip/deflate isn't used.

Supported for chunked encoding should be added (easy), you can already edit it manually, just remove the chunked header, the hex numbers and add a Content-Length to work around it.

Real support for gzip / deflat should be added. I already have code for gzip / deflate support laying around somewhere.

Download

As an Ubuntu/debian package available here.

It uses wx2.8. In wx3.0 the event model has changed a bit, and I believe there is an error in the event handling w/regards to sockets. I will post some more details on this later, and if I have the time see if I can fix it..

Install NVIDIA proprietary drivers in Ubuntu 13.10 or Linux mint 15/16

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 26 Dec 2013 00:00:00 +0000

December 26 2013

Install NVIDIA proprietary drivers in Ubuntu 13.10 or Linux mint 15/16

My super awesome NVIDIA Quadro K600 doesn't work properly with the default video drivers in Linux mint 15, 16 or Ubuntu 13.10. Especially in mint it was especially unstable. In Ubuntu everything seems fine for a few days, until the GPU finally crashed as well.

Linux mint 15 / 16

You disable the default driver called nouveau, to be loaded by the kernel with nouveau.blacklist=1.

In mint I've tried editing GRUB_CMDLINE_LINUX etc. in /usr/share/grub/default/grub and all of /etc/grub.d/*. Somehow update-grub didn't parse it, I was not so patient, so I ended up simply editting /boot/grub/grub.cfg.

trigen@Firefly21 ~ $ vim /boot/grub/grub.cfg
...
:g/^\s*linux
        linux   /boot/vmlinuz-3.8.0-19-generic root=UUID=60f8754f-f688-461e-b120-bf8402c1e2a9 ro nouveau.blacklist=1
        linux   /boot/vmlinuz-3.8.0-19-generic root=UUID=60f8754f-f688-461e-b120-bf8402c1e2a9 ro recovery nomodeset nouveau.blacklist=1
        linux16 /boot/memtest86+.bin
        linux16 /boot/memtest86+.bin console=ttyS0,115200n8
Press ENTER or type command to continue

Reboot the system. CTRL+ALT+F1, login, sudo service mdm stop, ./NVIDIA-Linux-x86_64-331.20.run and follow instructions. Reboot.

Ubuntu 13.10

In ubuntu I attempted to directly edit /boot/grub/grub.cfg again. Adding the blacklist parameter, somehow this failed, the NVIDIA installer still complaining about nouveau being loaded.

So I attempted the 'normal approach' again: vim /etc/default/grub, modified this line: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset nouveau.blacklist=1". I also googled and found this answer on stackoverflow, suggesting the nomodeset is necessary as well. (So I added both). sudo update-grub and EAT REBOOT INSTALL REPEAT.

Reboot the system. CTRL+ALT+F1, login, sudo service lightdm stop, ./NVIDIA-Linux-x86_64-331.20.run and follow instructions. Reboot.

Some notes: Install grub into specific partition

This is a "Note to self", how to install grub for specific partition (source). I needed this command fixing my dual boot (linux and windows on one ssd).

sudo mount /dev/sda5 /mnt
sudo grub-install --root-directory=/mnt/ /dev/sda
sudo update-grub

Somehow linux mint f*cked up my boot to windows 8 partition. It had some problems recognizing my partition table or something. (At work I have the exact same setup, and there were no problems.) I ended up fixing it with the above command, and from windows (had to restore an image) using this tutorial that uses EasyBCD.

A proper ksh profile with proper handling of history over subshells.

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 25 Dec 2013 00:00:00 +0000

December 25 2013

A proper ksh profile with proper handling of history over subshells.

In shell scripting I prefer the Kornshell. A while ago I experimented with "oh my zsh", but I switched back to ksh. Their auto completion for program commands is really unsurpassed (tab completion on program parameters for grep for example). The auto-incorrect however, is quite annoying . There is also a git plugin that visualized the active git branch in the $PS1 prompt. I liked these features and I want to add them to ksh.

Apparently I like old things

I fixed a few things for ksh in my .kshrc at bitbucket.

Separated history amongst the different ksh shells.

My problem with the default behaviour: by default history is globally shared amongst all shells. I tend to work in a screen and do different stuff in each buffer. So it's annoying if stuff from one buffer magically appears in the other buffer.

So what I do is I make sure there is one history file ~/.ksh_history which contains all history from all shells. When starting a new shell I copy this file into a history file specific for that ("sub")shell, i.e. ~/.ksh_history_files/<shellpid>. Each new shell does this.

When a shell is started all history files from ~/.ksh_history_files/* are merged back into ~/.ksh_history. And the ones that are no longer in use (shells have exited) are removed. This is done with a simple lsof call.

Commands are processed through a simply ksh function that makes sure all history lines are unique, without changing the order.

Some funny caveats were:

A history file should start with the character sequence \x81\01.
Each command in history file should end with a \x00 character.

A prompt that embeds current git branch

If you are inside a git clone).
Maybe the oh-my-zsh git integration is more advanced, no idea. Luckily this visualization in $PS1 is very fast.

trigen@Firefly21:/projects> cd smashbattle
trigen@Firefly21:/projects/smashbattle[network_multiplayer]> git branch
  master
* network_multiplayer
trigen@Firefly21:/projects/smashbattle[network_multiplayer]>

Fix "cd" (with no params) to be the equivalent of "cd ~" again

Fixed with a simple alias around cd provided by /usr/share/ksh/functions/dirs. For dirs usage check here.

trigen@Firefly21:/usr/local/src/wxWidgets-3.0.0> cd <<<< this didn't work on Ubuntu 13.10 anyways.
trigen@Firefly21:/home/trigen>

Integrated my "launcher tool" I wrote for windows a long time ago.

The browser is assumed to be "chromium-browser" (sudo apt-get install chromium-browser).

The result for gi dan flavin is opened in a new chromium tab.

i <url> - open url in browser
g <search terms> - search with google.com
gi <search terms> - search with google images
gv <search terms> - search with google videos
gnl <search terms> - search with google.nl
gs <search term> - search google scholar
w <search term> - search wikipedia
wa <search terms> - search wolfram alpha
yv <search terms> - search youtube
y <search terms> - search yahoo
yi <search terms> - search yahoo images
yv <search terms> - search yahoo videos
imdb <search terms> - search imdb
h <search term> - search hyperdictionary
v <search term> - search vandale (dutch dictionary)

Usage example: Open run menu (ALT+F2). Then type gi dan flavin to get the example result from screenshot.

I had to abandon my pure-ksh-functions approach to make the launcher commands available everywhere. So you have call install_launcher as root to install the shortcuts as scripts in /usr/local/bin/. Ubuntu does not respect shell functions in run unfortunately.

Free HTTP proxy for debugging purposes with GUI

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Tue, 03 Dec 2013 00:00:00 +0000

December 3 2013

Free HTTP proxy for debugging purposes with GUI

As a free alternative for Charles proxy and/or Fiddler I developed wxHttpProxy in one (long) night! Unfortunately I had to spend the next day making it stable, and support https traffic as well

The mentioned alternatives are really great by the way, and have quite a few additional features, I just needed a simple http proxy that I could use in metalogmon, that's why I developed my own. It is opensource, find it on bitbucket.

The idea is to integrate this http proxy into metalogmon, so that you would be able to process your output through javascript. The reason that request/response are both first class citizens is that where I currently develop software I plan to use this proxy between all subsystems communication, and here multiple services communicate with eachother using REST calls.

I want to visualize the call sequences/hierarchy, like [A -> B -> C], [A <- B <- C]. And not group them together like: [A <> B], [B <> C].

Install binary (64 bit and ubuntu/debian based system only)

Install my key

wget -O - http://cppse.nl/apt/keyFile | sudo apt-key add -

Add my repo to apt's sources.list.

sudo sh -c "echo deb http://cppse.nl/apt/dists/stable/main/binary / >> /etc/apt/sources.list"

Install the package:

sudo aptitude update
sudo aptitude install wxhttpproxy

Usage

wxHttpProxy

Tip of the day, add a shortcut for starting the proxy (like WINKEY+2 or something)

Available shortcuts

alt+i - Toggle intercept mode (capture/display request/response, or be silent)
alt+c - Clear buffer(s)

Catch library

With Catch I developed a few helper classes in a test-driven way, it's a lightweight single-header c++ library for creating unit tests. Really liked working with it and look forward to using it in the future some more.

The apt repository

The shiny new apt repository was also really easy to setup, once you have made your packages, simply follow this wiki.

Other platforms

Should you wish to compile it for windows, it is wxWidgets 3.0.0 based and should already be cross-platform, just didn't test it yet. Use Dialogblocks to compile it (and also have it compile wxWidgets for you (as that's just convenient )).

Away- nicknames coloring in nicklist mIRC

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 20 Oct 2013 00:00:00 +0000

October 20 2013

Away- nicknames coloring in nicklist mIRC

Implemented this feature I saw in XChat

Yes, having switched to Linux and using XChat, I now first realize how slow mIRC is (connecting to my bouncer, joining all kinds of channels, replaying scrollback, etc.-- mirc takes a few seconds, where XChat is "instant"), but I don't care. It's still my client of choice for Windows. On Tweakers IRC network, it is custom for a lot of people to indicate their away status with /nick <nickname>|afk and such. That makes sense, but XChat does something "smart", it periodically /who's all channels to know whom are away so it can color those in light grey in the nicklist. Cool feature I thought! So I "extended" my nicklisting coloring mirc script, by adding aways.ini. It's no longer necessary to keep changing your nickname with an away status (on freenode it isn't even allowed in some channels ).

Away nicks in light-grey in XChat

The same feature in mIRC

It's a pity that such an inefficient implementation is required ("pull"ing /who periodically will result in a lot of extra network data), but the IRC protocol does not currently provide something that "push"es away info more intelligently. You can download the script here, note that aways.ini depends on nicklist.ini, also downloadable there. (nicklist.ini still works without aways.ini, it will then use light grey at random instead.)

Away nicks in light-grey

Away nicks in light-grey, other nicks colored.

Visual studio shortcut keys / settings

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 07 Oct 2013 00:00:00 +0000

October 7 2013

Visual studio shortcut keys / settings

These are the ones I keep forgetting about:

Enabling line numbers: Tools, Options, Text Editor, All Languages, Show Line Numbers (source: http://stackoverflow.com/questions/5905687/can-i-see-line-numbers-in-the-vs2010-editor)
Toggle whitespace/tabs: CTRL+R, CTRL+W. (source: http://stackoverflow.com/questions/4065815/how-to-turn-off-showing-whitespace-characters-in-visual-studio-ide)
Navigate cursor to previous+next position: CTRL+- and forward CTRL+SHIFT+-.

These are a few you can't do without:

Step into function: F12 (CTRL+- to return cursor to previous position)

If you can afford it you can't do without Visual Assist X either.

Raspberry pi camera streaming with crtmpserver test

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 18 Sep 2013 00:00:00 +0000

September 18 2013

Raspberry pi camera streaming with crtmpserver test

Click the play button to start stream. (Will only work if I have the script running on my raspberry pi though. )

raspivid + ffmpeg

# test chaining raspivid and ffmpeg
raspivid -t 5000 -w 960 -h 540 -fps 25 -b 500000 -vf -o - | ffmpeg -i - -vcodec copy -an -r 25 -f flv test.flv

# stream to tcp endpoint
raspivid -n -t 0 -w 1920 -h 1080 -fps 25 -b 2000000 -o - | ffmpeg -i - -vcodec copy -an -f flv -metadata streamName=video2 tcp://cppse.nl:6666

Port 6666 is the same with the sample that is delivered with crtmpserver (the flvplayback.lua sample).

Turn off pi camera led

#!/usr/bin/env python
import time
import RPi.GPIO as GPIO

# Use GPIO numbering
GPIO.setmode(GPIO.BCM)

# Set GPIO for camera LED
CAMLED = 5

# Set GPIO to output
GPIO.setup(CAMLED, GPIO.OUT, initial=False)

# Five iterations with half a second
# between on and off
#for i in range(5):
#  GPIO.output(CAMLED,True)  # On
#  time.sleep(0.5)
#  GPIO.output(CAMLED,False) # Off
#  time.sleep(0.5)

GPIO.output(CAMLED,False) # Off

Think I got this script from here.

Video streaming from code with SFML and ffmpeg

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 01 Sep 2013 00:00:00 +0000

September 1 2013

Video streaming from code with SFML and ffmpeg

Live video streaming

I was interested in specifically the generation of H264 videos directly from code (without audio). This was something I used to do by generating separate .BMP files that I then convert with mencoder. This costs quite a lot of hdd space and was performance-wise not optimal. Of course I already figured that out, but figuring out how to encode directly to video was something on my To Do list.

Believe it or not, I actually had to search a while for libraries and SDK's, before concluding that ffmpeg was the best option. I've had a bad experience with ffmpeg 5 years ago, so I was probably ignoring ffmpeg in the google search results. You can link to ffmpeg's libraries and use them in your own program, they also provide code samples. This made it all pretty easy to setup. So when I completed generating H264 videos from code I wondered how difficult it would be to directly stream video with ffmpeg.

One cool application would be to use it in games, f.i. make a small livestream for actual games being played on the server.

Ffmpeg supports a few protocols, I chose an RTMP server for my setup, which is Flash only. If that works it'll probably work for other protocols as well. The best alternative for HTML5 is probably Apple's HLS Streaming, which actually is pretty awesome, as a side-note from source:

If multiple qualities of a stream are available, the player will continuously monitor the current bandwidth and pick the next fragment from the highest quality it can load. This is called Adaptive Streaming:

For the videoplayer I chose JWplayer version 6.

Setting up an rtmp server with Jwplayer

The servers I tested: rtmplite on linux, crtmpserver on linux and Red5 Media Server on windows. The linux server I used was a Suse (SLED11) micro EC2 instance on Amazon AWS. But these should all work on linux and windows.

RTMP is flash-only, traffics over port 1935.

Some notes on using rtmplite with JWplayer

Eventually I chose not to use rtmplite, but crtmpserver instead, as you have more output there as to what's going on in the server. rtmplite output is less useful to me, by default it outputs nothing, with verbose you get messages at packet level. rtmplite used quite a bit more of my CPU compared to crtmpserver (not very accurate due to EC2, but crtmpserver at 0.3% CPU versus > 10% CPU for rtmplite) and it does not understand aspect ratio other than 4:3, so 16:9 will be "squished" into 4:3. It's not my intent to be negative about rtmplite, if you're a python guy It's probably very easy to tweak and I must say it works out-of-the-box with a simple python rtmp.py, ready for video at rtmp://servername/app/video.

With JWplayer you can view the stream with:

jwplayer("myElement").setup({
    file:       "rtmp://localhost/appflv:video",
    autostart:  true,
    width:      480,
    height:     320
});

If you attempt to use that config, please note that file URL! It took me an hour to get that correct, in all JWPlayer samples you see rtmp://server/app/flv:video.flv or rtmp://server/app/mp4:video.mp4. The prefix flv: or mp4: being just for JWplayer so that it knows what encoding to expect. As the .mp4 or .flv extension in URL's is not mandatory. Eventually JWplayer will strip away flv:. An rtmp url like rtmp://server/app/video consists of an application called app and a video called video. Stripping away the prefix from rtmp://cppse.nl/app/flv:video will provide app/ for the application which looks like app but isn't quite the same.

So... crtmpserver then!

After compiling you fire it up with default settings with: ./crtmpserver/crtmpserver crtmpserver/crtmpserver.lua. You should see something similar to the following output.

ip-10-224-83-43:/usr/local/src/crtmpserver/builders/cmake # ./crtmpserver/crtmpserver ./crtmpserver/crtmpserver.lua
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:203 Initialize I/O handlers manager: epoll
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:206 Configure modules
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:84 Module /usr/local/src/crtmpserver/builders/cmake/applications/appselector/libappselector.so loaded
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:84 Module /usr/local/src/crtmpserver/builders/cmake/applications/flvplayback/libflvplayback.so loaded
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:212 Plug in the default protocol factory
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:219 Configure factories
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:225 Configure acceptors
/usr/local/src/crtmpserver/sources/thelib/src/netio/epoll/iohandlermanager.cpp:100 Handlers count changed: 0->1 IOHT_ACCEPTOR
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:231 Configure instances
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:237 Start I/O handlers manager: epoll
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:240 Configure applications
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:177 Application appselector instantiated
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:177 Application flvplayback instantiated
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:246 Install the quit signal
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:257
+-----------------------------------------------------------------------------+
|                                                                     Services|
+---+---------------+-----+-------------------------+-------------------------+
| c |      ip       | port|   protocol stack name   |     application name    |
+---+---------------+-----+-------------------------+-------------------------+
|tcp|        0.0.0.0| 1935|              inboundRtmp|              appselector|
+---+---------------+-----+-------------------------+-------------------------+
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:258 GO! GO! GO! (19664)

With the default configuration anyone can stream a video to the RTMP server, as inbound and outbound are over port 1935. I tried separating that with the configs, but I couldn't find any configuration setting to do this, so I hacked something in the source to only allow inbound RTMP for 127.0.0.1 (see patch.txt). With crtmpserver you don't have the URL problem in JWPlayer.

jwplayer("myElement").setup({
    file:       "rtmp://server/flvplayback/flv:video",
    autostart:  true,
    width:      480,
    height:     320
});

Sending video to the server with "sendstream"

The proof-of-concept tool I set out to make is now a useful test tool you can use to stream a test video with optional audio at a given resolution to a given streaming server. I have compiled it for windows and linux 32 bit (opensuse 12) and also have that in a working chroot for download (~28 mb zipped).

usage: sendstream.exe < RTMP URL > [ < width > < height > < sound > < max_fps >]
 params: width default = 320, height default = 240 pixels, sound default = 1, max_fps default = 25.
 i.e: sendstream.exe "rtmp://localhost/app/video" 800 600

Example output for sendstream.exe rtmp://cppse.nl/flvplayback/video 1920 1080 1:

[libmp3lame @ 058a6320] Channel layout not specified
Output #0, flv, to 'rtmp://cppse.nl/flvplayback/video':
    Stream #0:0: Video: flv1, yuv420p, 1920x1080, q=2-31, 100 kb/s, 90k tbn, 20
tbc
    Stream #0:1: Audio: mp3, 44100 Hz, 2 channels, s16p, 24 kb/s
HandShake: client signature does not match!
writing frame 0 at 0.044793... +V
writing frame 1 at 0.112187... +A+A+A[flv @ 058a2b60] Encoder did not produce pr
oper pts, making some up.
+A+V
writing frame 2 at 0.324518... +A+A+A+V
writing frame 3 at 0.813168... +A+A+A+A+A+A+A+A+A+V
writing frame 4 at 1.03129... +A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+V
writing frame 5 at 1.16484... +A+A+A+A+A+A+A+A+A+V
writing frame 6 at 1.23609... +A+A+A+A+A+V
writing frame 7 at 1.28777... +A+A+A+V
writing frame 8 at 1.33955... +A+A+V
writing frame 9 at 1.38883... +A+A+V
writing frame 10 at 1.43901... +A+V
writing frame 11 at 1.48954... +A+A+V
writing frame 12 at 1.53965... +A+A+V
writing frame 13 at 1.58858... +A+A+V
writing frame 14 at 1.6456... +A+A+V
writing frame 15 at 1.69497... +A+A+V
writing frame 16 at 1.74456... +A+A+V

Audio and video are combined with interweaving and if rendering of video frames is relatively slow compared to audio it may require more audio frames in between and vica versa. In the output +A is a written audio and +V a written video frame.

I use SFML for generating the graphics and for audio something similar to the example code by ffmpeg. The interweaving part was the most difficult but important thing to get right. This is the easiest version of the code I could come up with that works properly. Something I didn't know in the beginning was that you don't need to keep your framerate of sending frames to the server consistent at a given frames per second-rate.

sfml_init(&pixels, &pixelSize);

auto timer = TimerFactory::factory(TimerFactory::Type::WindowsHRTimerImpl);
timer->start();

frame->pts = timer->end();

int64_t currentframe = 0;
while (true) {

    sfml_generate_frame();

    std::cout << "writing frame " << currentframe++ << " at " << (timer->end() / 1000.0) << "... ";

    // Make sure to write enough audio frames until the audio timer lines up
    while (true) {
        double audio_pts = (audio_st) ? (double)audio_st->pts.val * audio_st->time_base.num / audio_st->time_base.den : 0.0;
        double video_pts = (video_st) ? (double)video_st->pts.val * video_st->time_base.num / video_st->time_base.den : 0.0;

        if (audio_pts >= video_pts)
            break;

        std::cout << "+A";
        write_audio_frame(oc, audio_st);
    }

    // Write video frame
    write_video_frame(oc, video_st);
    std::cout << "+V" << std::endl;

    // Set elapsed time as frame time
    frame->pts = timer->end();
}

One final side-note if you are going to use sendstream on linux with Xvfb (X virtual framebuffer). On SUSE it was a pain to get it working, I ended up installing it on an openSuse VMWare VM where it did work and then made a minimal chroot for it, one that includes Xvfb and sendstream and a run.sh script (you might want to edit that). Not all screen resolutions you can think off that work fine with RTMP server will work, i.e.: I wanted a 980 x 100 resolution video, that didn't work for Xvfb, it does work if you create a larger screen with a more common resolution like 1280 x 768. Also 980 x 100 didn't work for SFML, 1000 x 100 did, probably the OpenGL underneath complaining or something. AFAIK these problems do not exist without Xvfb.

The result

Sendstream runs on my webserver capped at 2 FPS and without audio, so that it doesn't hurt the CPU too much.

Download: sendstream-1.0.zip for windows. 11.2MB

Download: sendstream-1.0-chroot.zip for linux (chroot, use: "chroot /path/to/sendstream /run.sh"). 27.2MB

a930805b8867232755da195d34587476 *sendstream-1.0-chroot.zip
6ed8cf5102dbe10c5f4722c80d38cbbd *sendstream-1.0.zip

![nl][] Dutch PHP Conference 2013

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 28 Jun 2013 00:00:00 +0000

June 28 2013

Dutch PHP Conference 2013

Het thema van de Dutch PHP Conference 2013 was Respecteer het eco-systeem.

Dag één begon voor mij (en mijn collega's) 7 juni en Ade Oshineye opende met het pleiten voor User Journeys, de flows over meerdere devices die gebruikers volgen zoals openen van een nieuwsbrief op een telefoon en naar doorklikken naar product om deze uiteindelijk te kopen op een werkstation.

User Journey

Denk ook na over dat een website geopend in een LinkedIn app nog kleiner getoond wordt (aangezien het in een webview geladen wordt, soort een frame). Hoe ziet je site er dan nog uit? Hoe vaak komt het niet voor dat je vanuit een nieuwsbrief door wilt klikken naar een product om vervolgens geconfronteerd te worden met een inlogscherm? Kan dat niet gebruiksvriendelijker?

Unbreakable Domain Models van Mathias Verraes bevatte tips die mij doen denken aan OO in C++: goed gebruik maken van types. I.p.v. een string voor een e-mail echt een Email class maken die eventueel ook validatie regelt. Wat ik interessant vind om te zien is dat de spreker in zijn sheets gebruik maakt van exceptions in de constructor. Waar ik persoonlijk ook groot fan van ben maar veel mensen in mijn omgeving niet . Naar mijn mening moet je een type niet half kunnen aanmaken (constructen) want dan is het geen volledig type, maak dan een liever een ander type die die state representeert. Zijn slides over "Encapsulate operations" tonen gebruik van een CustomerSpecification interface en het maken van classes zo dicht mogelijk bij het domein.

Surviving a Prime Time TV Commercial - David Zuelke. Zeer interessante manier van vertellen met veel tips en info, triggert mij te kijken naar e.e.a., zoals genoemde supervisord, transactional e-mail of elastic search. Voor hun use-case was het zinvol om zo'n beetje alles in Elastic Search op te slaan.

PHP, Under The Hood - Anthony Ferrara. Ook goede spreker en presentatie was humorvol. Hij heeft een PHP compiler gemaakt in PHP en wist leuke dingen te vertellen over opcodes en liet bijbehorende code hier en daar zien. Helaas talk maar voor de helft gezien. Dat vm_execute.h gegenereerd wordt tijdens compileproces door PHP vind ik niet zo schokkend, dat je heel g++ compiled met g++ vind ik pas schokkend!

Mikado!

Uncon: The Mikado method - Pascal de Vink (slides here). Hele korte talk, maar wel interessant. Een methode omtrent een leuk idee, met als onderliggende toon misschien wel het tegenovergestelde van "Respecteer het eco-systeem":

Formuleer een doel, schrijf deze op papier.
Naief implementeren van je doel in de code, 'fuck' de details of de dingen die je stuk maakt. Gewoon coden hoe het moet worden.
Als je dit foutloos hebt gedaan en alles werkt nog: klaar. Maaarrr waarschijnlijk heb je wel een aantal dingen stuk gemaakt, heb je een lijstje compile errors of heb je in je hoofd een aantal TODO's of FIXME's
Je schrijft deze als dependencies onder je doel (alles met pen en papier). Dit zijn dan de prerequisites voor je doel in een boomstructuur.
Je fixed ze als het simpel is of shelved je changes en gaat één voor één die prerequisites fixen en comitten.
Je streept alles van je blaadje af en werkt zo gericht naar je doel toe.

Idee achter deze methode is dat je wellicht code weg moet gooien of shelven, maar daar tegenover staat dat je snel je prerequisites inzichtelijk hebt. Deze kun je ook aan een andere developer geven als een duidelijkere roadmap voor implementatie van dit doel.

Dependency Injection Smells - Matthias Noback. Volgens mij vond ik dit een saaie talk, weet het niet meer zeker. Maar wel veel herkenbare issues qua design en soort van eigenaardigheden in PHP. Zo is in PHP public function foo(Bar $baz) alleen maar afdwingen dat $baz de interface Bar implementeert. Maar is $baz gewoon de implementatie, en als je niet uitkijkt programmeer je tegen de implementatie aan i.p.v. de interface. (Gelukkig is phpstorm slim genoeg om dit snel herkenbaar te maken)

Getting your toolbox together - Daan van Renterghem. Was een leuke talk, ging over Vagrant. Persoonlijk nooit een hobby van me geweest het configureren van dit soort dingen, maar hij liet Vagrant zien en vertelde over Chef en dat was zeker wel interessant.

Mariokarten (N64)

De volgende dag...

Growth Hacking for Humans - Eamon Leonard. (video) Zeer interessante keynote. Een man die behoorlijk ondernemend is en een aantal fasen heeft doorlopen die hij wilde delen met de rest. Benadrukte de belangrijke rol van communities.

Scenario Driven API Design - Ivo Jansch. Sta je op het punt een (REST) API te ontwikkelen, neem dan deze talk even door. Leuke anecdote, spreker: "Wie heeft er wel eens wat gehoord over Scenario driven design, buiten deze talk om dan natuurlijk?". Eén of twee mensen steken hun vinger op. "Oh, dat is grappig want ik heb die term dus zelf verzonnen.".

Introduction to Django - Travis Swicegood. Was eerlijk gezegd best nieuwsgierig naar dit framework. Omdat ik binnenkort ook eens wat met Python wil gaan doen. Wel veel goede dingen over Django gehoord maar de spreker heeft niet echt goed de kracht van het framework over weten te brengen naar mij. Bleef overigens ook allemaal erg basic en na elke slide kwam een slide met daarop: "Are there any questions?".

Emergent Design with phpspec - Marcello Duarte. Deze meneer was er erg handig mee, hij is dan ook de man achter het project. Het is denk ik PHPUnit done right, maar dan niet zoals het ellendige SVN "CSV done right" zou moeten zijn. Het is iets breder, je werkt meer vanuit de specificaties (die geformuleerd zijn als tests). phpspec is gefocust om zo zinvol mogelijke foutmeldingen/feedback te geven en slimmer te zijn. Dat laatste wil zeggen dat het gespecte methoden of classes voor je kan aanmaken die nog niet bestaan. Het ondersteund je beter in Test of Behaviour driven development.

Measuring and Logging Everything in Real Time - Bastian Hofmann. Veel zinvolle tips m.b.t. logging. Veel voorkomende problemen is dat je meerdere servers en dus logs hebt, welke log entries overal horen nou bij elkaar. PHP heeft leuke apache_note. Andere trefwoorden: graylog2, elastic search, AMQP, logstash, graphite, statsd, boomerang.js, X-trace-id, monolog, error_log, set_error_handler, set_exception_handler. Zat een heel verhaal omheen, wat ik hier niet zal reproduceren.

Conclusie

Worse Is Better, for Better or for Worse - Kevlin Henney. Zeer boeiende spreker, kan ik echt lang naar luisteren zulke talks. Alan Kay meerdere keren geciteerd gedurende deze conference, zo ook in deze talk, zie ook phpspec slides, slide 40. Het messaging aspect tussen de objecten is onderbelicht geweest, zie slides 13, 14, 15 (erg leuk). Ik vind het "jammer" dat ik geen ervaring heb met COM, dus zal eens een keer uit nieuwsgierheid naar kijken. Hij ging in deze talk in op een uitspraak "Worse is better":

Over two decades ago, Richard P Gabriel proposed the thesis of "Worse Is Better" to explain why some things that are designed to be pure and perfect are eclipsed by solutions that are seemingly limited and incomplete. This is not simply the observation that things that should be better are not, but that some solutions that were not designed to be the best were nonetheless effective and were the better option. We find many examples of this in software development, some more provocative and surprising than others. In this talk we revisit the original premise and question in the context of software architecture.

Wat is het antwoord op deze vraag? Als Smalltalk zoveel "mooier" is, waarom programmeren we nu inmiddels niet allemaal in Smalltalk? Waarom werd C++ zo populair? Het antwoord is dat C++ het ecosysteem respecteerde. In de tijd van Smalltalk waren thuis computers nog zo snel niet en smalltalk was een stuk zwaarder, daarbij draaide het afgezonderd in een virtual machine in zijn soort van eigen ideale universum. Hierbuiten smalltalk is echter een heel eco-systeem wat in dit geval een "trage" computer is met een printer, muis, toetsenbord, etc. De spreker vertelde over hoe practisch hij Turbo C++ compiler vanaf een floppy op zijn computer zette en daarmee gelijk kon programmeren en echt dingen voor elkaar kon krijgen en dat de geschreven programma's snel waren. Dit is zijn waarheid, maar ik denk dat de kern van het ecosysteem respecteren wel een goed punt is, en voor mij zeer herkenbaar.

Om terug te komen op de keynote van dag één, waarom is het web zoals we dat nu kennen, met HTTP + HTML zo succesvol geworden? Er waren ook daar (geavanceerdere?) protocollen of in iedergeval alternatieven, zoals gopher. De rede is natuurlijk dezelfde, HTML "embraces the ecosysteem", het groeit mee met de ontwikkelingen, integreert en past zich aan. Denk aan Java applets, Flash, talloze file formats, VRML, WebGl maar ook juist integreert met andere protocollen zoals FTP--en ook al weet ik niet--vast ook gopher!

![nl][] Tweakers fotoalbum images uploader

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 07 Jun 2013 00:00:00 +0000

June 7 2013

Tweakers fotoalbum images uploader

Wat ik handig vind is om snel een plaatje te kunnen delen, op IRC bijvoorbeeld. Naar tweakers.net gaan, {inloggen, }naar fotoalbum, file upload knop, naar juiste map bladeren, plaatje selecteren en dan uploaden... is voor mij dan niet snel genoeg. Vooral als ik het plaatje voor me heb staan, negen van de tien keer in Windows Explorer®™.

Ik wil gewoon de files selecteren, een sneltoets indrukken en aangeven in welke map ze moeten. Daarna wil ik de link hebben naar het plaatje en klaar. Geen frustratie. :)

Voorbeeld van upload naar tweakers.net fotoalbum

Leuke bijkomstigheid is dat je gelijk een heleboel files in één keer kunt uploaden op deze manier.

Toelichting

sanitizer.exe moet gestart zijn en luistert (standaard) o.a. op CTRL + ALT + 6. Zodra die hotkey ingedrukt wordt zal sanitizer (1) de geselecteerde files uit het actieve explorer venster lezen en wegschrijven naar het bestand selected_files.txt en (2) het php script explorer_call.php aanroepen die met dit bestand kan doen wat je wilt, in dit geval de files uploaden met curl.

Uploaden naar tweakers was het doel, maar omdat ik het überhaupt handig vind om ook andere dingen te kunnen doen met geselecteerde files in explorer heb ik het flexibel gehouden met een script. Ook vind ik het handig om allerlei dingen onder ctrl+alt+{nummer} te kunnen stoppen: zoals text in het clipboard of system calls uitvoeren. Dus ook dat heb ik wat ruimer opgezet en daar is sanitizer uitgekomen.

Sanitizer configgen voor Tweakers.net image uploads

Je moet dus wel 1337 genoeg zijn om het fotoalbum unlocked te hebben binnen tweakers, anders zul je de optie onder je profiel niet hebben. :7 Als je mij genoeg vertrouwd kun je de installer van sanitizer downloaden en uitvoeren. De installer maakt ook een example_tweakers/ directory aan met daarin een explorer_call.php, die kun je bewerken en over explorer_call.php heenplakken die een directory hoger staat. Out of de box werkt deze dus nog niet.

Ten eerste zitten in explorer_call.php ook nog wat andere probeersels van mij die je kunt negeren (opties 'c' en 's').

Voorbeeld output:

Listing of files:

 - C:\Program Files (x86)\sanitizer\example_scripts\explorer_call.php.tweakers


Please select from the following list
 - s = move to unsorted directory with extensions as subdir
 - p = upload to tweakers fotoalbum public folder
 - r = upload to tweakers fotoalbum private folder
 - c = create command prompt here
 (other) = exit


You choose (press RETURN):

Ten tweede ga ik even uit van de standaard situatie dat je een private en een public folder hebt in je fotoalbum. In mijn geval is 'r' -> upload naar m'n album 'private' en 'p' naar 'public'. Je kunt het script makkelijk uitbreiden met een extra curl request om de albums eerst dynamisch op te halen. Vergeet het dan niet in de reacties te delen ;)

Als je dit huidige script werkend wilt maken voor jezelf moet je momenteel:

1. Een TnetID (sessie id) in de sourcecode op regel 29 zetten van een actieve sessie naar tweakers.net (je zou een extra sessie kunnen maken speciaal voor dit script). Dat kun je doen door in te loggen en met Firebug je cookie uit te lezen:

2. Je fotoalbum id's in de source zetten op regels 32 en 35.

Deze staan in de URL's van de albums, in het voorbeeld 1001:

3. sanitizer.exe moet wel gestart zijn natuurlijk

Troubleshoot

Image upload werkt niet op geselecteerde files op je desktop. Open dan eerst explorer en ga via dat ding naar de desktop..
Lees ook de troubleshoot van sanitizer. Huidige PHP script is namelijk niet geheel fout ongevoelig opgezet. :X

sanitizer - a shortcut key app with explorer integration

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 06 Jun 2013 00:00:00 +0000

June 6 2013

sanitizer - a shortcut key app with explorer integration

Hook shortcut keys Ctrl+Alt+{0,1,2,3,4,5,6,7,8,9}

sanitizer.exe is a program that reads a config, and then registers the hotkeys Ctrl+Alt+{number}, where {number} is {0, 1, ..., 9}.

Example contents for config.ini:

[1]
type=command
value=putty -load SomeSession

[4]
type=clipbard
value=MyMysqlPasswd

[6]
type=explorer

...

(Only [0] through [9] are valid entries in the config.)

As shown in the example, available action types are:

action types	description	example
command	execute system call	`putty.exe -load amazon`
clipboard	replace clipboard with text	some database password perhaps
explorer	dispatch selected files in windows explorer to script	see `explorer_call.php` source

The explorer action

This action type is special in the config: the value key in the config.ini for it is not used, so you don't need to define it.

Example use: If you assign it under hotkey Ctrl+Alt+6 as I did in the example, you can use this hotkey whenever a (Windows file) explorer is active/has focus (explorer.exe). If you trigger the hotkey it fetches the selected files and writes them to selected_files.txt in the program installation directory. As long as this file exists, the hotkey won't trigger again! It will then immediately call the explorer_call.php PHP-script, also in the installation directory, that can be defined to do whatever you want on this list of files. As long as it removes the selected_files.txt afterwards.

Contents of the default explorer_call.php script:

<?php
$files = explode("\n", trim(file_get_contents('selected_files.txt')));
unlink('selected_files.txt');

print_r($files);

system("pause");
?>

Small troubleshoot

If you crash the script, and selected_files.txt is not deleted, your shortcut key won't appear to work anymore, as the existance of this file also serves as a lock. Only when the file was processed and deleted can the explorer hotkey fire and launch the PHP script again. So when your hotkey appears non-responsive, chances are you need to delete selected_files.txt, or make the script more advanced .

Download

Download setup file: sanitizer-1.0-setup.exe.

The executable is compiled with VS2012 with platform target "Visual Studio 2012 - Windows XP (v110_xp)". VS2012 x86 redistributable is included in the setup. After install make sure to edit config.ini first, and realize that there is no GUI, the executable will run in the background. If you want to restart it use the restart_sanitizer.bat. If you want to kill it, use taskmgr or taskkill /IM sanitizer.exe.

Verify

If you start it with the sample config: Ctrl+Alt+4 should yield SomeOtherPassword in your clipboard; Ctrl+Alt+6 should print selected files in Windows explorer (do this from within explorer); Ctrl+Alt+1 should start putty.exe if you have that installed (and available in your PATH).

Windows 8 (Vista?) Disable UAC

If your system complains about not being able to write to selected_files.txt, just make it run as Administrator:

Explorer call use case: Tweakers.net foto uploader script

Update 10 jun 2013:

It's included in the installer in the example directory after install, an explorer_call.php script that makes it possible to upload images to your Tnet foto-album. Wrote a blog-post about it in Dutch here: http://cppse.tweakblogs.net/blog/9047/tweakers-fotoalbum-images-uploader.html.

Update 20 oct 2013:

Made a few changes to the tweakers script again, another Tweakers user wrote a blog about this: http://perkouw.tweakblogs.net/blog/9474/t-punt-net-fotoalbum-script!.html.

Now the only thing you need to edit is the TnetID (session id).
It will fetch the available foto albums itself.
I've also improved the check whether a file upload succeeded, it fetches the last known image from the fotoalbum, before and after the upload, if they match, the upload failed.
A summary of the uploads is now opened after the last image upload, instead of opening an url every upload ;)

(I'll update the installer later to include this new explorer_call.php)

benchmarklib: a small benchmarking library

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 29 Apr 2013 00:00:00 +0000

April 29 2013

benchmarklib: a small benchmarking library

Background

Yet another C++ benchmark library I guess, but I couldn't find any that fulfilled my needs properly The requirements: output to text into a format easily parsable (for generating grahps afterwards). Low coupling, non intrusive, performant, provide basic statistics, ... .

So this one is easy to use, at first I implemented the timer using boost::posix_time::microsec_clock, only to discover that the resolution was not as good on windows. [Edit 15-02-2015: I now think the individual measures may be off a bit, but overall it does seem to average out to correct timings. I now added a boost::chrono based timer, which seems to offer good results on all platforms.] I don't understand why exactly, so I searched for a better timer (for windows) and found an example for an HRTimer [Edit 15-02-2015, forgot the link, but I found this]. So now you can choose Timer implementations ^_^

Available Timer implementations are:

BoostTimerImpl - microseconds resolution on *nix systems, at most milliseconds on windows.
BoostChronoTimerImpl - microseconds resolution on *nix systems and windows.
WindowsHRTimerImpl - microseconds resolution on windows, windows specific.

Available Measure classes are:

MeasureCounts - group individual measures per time interval.
MeasureInterval - individual measures.

With both classes you can set the number of measures you require before the report file should be generated.

Download

Currently compiled as a static library for platform toolset Visual Studio 2012 - Windows XP (v110_xp). It is 32 bit, with flags /MD (Multi-threaded DLL).

Download link: benchmarklib-1.0.zip (md5 37ce3698a5af8b25555913f5151c85c9).

Using the Benchmark and Timer classes

Using the library

#include "benchmark.h"
Link against benchmarklib.lib

Using the MeasureCounts class

MeasureCounts counter(TimerFactory::Type::WindowsHRTimerImpl);

// Measure FPS
counter.setDescription("number of frames");
counter.setBatchSize(1000);

// Measure 30 times at 500 milliseconds interval (duration of 15 seconds)
counter.require(30);
counter.setBatchGroupSize(500);

// Results will be written to RESULT_EXAMPLE.TXT
counter.setOutput("EXAMPLE");

while ( ! counter.complete()) {

    /* do something */

    counter.measure();
}

Using the MeasureInterval class

MeasureInterval counter2(TimerFactory::Type::WindowsHRTimerImpl);

counter2.setDescription("milliseconds per frame");

// Measure 500 frames
counter2.require(500);

// Results will be written to RESULT_EXAMPLE2.TXT
counter2.setOutput("EXAMPLE2");

while ( ! counter2.complete()) {

    /* do something */

    counter2.measure();
}

Using the timer class

auto timer = TimerFactory::factory(TimerFactory::Type::WindowsHRTimerImpl);
timer->start();

/* do something */

std::cout << "Run time CPU : " << timer->end() << " milliseconds." << std::endl;

Example output (`RESULT_EXAMPLE.TXT`)

@@@ BENCHMARK @@@

profile "EXAMPLE"

measure "number of frames" / 1000 ms.

measure interval = 500 ms.


@@@ STATISTICS @@@

N          30
Mean       122.5427
S.E. Mean  0.2686
Std. Dev   1.4714
Variance   62.7829
Minimum    118.8119
Maximum    124.5059
Median     122.7723


@@@ HISTOGRAM @@@

                      #       
                      #       
                      #       
                      #       
                      #       
                      #       
                      #       
                #  #  #       
          #     #  #  #       
 #        #  #  #  #  #  #    
 #  #  #  #  #  #  #  #  #  # 
------------------------------
 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.

0.  118.8119 <> 119.4446  =  2
1.  119.4446 <> 120.0772  =  1
2.  120.0772 <> 120.7099  =  0
3.  120.7099 <> 121.3426  =  3
4.  121.3426 <> 121.9752  =  2
5.  121.9752 <> 122.6079  =  4
6.  122.6079 <> 123.2406  =  4
7.  123.2406 <> 123.8733  =  11
8.  123.8733 <> 124.5059  =  2
9.  124.5059 <> 125.1386  =  1

Troubleshoot

Use this if your compiler does not support typed enums:

auto timer = TimerFactory::factory("WindowsHRTimer");

~~And for the benchmark classes as well..~~ (Edit: not yet supported for the Measure classes)

~~MeasureCounts counter("WindowsHRTimer");~~

A short reminder that sequential memory I/O is faster :P

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 08 Apr 2013 00:00:00 +0000

April 8 2013

A short reminder that sequential memory I/O is faster :P

Faster than random reads that is. I knew this to be true for harddisks and SSD drives because of this presentation. (For average devices: Random I/O 2.7 MB/s vs. Sequential I/O of 213 MB/s for HDD's, 60-300 MB/s vs. 293-366 MB/s for SSD's).

But I never realized it was similarly true for RAM access, or that the impact would be this big. Now that I do however, I realize where I must have destroyed performance in the past.. and why something else surprised me at the time in being unexpectedly fast!

SFML fast pixel read/write access!

Anyway, my goal was finding the fastest possible read and write access for pixels inside an image. In this case I used SFML and desired for no overhead by getters or setters (providing f.i. bounds checking). Just raw access.

Found out you can get a (read only) pointer to the underlying pixel data for an sf::Image object (that I force non-const):

sf::Image image;

image.create(1280, 720, sf::Color(0, 0, 0, 255));

auto *pixels      = const_cast<sf::Uint8 *>(image.getPixelsPtr());

Wrote a simple loop to initialize an image with fancy colors (you can probably guess where this is going.. ):

for (int x = 0; x < width; x++) {
    for (int y = 0; y < height; y++) {
        int index = (x + y * width) * 4;
        // My trademark color setup
        int index = x + y;
        pixels[index + 0] = index         % 255; // R
        pixels[index + 1] = (index + 100) % 255; // G
        pixels[index + 2] = (x     + 200) % 255; // B
        pixels[index + 3] = 80;                  // A
    }
}

In the mainloop I have similar code: for each pixel, increment the RGB color values a bit. You can view the code in the screenshot a few paragraphs from now. The result was 42.65 FPS (frames per second).

Measuring FPS every 0,5 seconds 30 times, results in this average of 42.65 fps with a Standard Error of 0.08. See [1] in the following table.

*Table: performance results*
[1]		[2]		[3]
N	30	N	30	N	30
Mean	42.6518	Mean	122.4701	Mean	125.8626
S.E. Mean	0.0801	S.E. Mean	0.2189	S.E. Mean	0.3322
Std. Dev	0.4387	Std. Dev	1.1991	Std. Dev	1.8193
Variance	5.5810	Variance	41.6968	Variance	95.9866
Minimum	42.1456	Minimum	119.8428	Minimum	120.3156
Maximum	44.7471	Maximum	124.7525	Maximum	128.2051
Median	42.6357	Median	120.7921	Median	125.0000

The fix: Changing the order of pixel access

I don't have the fastest PC so initially I thought it wouldn't get that much faster, but then I ran the profiler and discovered the first write to the color values was extremely slow. Once the pointer was in position for the pixel however, successive writes to green (G) and blue (B) (of RGBA) are fast. This made me realize it was seeking for each pixel.

So I swapped the two for loops (to first Y then X), thus aligning the loop with the memory representation of the pixels to get the much better 122.47 FPS! (see [2]).

Another minor improvement was by making the intermediate "index" variable obsolete (see [3]).

Figure: Visual studio 2012's awesome profiler output.

Note that you don't really need two for loops if you don't do stuff with the colors related to x or y.

This fix may seem obvious now, but for me this mistake of swapping the for loops was one easily made. I found it subtle and it resulted in unnecessarily poor performance. That's why I hope others to find this reminder useful!

Figure: If you didn't want poor performance, you shouldn't have swapped the for loops! (image source: here)

Also, SFML stores the RGBA values this way, other libraries may do so differently.

![nl][] watermarker tool

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 29 Mar 2013 00:00:00 +0000

March 29 2013

watermarker tool

Installer

Hier te downloaden: watermarker-1.0-setup.exe

Hoe te gebruiken

Na installatie gewoon "watermarker" starten.

De plaatjes die je wilt watermerken kun je op het grijze vlak slepen. (Zoveel je wilt tegelijk) Je kunt zien hoe ze eruit komen te zien, en dan op "Write to images!" klikken om ze daadwerkelijk te watermerken.

Let wel op: de plaatjes worden overschreven.

Dus je zult wel de plaatjes eerst moeten kopieeren en dan de kopieen watermerken. Want het watermerk krijgt je er niet meer uit, en de originele foto's wil je wel behouden lijkt me

Voorbeeld:

Update 29 mei 2014: Install it on Ubuntu using wine!

Nu ik geen linux meer gebruik, heb ik gekeken of watermarker goed installeerd d.m.v. wine. De installatie duurt erg lang, en de shortcut die gemaakt werd werkte niet...

Het probleem was blijkbaar dat de installer niet de C++ runtime dlls heeft geinstalleerd:

trigen@Firefly21:/home/trigen/.wine/drive_c/Program Files (x86)/watermarker> wine watermarker.exe
err:module:import_dll Library MSVCR110.dll (which is needed by L"C:\\Program Files (x86)\\watermarker\\watermarker.exe") not found
err:module:import_dll Library MSVCP110.dll (which is needed by L"C:\\Program Files (x86)\\watermarker\\watermarker.exe") not found
err:module:LdrInitializeThunk Main exe initialization for L"C:\\Program Files (x86)\\watermarker\\watermarker.exe" failed, status c0000135

Deze heb ik handmatig in de watermarker map moeten plaatsen. Download ze hier: msvcp110.dll en msvcr110.dll.

De shortcut doet het daarna wel. Het programma doet het alleen nog niet omdat convert.exe ook problemen geeft.

trigen@Firefly21:/home/trigen/.wine/drive_c/Program Files (x86)/watermarker> wine convert.exe
err:module:import_dll Library VCOMP100.DLL (which is needed by L"C:\\Program Files (x86)\\watermarker\\convert.exe") not found
err:module:LdrInitializeThunk Main exe initialization for L"C:\\Program Files (x86)\\watermarker\\convert.exe" failed, status c0000135

Deze dll ook hier handmatig neerzetten fixed het probleem, download: vcomp100.dll

Op die manier werkt het (inclusief de drag & drop functionaliteit) prima onder Ubuntu (versie 13.10).

Fixing slow phpmyadmin by rewriting queries using mysql-proxy

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 24 Mar 2013 00:00:00 +0000

March 24 2013

Fixing slow phpmyadmin by rewriting queries using mysql-proxy

The performance problem

Opening phpmyadmin becomes quite annoying if the initial loading takes around 30 seconds, after logging in. Viewing the queries being executed by phpmyadmin with meta log monitor easily shows the bottleneck is "SHOW TABLE STATUS FROM '<DATABASE>'". It is the only query requiring > 14 seconds in this case.

Phpmyadmin executes this query twice: in it's navigation frame for a listing of tables and in the right frame for the detailed listing.

Create cache for: SHOW TABLE STATUS FROM <DATABASE>

The output of this query is not that accurate. Running it twice, and comparing the output shows some columns are estimates. So if we would cache it for two minutes that would probably not harm phpmyadmin.

Then I created a table where to store the SHOW TABLE STATUS output into (I realize the types I used are probably overkill for some fields ).

CREATE TABLE showtablecache (
  Database_       VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci,
  Name_           VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci,
  Engine_         VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci,
  Version         Integer(11),
  Row_format_     VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci,
  Rows_           BigInt(20),
  Avg_row_length  BigInt(20),
  Data_length     BigInt(20),
  Max_data_length BigInt(20),
  Index_length    BigInt(20),
  Data_free       BigInt(20),
  Auto_increment_ BigInt(20),
  Create_time     DateTime,
  Update_time     DateTime,
  Check_time      DateTime,
  Collation_      VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci,
  Checksum        VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci,
  Create_options  VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci,
  Comment_        VarChar(255) CHARACTER SET latin1 COLLATE latin1_swedish_ci
) ENGINE=InnoDB DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci;

This doesn't work by the way: INSERT INTO showtablecache SHOW TABLE STATUS FROM '<DATABASE>' (leaving the "Database_" column from above CREATE TABLE). I didn't really expect that to work, but I would have been pleasantly surprised if it did .

I created a resultset with similar output to the SHOW TABLE STATUS query with a SELECT query on INFORMATION_SCHEMA.TABLES. And perform INSERT's with the following cron:

<?php
$res = $conn->select('SHOW DATABASES');

while ($row = $res->assoc())
{
    $dbname = current($row);
    print "Caching $dbname...";

    $conn->startTransaction();
    $conn->update('DELETE FROM showtablecache WHERE Database_ = \'' . $conn->escape($dbname) . '\'');
    $conn->update('INSERT INTO showtablecache
                   SELECT \''.$conn->escape($dbname).'\'
                     ,TABLE_NAME
                     ,ENGINE
                     ,VERSION
                     ,ROW_FORMAT
                     ,TABLE_ROWS
                     ,AVG_ROW_LENGTH
                     ,DATA_LENGTH
                     ,MAX_DATA_LENGTH
                     ,INDEX_LENGTH
                     ,DATA_FREE
                     ,AUTO_INCREMENT
                     ,CREATE_TIME
                     ,UPDATE_TIME
                     ,CHECK_TIME
                     ,TABLE_COLLATION
                     ,CHECKSUM
                     ,CREATE_OPTIONS
                     ,TABLE_COMMENT
                   FROM
                     INFORMATION_SCHEMA.TABLES
                   WHERE
                     table_schema = \'' . $conn->escape($dbname).'\'');
    $conn->endTransaction(true);
    print "done.\n";
}
?>

I run this script every 2 minutes. You really want the transaction because you don't want the cache to be empty between the DELETE and the time the INSERT finishes.

Rewrite query to use our cache using mysql-proxy

The final step was to rewrite the SHOW TABLE STATUS queries to SELECT queries on our new cache table. This turns out to be quite simple using a neat Lua script for mysql-proxy:

   0 ---
   1 -- Uses MySQL-Proxy to rewrite annoyingly slow SHOW TABLE STATUS FROM `<DATABASE>` queries to a SELECT from our periodic cache
   2 --
   3 -- @author trigen
   4 --
   5 function read_query( packet )
   6     if string.byte(packet) == proxy.COM_QUERY then
   7         local query = string.sub(packet, 2)
   8
   9         -- attempt to match the query, and more specifically extract the requested database
  10         local dbname = string.match(string.upper(query), "^SHOW TABLE STATUS FROM \`(%a*)\`")
  11         if dbname then
  12             local newquery = string.format('SELECT' ..
  13                 '  Name_ AS `Name`,' ..
  14                 '  Engine_ AS `Engine`,' ..
  15                 '  Version,' ..
  16                 '  Row_format_ AS `Row_format`,' ..
  17                 '  Rows_ AS `Rows`,' ..
  18                 '  Avg_row_length,' ..
  19                 '  Data_length,' ..
  20                 '  Max_data_length,' ..
  21                 '  Index_length,' ..
  22                 '  Data_free,' ..
  23                 '  Auto_increment_ AS `Auto_increment`,' ..
  24                 '  Create_time,' ..
  25                 '  Update_time,' ..
  26                 '  Check_time,' ..
  27                 '  Collation_ AS `Collation`,' ..
  28                 '  Checksum,' ..
  29                 '  Comment_ AS `Comment`,' ..
  30                 '  Create_options' ..
  31                 ' FROM' ..
  32                 '  tweakers.showtablecache' ..
  33                 ' WHERE Database_ = %q', dbname);
  34             print('CONVERTING ' .. query .. '  ===>  ' .. newquery);
  35             proxy.queries:append(1, string.char(proxy.COM_QUERY) .. newquery )
  36             return proxy.PROXY_SEND_QUERY
  37         end
  38     end
  39 end

Save that script to pma.lua, start proxy with:

mysql-proxy -P :4041 --keepalive --proxy-backend-addresses=YOUR_SERVER:3306 --proxy-lua-script=pma.lua

Example output of mysql-proxy:

CONVERTING SHOW TABLE STATUS FROM `tweakers`  ===>  SELECT  Name_ AS `Name`,  Engine_ AS `Engine`,  Version,  Row_format_ AS `Row_format`,  Rows_ AS `Rows`,  Avg_row_length,  Data_length,  Max_data_length,  Index_length,  Data_free,  Auto_increment_ AS `Auto_increment`,  Create_time,  Update_time,  Check_time,  Collation_ AS `Collation`,  Checksum,  Comment_ AS `Comment`,  Create_options FROM  tweakers.showtablecache WHERE Database_ = "TWEAKERS"
CONVERTING SHOW TABLE STATUS FROM `tweakers`  ===>  SELECT  Name_ AS `Name`,  Engine_ AS `Engine`,  Version,  Row_format_ AS `Row_format`,  Rows_ AS `Rows`,  Avg_row_length,  Data_length,  Max_data_length,  Index_length,  Data_free,  Auto_increment_ AS `Auto_increment`,  Create_time,  Update_time,  Check_time,  Collation_ AS `Collation`,  Checksum,  Comment_ AS `Comment`,  Create_options FROM  tweakers.showtablecache WHERE Database_ = "TWEAKERS"
...

Example output for metalogmon:

Some notes

I never did any Lua scripting before, I find this website quite useful as it provides a console for Lua (but for other languages as well). This way you can test some stuff without constantly restarting mysql-proxy.

With regards to the caching, if you create a new table, this table will become visible once the cron updated the cache. So you don't want to set the delay for too long. You could do some extra scripting for encountered CREATE TABLE statements in mysql-proxy, or make a more advanced cron script that checks the faster "SHOW TABLES;" more often to see if any caching needs more priority.

ksh "pushd .", "popd" and "dirs" commands

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 23 Mar 2013 00:00:00 +0000

March 23 2013

ksh "pushd .", "popd" and "dirs" commands

The New KornShell

I prefer ksh (Version JM 93u+ 2012-02-29) for my shell (with "set -o vi"). Not that it's soooo much better than bash as a cli (and it's probably pwnd by some of zsh's features like programmable autocomplete). But I do find it alot cleaner than bash for scripting. Anyway, currently I've given all machines I work with a /bin/ksh and chsh'd it for my user, but I noticed I missed bookmarking the current directory with "pushd ." for returning to it later with "popd" (after i.e. some subtask that will make you endup /nowhere/near/your/old/far/away/path).

Sometimes you just don't want to open a new tab in screen. (And you are right, you could of course also use goto.cpp for bookmarking ) An alternative solution would be starting- and exiting a subshell.

Found: dirstack.ksh by Eddie

So I googled and found this stack overflow answer which has a pretty nice pushd/popd/dirs implementation. But it behaves a little different: "pushd /usr" bookmarks and changes to that directory (the normal behaviour). But what I often want is to store a directory right before I'm about to leave it. (Chances are I didn't use "pushd" but "cd" to get there in the first place.) Normally you simply use "pushd ." to put it on the stack (and ignore the useless changedir on the side ). But this implementation is explicitly designed so that the current directory is already (and always) the first (or zeroth position) on the stack and from that line of thought it would be "useless" to put it as a "duplicate" in the list.

I still want to use pushd on $PWD, so I commented these four lines in dirstack.ksh:

     # check for duplicate stack entries
     # current "top of stack" = ids; compare ids+dsdel to $PWD
     # either "ids" or "dsdel" must increment with each loop
     #
     (( ids = 1 ))          # loop from bottom of stack up
     (( dsdel = 0 ))        # no deleted entries yet
     while [ ids+dsdel -le sd ] ; do
>>>>    #if [ "${dir_stack[ids+dsdel]}" = "$PWD" ] ; then
>>>>    #   (( dsdel = dsdel + 1 ))  # logically remove duplicate
>>>>    #else
           if [ dsdel -gt 0 ] ; then        # copy down
              dir_stack[ids]="${dir_stack[ids+dsdel]}"
           fi
           (( ids = ids + 1 ))
>>>>    #fi
     done

Then I remembered my book (that has caused me many jealous colleagues :+) also provided pushd and popd as examples in the appendix .

So I was curious to see if these were usable (the book is from 1995).

Found: fun/{pushd|popd|dirs} by David G. Korn himself*

* by my guess

SuSE provides these scripts in /usr/share/ksh/fun so I didn't need to type them in. If you need them, I tarballed them into kshfun.tar.gz (md5=7173831211d3d54f26f630f3cc720282). I was surprised by /usr/share/ksh/fun/dirs, they re-alias "cd" with a custom "_cd" function that does basically "pushd" all the time, with a stack of max. 32 dirs. That's a cool idea, you can view your stack with "dirs" or even use the "mcd" (menu change dir) command. You use this "cd" alias like "cd N" as well, where N is the index on the stack (given by "dirs" output). And pushd and popd work on this same stack.

For your .kshrc profile:

. /usr/share/ksh/fun/popd <<<< includes pushd as well **
. /usr/share/ksh/fun/dirs <<<< if you also want the custom cd + dirs

**ls -althr /usr/share/ksh/fun:

total 16K
-rwxr-xr-x 1 root root 2.5K Jun  6  2012 pushd
-rwxr-xr-x 1 root root 2.3K Jun  6  2012 dirs
lrwxrwxrwx 1 root root    5 Jan 19 23:43 popd -> pushd     <<<< symlink

These work great! (example)

trigen@ip-10-235-45-12:/etc/apache2> cd /var/log/
/var/log
trigen@ip-10-235-45-12:/var/log> dirs
1) /var/log
2) /etc/apache2
3) /usr/local/src
4) ~
trigen@ip-10-235-45-12:/var/log> cd 2             <<<< If you have a directory named '2', use "cd ./2" :P
/etc/apache2
trigen@ip-10-235-45-12:/etc/apache2> mcd
1) /etc/apache2
2) /var/log
3) /usr/local/src
4) ~
Select by number or enter a name: 3
/usr/local/src
trigen@ip-10-235-45-12:/usr/local/src>

or what I was talking about:

trigen@ip-10-235-45-12:/home/trigen> cd /usr/local/lib
/usr/local/lib
trigen@ip-10-235-45-12:/usr/local/lib> pushd .
/usr/local/lib ~ /usr/local/lib
trigen@ip-10-235-45-12:/usr/local/lib> cd /
/
trigen@ip-10-235-45-12:/> popd                 <<<< In this trivial example "cd -" (where - is previous dir) would also work.
/usr/local/lib ~ /usr/local/lib
trigen@ip-10-235-45-12:/usr/local/lib>

Generating gradients

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 13 Jan 2013 00:00:00 +0000

January 13 2013

Generating gradients

Code that generates following images script here

More elaborate version here

spf13 vim distribution

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 29 Dec 2012 00:00:00 +0000

December 29 2012

spf13 vim distribution

It's really nice that at least one of my creations (phpfolding.vim) is really being picked up by others. I.e.: on Stackoverflow [1], [2], in China [1], Japan [1], [2], France [1], Holland [1], someone made a package for archlinux, creator of spf13 distribution listed as one of 15 best vim plugins. Also found a few forks on github which is really awesome, will check out if I should merge their changes back soon. These are just a few links I found now with a simple google search. This makes me regret being a little reluctant with reading my hotmail account.

I'm a bit proud it's part of the excellent spf13 vim distribution. I now use spf13 myself on my linux servers, it suprised me with some features I think are extremely smart: Like the use of numbers for each line relative to your cursor, so you can easily make X line(s) jumps or yank the correct amount of lines easily! Also the CTRL + P quick navigation that works with acronyms as well, ,, + direction key for quick jumping. And the included themes are also very nice.

phpfolding.vim : Automatic folding of PHP functions, classes,.. (also folds related PhpDoc)

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 29 Dec 2012 00:00:00 +0000

December 29 2012

phpfolding.vim : Automatic folding of PHP functions, classes,.. (also folds related PhpDoc)

To everybody that contacted me, and whom I didn't reply to from my hotmail address: please read this, I had problems with the Hotmail Junk filter *again* sorry!

This script can fold PHP functions and/or classes, properties with their PhpDoc, without manually adding marker style folds ({{{ and }}}). It will fold stuff into something like the following screenshot.

Features

It remembers fold settings. If you add functions and execute the script again, your opened folds will not be closed.
It will not be confused by brackets in comment blocks or string literals.
The folding of class properties with their PhpDoc comments.
The folding of all class properties into one fold.
Folding the original marker style folds too.
An "**" postfixing the fold indicates PhpDoc is inside (configurable).
An "**#@+" postfixing the fold indicates PhpDocBlock is inside (configurable).
Empty lines postfixing the folds can be configured to be included in the fold.
Nested folds are supported (functions inside functions, etc.)

Future

Better 'configurability' as opposed to editting the PHPCustomFolds() function and some "Script configuration" global variables.

Compatibility

This script is tested successfully with Vim version >= 6.3 on windows and linux (With 6.0 it works sometimes, I don't recommend using it in that version)

But I'd recommend using a new vim, at the time of writing 7.3 works fine with it!

spf13's vim distribution

You can also install spf13's vim distribution (if you like it). phpfolding.vim is included in this package too. I'm testing it out at the moment, like it so far. Kind of missed my mappings though, so I still added <F5>, <F6> and <F7> to my .vimrc.

Also I noticed EnableFastPHPFolds gives better results now better than EnablePHPFolds, this is what I experienced at work anyway, with vim 7.3. This used to be the other way around..

INSTALL

Put phpfolding.vim in your plugin directory (~/.vim/plugin)
i.e., wget: http://www.vim.org/scripts/script.php?script_id=1623

You might want to add the following keyboard mappings to your .vimrc:

map <F5> <Esc>:EnableFastPHPFolds<Cr>
map <F6> <Esc>:EnablePHPFolds<Cr>
map <F7> <Esc>:DisablePHPFolds<Cr>

You might want to add the following lines to php.vim in your after/ftplugin directory (~/.vim/after/ftplugin/php.vim), this will be executed after opening a .php file:
```
" Don't use the PHP syntax folding
setlocal foldmethod=manual

" Turn on PHP fast folds
EnableFastPHPFolds
```

It might be necessary that you load the plugin from your .vimrc, i.e.:

let php_folding=0
  (if you can't use the after directory in step 3)
source ~/path/to/phpfolding.vim
  (if you're not using the default plugin directory)

Known issues

C++ style commented brackets can still interfere with the bracket matching.
For example comments like are not recognized as comments: // old: for (...) {
Whereas C-style comments are, e.g.: /* old: for (...) { */
The following won't be matched with the Regex in FindFoldStart():
function (...,
....,
....,
) {
Though Steve McConnell (writer of Code Complete) would ask why you'd need more then two lines for function parameters :)

Note: My Hotmail Junk filter probably ate a some e-mails I received from vim.org

I use my hotmail address for accounts where my e-mail is publicaly displayed, like in my profile page on vim.org. (I don't want spam in my primary e-mail inbox). So I only check my e-mail this hotmail address every now and then, probably every 6 months and it's full of stuff spam so I could have easily missed e-mails regarding phpfolding. I discovered in 2010 that there was feedback in my Junk filter, so I turned the Junk filter off, anyways that's how I remember it (and how I wrote it down). Now in 2012 I check the Junk filter again, and there is another email. Junk is automatically deleted after X days, so who knows how many mails I might have missed.

Really don't understand why it got into Junk, this specific person wrote hello, introduced himself, said something nice and signs off the e-mail. All in normal letter formatting, no URL's or e-mail addresses used, no images, just plain text. It was a valid gmail address, there were no weird Headers, and it was sent from gmail itself not some other smtp server...

You can contact me here by commenting, or send me an e-mail to the new one I use in my profile at vim.org.

Visualizing (inverse) square root optimizations

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Tue, 25 Dec 2012 00:00:00 +0000

December 25 2012

Visualizing (inverse) square root optimizations

A few years ago I experimented with a few of these approximation functions, and then I decided not to use them for some reason. (I forgot--, maybe they weren't a performance gain at the time.) I remember vaguely getting weird (but cool) results with some approximation formula.

Then I encountered David Hulsman's RSqrtBenchmark by accident: a collection of sqrt approximations he benchmarked for performance and accuracy. I found this a very cool idea, and wondered how these different sqrt approximations would visualize.. (having this faint memory of cool visual artifacts).

But apparently all these approximations are extremely good! :P Visually indistinguishable even! In this image each row represents a different sqrt implementation, and a white to black gradient is drawn with it:

I tried different visual representations of course, where for some functions you can see very small differences along the edges (if you swap them real fast). If you view the images side-by-side you won't see anything.

Of course there are more differences, you see them when comparing, like for example.. (left: compare sqrt <> lomont's InvSqrt, right: sqrt <> carmack D3RSqrt)

But I wanted my anomolies! :P so I tried tracking down the function I might have used back then, but I couldn't find it anywhere in my code repositories but after searching the internet for a while as well I now think it probably was something based on this very crude approximation:

(And yes it really is a very crude approximation :P)

So I kind of wasted way too much of my time on this already, but while searching the internet I encountered a few functions and got curious how they would visualize. I added a few, and maybe i'll add them to RSqrtBenchmark later:

Some of these do actually have visual artifacts. You can see those more clearly in the zipped uncompressed images. << I included radial blur images (1024x1024 px) for all functions.

By the way--I thought this one I generated while with a programming mistake was also pretty cool, almost fractal-like

(Edit 29-12-2012: uploaded source to bitbucket repository)

Configure highlight weechat in screen and putty

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 09 Dec 2012 00:00:00 +0000

December 9 2012

Configure highlight weechat in screen and putty

Recently discovered weechat, and I'm quite impressed with it!

Out of the box nicklist, nick colouring.., it just looks awesome: http://www.weechat.org/screenshots/

I am used to my IRC client to flash it's window when my name is highlighted. You can achieve this with the script bleep.pl As qbi comments over here:

You can download beep.pl: cd ~/.weechat/perl && wget http://www.weechat.org/files/scripts/beep.pl. Now you can use in weechat: /perl load beep.pl

If you use multiple windows in screen, and your weechat client is not the active one, you get a "Bell in window %n" message instead of an actual bell in the terminal. You can change that message with CTRL+A, :bell_msg "^G". Just as %n will be replaced with the window number, ^G is replaced with an actual bell. (source: http://www.delorie.com/gnu/docs/screen/screen_64.html)

In PuTTY: Change settings->Terminal -> Bell -> Taskbar/caption indication on bell: (x) Flashing. Also don't forget to use FauxFaux build of putty as it has lots of cool additional features like clickable links, ctrl+scrollwheel for changing font size, minimize to tray, ... .

Meta log monitor (or scriptable tail with GUI)

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 28 Nov 2012 00:00:00 +0000

November 28 2012

Meta log monitor (or scriptable tail with GUI)

I created the next best thing in the log monitoring world! It's like unix 'tail' but with additional features!! The tool pipes all data from the logfile through a customizable java-script before presentation. So you can change the behaviour on the fly.

The ability to script the behaviour allows you to use the following features in your logfiles (or custom logfiles). You could for example pretty-print in the "main log" output using colors, extra indenting, or enrich it with related information. But the main idea for the tool is that you can add references or bookmarks to important parts inside the log in a separate listing, the 'meta log', and thus create a more general overview of the log's output for yourself.

It displays (and parses) the logfile realtime as it behaves like tail in that sense, and it's fast because it is written in C++ but also because it uses Google's V8 engine for Javascript, which is known to be a very fast implementation. It's also quite handy that you can script and reload at runtime, so you can tweak your logfile easily with real data. The program takes around 4MB of memory, initially and grows as it logs the file of couse, it doesn't remove stuff from its outputs. There are shortcuts in case you wish to flush current output (ALT+C = clear log).

Example: Webdevelopment with MySQL (and other stuff)

While developing it can be useful to see all queries that are being executed in a specific request. Especially if you are working with software you don't know very well yet. If you have a proper database layer in your system you can probably log the queries to a file from one specific place in the code. But if you don't have that, and legacy PHP code for instance uses mysql_** functions, you cannot make a centralized change. In that case you can use mysql-proxy to sit between your software and the actual server.

You have to start it with a script like this:

local log_file = 'mysql.log'
local fh = io.open(log_file, "a+")
function read_query( packet )
    if string.byte(packet) == proxy.COM_QUERY then
        local query = string.sub(packet, 2)
        fh:write(string.format("@@begin@@\n%s\n%6d\n%s\n@@end@@\n",
            os.date('%Y-%m-%d\n%H:%M:%S'),
            proxy.connection.server["thread_id"],
            query))
        fh:flush()
    end
end

To have it output log entries in the following format:

@@begin@@
2012-11-25     << date
19:48:58       << time
786455         << thread id (probably useless)
SELECT         << query
  *            <<  ,,
FROM           <<  ,,
  some_tabl;   <<  ,,
@@end@@

So If you use my tool to tail an output file like that, with the script 'sqlonly.js' loaded. If you were to make a request that would send data to the database, It would display something like this:

I have obfuscated the actual queries though ( line.replace(/[a-z]/g, 'x') IIRC).

In the screenshot, in the meta log specific query types have been given different colours, by focusing on the blue text for example you can see what was executed inside a transaction. Also I hadded a column "Origin": at work I use a different script because I abuse the mysql-proxy and send it other types of data as well. Like MongoDB queries, engine calls, memcached calls and JML queries. Like this for example: $db->select('/**@mongodb <some stuff>*/'); It will be proxied to MySQL, which is the dirty part, but it is considered a comment so nothing happens. But I parse this comment and so something cool with everything between the comments. You can dump a lot of stuff between the C-style comments, for example a print_r if you like, and simply add a single meta log line with "MongoDB" as the "Origin".

Another thing I setup is a .htaccess files in my development environment that sets the php error_log to the same file. I write this down just to give you some ideas. I even use it for debugging now: $something->select('/* contents: ' . print_r($obj,1) . '*/'); It was not why I made this tool initially.

"metalogmon" Usage

Personally I prefer a quickstart link (those you can start with WINKEY+{1,2,3,...}). On Windows 7 it is really nice that if you already started to log monitor it makes the existing window active. Which allows for even easier navigation to it then alt+tab.

Usage: metalogmon.exe [/h] [/s ] /t 
  /h, --help            displays help on the command line parameters
  /s, --script=    javascript file that handles the parsing
  /t, --tail=      the file to tail
The value for the option 't (or tail)' must be specified.

Example: "metalogmon.exe /t \\networkshare\something\mysql.log /s C:\path\to\sqlonly.js"

Keyboard:

CTRL+C (copy selected lines from main log to clipboard)
ALT+C (clear output in main- and meta log)
ALT+F4 (exit program :P)

Toolbar:

	Currently does not do anything.
	Copy selected lines to clipboard (or CTRL+C)
	Clear all output (or ALT+C)
	Process entire logfile (default metalogmon will seek to end of log and tail there)
	Stop tailing, halts tail command.
	Open the active logfile in gvim.
	Open the active script in gvim.

Features

You can enable an 'idle' bar, if the log is idle for more than two seconds it adds a marker. For webdevelopment this is (for me anyways) good enough to separate consecutive requests. (Note that you could log a request start explicitely)
It detects log rotation/truncation. It will seek to the beginning of the file and simply continue.
When resizing the window it splits the main- and meta log 50/50.
Mainlog: customizable background and foreground color per line. Lines > 512 are truncated, but when you copy & paste them to clipboard they won't be truncated.
Metalog: customizable background and foreground color per line, customizable column names and sizes. (largest column gets extra remaining space)
Reload your script at runtime.
Search case sensitive or insensitive in the main log.

Scripts

Some included scripts

sample.js: simple example that you can use as a base for your own script.
sqlonly.js: simple example that parsers sql queries from my log.lua mysql-proxy output.
sqlphplog.js: sqlonly.js extended with parsing for php's error_log.

Contents of sample.js:

/**
 * Example script, display all lines prefixed with line number in main log, and
 *   create a meta log entry for every 100th line.   Meta log has two columns.
 *
 * $author: Ray Burgemeestre$
 *
 * $date: 2012-12-06$
 */


// Implemented API functions

function getColumns()
{
    return [['Line', 75], ['Message', 200]];
}

function handleLine(num, line)
{
    var newline = log(num + ': ' + line);

    if ((num % 100) == 0)
        metalog(newline, ['' +num, 'Shortcut to ' + num + 'th line', '']);
}

The API

Expects you to implement mandatory functions (see sample.js):

function getColumns()
- return: (string[]) array: [[colname, size], [another, size], ...]
function handleLine(num, line)
- param: (int) num, current line number
- param: (string) line, the current line from log
- return: (void)

Optionally (see sqlonly.js):

function getIdleColors() - implementing enables the idle bar I mentioned
- return: (string[]) array: [forground color, background color]
function getIdleMetaLog() - what text to display in idle bar in meta log, @message@ is replaced with "Idle for xx seconds".
- return: (string[]) array: [column value, column value, ...]

You have at your disposal:

function log(message, foreground color, background color) -- add log in main log
- param: (string) message or array of format: [message, clipboard-message].
- if you want a different message to be printed in the log and another message to be copied to clipboard you can use the array version.
- param: (string) foreground color (i.e., 'red', or '#ff0000')
- param: (string) background color ,, ,,
- return: (int) the line number in the main log
function metalog(num, colvalues, foreground color, background color) -- add log in meta log
- param: (int) num -- line number in main log, if you click this item in the meta log, it will scroll to this line in the main log.
- param: (string[]) array: [first column value, second column value, ...].
- param: (string) foreground color (i.e., 'red', or '#ff0000')
- param: (string) background color ,, ,,
function md5(str) -- calculate md5 hash for given string
- param: (string) input
- return: (string) md5 hash.

Roadmap

I used to use a version of this tool to monitor a debug log for a multithreaded transaction processing system. Each thread would get their own text colour which was cool, and important parts in the log were logged in the meta part. It wasn't scriptable then, so no javascript was used, it had a few more features that may be reinstated in the future (If there is demand for it):

Tree view for grouping log messages per thread
Multiple meta log controls
Toggable filters for what-not to log in meta log.
Tail multiple files, combine output in log view.
Tail database tables through ODBC connection.

TODO:

Copy & paste from the meta log.

Download

Download here: metalogmon-1.0.zip.

In order to run the executable you may need to install the Visual C++ Redistributable for Visual Studio 2012 from microsoft. (My guess is that you don't need to do that if you run windows 8.)

Other platforms: both these libraries compile to windows, linux and osx. But currently I only made binaries for windows. If somebody is interested in another OS I will create the binaries, but until then I'll be lazy

Adding network multiplayer to Smash Battle

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 21 Oct 2012 00:00:00 +0000

October 21 2012

Adding network multiplayer to Smash Battle

Smash Battle is a really cool game made by Bert Hekman and Jeroen Groeneweg. Of which Jeroen is now a collegue of mine at Tweakers.net. It supports up to four players, you can use gamepads and the multiplayer is best out of five. You can get powerups in the game like extra damage, health pack, armor etc. :) (I see that in the codebase also a single player is under development!)

Network version

I decided to add a network multiplayer gametype to it, and I develop that in a separate branch. It supports more than four players.

Currently the network multiplayer supports only bullets and mines (your default equipment). Powerups do not yet appear. All damage to players or tiles is determined on the server. The clients are trusted though, to send their correct player positions, shots fired etc. You could theoretically cheat (up to a certain level) by modifying and compiling your own client, but it is far easier to implement a network multiplayer if I can trust clients somewhat. This can easily be rewritten though, and if you play with low lags you will probably not notice any difference. But I think you will notice if someone is cheating.

Pre-alpha test release

My fork is https://bitbucket.org/rayburgemeestre/smashbattle

It's a pre-alpha because the gametype is not completely finished yet, if there are more than two players a normal best out of five multiplayer starts. Once the game has started, you cannot join the server anymore. You can already test it out simply install the current release of Smashbattle.

On Windows:

It will default install to C:\Program Files\Smash Battle\
Extract the contents of smashbattle-pre-alpha.zip into your installation dir. (Use BattleXP.exe if you're running Windows XP)
If you changed the install location of it, modify the path in register_server.reg.
Double click the .reg file, now smashbattle:// protocol should be available.

On Ubuntu:

Put this line in your /etc/apt/sources.list: deb http://repository.condor.tv lucid main
apt-get update
apt-get install battle
cd /usr/share/games/smashbattle/ <<<<< important! the game expects level data in ".".
./battle

Note that the update command might give you this if you are running 64 bit:

Ign http://us.archive.ubuntu.com quantal-backports/universe Translation-en_US
Fetched 1,032 kB in 30s (33.7 kB/s)
W: Failed to fetch http://repository.condor.tv/dists/lucid/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.

You can ignore this error and continue installing the 32 bit version.

The game should now run, but to use the pre-alpha, you have to replace the 'battle' binary with mine:

rm -f battle && wget http://blog.cppse.nl/battle && chmod +x battle
Edit: you may need to: apt-get install libsdl-net1.2

Server usage

You can start your server with these parameters:

Battle.exe -s 
Example: Battle.exe -s

Battle.exe -s <listen port>
Example: Battle.exe -s 1100

Battle.exe -s <levelname> <listen port>
Example: Battle.exe -s "TRAINING DOJO" 1100

In case no level is given as a parameter, the server will start with level selector. In case no port is given, default will be used (1100).

Client usage

To connect to a client, you need to have registered the .reg file. You can click links like:

smashbattle://<domain.tld>:<port>
Example: smashbattle://cppse.nl:1100

You could type such an url in your WINDOWS+R (Run command) or in command prompt start <url>.

If you do not like to register the .reg file, you can also give it to Battle.exe as a parameter:

Battle.exe <url>
Example: Battle.exe smashbattle://cppse.nl:1100

After you have set a server on your machine, you should be able to connect using ---> smashbattle://localhost:1100

The level names are

"TRAINING DOJO"
"PLATFORM ALLEY"
"PITTFALL"
"DUCK'N'HUNT"
"COMMON GROUNDS"
"POGOSTICK"
"LA MOUSTACHE"
"THE FUNNEL"
"BLAST BOWL"
"PIT OF DEATH"
"RABBIT HOLE"
"STAY HIGH"
"PIE PIT"
"SLIP'N'SLIDE"
"BOULDERDASH"
"SNOW FIGHT"

In game controls

F1 - toggle console (for debugging)
F10 - toggle fullscreen
F11 - toggle FPS
ESCAPE - pause menu

Default your keyboard controls are

a, s, d, w - left, duck, right, jump.
CONTROL - shoot
ALT - mine
SHIFT - run

Roadmap

~~Link gameloop to time~~
~~Create client and server class~~
~~Design efficient protocol~~
~~Create solution for lag~~
Introduce nicknames
Introduce ingame chat
~~Support powerups~~
Introduce more gametypes?
- Red team vs Blue team
- King of the Hill
- Capture the flag
Refactor*
Merge with master

Refactoring:

While developing I sometimes put #include's above the function where I use stuff from it. This is when I feel like I might refactor the code, I can easily remove the #include again. Works for me, but it results in some stray #include's. Also I'm not sure about my design choice of making server and client singleton's (basically global classes). It was easy so I could weave the client/server code into the game rapidly, but I think it may need to integrate with the existing classes better, and use polymorphism a bit more here and there. Example: I have a few places in the code where I do different stuff based on Main::runmode static global, for server do this, for client do this..

Firefox select behaviour and H264 support

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 21 Oct 2012 00:00:00 +0000

October 21 2012

Firefox select behaviour and H264 support

Select behaviour in address bar in Linux

I use control + arrow keys and control + shift + arrow keys for selecting a lot. And as a webdeveloper especially in the address bar. I think it is somehow the default under linux distributions, under OpenSuse anyways, that always all text is selected. I find that very VERY annoying. Because you cannot quickly select (a) piece(es) from the URL. But luckily I found the config setting where you can change this!

Enable H264 support in Windows

In about:config, enable the value media.windows-media-foundation.enabled. Especially useful if you disable Flash. A lot of video players use a HTML5 player as fallback support only the H264 codec.

Cool way to provide multiple iterators for your class (C++)

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 19 Oct 2012 00:00:00 +0000

October 19 2012

Cool way to provide multiple iterators for your class (C++)

Multiple iterators that use the same begin() or end() functions.

In C++ you cannot differentiate based on the type-to-return. Like have two begin() methods in a class that return different iterators.

class foo 
{
    public:
        some_iterator begin() 
        {
            return some_iterator(); 
        }

        // Not possible
        other_iterator begin() 
        {
            return other_iterator(); 
        }
};

some_iterator it = fooinstance.begin();

There is also no template syntax to implement a begin() method for this purpose. Note that you cannot use "straightforward" polymorphism because the subclasses are on the LHS of the assignment.

I still wanted it though and came up with the following solution.

For example an instance of a NumberRange class provides two iterators the default "iterator" simply outputs all the numbers. The "cumulative_iterator" outputs all numbers cumulatively.

int main(int argc, char **argv)
{
    NumberRange range(1, 10);

    cout << "NumberRange::iterator:" << endl;
    for (NumberRange::iterator iter = range.begin(); iter != range.end(); iter++)
        cout << *iter << endl;

    cout << "NumberRange::cumulative_iterator:" << endl;
    for (NumberRange::cumulative_iterator iter = range.begin(); iter != range.end(); iter++)
        cout << *iter << endl;

    return EXIT_SUCCESS;
}

/**
 * Desired output:
 * -----------------------------------------------------------
 * ksh$ g++ iterators.cpp &&./a.out
 * Constructing NumberRange object with numbers 1 to 10
 * NumberRange::iterator:
 * 1
 * 2
 * 3
 * 4
 * 5
 * 6
 * 7
 * 8
 * 9
 * 10
 * NumberRange::cumulative_iterator:
 * 1
 * 3
 * 6
 * 10
 * 15
 * 21
 * 28
 * 36
 * 45
 * 55
 */

I really like this as an API because if you want to change the way of iterating through the range (in this example), you only change NumberRange::iterator to something else.

Implementation of NumberRange

class NumberRange
{
public:
    NumberRange(int rangebegin, int rangeend)
    {
        cout << "Constructing NumberRange object with numbers " << rangebegin << " to " << rangeend << endl;

        for (int i=rangebegin; i<=rangeend; i++)
            numbers_.push_back(i);
    }

    NumberIter<void> begin()
    {
        return NumberIter<void>(numbers_, 0);
    }

    NumberIter<void> end()
    {
        return NumberIter<void>(numbers_, numbers_.size());
    }

    typedef NumberIter<Traits_Normal> iterator;
    typedef NumberIter<Traits_Cumulative> cumulative_iterator;

private:
    vector<int> numbers_;
};

It's a very simple implementation. It stores the numbers in a vector. The functions begin() and end() provide iterators of NumberIter. NumberIter is a templated class with traits. Possible traits that we are going to define are: void, Traits_Normal, Traits_Cumulative. I prefer to use void where the specific Trait is not yet known. I could have also have made a Traits_Null.

NumberRange only works with NumberIter<void> because begin() does not know what specifc NumberIter instance to return (Traits_Normal or Traits_Cumulative). In the assignment "NumberRange::iterator iter = range.begin()" the NumberIter<void> is converted into a NumberIter<Traits_Normal>. (NumberRange::iterator is a typedef for NumberIter<Traits_Normal>.)

Implementation of NumberIter

This class is templated to provide multiple kinds of iterators, by using NumberIterTraits. These traits provide the implementation of the specific iterators. So this class only provides the API.

template <typename T, typename Traits = NumberIterTraits<T> >
class NumberIter: public std::iterator< std::forward_iterator_tag, string >
{
public:

    // Constructors
    NumberIter(const vector<int> &numbers, size_t seq)
        : sequence_(seq), numbers_(numbers)
    {}

    // Copy constructor
    NumberIter(const NumberIter<void> &other)
    {
        numbers_ = other.numbers_;
        sequence_ = other.sequence_;
    }

    // Operators
    const int operator*() const
    {
        return Traits::next(numbers_, sequence_);
    }

    NumberIter & operator++(int)
    {
        sequence_++;
        return *this;
    }

    template <typename N>
    bool operator==(const NumberIter<N>& other)
    {
        return sequence_ == other.sequence_;
    }

    template <typename X>
    bool operator!=(const NumberIter<X>& other)
    {
        return !((*this) == other);
    }

private:

    vector<int> numbers_;
    size_t sequence_;

    friend class NumberIter<Traits_Normal>;
    friend class NumberIter<Traits_Cumulative>;
};

Constructor takes a copy of the numbers vector, which is really inefficient. But I wanted to keep the example simple. Sequence parameter is the current position of the iterator.
There is a copy constructor (used in the assignment "NumberRange::iterator iter = range.begin();")
operator* returns the current value of the operator. Note that the traits implement different processing and return.
operator++ increments the sequence.
operator== and operator!= are required for "i != range.end()".

Traits classes

template<typename T> class NumberIterTraits;
template<> class NumberIterTraits<void>
{
public:
    static int next(const vector<int> &numbers, size_t sequence)
    {
        throw logic_error("NumberIterTraits<void>::next should not be used.");
    }
};


class Traits_Normal;
template<> class NumberIterTraits<Traits_Normal>
{
public:
    static int next(const vector<int> &numbers, size_t sequence)
    {
        return numbers[sequence];
    }
};


class Traits_Cumulative;
template<> class NumberIterTraits<Traits_Cumulative>
{
public:
    static int next(const vector<int> &numbers, size_t sequence)
    {
        if (sequence < 0)
            return 0;

        int value = 0;

        for (int i=0; i <= sequence; i++)
            value += numbers[i];

        return value;
    }
};

The Traits_Normal version simply returns the number at the index. The Traits_Cumulative sums all numbers from first to current index.

Note that to add another iterator you only need to add another Traits class. (Well in my case another typedef in NumberRange for consistency as well. But you could do without and omit them like "for (NumberIter<Traits_Something> i = range.begin(); ...)".)

[Edit: also a friend class declaration in NumberIter. That's so that the "generated" NumberIter classes can reference internals. Personal preference over adding more class functions.]

[Edit2: You could add a Traits_Reverse with "return numbers[numbers.size() - ++sequence];"]

Final notes

IIRC there are some compilers that require an implementation of "operator=" for the conversion in "NumberRange::iterator = range.begin()". They refuse to use the copy constructor for this statement. In that case use this on the class.

NumberIter operator=(NumberIter<void> val)
{
    numbers_ = val.numbers_;
    sequence_ = val.sequence_;
    return *this;
}

Complete source code can be downloaded here.

There are probably more alternatives for this, i.e. you could probably do without templates.

The iterators in this example are not fully std compliant. I.e. you cannot use them in functions from #include <algorithm>.

Code tested on gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux).

X, Y to Latitude + Longitude functions for (google) maps

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 23 Sep 2012 00:00:00 +0000

September 23 2012

X, Y to Latitude + Longitude functions for (google) maps

I have an unfinished project with--in general--some really horrible sourcecode, but with some cool functions and solutions I came up with as well. One thing I needed for example was to calculate latitude and longitude coordinates from X and Y positions on a google maps canvas, taking zoom level into account. I could not find these conversion functions (around august 2011 anyway).

[Edit, now since May 21, 2015 Google Maps Api V3 was released, which makes it possible with the Google API. I also found an example gist here]

Aligning markers to a grid

The reason I needed these convertion functions in the first place was for creating an align feature for (custom) markers on a google maps canvas. This is how it works before/after aligning:

The (very simple) algorithm I came up with divides the map in slots. A grid with a width of 100 for example, only positions markers on 100, 200, 300, 400 pixels. In this example, the 'nearest' slot's width of a marker at position 220,50 pixels wouuld be 200.

The algorithm in pseudocode:

INITIALIZE GRID HEIGHT AND WIDTH ACCORDING TO MAP'S ZOOMLEVEL
    /* gridwidth = 100 << (21 - map.getZoom()) */
    /* gridheight = 30 << (21 - map.getZoom()) */

FOREACH MARKER

    CONVERT MARKER LAT,LON TO X,Y COORDINATES
        /* y = latToY(marker.getPosition().lat()); */
        /* x = lonToX(marker.getPosition().lng()); */

    CONVERT X,Y TO NEAREST SLOT X,Y
        /* slot_y = Math.round(y - (y % gridheight)) */
        /* slot_x = Math.round(x - (x % gridwidth)) */

    SET MARKER X,Y TO SLOT X,Y

    WHILE SLOT POSITION IS OCCUPIED BY ANOTHER MARKER

        MOVE TO NEXT SLOT POSITION
            /* Next slot position is according to a simple spiral movement [1] */

        SET MARKER X,Y TO SLOT X,Y

    ENDWHILE

    CONVERT MARKER X,Y TO LAT,LON
        /* marker.setPosition(new google.maps.LatLng(lat, lon)); */
        /* marker.setPosition(new google.maps.LatLng(lat, log)); */

    STORE SOMEWHERE THAT MARKER IS IN THIS SLOT POSITION

ENDFOREACH

[1]: The search for next slot position is according to this pattern:

up, right, down, down, left, left, up, up, up, right, right, right, etc.

The change of direction is continious (a spiral): {up, right, down, left, ..}
The number of 'steps' in each direction is {1, 1, 2, 2, .... n, n}

latToX() and lonToY()

I'm not an expert in math but I was able to find some expressions online that resolved lat+lon for x+y (the other way around). I simply replaced all the constants with their values and put them in a solver to solve them for the variables I was interested in (e.g. longitude for XtoLon). I probably have the sites bookmarked somewhere but I can't find them.

var glOffset = 268435456;
var glRadius = 85445659.4471;// offset / pi

function lonToX(lon) 
{
    var p = Math.PI / 180;
    var b = glRadius * lon;
    var c = b * p;
    return Math.round(glOffset + c);
}

function XtoLon(x) 
{
    return -180 + 0.0000006705522537 * x;
}

function latToY(lat)
{
    return Math.round(glOffset - glRadius *
            Math.log((1 + Math.sin(lat * Math.PI / 180)) /
                (1 - Math.sin(lat * Math.PI / 180))) / 2);
}

function YtoLat(y) 
{
    var e = 2.7182818284590452353602875;
    var a = 268435456;
    var b = 85445659.4471;
    var c = 0.017453292519943;

    return Math.asin(Math.pow(e,(2*a/b-2*y/b))/(Math.pow(e,(2*a/b-2*y/b))+1)-1/(Math.pow(e,(2*a/b-2*y/b))+1))/c;
}

They are not pretty but I like them because they work really well

Edit 10-AUG-2015: deltaLonPerDeltaX(), deltaLatPerDeltaY()

I found out somebody on Stackoverflow elaborated my functions with a deltaLonPerDeltaX() and deltaLatPerDeltaY(). The original poster's image is no longer available, so I'm not sure if I understand the question correctly, and therefore these additional functions. But there is a nice extra info cited from Google, which I will copy here:

At zoom level 1, the map consists of 4 256x256 pixels tiles, resulting in a pixel space from 512x512. At zoom level 19, each x and y pixel on the map can be referenced using a value between 0 and 256 * 2^19

(See [https://developers.google.com/maps/documentation/javascript/maptypes?hl=en#MapCoordinates][https://developers.google.com/maps/documentation/javascript/maptypes?hl=en#MapCoordinates])

PhpStorm and Ideavim {Escape,C-c,C-[} responsiveness

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 30 Jul 2012 00:00:00 +0000

July 30 2012

PhpStorm and Ideavim {Escape,C-c,C-[} responsiveness

Just one thing that was annoying me for a long while, and how I fixed it. I tend to switch back and forth between insert and command mode in vim. And somehow PhpStorm with IdeaVim plugin enabled felt non-responsive. I press escape, start hitting :wq, and I have :wq in my code.

I got accustomed hitting Escape twice, and later even three times, by default so that I was more certain I was out of insert mode. I also tried Control+C, and Control+[, but they have the same problem.

I know the 'problem' always occured when i.e. PhpStorm started rendering an Intellisense popup: press '.' somewhere, in a large file it may take a few moments before that popup appears (maybe due to parsing etc.), so you don't see it. Assuming you are now in command mode, the escape press was actually consumed by the popup. Then of course you do escape to command, and try to undo, but it undo's a lot more than the chars you now accidentally sprayed in the code (also not exactly the same behaviour as Vim, but alas :D)

Fix

Right mouse click -> Remove Escape:

Go to Plug-ins -> IdeaVIM ->

Find the row with all the keybindings on it.. right click on it -> Add Keyboard Shortcut

Hit escape, save that. -> Apply -> Ok.

Annnnnd you're done!

Tweakers logo test

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 12 Jul 2012 00:00:00 +0000

July 12 2012

Tweakers logo test

Firebird IBPP use in non-unicode project

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 12 May 2012 00:00:00 +0000

May 12 2012

Firebird IBPP use in non-unicode project

I posted this in a comment here, a long while ago. I forgot about it but yesterday someone posted in the same topic, and therefore I received an e-mail.

As my comment somehow isn't visible on the blog (maybe it was never moderated?) i'll post it here. I was using it in a non-unicode project myself, so I encountered the same problem. According to the mail I recvd this is what I wrote:

Nice fixes.

How I get the sourcecode to work in my unicode program however, without modifying the source is as follows.

Simply don't compile the file all_in_one.cpp (or all ibpp/core/*.cpp files individually) with the defines/"preprocessor definitions" _UNICODE and UNICODE enabled.

I didn't look at the Flamerobin source, but my guess is that they do the same.
Posted by rayburgemeestre to Untouched at 6:05 PM

IBPP in a Qt Creator project

Add the following in your .pro file:

DEFINES += IBPP_WINDOWS=value

LIBS += Advapi32.lib

the lib is for fixing

all_in_one.obj : error LNK2019: unresolved external symbol __imp__RegCloseKey@4 referenced in function "public: struct ibpp_internals::GDS * __thiscall ibpp_internals::GDS::Call(void)" (?Call@GDS@ibpp_internals@@QAEPAU12@XZ)
all_in_one.obj : error LNK2019: unresolved external symbol __imp__RegOpenKeyExA@20 referenced in function "public: struct ibpp_internals::GDS * __thiscall ibpp_internals::GDS::Call(void)" (?Call@GDS@ibpp_internals@@QAEPAU12@XZ)
all_in_one.obj : error LNK2019: unresolved external symbol __imp__RegQueryValueExA@24 referenced in function "public: struct ibpp_internals::GDS * __thiscall ibpp_internals::GDS::Call(void)" (?Call@GDS@ibpp_internals@@QAEPAU12@XZ)

Also added the following two #undefs to all_in_one.cpp.

#undef _UNICODE
#undef UNICODE

Automatic nickname in channel and nicklist colouring in mIRC

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 18 Apr 2012 00:00:00 +0000

April 18 2012

Automatic nickname in channel and nicklist colouring in mIRC

I find it pleasant to have nicknames coloured in busy channels, that's why I made this. It simply generates colours by hashing the nicknames. This ensures that a given nickname will always be the same colour.

(more-begin)

The script

;;;
;;; Lazy nickname coloring script
;;;
;;; Color all nicknames automatically by calculating a numeric hash over the nickname.
;;; The calculated number is used to pick a (space delimited) color from the %colors variable 
;;;  (set in "on START" event).
;;; Colors are made configurable because yellow on white is annoying, and you may want to use
;;;  black or white depending on your background color.
;;;

;; Initialize

on 1:START: {
  .initialize_coloring
}

alias initialize_coloring {
  ; use the following colors only
  .set %colors 1 2 3 4 5 6 7 9 10 11 12 13 14 15

  ; reset all entries in the clist
  while ($cnick(1)) {
    .uncolor_nick $cnick(1)
  }
}

;; Events

; Parse the /names <channel> response(s)
raw 353:*: {
  var %names = $4-  
  var %i = 1
  var %n = $gettok(%names,0,32)
  while (%i <= %n) {
    var %current_nick = $gettok(%names,%i,32)
    var %firstchar = $mid(%current_nick, 1, 1)
    while (%firstchar isin @+%) {

      %current_nick = $mid(%current_nick, 2)
      %firstchar = $mid(%current_nick, 1, 1)
    }
    .color_nick %current_nick

    inc %i
  }
}

; Handle nick changes/joins/quits
on 1:NICK: { 
  .uncolor_nick $nick
  .color_nick $newnick 
}

on 1:JOIN:*: { 
  .color_nick $nick
}

on 1:QUIT: { 
  .uncolor_nick $nick
}

;; Helper functions

; usage: color_nick <nickname>
alias color_nick {
  if (!%colors) {
    .initialize_coloring 
  }
  var %colors_idx = $calc($hash($1, 16) % $numtok(%colors, 32)) + 1
  var %nick_color = $gettok(%colors, %colors_idx, 32)
  .cnick $1 %nick_color 
}

; usage: uncolor_nick <nickname>
alias uncolor_nick {
  .cnick -r $1
}

Copy & paste it in your remote (open with alt + r).

You may need to enable nicklist colouring in general. Use alt + b, Nick colors, choose "Enable".

Update!!!

Note that I have a new version of this available, see this blogpost.. It also provides a script that makes nicks marked as away light-grey!

(more-end)

Allegro 5 separate process for rendering

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 08 Apr 2012 00:00:00 +0000

April 8 2012

Allegro 5 separate process for rendering

The following image was rendered by another process

Motivation for using separate processes for rendering is if you wish to have multiple threads rendering. I do a lot of set blending type, put pixels, set blender type again, more pixels, etc. If I use async() to render multiple images at once these function calls might interfere as race conditions.

(more-begin)

Probably a noobish moment, but I never realized the "stack" was this limited. I tried declaring something like

struct structw800h600
{
    ...
    Pixels pixels[800 * 600].
};

message_queue mq (create_only, "pixels",
    1, //max message number
    sizeof(structw800h600)); //max message size

structw800h600 img;
memset(&img, 0x00, sizeof(structw800h600));

This code caused an exception while constructing the object that declared an instance of the struct on the stack:

Unhandled exception at 0x003E5017 in Starcry.exe: 0xC00000FD: Stack overflow (parameters: 0x00000000, 0x00702000).

Shows break here in chkstk.asm (because I am in debug mode):

[...]
; Find next lower page and probe
cs20:
        sub     eax, _PAGESIZE_         ; decrease by PAGESIZE
        test    dword ptr [eax],eax     ; probe page. <<<<<<<<<<<<<<<<<<<< here
        jmp     short cs10

_chkstk endp

        end

I did not find out the exact threshold but the the crash occured when the size of the struct was above ~1024972 bytes or ~1000 kB. (Size of each pixel object is 16 byte). If I understand it correctly the stack is only several MB so I was simply storing too much data on it.

Still posting this because I almost jumped to the false conclusion that it was a windows platform shared memory limitation. Simply allocate the Pixel objects from the free-store and send that through the message queue. Something like: Pixel *pixels = new Pixel[800 * 600];

(more-end)

Allegro 5 and wxWidgets example

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 08 Apr 2012 00:00:00 +0000

April 8 2012

Allegro 5 and wxWidgets example

I have posted on using allegro 4 with wxWidgets before. Allegro 5 is more easy.

Just the stuff I encountered and how to fix

(more-begin)

Fix main conflict

#define ALLEGRO_USE_CONSOLE 1

Avoids the following error.

1>MSVCRTD.lib(crtexe.obj) : error LNK2019: unresolved external symbol _main referenced in function ___tmainCRTStartup

#define ALLEGRO_USE_CONSOLE 1

#include <allegro5/allegro.h>
#include <allegro5/allegro_image.h>
#include <allegro5/allegro_primitives.h>
#include <allegro5/allegro_font.h>
#include <allegro5/allegro_ttf.h>

Draw ALLEGRO_BITMAP on static canvas

Note that there is no equivalent of Allegro 4's draw_to_hdc() function. With a little grepping in the source code I found out that (for windows anyway) you have functions that do the same in C:\allegro5\src\win\wmcursor.c

Just borrow local_draw_to_hdc from there and use it in the paint event.

staticbitmap->Connect(wxID_STATIC, wxEVT_PAINT, wxPaintEventHandler(SharedMemoryTest::OnPaint), NULL, this);

void SharedMemoryTest::OnPaint( wxPaintEvent& event )
{
    wxPaintDC dc(wxDynamicCast(event.GetEventObject(), wxWindow));
    WXHDC wxHDC = wxPaintDC::FindDCInCache((wxWindow*) event.GetEventObject());
    HDC hDC = (HDC) wxHDC;

    local_draw_to_hdc(hDC, bmp, 0, 0);
}

(more-end)

Proxytunnel HOWTO

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 26 Jan 2012 00:00:00 +0000

January 26 2012

Proxytunnel HOWTO

If you are behind a firewall, chances are you can tunnel through it with Proxytunnel. This post does not describe anything new, but I think is still useful because it includes configuration of apache and ssh client examples.

The goal is being able to tunnel through a (corporate) firewall/proxy. And even more important, have your communication encrypted. This also has the advantage that even if you are not restricted, a corporate firewall/proxy can still not cache the websites you visit.

We do this by establishing an ssh session to some machine, and useing ssh portforwarding from there. This target machine may be your home computer or some server on the internet.

(more-begin)

If you are able to run your SSH server on port 80 or 443, you might want to do that because then you can simply define the firewall as a proxy in PuTTY. The firewall should probably allow the communication, especially on 443 as this is normally for HTTPS and encrypted (as is SSH). I haven't tested this, but I believe you should be able to skip the proxytunnel stuff.

I assume you already have Apache running on port 80 and 443, so switching SSH to one of those ports won't be possible. We simply configure Apache so that it becomes itself another proxy that can make the connect to port 22, or 42 in the example I'm going to use. If you do not want to use apache, you can put your webserver of choice on a different port and use Apache's mod_proxy to redirect a virtual host to it.

In short how it works:

Your ssh client will NOT communicate directly to your ssh server. Instead it will communicate with proxytunnel, and proxytunnel establishes the actual connection. Proxytunnel will first connect to the "corporate" firewall/proxy and request a connection to your server on the HTTPS port, The firewall will then consider all communication HTTPS encrypted traffic and therefor allow it. But actually a mod_proxy is configured to respond to connection requests to specific destinations (using CONNECT dest:port HTTP/1.1). So we issue another CONNECT connection to the destination + SSH port. From that moment on proxytunnel simply redirects all read/write to the ssh client.

Once connected to your SSH server you can simply use the Port forwarding stuff that the SSH protocol supports.

Example config

I will be using this hosts throughout the post, you will have to replace these.

Ip	Host	Description
46.51.179.218	ext.cppse.nl	My server machine, runs the apache @ port 80 and destination ssh @ 42
NA	whatismyipaddress.com	Some website @ port 80 that displays remote host (optional for testing)
172.18.12.11	NA	The firewall @ port 8080, accepts only connections to ports 80,443.

Configure proxy on some Apache server

You need mod_proxy, mod_proxy_http, mod_proxy_connect modules enabled in Apache. (Not 100% sure about mod_proxy_http.)

Create a VirtualHost like this:

<VirtualHost *:80>
    ServerAdmin no-reply@ext.cppse.nl
    ServerName ext.cppse.nl
    ErrorLog /var/log/apache2/error_log
    TransferLog /var/log/apache2/access_log

    # Allow proxy connect (forward-proxy) to servers only on port 80 (http) and 42 (at my box SSH)
    ProxyRequests On
    AllowConnect 80 42
    # Deny all proxying by default...
    <Proxy *>
        Order deny,allow
        Deny from all
    </Proxy>
    # This directive defines which servers can be connected to.
    # Access is controlled here via standard Apache user authentication.
    <ProxyMatch (46\.51\.179\.218|ext.cppse.nl|whatismyipaddress.com|www.whatismyipaddress.com)>
        Order deny,allow
        Allow from all

        #You should replace the above two rules with something like this:
        # Deny from all
        # Allow from <some_host>
        # Allow from <some_host>
    </ProxyMatch>
</VirtualHost>

This example will allow from any source to CONNECT to four locations: 46.51.179.218, ext.cppse.nl, whatismyipaddress.com and www.whatismyipaddress.com. Only destination ports 80 and 42 are allowed. We'll be using 46.51.179.218 on port 42 (SSH server), and {www.}whatismyipaddress.com on port 80 (plain HTTP) for testing.

Add this VirtualHost as the first virtual host. Loading it /after/ other vhosts made the proxy deny all CONNECT's on my machine.
Port 443 would be nicer, again, on my machine I couldn't do this because I have other HTTPS sites configured, and couldn't get it to use the proxy "as HTTP on port 443". My apache seems to expect SSL communication although I didn't enable SSL on the vhost.
The vhost name "ext.cppse.nl" seems unimportant, the Proxy settings appear not to be specifically bound to this vhost. This might explain why using port 443 didn't work.
I can imagine there would be some more complicated trick to make it possible to configure "unencrypted" traffic over port 443 for a specific vhost, butthis works well enough for me.

Test if this proxy works

You might want to test this from some location where you are not behind the firewall. Configure it as a proxy in your browser:

This is why I added [www.whatismyipaddress.com][] and port 80 in the Virtual Host, open it:

You can also test the SSH connection if your client supports usage of an HTTP proxy.
You also might want to replace the default allow by the default deny config in the vhost.
You might want to remove port 80 from the AllowConnect parameter in the vhost, and the whatismyipaddress domain(s).

Configure proxytunnel for PuTTY

In our example we have the proxy "172.18.12.11:8080", with no-authentication required. If you have a proxy that requires a username and password use the -P "username:password" parameter on proxytunnel. Also see the help for more available options.)

Install proxytunnel on windows

~~I made a zip file with Putty "Development snapshot 2012-01-16:r9376" because it supports "local proxy" feature we need to use for Proxytunnel, also included version 1.9.0.~~ You can download PuTTY Tray a version of PuTTY that supports local proxy and some more very nice additional features!!

When PuTTY is configured to use Proxytunnel it delegates the connection to proxytunnel, which will first connect to our newly configured proxy "46.51.179.218:80" (the one we configured in apache) using the firewall/proxy 172.18.12.11:8080. Once connected to our proxy we connect to our intended destination "46.51.179.218:42". In PuTTY you use %host:%port (these values get replaced).

This is a command you can use for testing at commandline:

C:\proxytunnel>proxytunnel -v -p 172.18.12.11:8080 -r 46.51.179.218:80 ^
         -d 46.51.179.218:42 -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n"
Connected to 172.18.12.11:8080 (local proxy)

Tunneling to 46.51.179.218:80 (remote proxy)
Communication with local proxy:
 -> CONNECT 46.51.179.218:80 HTTP/1.0
 -> Proxy-Connection: Keep-Alive
 -> User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n
 <- HTTP/1.1 200 Connection established

Tunneling to 46.51.179.218:42 (destination)
Communication with remote proxy:
 -> CONNECT 46.51.179.218:42 HTTP/1.0
 -> Proxy-Connection: Keep-Alive
 -> User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n
 <- HTTP/1.0 200 Connection Established
 <- Proxy-agent: Apache/2.2.12 (Linux/SUSE)

Tunnel established.
SSH-2.0-OpenSSH_5.1

You give exactly the same command to PuTTY although, instead of the -v flag and hardcoded destination you use the -q (quiet mode) (and %host:%port). PuTTY then communicates by reading/writing to the started proxytunnel process, instead of a socket.

This is how you configure PuTTY

Note that the Keep-alive may be necessary if the firewall we're going to tunnel through actively closes connections if they are idle for longer than xx seconds.

You can configure all kinds of portforwarding.

Install proxytunnel on linux

Download proxytunnel and "make" like any other tool. If you are missing development packages, I may have a precompiled 32 bit version available that might work on your box. Todo: Add download link.

linux-yvch:/usr/local/src # tar -zxvf proxytunnel-1.9.0.tgz 
...
linux-yvch:/usr/local/src # cd proxytunnel-1.9.0
..
linux-yvch:/usr/local/src/proxytunnel-1.9.0 # make
..
linux-yvch:/usr/local/src/proxytunnel-1.9.0 # make install
..
linux-yvch:/usr/local/src/proxytunnel-1.9.0 # cd

Just as with PuTTY you need to configure your ssh config: In linux I prefer to keep it verbose (the -v setting, you can use -q for quiet mode). Note that openssh uses %h:%p for host / port replacement.

linux-yvch:~ # cat ~/.ssh/config
Host 46.51.179.218 ext.cppse.nl ext.cppse.nl 
        DynamicForward 1080
        ProxyCommand proxytunnel -v -p 172.18.12.11:8080 -r 46.51.179.218:80 \
               -d %h:%p -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n"
        ServerAliveInterval 30

Connecting with openssh should yield something like:

linux-yvch:~ # ssh -l proxy -p 42 46.51.179.218
Connected to 172.18.12.11:8080 (local proxy)

Tunneling to 46.51.179.218:80 (remote proxy)
Communication with local proxy:
 -> CONNECT 46.51.179.218:80 HTTP/1.0
 -> Proxy-Connection: Keep-Alive
 -> User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n
 <- HTTP/1.1 200 Connection established

Tunneling to 46.51.179.218:42 (destination)
Communication with remote proxy:
 -> CONNECT 46.51.179.218:42 HTTP/1.0
 -> Proxy-Connection: Keep-Alive
 -> User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n
 <- HTTP/1.0 200 Connection Established
 <- Proxy-agent: Apache/2.2.12 (Linux/SUSE)

Tunnel established.
Password: *****
Last login: Thu Jan 26 15:55:40 2012 from 46.51.179.218

  __|  __|_  )  SUSE Linux Enterprise
  _|  (     /       Server 11 SP1
 ___|\___|___|       x86 (32-bit)

For more information about using SUSE Linux Enterprise Server please see
http://www.novell.com/documentation/sles11/

Have a lot of fun...
YOU ARE IN A RESTRICTED SHELL BECAUSE THIS ACCOUNT IS ONLY FOR TUNNELING
proxy@ip-10-235-45-12:/home/proxy>

After the "Tunnel established" you continue as with any other SSH connection.

Using SSH port forwarding

It would have been more elegant if the first connect would have been to port 443. Because then the communication, although when sniffing you see the CONNECT statement and the SSH banner in plain text. From the firewall perspective it is all encrypted data. It just coincidentally happens to be readable . But after the initial stuff everything is encrypted as we're tunneling SSH. I'm not sure if it is possible to communicate in SSL to the second proxy, because then it won't be detectable at all.. the SSL communication would be encrypted twice!

I already included in the PuTTY screenshots and OpenSSH example a Dynamic Forward (socks) proxy on 1080. This means that SSH will start a listener socket on port 1080 accepting connections and tunneling it through the established connection. The SSH protocol supports this, and this feature is (I think) enabled by default, it is configurable on the server in your sshd config.

You can then configure your browser to use the socks proxy, localhost:1080 and all communications will go through the established tunnel. Remote desktop, at the time of writing, doesn't support the use of a proxy, but you can create a "normal" port-forward as for this to a specific destination & port.

If your firewall does not support CONNECT you might want to try cURLproxy, a proxy program I wrote that works simply by downloading and POSTing HTML. Available here: curlprox[cURLproxy].

(more-end)

Proxy securely through ANY corporate proxy/firewall

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 14 Jan 2012 00:00:00 +0000

January 14 2012

Proxy securely through ANY corporate proxy/firewall

DISCLAIMER: Okay, probably still almost any firewall. There are a few posts on the internet about how SSH tunnels bypass "almost any firewall", I believe this proxy will probably bypass a whole lot more firewalls. So I had to do come up with something better than "almost any" .

When is this useful?

ProxyTunnel is awesome as it allows you to tunnel to SSH through--for example--port 443. And due to SSH supporting port forwards you can go from there to wherever you want. If I am correct, it requires that the proxy in question supports the CONNECT syntax.

Sometimes however, proxies are more restricted than that: CONNECT may not be supported; connections are not allowed to stream (i.e., file downloads are first downloaded by the proxy server, scanned for viruses, executables and other filetypes may be blocked); base64 may actually be decoded to see if it contains anything that isn't allowed, it may go as far as to inspect content of zip files and may have restrictions on the maximum file size for downloads (XX MB limit). In that case ProxyTunnel won't suffice.

If you're unfortunate enough to be behind such a firewall, no worries because now there is a way to tunnel through it! The only requirement for it to work is that you can receive plain text from a webpage, and post data to it. One that you own or have access to. Well If you can't do that, I suggest you look for another Job, because this is REALLY important!!!!1 (Not really but then this proxy solution won't work). Do not expect it to be very performant with broadband type of stuff by the way.

(more-begin)

How it works in short

It works with three PHP scripts. And just like with Proxytunnel you need to run one of them on your local computer: localclient.php. This script binds to a local port, you connect with your program to this local port. Each local client is configured to establish a connection with some destination host + port. But the cool part is, it does so by simply reading plain old HTML from an url, and posting some formdata back to it. Well actually it appears to be plain old HTML, because it's the data prefixed with an HTML tag, followed by the connection identifier and the DES encrypted data (converted into base64).

The curl proxy (as I call it, because I use the cURL extension in PHP) retrieves HTML pages like this:

Example of packet with data "PONG :leguin.freenode.net", is sent as the following HTML:

<PACKET>a5bc97ba2f6574612MNIoHM6FyG0VuU6BTF/Pv/UcVkSXM5AbiUrF4BDBB4Q=
|______||_______________||__________________________________________|
       |                |                                           `=BASE64 OF ENCRYPTED DATA
       |                `=Session id / socket id
       `=Fake HTML tag

POSTing back sends a string with the same syntax back, basically only prefixed with "POST_DATA=".

In order for this to work, a second script has to be callable on the web, you must be able to access it, and the same machine has to be able to make the connections you want. For example: http://your-server/proxy.php (you could rename it to something less suspicious; there are some smart things you can do here, but I'll leave that to your imagination ). All proxy.php does is write and read files from a directory, nothing more.

Then a shellscript has to be started to run in background, with access to the same directory. This script scans that directory for instructions, specifically starting server.php processes for new connections. The actual connection is made in the server.php script. And all this script does is read from the same directory for packets received, which it will send to it's socket, any data read from the proxy is written back to the directory, which proxy.php will eventually sent back to the client.

Graphical explanation

You should follow the arrows in the same order as presented in the Legend. Click to enlarge the image.

Design decisions

When I had the idea to make it, I didn't feel like spending alot of time on it, so I hacked it together in a few hours. Then I tested it, it worked and it got me exited enough to refactor it and make a blog post out of it.

After the encryption of the packets I use base64 encoding, which increases the size of the messages, but it looks more HTML-like. If I wanted to send the encrypted data raw I'd have to do some more exotic stuff, maybe disguise it as a file upload, because AFAIK a plain old POST does not support binary data.
I use BASE64 and not urlencode on the encrypted data, because when I tested it urlencode produced even more overhead. Of course the BASE64 string is still "urlencoded" before POST, but only a few chars are affected.
I don't use a socket for communicating between proxy.php and server.php, but files and some lock-files because I preferred them somehow. A database would be nicer, but implies more configuration hassle.

Encryption used

define('CRYPT_KEY', pack('H*', substr(md5($crypt_key),0,16)));

function encrypt_fn($str)
{
    $block = mcrypt_get_block_size('des', 'ecb');
    $pad = $block - (strlen($str) % $block);
    $str .= str_repeat(chr($pad), $pad);

    return base64_encode(mcrypt_encrypt(MCRYPT_DES, CRYPT_KEY, $str, MCRYPT_MODE_ECB));
}

function decrypt_fn($str)
{   
    $str = mcrypt_decrypt(MCRYPT_DES, CRYPT_KEY, base64_decode($str), MCRYPT_MODE_ECB);

    $block = mcrypt_get_block_size('des', 'ecb');
    $pad = ord($str[($len = strlen($str)) - 1]);

    return substr($str, 0, strlen($str) - $pad);
}

If you prefer something else, simply re-implement the functions, you'll have to copy them to all three scripts (sorry, I wanted all three scripts to be fully self-contained).

I found my "ASCII key → md5 → 16 hexadecimal display chars → actual binary" a pretty cool find by the way. Did you notice it?

Demonstration

Note that first I demo it where the server is running on an Amazon AMI image. Appended to the video is a short demo where I run the server on my local windows pc (just to show how it it'd work on windows). This second part starts when I open my browser with the google page.

Remote desktop actually works pretty good through the curl proxy by the way. Establishing the connection is a little slow like with WinSCP, but once connected it performs pretty good. I could't demo it because I don't have a machine to connect to from home.

Sourcecode & downloads

Put it here on bitbucket: https://bitbucket.org/rayburgemeestre/curlproxy Placed it under MPL 2.0 license, which seamed appropriate. Basically this means that when you distribute it with your own software in some way, you'll have to release your code changes/improvements/bugfixes (applicable to curlproxy) to the initial developer. This way the original repository will also benefit and you're pretty much unrestricted.

(more-end)

GNU Screen Navigator V2

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 25 Nov 2011 00:00:00 +0000

November 25 2011

GNU Screen Navigator V2

Work in progress...

It will be a lot easier to compile. No longer dependant on the json lib. A single .cpp file (as the code is quite small).

No makefile, just a g++ goto.cpp -o goto -lncurses

Get the source code here

(more-begin)

P.S. I added colours:

Updates

24-feb-2013: Now listens for ncurses KEY_RESIZE event so changing window size will redraw.

(more-end)

Some CGI "tips"

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 20 Nov 2011 00:00:00 +0000

November 20 2011

Some CGI "tips"

Bash wrapper script

With Apache (2.2) you could get an generic "Internal Server Error" error message in case the cgi sends the wrong headers. There is probably a setting for this in Apache as well, but I always create a bash wrapper script. For example someapp.cgi:

#!/bin/bash
printf "Content-type: text/html\n\n"
/path/to/actual_appl

This immediately makes the output visible and you can comment the printf statement once fixed. This trick only makes sense if you don't have quick access to a debugger or a core dump.

Running application in chroot

There are plugins for apache AFAIK for running cgi applications in a chroot. I didn't experiment with these, as I simply use my (probably lame) bash wrapper here as well:

#!/bin/bash
sudo -E /usr/bin/chroot /usr/local/src/some_jail /usr/bin/some_appl 2>&1

The -E flag means "preserve environment". To allow this you have to configure sudoers properly (visudo). Something like this:

wwwrun ALL=(ALL) SETENV: ALL, NOPASSWD : /usr/bin/chroot

Inline printf compatible with char *

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 11 Nov 2011 00:00:00 +0000

November 11 2011

Inline printf compatible with char *

This is no rocket science but I thought this was a really cool solution to the problem.

First I created a helper function Xprintf to interface with an existing C API that works with (non const) char arrays. Hence its char * return value.

char *Xprintf(const char *format, ...);

// This function works in the following situations

foo1(Xprintf("Hello world: %d", 1001)); // void foo1(char *);
foo2(Xprintf("Hello world: %d", 1001)); // void foo2(const char *);
foo3(Xprintf("Hello world: %d", 1001)); // void foo3(const string);
foo4(Xprintf("Hello world: %d", 1001)); // void foo4(const string &);
foo5(Xprintf("Hello world: %d", 1001),
     Xprintf("...", ...));              // void foo5(char *, char *);

Xprintf cannot use just one buffer because the case of 'foo5' would fail (it would get the same pointer twice).

I needed a different return value, like std::string, so that copies could be returned which would clean themselves up as soon as they went out of scope. But std::string does not provide implicit casting to const char *, only explicit casting through .c_str(). The call to foo1 would become: foo1(const_cast(Xprintf("").c_str())), which is kind of ugly!

The following fixes it, creating a tmp_str class that extends std::string and simply provides the implicit cast:

class tmp_str : public std::string
{
public:
    tmp_str(const char *str)
        : std::string(str) {}

    // g++ is fine with adding this one, xlC isn't
    //operator const char *() const { return c_str(); }

    operator char *() const { return const_cast<char *>(c_str()); }
};

tmp_str cHelperCharArray::Xprintf(const char *format, ...)
{
    char buffer[512] = {0x00};

    va_list args;
    va_start(args, format);
    vsprintf(buffer, format, args);
    va_end(args);

    return tmp_str(buffer);
}

A note why tmp_str is-a std::string and not an is-implemented-in-terms-of: the call to foo4 would fail as it would not accept tmp_str as a reference to string (A parameter of type "const std::basic_string,std::allocator > &" cannot be initialized with an rvalue of type "tmp_str".). )

g++ accepts all these foo* functions, but IIRC xlC doesn't like foo2. In that case I had to cast to const. Adding the const char * operator overload would make some casts for that compiler ambiguous.

Render on top of mplayer using custom window

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 27 Oct 2011 00:00:00 +0000

October 27 2011

Render on top of mplayer using custom window

mplayer can easily be instructed to render on a custom window with the -wid (window handle) parameter.

// On windows
long targetWindowId = reinterpret_cast<long>(canvas->GetHWND());

// On Linux
long targetWindowId = GDK_WINDOW_XWINDOW(canvas->GetHandle()->window);

Now that I got it to render on my canvas, I cannot render on top of it without flickering, because I cannot do double buffering. (I cannot control when mplayer renders frames on the window). That's why I add a second window that reads the first window to a bitmap, I can do whatever I want to that bitmap, and display it . This meant that I could no longer use my preferred video renderer on windows -vo direct3d because somehow that setting doesn't actually draw on the window, just in the same region. When reading the first window I'd get an empty bitmap and not the video. I ended up using -vo directx:noaccel in order to properly read it.

Fix overlap problem

This posed another problem, when hovering the second window on top of the first, it interferes with the video as it renders itself in window1 first. I only encountered this on my windows pc:

I decided to ignore this problem and try to find a way to hide the first window so that it wouldn't interfere. I tried minimizing it, Hide(), move it outside the screen, etc. But mplayer would not render the video in these cases. I then tried making the window 100% transparent and this worked. It also fixed my overlap-problem as I could now overlap the windows without problems. Somehow making the windows transparent forces the no-hardware-acceleration-directx renderer to behave differently. Making the window 1% transparent also fixes the overlap-problem.

Fix linux support

On Linux I use the -vo X11 video output, and overlap wasn't a problem. The only annoying thing is that in order to get the GTK window handle you have to include a GTK header in C++, which requires adding a lot of include directories to your include path. Because you need to cast the window handle to a GTKWidget instance, and ask it for the xid.

(more-begin)

Result

The code is available on bitbucket and works on Windows (tested Windows vista with aero theme) and Linux (openSUSE 11.4). Makefile and Visual studio project files included.

Where I used this for..

All texts and images are rendered on top of the background video.

(more-end)

Starcry rendered videos

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 18 Sep 2011 00:00:00 +0000

September 18 2011

Starcry rendered videos

Behaviours engine

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 22 May 2011 00:00:00 +0000

May 22 2011

Behaviours engine

(more-begin)

Milestone 1

Milestone 2

(more-end)

DialogBlocks Howto

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 30 Apr 2011 00:00:00 +0000

April 30 2011

DialogBlocks Howto

Putting source files in separate directory

View youtube video
(don't forget to enable 720p and fullscreen)

How to put source and image files in subdirectories in DialogBlocks..
NOTE for git users: If you're going to do this with an existing git repository. My advice is to do a move through git as well: "git mv source dest". So that git knows it was a file move, not a delete/create.

Seperate implementation files on a few panels

View youtube video
(don't forget to enable 720p and fullscreen)

We put two panels (with their contents) into separate implementation files.
Show some examples you can do with it (or more easily than before).
Events and other functions can be nicely grouped into that implementation.

Compiling with DialogBlocks

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 30 Apr 2011 00:00:00 +0000

April 30 2011

Compiling with DialogBlocks

Compiling with MinGW

View youtube video
(don't forget to enable 720p and fullscreen)

How to add configuration for MinGW

Compiling with Visual C++ compiler with project files

View youtube video
(don't forget to enable 720p and fullscreen)

Shows how to set the platform SDK.
Some specific errors you can get w/ DialogBlocks.
How to use generated project file(s) for visual studio.
- How they handle file changes
- Why to use: autocomplete, debugger, etc.

![nl][] Urenregistratie

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 17 Apr 2011 00:00:00 +0000

April 17 2011

Urenregistratie

Werkzaamheden tijden reverse engineering

Indien je urenregistratie moet invullen met begin- en eindtijden is het soms lastig als je dat achteraf doet. Soms weet je nog wel uit het hoofd wat je gedaan hebt, soms zoek je dat op in (verstuurde) e-mails, of aantekeningen, agendapunten, enz.

Als je dan toch de tijden moet reverse-engineeren, kun je het jezelf natuurlijk ook wat makkelijker maken door in een simpele lijst je werkzaamheden te tikken. Op te geven hoeveel tijd het bij elkaar moet zijn, van een aantal zelf de tijd invullen en vastzetten ('freeze'), een beetje aan knopjes draaien om aan te geven wat meer en wat minder werk is, af en toe op F5 drukken om te zien hoe de taart verdeeld is.

Indien tevreden, kun je het hieruit overkopieeren. Mij heeft het al meerdere malen geholpen

Download

Executable hier te downloaden.

Git, github, Mercurial, bitbucket

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 16 Apr 2011 00:00:00 +0000

April 16 2011

Git, github, Mercurial, bitbucket

I like both equally but am a little more familiar with git. Although now I prefer bitbucket over github due to private repositories being free . Actually I think currently mercurial tool support on windows seems better too (I didn't expect TortoiseHG to be so neat, as I had only used Tortoise CVS in the past, and I didn't like it, but thats probably due to the nature of CVS).

Some notes, small annoyances I encountered on my system and how I fixed them.

Git usage on Windows Vista notes
Git removing contents permanently from repository
Converting from Git to Mercurial
Mercurial through proxy

SuperMouser - mouseless navigation

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 08 Apr 2011 00:00:00 +0000

April 8 2011

SuperMouser - mouseless navigation

I'm actually using Adam Pash' mouser for a few years now on vimkeys with a small patch to improve the performance. A few days ago me and Marijn Koesen talked about an OSX / platform independent version. We used to talk about this before, but now being familiar with wxWidgets I realised creating a clone in C++ was probably easy.

My first windows-only version used a simple timer and a few tricks to prevent keypresses from escaping to the active window. The current version is now improved, and no longer uses a timer. I was also able to minimize an artifact the original mouser also had with the flickering of the blue shapes; it is really fast and transparency appears not to slow it down either (it did on my PC with the original mouser).

~~The only thing missing is multiple-monitor support at this moment, but give us a few days. Having only one monitor isn't helping .~~

EDIT 11-04-2010: A nice SuperMouser milestone day: Marijn added undo key 'U'. I added multiple monitor support for windows, 'M' key. The latter in need of some testing (I tested it by hardcode-dividing my screen into two halfs).

Marijn Koesen is collaborating and also adding OSX support. The GTK2 version is coming soon. ~~And multiple-monitor support currently has highest priority.~~

It's all on github for download. The binary for windows is on github as well.

Roadmap:

80% Multiple monitor support (working on it)
100% ~~Undo history key (for correcting mistakes)~~
0% Remember settings in config.
80% OSX compatibility
60% GTK2 compatibility

Collection of wallpapers

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 27 Mar 2011 00:00:00 +0000

March 27 2011

Collection of wallpapers

BASE64 COMMANDLINE

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 02 Mar 2011 00:00:00 +0000

March 2 2011

BASE64 COMMANDLINE

Probably lots of them on the internet, but I made one myself (don't ask me why) Just searched for some encode and decode functions w/google codesearch, then added a simple main().

Usage: base64conv.exe [OPTION] [SOURCE] [DEST]
 where [OPTION] is either --to-base64 or --from-base64,
  [SOURCE] is the file to convert
  [DEST] is the target/output file (will be overwritten)

Examples: base64conv.exe --to-base64 somefile.exe somefile.txt
          base64conv.exe --from-base64 somefile.txt somefile.exe
(something like ./base64conv on linux)

(more-begin)

Here you have it:

/*
   base64.cpp and base64.h

   Copyright (C) 2004-2008 Rene Nyffenegger

   This source code is provided 'as-is', without any express or implied
   warranty. In no event will the author be held liable for any damages
   arising from the use of this software.

   Permission is granted to anyone to use this software for any purpose,
   including commercial applications, and to alter it and redistribute it
   freely, subject to the following restrictions:

   1. The origin of this source code must not be misrepresented; you must not
      claim that you wrote the original source code. If you use this source code
      in a product, an acknowledgment in the product documentation would be
      appreciated but is not required.

   2. Altered source versions must be plainly marked as such, and must not be
      misrepresented as being the original source code.

   3. This notice may not be removed or altered from any source distribution.

   Rene Nyffenegger rene.nyffenegger@adp-gmbh.ch

*/

#include <iostream>

static const char base64_chars[] =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    "abcdefghijklmnopqrstuvwxyz"
    "0123456789+/";

static char *
    base64_encode(const unsigned char *input, int length)
{
    /* http://www.adp-gmbh.ch/cpp/common/base64.html */
    int i=0, j=0, s=0;
    unsigned char char_array_3[3], char_array_4[4];

    int b64len = (length+2 - ((length+2)%3))*4/3;
    char *b64str = new char[b64len + 1];

    while (length--) {
        char_array_3[i++] = *(input++);
        if (i == 3) {
            char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
            char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
            char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
            char_array_4[3] = char_array_3[2] & 0x3f;

            for (i = 0; i < 4; i++)
                b64str[s++] = base64_chars[char_array_4[i]];

            i = 0;
        }
    }
    if (i) {
        for (j = i; j < 3; j++)
            char_array_3[j] = '\0';

        char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
        char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
        char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
        char_array_4[3] = char_array_3[2] & 0x3f;

        for (j = 0; j < i + 1; j++)
            b64str[s++] = base64_chars[char_array_4[j]];

        while (i++ < 3)
            b64str[s++] = '=';
    }
    b64str[b64len] = '\0';

    return b64str;
}

static inline bool is_base64(unsigned char c) {
    return (isalnum(c) || (c == '+') || (c == '/'));
}

static unsigned char *
    base64_decode(const char *input, int length, int *outlen)
{
    int i = 0;
    int j = 0;
    int r = 0;
    int idx = 0;
    unsigned char char_array_4[4], char_array_3[3];
    unsigned char *output = new unsigned char[length*3/4];

    while (length-- && input[idx] != '=') {
        //skip invalid or padding based chars
        if (!is_base64(input[idx])) {
            idx++;
            continue;
        }
        char_array_4[i++] = input[idx++];
        if (i == 4) {
            for (i = 0; i < 4; i++)
                char_array_4[i] = strchr(base64_chars, char_array_4[i]) - base64_chars;

            char_array_3[0] = (char_array_4[0] << 2) + ((char_array_4[1] & 0x30) >> 4);
            char_array_3[1] = ((char_array_4[1] & 0xf) << 4) + ((char_array_4[2] & 0x3c) >> 2);
            char_array_3[2] = ((char_array_4[2] & 0x3) << 6) + char_array_4[3];

            for (i = 0; (i < 3); i++)
                output[r++] = char_array_3[i];
            i = 0;
        }
    }

    if (i) {
        for (j = i; j <4; j++)
            char_array_4[j] = 0;

        for (j = 0; j <4; j++)
            char_array_4[j] = strchr(base64_chars, char_array_4[j]) - base64_chars;

        char_array_3[0] = (char_array_4[0] << 2) + ((char_array_4[1] & 0x30) >> 4);
        char_array_3[1] = ((char_array_4[1] & 0xf) << 4) + ((char_array_4[2] & 0x3c) >> 2);
        char_array_3[2] = ((char_array_4[2] & 0x3) << 6) + char_array_4[3];

        for (j = 0; (j < i - 1); j++)
            output[r++] = char_array_3[j];
    }

    *outlen = r;

    return output;
}


/**
 * This is the interface I added to these two convert functions, 
 *  that were originally written by Rene Nyffenegger.
 * Found them via google codesearch, specifically in
 *  http://jacksms.googlecode.com/svn/base64.cpp and
 *  http://jacksms.googlecode.com/svn/base64.hh
 * 2011, Ray Burgemeestre.
 */
#include <string>
#include <fstream>

using namespace std;

void display_usage(const char *programname)
{
    cerr << "Usage: " << programname << " [OPTION] [SOURCE] [DEST]" << endl
         << " where [OPTION] is either --to-base64 or --from-base64," << endl
         << "  [SOURCE] is the file to convert" << endl
         << "  [DEST] is the target/output file (will be overwritten)" << endl
         << "" << endl
         << "Examples: base64conv.exe --to-base64 somefile.exe somefile.txt" << endl
         << "          base64conv.exe --from-base64 somefile.txt somefile.exe" << endl
         << "(something like ./base64conv on linux)" << endl;
}

int main(int argc, char *argv[]) 
{
    enum Arguments { PROGRAM, OPTION, SOURCE, DEST, ARGS_LEN };
    if (argc != ARGS_LEN) {
        display_usage(argv[PROGRAM]);
        return -1;
    } else {
        char *sourcedata = NULL;
        ifstream::pos_type sourcedataLen = 0;
        ifstream sourcefile (argv[SOURCE], ios::in|ios::binary|ios::ate);
        if (sourcefile.is_open()) {
            sourcedataLen = sourcefile.tellg();
            sourcedata = new char [static_cast<int>(sourcedataLen)];
            sourcefile.seekg(0, ios::beg);
            sourcefile.read(sourcedata, sourcedataLen);
            sourcefile.close();
            cout << "source file " << argv[SOURCE] << " loaded" << endl;
        } else {
            cerr << "unable to open source file, " << argv[SOURCE] << endl;
            return -1;
        }

        if (strcmp(argv[OPTION], "--to-base64") == 0) {
            cout << "converting to base64" << endl;
            char *base64string = base64_encode(reinterpret_cast<unsigned char *>(sourcedata), 
                static_cast<int>(sourcedataLen));
            ofstream outputfile(argv[DEST], ios::out|ios::binary|ios::ate);
            if (outputfile.is_open()) {
                outputfile.write (base64string, strlen(base64string));
                outputfile.close();
                cout << "base64 string written to dest file, " << argv[DEST] << endl;
            } else {
                cerr << "unable to open dest file, ", argv[DEST];
                return -1;
            }
        }
        else if (strcmp(argv[OPTION], "--from-base64") == 0) {
            cout << "converting from base64" << endl;
            int outlen = 0;
            unsigned char *decodedbase64string= base64_decode(sourcedata, 
                static_cast<int>(sourcedataLen), &outlen);
            ofstream outputfile(argv[DEST], ios::out|ios::binary|ios::ate);
            if (outputfile.is_open()) {
                outputfile.write(reinterpret_cast<const char *>(decodedbase64string), outlen);
                outputfile.close();
                cout << "decoded base64 string written to dest file, " << argv[DEST] << endl;
            } else {
                cerr << "unable to open dest file, ", argv[DEST];
                return -1;
            }
        } else {
            display_usage(argv[PROGRAM]);
            cerr << "@@ invalid [OPTION] specified.";
            return -1;
        }
        delete[] sourcedata;
    }
}

(more-end)

Singleton notepad.exe

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 27 Feb 2011 00:00:00 +0000

February 27 2011

Singleton notepad.exe

If you need to protect yourself from yourself, like I needed to do. Then this simple notepad replacement executable may be for you . Click here to download. What it does is:

Always instantiates exactly one blank/new/untitled notepad instance at a time.
The untitled stuff is kept in a user configurable file. So you are not constantly bothered with a Save as dialog..
Easily 'dispatch' selections within your notepad to separate files.
When called to open a file, does open a new editor for that file. That editor is configurable.
Supports better CTRL+Z, CTRL+Y than default notepad.
Supports CTRL+F and search next with F3.
(For goto-line with CTRL+G you'll have to dispatch the text to another editor!)

When you own New text document (1).txt until New text document (19).txt, asdf.txt, jlksdjflkajsd.txt, etc., on your desktop. Then you have the same problem I had: using notepads for storing temporary buffers and things like that. With this replacement you are forced to keep just one buffer, which is automatically saved with CTRL+S. And when you do want to keep specific stuff, you select that text and CTRL+D (dispatch to file).

(more-begin)

To install it you need to do the following:

Disable UAC (on Vista for example)
Reboot the system
Execute a batch file like the following. Got this copy from here.

Here's the copy (it is included in my .zip). This is not the same script I used, but my guess is it'll probably work:

@echo off

takeown /f c:\windows\syswow64\notepad.exe
cacls c:\windows\syswow64\notepad.exe /G Administrators:F

takeown /f c:\windows\system32\notepad.exe
cacls c:\windows\system32\notepad.exe /G Administrators:F

takeown /f c:\windows\notepad.exe
cacls c:\windows\notepad.exe /G Administrators:F

copy c:\windows\syswow64\notepad.exe c:\windows\syswow64\notepad.exe.backup
copy c:\windows\system32\notepad.exe c:\windows\system32\notepad.exe.backup
copy c:\windows\notepad.exe c:\windows\notepad.exe.backup

rem this has to be executed from within the installation dir.
copy notepad.exe c:\windows\syswow64\notepad.exe
copy notepad.exe c:\windows\system32\notepad.exe
copy notepad.exe c:\windows\notepad.exe

You can read the INSTALL file in the .zip bundle for details.

Example configuration

; configuration for notepad.exe replacement
editor=C:\Program Files\Vim\vim72\gvim.exe
file=C:\temp.txt
title=Untitled
maximized=0

(more-end)

Starcry rendered frame images

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 24 Feb 2011 00:00:00 +0000

February 24 2011

Starcry rendered frame images

These are some examples of rendered frames by the starcry rendering engine.

(more-begin)

(more-end)

GNU Screen Navigator

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sat, 15 Jan 2011 00:00:00 +0000

January 15 2011

GNU Screen Navigator

When navigating directories, every now and then pushd . and popd just aren't sufficient. When directory structures tend to become quite deep, I get really annoyed with typing change-directory’s and wish I had some bookmarks. That's why some time while ago I created a simple tool for usage within (GNU) screen: CTRL+X pops up a menu for selecting common directories and files ("bookmarks"!). This makes switching between those long directory structures a lot more pleasant !

Create bookmarks file

// Configuration file ($HOME/.launcher)
[
   {"Websites, configuration" : [
      {"dir" : "/srv/www/vhosts/ray.burgemeestre.net"},
      {"dir" : "/srv/www/vhosts/www.burgemeestre.net"},
      {"dir" : "/usr/local/src/mongoose/am"},
      {"dir" : "/usr/local/src/mongoose/am/output/ray-blog-burgemeestre-net"},
      {"file" : "/etc/apache2/vhosts.d/www-burgemeestre-net.conf"},
      {"file" : "/etc/apache2/vhosts.d/ray-blog-burgemeestre-net.conf"},
      {"file" : "/etc/apache2/vhosts.d/ray-burgemeestre-net.conf"}
   ]},
   {"Common Lisp" : [
      {"dir" : "/var/chroot/lispbot/home/trigen"},
      {"file" : "/var/chroot/lispbot/home/trigen/bot.lisp"}
   ]},
   {"C++ projects" : [
      {"dir" : "/usr/local/src/launcher"},
      {"dir" : "/usr/local/src/launcher/src"},
      {"file" : "/usr/local/src/launcher/src/parser.cpp"}
   ]}
]

Use in screen with C-x

Choosing a directory in the menu opens it in a new screen tab. Choosing a file opens a new menu which allows you to dispatch it to specific editors or e-mail it (options currently all hardcoded in the source).

Installation

Most of the code is still from 2007, but these last two days I refactored the code a bit. And added some build scripts. (Needless to say I didn't get to implementing all features I had in mind.) It's all on github for download.

It installs /usr/local/bin/launcher and two helper bash scripts (/usr/local/bin/launch_scr.sh and /usr/local/bin/launch_scr2.sh). You bind it to screen in your .screenrc with this line: bindkey ^x exec /usr/local/bin/launch_scr.sh.

(Also for your .screenrc I recommend using a caption if you don't already with: caption always "%{Yb} %D %Y-%02m-%02d %0c %{k}|%{G} %l %{k}|%{W} %-w%{+b}%n %t%{-b}%+w")

PHP debugging in practice!

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 14 Jan 2011 00:00:00 +0000

January 14 2011

PHP debugging in practice!

Just wanted to use the 'in practice' part because it sounds so cool . At work I used to have some cool tricks I'd use at customers (test-environments) or local setups to debug. One of my favourite snippets I use all the time is the following.

Record and re-create requests..

/**
 * Debug helper function for logging and restoring requests. Do not use in 
 * production environments.
 *
 * - Logs the URL and request data.
 * - With __STATE_DEBUGGER__ parameter in the URL it displays a listing
 *   of logged requests.
 * - Clicking an item in that listing, re-creates that request.
 *
 * I know it is usually not good practice to have *one* function do multiple 
 * things. Make sure you know what you're doing ;)
 * 
 * @param writableDirectory directory for saving the requests in files
 * @return void (call it solely for it's side-effects ;-))
 */
function statedebugger($writableDirectory = '/tmp')
{
    if (!is_dir($writableDirectory) || !is_writable($writableDirectory)) {
        trigger_error('parameter writableDirectory needs to exist and be
            writable', E_USER_ERROR);
    }
    $indexFile = $writableDirectory . '/index.txt';
    if (!isset($_GET['__STATE_DEBUGGER__'])) {
        // Write state file for request
        $stateFileName = tempnam($writableDirectory, 'p');
        $fd = fopen($stateFileName, 'wb');
        fwrite($fd, serialize(array($_SERVER, $_GET, $_POST, $_REQUEST,
            !isset($_SESSION) ? array() : $_SESSION)));
        fclose($fd);
        // Rewrite index
        $indexFileContents = file_exists($indexFile) ?
            file_get_contents($indexFile) : '';
        $fd = fopen($indexFile, 'wb');
        fwrite($fd, implode(array($indexFileContents, //INEFFICIENT
            $stateFileName, ' ', $_SERVER['REQUEST_URI'], "\n")));
        fclose($fd);
    } else {
        if (!isset($_GET['set_state'])) {
            // Show index/listing of states
            $indexFileLines = array_reverse(explode("\n",
                file_get_contents($indexFile)));
            foreach ($indexFileLines as $line)  {
                if (empty($line))
                    continue;
                list($filename, $requestUri) = explode(" ", $line);
                printf("<a href=\"%s%s__STATE_DEBUGGER__&set_state=%s\">%s</a><br/>\n",
                    $requestUri, (strpos($requestUri, "?") === FALSE ? "?" :
                                    "&"), $filename, $requestUri);
            }
            exit(0);
        } else {
            // Restore a specific state
            list ($_SERVER, $_GET, $_POST, $_REQUEST, $_SESSION) =
                unserialize(file_get_contents($_GET['set_state'])); //DANGEROUS
        }
    }
}
statedebugger('E:/TPSC/htdocs/CRMS-Test/webframe/templates_c/');

Just paste that in the config, or whatever global headerfile (your software probably has), and it logs all requests to some specified directory. Usually when something goes wrong within a very specific context (a specific user with rights, session variables, page, ..), it would be nice to just log the requests (e.g. with post data), and use that information to re-create the (failing) request. So you can keep pressing F5 in your browser window while fixing the bug. This function does that. Also this can be especially useful when debugging AJAX requests, i.e. without firebug, or you can let the user create the very specific context (with privileges and user settings) that causes some failure, and in that way create the test-case.

(Add __STATE_DEBUGGER__ to the URL as a parameter, and it will show a listing with clickable requests. Clicking redirects to that page and initializes it with the recorded $_GET, $_POST, $_SESSION, ...)

(By the way I didn't bother to make this very secure (see file_get_contents($_GET['set_state']) for example), because this should only be used when debugging. This could easily be improved with some numerical parameters, optional descriptions, limit to fixed number of requests...)

Photoshop blenders for allegro

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 26 Nov 2010 00:00:00 +0000

November 26 2010

Photoshop blenders for allegro

One of the reasons I was using allegro for my project was that they already had (apparantly) a lot of blending modes available. You have set_blender_alpha(), set_blender_hue(), set_blender_color(), but these are not the same as in photoshop. I got some unexpected behaviour from these, and I will edit this post later and elaborate on that some more.

Most of them ignore the r, g, b parameters as well. So it appears they were a big disappointment. By the way, a quick glance at the Allegro 5 source made me think they aren't implemented there either.

That's why I did a quick search and found someone who already collected macro's for all photoshop blending types. By Nathan Moinvaziri. Today, or I should say yesterday (now at 3:17am), I implemented some wrapper functions around it that make it available to allegro, with an API that looks and feels the same as allegro.

I put it on github with a demo program. From the README:

Put al_blend.h in your project somewhere and include it.  Or put it in the
allegro include directory, i.e. /usr/local/src/allegro/include or
C:\allegro\include.

Now instead of using the allegro blenders with getpixel() and putpixel(), you
can use put_blended_pixel().

    put_blended_pixel(): 
        Writes a pixel into a bitmap with specific blender.

    Description:
        void put_blended_pixel(BITMAP *bmp, int x, int y, int color,
            (*blender_func)(int &basecolor, int &blendcolor)); 

    Example:
        put_blended_pixel(screen, x, y, somecolor, blender_lighten);

Available blenders are: blender_darken, blender_multiply, blender_average, blender_add, blender_subtract, blender_difference, blender_negation, blender_screen, blender_exclusion, blender_overlay, blender_softlight, blender_hardlight, blender_colordodge, blender_colorburn, blender_lineardodge, blender_linearburn, blender_linearlight, blender_vividlight, blender_pinlight, blender_hardmix, blender_reflect, blender_glow, blender_phoenix, blender_hue, blender_saturation, blender_color, blender_luminosity,

wxWidgets code snippets

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 26 Nov 2010 00:00:00 +0000

November 26 2010

wxWidgets code snippets

There are a few things I figured out how to do, and I seem to use them on a regular basis. For these snippets I have to search through my projects every now and then, to find them, and it would be a lot easier if I kept them somewhere listed. Hopefully you will find something useful as well.

Clearing a sizer

I usually make master/detail screens where the detail list is a vertical sizer with stuff on it. Usually a panel with more stuff on it. Every now and then you may want to clear it. I use:

while (theSizer->GetChildren().GetCount() > 0) {
    theSizer->GetItem(static_cast<size_t>(0))->DeleteWindows();
    theSizer->Remove(0);
}

// populate the sizer again maybe

sizerChangelogItems->Layout();

You might want to surround the code with Freeze() and Thaw() for performance.

Hiding and showing a panel on an AUI frame

This is more of a note I keep forgetting about: Instead of calling the Hide() and Show() on the window (panel) itself (which compiles). Use the following:

someWindow->GetAuiManager().GetPane(someWindow->splitterwindow1).Show(false);
someWindow->GetAuiManager().GetPane(someWindow->panel1).Show(true);

someWindow->GetAuiManager().Update();

Using dropdowns with object data.

I use wxChoice a lot, I especially like that the items themselves can refer to anything. Typically how I initialize a choicebox:

wxChoice *choice = ...;
Object *someObject = ...;
for (...) {
    choice->Append("object 1", (void *)someObject);
}
choice->SetStringSelection("object 1");

Then usually I have an event that triggers on the selection of an item in the choicebox. In the event I first get the wxChoice, then cast the client data for the selected item to the object pointed to:

void SomeWindow::OnChoiceSelected( wxCommandEvent& event )
{
    wxChoice *choice = static_cast<wxChoice *>(event.GetEventObject());
    Object *someObject = static_cast<Object *>(choice->GetClientData(choice->GetSelection()));
    if (someObject != NULL) {
        //...
    }
}

Directory functions

wxString resdir(RESOURCE_DIR);
if (!::wxDirExists(resdir)) {
    throw std::runtime_error("resources dir not found");      
}
wxDir dir(resdir);
if ( !dir.IsOpened() ) {
    throw std::runtime_error("resources directory could not be opened");
}
wxString filename;
bool cont = dir.GetFirst(&filename);
while ( cont ) {
    int lastdot = filename.Find('.', true);
    if (lastdot != wxNOT_FOUND) {
        wxString name(filename.SubString(0, lastdot - 1));
        wxString ext(filename.SubString(lastdot + 1, filename.Length()));
        if (!ext.compare(_T("cpp"))) {
            ...
        } else {
            ...
        }
    }
    cont = dir.GetNext(&filename);
}

Write to file function

wxFile article(markdownfilename, wxFile::write);
if (!article.IsOpened()) {
    throw std::runtime_error("main.md could not be opened for writing");
}
article.Write(textctrlArticleBody->GetValue());
article.Close();

Launch of yet another blog ii

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Fri, 19 Nov 2010 00:00:00 +0000

November 19 2010

Launch of yet another blog ii

Today I moved from my "127 machine" (localhost) to an Amazon EC2 server.

First I tried the Amazon Linux AMI (32-bit) micro, but it was to difficult to get GTK working. Package manager was yum, which is nice, but a few packages I needed weren't available, decided to check another AMI out.

Found SUSE AMI (32-bit) on micro, package manager is zypper, and works like a charm. GTK was already installed on this one. Everything is up and running.

I've used Debian for a lot of years for personal use and at work, and henceforth became accustomed to Debian. I'm actually finding out SUSE isn't that bad either!

On my local machine GTK was ugly by default, so I immediately changed the theme to something less hideous. On this SUSE AMI the default GTK is pretty fancy, although the fonts were missing ;)

What annoys me is that forwarding X11 over the internet is slower than I expected. Editting text inside the forwarded "ArticleManager" isn't particularly fast. Still love my weird blogging system though!

Common Lisp wallpaper

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 17 Nov 2010 00:00:00 +0000

November 17 2010

Common Lisp wallpaper

The original wallpaper by ~Pocket7878 on devianART was designed for a 1024 x 768 pixels resolution. I have 1280 x 1024. And although even that is probably considered small, created a 1280 x 1024 version. Just by creating a new canvas with the surrounding color and centered his wallpaper on top of it (so not by upscaling). The mascotte is of course by Conrad Barski (author of Land of lisp)

Check it!

Improving the outline for the Adornment of the Middle Way

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 17 Nov 2010 00:00:00 +0000

November 17 2010

Improving the outline for the Adornment of the Middle Way

This is how the text is structured (excerpt)

The Commentary 147
1. The exposition of the root verses 147
2. The meaning of the title 147
2. The homage of the translator 150
2. The text itself 151
3. An examination and establishment of the two truths 151
4. The two truths identified 151
5. A demonstration that no entities exist on the ultimate level 151
6. The main argument of the Madhyamakalankara (stanza I) 151
7. An investigation of the subject of the probative argument 152
7. An investigation of the argument 153
8. A Prasangika or a Svatantrika argument? 153

You may think this isn't so bad, but this goes on for a few levels more, especially in the first part of the book. Sometimes this reaches a depth of 21 levels. That would mean that using more regular numbering you would get sections like 1.2.1.2.2.2.1.1.2, and worse.

I wrote a little Lisp program to parse this outline lines-of-text. And output it with proper indenting. Output for (print-tree 4)

The Commentary 147
   1 The exposition of the root verses 147
      2 The meaning of the title 147
      2 The homage of the translator 150
      2 The text itself 151
         3 An examination and establishment of the two truths 151
            4 The two truths identified 151
            4 Answers to objections made to this distinction of the two truths 293
            4 The benefits of understanding the two truths correctly 337
         3 The conclusion: a eulogy of this approach to the two truths 361
            4 An outline of the tradition in which the Chittamatra and Madhyamaka (..)
            4 In praise of this path 363
      2 The conclusion 377
         3 The author's colophon 377
         3 The colophon of the translators 377
         3 Colophon of Mipham Rinpoche 381
   1 The necessity for the explanation of the root verses 378

Will update this post as soon as I've made an interactive version of this tree, which I consider of great value maintaining an overview text's structure.

I am considering testing either Wt, the C++ Web toolkit or CppCMS, a C++ Web Development Framework in the process. Then i'll probably make some small C++ program for this, and use it as cgi in Mongoose (g++ outline.cpp -o outline.cgi!).

Using allegro with wxWidgets

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Wed, 10 Nov 2010 00:00:00 +0000

November 10 2010

Using allegro with wxWidgets

My two favourite libraries ever

Allegro is a game programming library for C/C++ developers distributed freely, supporting the following platforms: Unix (Linux, FreeBSD, etc.), Windows, MacOS X and Haiku/BeOS. (...) source

wxWidgets is a C++ library that lets developers create applications for Windows, OS X, Linux and UNIX on 32-bit and 64-bit architectures as well as several mobile platforms including Windows Mobile, iPhone SDK and embedded GTK+. (...) source

The required code summarized

I include allegro before wxWidgets, not sure if that matters, but what does matter is: the workaround regarding the RGB define, the use of winalleg.h instead of windows.h (needed to avoid conflict between allegro and windows.h); and disabling the 'magic' main() in allegro (wxWidgets' main is used):

#define ALLEGRO_NO_MAGIC_MAIN
#define RGB AL_RGB
#include <allegro.h>
#include <winalleg.h>
#undef RGB

Then I initialize allegro like this in OnCreate() of your wxWidgets application:

install_allegro(SYSTEM_NONE, &errno, NULL);
set_palette(desktop_palette); // example
set_color_depth(32); // example

You could draw allegro BITMAP's to wxStaticBitmap's in the OnPaint() event of the wxWidgets window. But you'll have to decide for yourself what method is most appropriate for your project.

wxPaintDC dc(wxDynamicCast(event.GetEventObject(), wxWindow)); 
    // you probably already have the wxPaintDC

// Found the following code by searching for a very long time in the 
//  wxWidgets source code ;)
WXHDC wxHDC = wxPaintDC::FindDCInCache((wxWindow*) event.GetEventObject());
HDC hDC = (HDC) wxHDC;

... // do your thing here

draw_to_hdc(hDC, allegroBitmap, 0, 0); 
    // where allegroBitmap is a valid "BITMAP *"

It took me a while to figure out how I could use draw_to_hdc like this. It's not that obvious (see allegro forums). Others have been struggling with drawing allego bitmaps on wxStaticBitmaps. This method will still yield you quite a high fps count.

Known issues

wxWidgets eventloop--or apparently random--crashes I have experienced some weird behaviour with using allegro with wxWidgets. There have been cases, that although allegro was initialized, in a separate window allegro seems to fail after calls to save_bmp(). Not in the function itself, but even weirder, after having called save_bmp() more than once, eventually the wxWidgets event handling system crashed, complaining about unhandled events. I guess something really wasn't properly initialized (although everything seemed to work, drawing, saving, etc.)

If you experience this, just initialize allegro in that window's OnCreate(), it should fix the problem. Maybe it has to do something with threading, that would explain why allegro has to be initialized (in that thread).

Assembly optimized {get|put}pixel crashes With _getpixel32() and _putpixel32() assembly, the optimized versions of getpixel() and putpixel(). For me these didn't work properly on 64-bit windows environments. After turning off some compiler optimizations the problem seemed to go away.

Non-assembly optimized {get|put}pixel crashes Continuing on the previous, if you replace _getpixel32 with getpixel, you may need to explicitely set a blending mode for 64-bit environments. Somehow I didn't do that, and as the non-assembly optimized getpixel function does bounds checking and blending modes, this became important.

[Edit 22-nov-2010: Just remembered this, but didn't verify it. It may have something to do with calling drawing_mode(..), and not setting a specific blending mode like set_trans_blender(..). Setting one of those, or instead of drawing_mode and set_trans_blender a call to solid_mode(), would fix it.]

[Edit 29-01-2012: By the way. Allegro 5 integrates a little easier with wxWidgets, although come to think of it, I had to grep the source, or more specifically the examples directory in order to get the functions needed to render on a wxStaticBitmap though. I'll create a post for it: TODO]

Launch of yet another blog

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 08 Nov 2010 00:00:00 +0000

November 8 2010

Launch of yet another blog

What's new about this blog??

Hopefully some ideas on this blog will be new, or just fun. It's also for myself to keep a track of certain stuff. About the blog itself, it doesn't use typical blog or CMS software. It uses C++, and has interfaces to other tools. I created this system in a few hours this weekend, it's quite minimal.

How does it work? I have to run Xming on my windows box, and request from the administration panel of the blog a management console. This opens a C++ program developed using DialogBlocks (my best software-buy ever!!) / wxWidgets. In this application I can add sites, and choose what categories should be dispatched to it. This is the weirdest part I guess, no regular login + management through a webinterface.

Why do I do it like this? First of all I don't like to write HTML. That's why I can define a (simple) site template with a HAML and SASS, and add some markers in it for replacement.

Demo snippet from the screenshot:

(defun factorial (n)
  (if (<= n 1)
    1
    (* n (factorial (- n 1)))))

The editor simply has a listing of articles, which are stored in multimarkdown syntax. (note: this format is actually easily converted to LaTeX pdf documents as well!(works like a charm)). I have made some facilities to make it easy to add i.e. C++ or other code-snippets (they can be editted in separate files). Using this code prettifyer by Mike Samuel. They will be represented by a string like (lisp-code filename), and if I use that in the markdown document it will place syntax highlighted code with the snippet there. I made something similar for images and some meta-data with regards to the articles is stored using TinyXML.

In the editor I can request gvim or xemacs to edit the markdown (or a snippet), or use the one build-in.

Lastly, I can (re)generate (parts of- or the entire-)website. It will convert HAML files to HTML. Markdown to HTML. Merge the snippets, merge articles with main html. And the website is updated.

Update 19-7-2012

Now it processes all JPG's through imageconvert to compress them better, something similar for PNG's
Minifies the CSS and Javascript (also combines them where possible)

Motion blur

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Mon, 08 Nov 2010 00:00:00 +0000

November 8 2010

Motion blur

The engine supports correct rendering of motionblur for objects. (view more about it at TGLTLSBFSSP: Motion Blur) The implementation is now pretty efficient, and works well for all kinds of objects with all kinds of textures, etc.. It isn't fast enough for realtime rendering, you'd probably need to use hardware acceleration for that, or false motionblur.

The settings used for the object in the movie:

Functional programming

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Sun, 07 Nov 2010 00:00:00 +0000

November 7 2010

Functional programming

Quote from Wikipedia:

In computer science, functional programming is a programming paradigm that treats computation as the evaluation of mathematical functions and avoids state and mutable data. It emphasizes the application of functions, in contrast to the imperative programming style, which emphasizes changes in state.[1]

In practice, the difference between a mathematical function and the notion of a "function" used in imperative programming is that imperative functions can have side effects, changing the value of already calculated computations. (...) Conversely, in functional code, the output value of a function depends only on the arguments that are input to the function, so calling a function f twice with the same value for an argument x will produce the same result f(x) both times.

Lisp e-books

Practical Common Lisp by Peter Seibel.
Common Lisp: A Gentle Introduction to Symbolic Computation by David S. Touretzky.
Succesful Lisp by David B. Lamkins..

I've started with gentle --actually a really old book--and have planned to read pcl second. [Edit 23-11-2010, now actually bought a used copy as well!]

Enable wake-on-lan on Linux Debian (4.0)

rayburgemeestre@no-spam.tld (Ray Burgemeestre) — Thu, 18 Dec 2008 00:00:00 +0000

December 18 2008

Enable wake-on-lan on Linux Debian (4.0)

Enable wake-on-lan on Linux Debian

Source = http://www.oger-partners.be/?q=node/60 I have an asus motherboard with an integrated NIC. Integrated NICs on recent motherboards have WOL capability. I hope yours has too, otherwise you may have to obtain some kind of wire to connect the NIC to the motherboard.

Read up on WOL on wikipedia.
Download ethtool (f.e. apt-get install ethtool)
The command 'ethtool eth0' should yield the output 'Supports Wake-on: g'. 'g' means the NIC can read the magic packet.
Make a shell script to enable WOL with ethtool, put this command in it: /usr/sbin/ethtool -s eth0 wol g
Change your /etc/network/interfaces so this script is executed each time the interface is brought up.
The tricky part was finding out why the settings set with ethtool were thrown away with each reboot. The answer lies in the /etc/init.d/halt script. Remove the '-i' from the halt command at the end of the script so the interfaces are not affected.

Good luck. :-)

Tool used in windows

Leeched from= http://www.depicus.com/download.aspx?product=gui

View attachments for word document I made with instructions for starting up ILLYRIA

Neither one nor Many

EBPF Flamegraphs C++ Ubuntu 20.04

EBPF Flamegraphs C++ Ubuntu 20.04

Flame Graphs

1) Install prerequisites

2) Install bcc tools (bpfcc-tools)

3) Do the profiling with EBPF

4) Generate the Flame Graph

Example

Settings for Tide graph Casio G-Shock GLX 150

Settings for Tide graph Casio G-Shock GLX 150

lunitidal interval

testing

Deploying owncloud via Helm on Kubernetes

Deploying owncloud via Helm on Kubernetes

OwnCloud

Deploy OwnCloud with Helm

Owncloud will require some storage.

Setup ingress (beware of caveat!)

DONE!

Deploying Kube on a single node machine notes

Unrelated notes about Owncloud itself

How to print an endlessly folding card

How to print an endlessly folding card

Figuring out how it works

Design and print the card!

Input & Output

How to fold?

Conclusion

i3 floating point window resize percentage wise and centered

i3 floating point window resize percentage wise and centered

Step 1: download, make, and make install wmutils.

Step 2: put this python script somewhere

Step 3: modify resize mode in i3

My visit to Meeting C++ 2016!

My visit to Meeting C++ 2016!

Talks I attended... DAY 1

DAY 2

The conference

Kerberizing Cloudera Manager

Kerberizing Cloudera Manager

Step 1: install Cloudera Manager

Step 2: install kerberos packages on the headnode

Step 3: modify /etc/krb5.conf

Step 4: modify /var/kerberos/krb5kdc/kdc.conf

Step 5: modify /var/kerberos/krb5kdc/kadm5.acl

Step 6: initialize the database

Step 7: add master root/admin user

Step 8: start services

Step 9: do the Enable security wizard in Cloudera Manager

How to screen capture in Windows 10 with HiDPI support

How to screen capture in Windows 10 with HiDPI support

HiDPI: Fix mouse pointer

Optional: Use H264 codec

Optional: Include sound in the video

The end result

Update 1> Add more keyframes for better editing

Update 2> Editing 4K on Windows 10...

Update 3> Support Adobe Premiere

Profiling and visualizing with GNU strace

Profiling and visualizing with GNU strace

Why are we doing this? (Other than because it's Awesome!)

First, let's take one step back & properly parse the strace output..

The grammer

Visitor example

A Proof of concept for Web

Final step, To ElasticSearch!

Remote debugging with NetBeans 8.1 for C/C++

Remote debugging with NetBeans 8.1 for C/C++

Example workflow

Wizard example to debug cmd unit test

You can use these settings to double check if everything is correct

How to "inspect element" XUL applications

How to "inspect element" XUL applications

How to debug XUL applications

How to debug XUL applications

First get a slightly older xulrunner

Enable a debug listener in the code

Start the application with this xulrunner

Attach debugger from Mozilla Firefox

Step 7: add master `root/admin` user

Step 9: do the `Enable security` wizard in Cloudera Manager

CLI example usage (`http_proxy=http://localhost:8888 curl http://cppse.nl/test.txt`)