Results tagged “ssh”

I know that title is mouthful. But, I occasionally use this blog as place to dump interesting configuration settings and it helps me remember configuration which helps me to remember it and might be useful to lone surfers who stumbles upon this page.

graph.png

This weekend our nice network admin upgraded VLAN setup to enables 1Gb/s network across whole private segment. That is clearly visible in green arrows on picture, especially if you compare it with last one. We still have a few slower links (100 Mb/s) which pass through NAT to Internet (red arrows), but we now have private segment on every server in library and system room in main building and each server has IP address in new 1 Gb/s segment (you can notice that by blue arrows which are loopback interface on same machine).

All is not well, however. I had to reconfigure my OpenVZ containers from point-to-point ip configuration using venet devices to ethernet bridge setup because multiple hops on OpenVZ machine produced havoc with connections from our private network to containers. It was very wired, I saw TCP retransmission requests, but first packet somehow managed to pass through. ICMP traffic worked (thanks to small packet sizes), but I can't honestly say why our new VLAN setup and/or OpenVZ had problem with it. Up to that point, I just had private IP addresses assigned to OpenVZ container using vzctl set --ipadd 10.60.0.12 and ping -R did show multiple public and private addresses, but it all somehow worked.

First, I had to make network bridge which will be new VLAN 60

root@koha-hw:~# cat /etc/network/interfaces

auto br60
iface br60 inet static
        bridge_ports eth1 veth212226
        bridge_fd 0
        address 10.60.0.10
        netmask 255.255.254.0
        gateway 10.60.0.1
Then, I removed old private IP addresses:
root@koha-hw:~# vzctl set 212226 --ipdel 10.60.0.12 --save
and add new veth device (my version of vzctl requires MAC addresses, so beware!):
root@koha-hw:~# vzctl set 212226 --netif_add veth1,00:12:34:56:78:9A,veth212226,00:12:34:56:78:9B,br60 --save
This create veth1 device inside container to which I can assign IP address:
koha:~# ifconfig veth1 10.60.0.12 netmask 255.255.254.0 up
I decided to put that in /etc/rc.local because /etc/network/interfaces is managed by OpenVZ.

Next, I needed to reconfigure my ssh VPN which was using point-to-point tunneling with tap into real ethernet tunnel:

root@brr:~# tail -10 /etc/network/interfaces 

# vpn lib -> home
auto tap0
iface tap0 inet static
        pre-up ssh -M -S /var/run/tap0 -v -f -i /root/.ssh/koha-hw-tun0 -w 0:0 -o Tunnel=ethernet -o ServerAliveInterval=5 -o ServerAliveCountMax=2 koha-hw.ffzg.hr true
        address 10.60.0.81
        netmask 255.255.0.0
        up iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o tap0 -j MASQUERADE
        post-down ssh -S /var/run/tap0 -O exit koha-hw.ffzg.hr
On remote side, force tunnel number and add new device to bridge:
root@koha-hw:~# head -1 /root/.ssh/authorized_keys 
tunnel="0",command="ifconfig tap0 up ; brctl addif br60 tap0" ssh-rsa AAAAB3Nza...

Finally, little cron job to start (and restart) VPN:

root@brr:~# cat /etc/cron.d/tap0 
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
*/1     * * * * root    fping -q 10.60.0.10 || ( ifdown tap0 ; ifup tap0 )

If you are system administrator this will sound familiar: you have to quickly fix something, and you know that you should document it somewhere (or keep backup) but it's so much work. You could install one of existing source control management tools on each box, but they usually come with huge dependencies, and having all files in central location would be so useful to co-relate configuration changes. To add insult to injury, existing SCMs don't do good job in tracking just few files spread across file-system.

So, what would be perfect tool for keeping remote files in central git repository look like?

  • no dependency on non-standard tools on clients allowing easy deployment
  • track individual files and ignore rest
  • central repository, one directory per hostname

I tried to solve this problem several times, writing wrappers around subversion to handle sparse checkouts and installing subversion and ssh authentication all over the place. But, all this should be simpler... Like this:

  1. add new client to track:
    dpavlin@klin:~/klin/bak-git$ ./bak-git-server.pl ~/backup/ 10.60.0.92 --install brr
    install on brr
    # lot of output stripped
    
    This will do several steps:
    • create git repository in ~/backup/ if it doesn't exist already
    • install root ssh authentication to brr using ssh-copy-id
    • install bak shell helper which uses netcat to connect back to 10.60.0.92
    • install rsync on client and use it as root over ssh to sync files
  2. Now we can login into brr and start tracking our files:
    dpavlin@brr:~$ bak add /etc/cron.d/tun0 
    dpavlin@brr:~$ bak add /etc/network/interfaces
    dpavlin@brr:~$ bak commit
    dpavlin@brr:~$ bak log
    commit df09dc5e19ef1d47311d701b4c63f0859b0b81c1
    Author: Dobrica Pavlinusic 
    Date:   Thu Feb 18 19:04:21 2010 +0100
    
        brr [commit] /home/dpavlin/
    
     create mode 100644 brr/etc/cron.d/tun0
     create mode 100644 brr/etc/network/interfaces
    
  3. change some configuration and review changes
    dpavlin@brr:~$ bak diff
    diff --git a/brr/etc/network/interfaces b/brr/etc/network/interfaces
    index 806c08e..c52c646 100644
    --- a/brr/etc/network/interfaces
    +++ b/brr/etc/network/interfaces
    @@ -2,8 +2,6 @@
     # and how to activate them. For more information, see interfaces(5).
     
     # The loopback network interface
    -auto lo
    -iface lo inet loopback
     
     # The primary network interface
     #allow-hotplug eth0
    
  4. Uups!! Where did loopback disappeared?
    dpavlin@brr:~$ bak revert /etc/network/interfaces 
    dpavlin@brr:~$ bak diff
    
  5. If we are content with changes, we can also commit them:
    dpavlin@brr:~$ bak commit /etc/network/interfaces optional note
    
As you guessed by now, it's very similar to git usage (expect revert which is from subversion) but with easy deployment on clients. It implements reduced subset of git commands:
  • bak add /path
  • bak commit [/path [message]]
  • bak diff
  • bak status
  • bak log
  • bak - push all local changes to server (without commit!)
If you need anything more complex, you can use git directly on ~/backup repository (even to commit changes from multiple hosts in one go).

Whole solution seems like ftp protocol, with data channel using ssh and rsync. File transfer should be encrypted (since we are trying to manage configuration files with sensitive information) and if you want to be really secure, just run server on 127.0.0.1 and tunnel port using RemoteForward 9001 localhost:9001 in .ssh/config.

For quote some time I have been using x2x to have single keyboard and mouse for multiple machines (EeePC or third monitor at work), but I had problem with selection buffers. x2x was written for scenario where all X servers are listening to network connections, so if you had two machines klin and t42 and you wanted to use keyboard and mouse on klin you would have to do something like this:

dpavlin@t42:~$ xhost klin

dpavlin@klin:~$ x2x -to klin:0 -west
However, recent Debian systems are not listening to TCP connections, so you will have to modify xserverrc for this to work:
dpavlin@t42:~$ cat /etc/X11/xinit/xserverrc 
#!/bin/sh

exec /usr/bin/X11/X #-nolisten tcp
However, it's much nicer to depend on ssh X11 forwarding to provide encryption using:
dpavlin@klin:~$ ssh -X t42 x2x -to :0 -west
To finish this post, I have to include small patch which allows to switch copy/paste direction (which I don't need any more, but it proved useful in at least one case):
diff -r 2ce6789a43da x2x.c
--- a/x2x.c     Tue Mar 18 22:47:34 2008 +0600
+++ b/x2x.c     Tue Nov 24 14:08:16 2009 +0100
@@ -1125,7 +1125,7 @@
       if (doSel) {
         // pDpyInfo->initialClipboardSeen = False;
         pDpyInfo->winSelText = NULL;
-        pDpyInfo->hwndNextViewer = SetClipboardViewer(pDpyInfo->edgewindow);
+        pDpyInfo->hwndNextViewer = SetClipboardViewer(pDpyInfo->bigwindow);
       }
 
       pDpyInfo->onedge = 0;

Interesting title, isn't it? It's just what you need when you have classroom full of Windows machines and yet you want to share terminal screen which you are projecting onto every screen for easy copy/paste from your Linux laptop. Can it be done?

So, what do we have?

  • presentation laptop running xterm (with white background) projecting picture to whiteboard
  • Windows PC in front of every student
  • local network
And what do we want?
  • copy/paste from session to own editor on student's computer
  • students session should be view only
At first, VNC came to mind. But, then again I didn't find easy way to relay single session to multiple computers. And I wasn't really sure how well it would work with my copy/paste requirement. And it seemed a bit too complex. And after all, it's not graphic session, but instead just terminal running presentation in vim for syntax highlighting...

Windows

Biggest requirement on Windows in minimal deployment under normal user privileges. Putty seems like perfect fit: single executable, good terminal emulation. No brainier.

Linux

On my laptop, I used screen with session sharing but with read-only twist (giving every student my account on personal laptop just... doesn't seem sane).

First I created new student user which will connect to screen which shares my session. This account has well-known password written on whiteboard together with IP address of my laptop.

student user is starting screen and attaching to my shared perl session using shell script configured in /etc/passwd as default shell.

exec screen -x dpavlin/perl
So, to participate, students just ssh using Putty from Windows machines. So far, so good...

To make session read-only I needed to remove write permission from student user in screen. This proved to be a challenge in this solution, but not something that reading man screen can't solve:

# screenrc
multiuser on
acladd student
aclumask student-w

After starting shared session named perl

screen -S perl -c screenrc
everything is ready. If you can thing of simpler thing to do, I would love to hear it :-)

Unexpected consequence of this setup is my ability to switch to other screen in shared session screen on my laptop which changes projected session on whiteboard (to show table of contents or some random example in middle of presentation), but students will still see perl session with presentation.

Detailed step-by-step instructions how to setup view only screen sharing are available in my Sysadmin Cookbook.

Let's assume that you want to create virtual network which spans sites (or continents :-). While we are at it, let's assume that you want to have layer 2 connectivity (because you want to run just single DHCP server for example).

At first, it seemed logical to use Virtual Distributed Ethernet for which kvm has support. However, this involves running multiple processes to support nodes on network, and it's really virtual -- you can't use familiar Linux tools (like brctl or arp) to configure it. And it's connected over ssh anyway, so why to add unnecessary complexity to setup?

Since we will use ssh to transfer traffic anyway (it easiest hole to drill over firewalls and you probably already have it for administration anyway), why do we need another layer of software in between, with new commands to learn if we already know how to make it using plain old Linux brctl?

So, let's take another look at ssh, especially option Tunnel=ethernet which provides Ethernet bridging between two tap devices. As I wrote before, ssh have point-to-point links using tun device which is great solution if you want to connect two networks on IP level using routing. However, tap devices provide access to Ethernet layer from user-space (so ssh, kvm, VDE and various others user-land programs can send and receive Ethernet packets). However, finding information on internet how to setup ssh to use tap devices is nowhere to be found and motivated me for this blog post.

Let's assume that we have two machines in following configuration:

  • t61p - laptop at home behind DSL link and nat which wants to run kvm virtual machine in virtual network 172.16.10.0/24
  • t42 - desktop machine at work which have network bridge called wire which has 172.16.10.0/24 network which provides network booting services
So, we need ethernet tunneling to remote client.
# install tunctl
dpavlin@t61p:/virtual/kvm$ sudo apt-get install uml-utilities

dpavlin@t61p:/virtual/kvm$ sudo tunctl -u dpavlin -t kvm0
Set 'kvm0' persistent and owned by uid 1000

dpavlin@t61p:/virtual/kvm$ kvm -net nic,macaddr=52:54:00:00:0a:3d -net tap,ifname=kvm0,script=no -boot n
This doesn't really boot our kvm from network because we didn't connect it together. Now we need to enable tunnels on t42 and setup remote tap device
dpavlin@t42:~$ grep -v PermitTunnel /etc/ssh/sshd_config > /tmp/conf
dpavlin@t42:~$ ( grep -v PermitTunnel /etc/ssh/sshd_config ; echo PermitTunnel yes ) > /tmp/conf
dpavlin@t42:~$ diff -urw /etc/ssh/sshd_config /tmp/conf
--- /etc/ssh/sshd_config        2009-04-20 12:50:27.000000000 +0200
+++ /tmp/conf   2009-08-14 20:42:40.000000000 +0200
@@ -75,3 +75,4 @@
 Subsystem sftp /usr/lib/openssh/sftp-server
 
 UsePAM yes
+PermitTunnel yes

# install and restart ssh
dpavlin@t42:~$ sudo mv /tmp/conf /etc/ssh/sshd_config
dpavlin@t42:~$ sudo /etc/init.d/ssh restart
Restarting OpenBSD Secure Shell server: sshd.
Now we can connect two machines using ssh ethernet tunnel
dpavlin@t61p:/virtual/kvm$ sudo ssh -w 1:1 -o Tunnel=ethernet root@10.60.0.94

t42:~# ifconfig tap1
tap1      Link encap:Ethernet  HWaddr fa:35:cb:9e:87:60  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

t42:~# ip link set tap1 up
t42:~# brctl addif wire tap1
t42:~# brctl show wire
bridge name     bridge id               STP enabled     interfaces
pan0            8000.000000000000       no
wire            8000.006097472681       no              eth2
                                                        eth3
                                                        tap0
                                                        tap1
                                                        tap94

t42:~# dmesg | grep tap1
[284844.064953] wire: port 5(tap1) entering learning state

t42:~# tshark -i wire
This created tap1 devices on both machines and added one on t42 to bridge and left us with dump from tshark on wire bridge.

Now we need to setup virtual bridge on t61p to connect ssh tunnel and kvm tap device.

dpavlin@t61p:/virtual/kvm$ sudo ifconfig tap1
tap1      Link encap:Ethernet  HWaddr 52:c5:f8:64:30:d4  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

dpavlin@t61p:/virtual/kvm$ sudo brctl addbr virtual
dpavlin@t61p:/virtual/kvm$ sudo brctl addif virtual kvm0
dpavlin@t61p:/virtual/kvm$ sudo brctl addif virtual tap1

dpavlin@t61p:/virtual/kvm$ sudo brctl show
bridge name     bridge id               STP enabled     interfaces
pan0            8000.000000000000       no
virtual         8000.4e1537af6cdc       no              kvm0
                                                        tap1

dpavlin@t61p:/virtual/kvm$ sudo ip link set kvm0 up
dpavlin@t61p:/virtual/kvm$ sudo ip link set tap1 up
dpavlin@t61p:/virtual/kvm$ sudo ip link set virtual up

dpavlin@t61p:/virtual/kvm$ dmesg | grep virtual
[31141.669760] virtual: port 1(kvm0) entering learning state
[31152.288025] virtual: no IPv6 routers present
[31156.668088] virtual: port 1(kvm0) entering forwarding state
[31211.699928] virtual: port 2(tap1) entering learning state
[31226.696070] virtual: port 2(tap1) entering forwarding state
dpavlin@t61p:/virtual/kvm$ kvm -net nic,macaddr=52:54:00:00:0a:3d -net tap,ifname=kvm0,script=no -boot n
This will boot our kvm using ethernet bridge from remote server using nothing more than brctl and ssh !

If you wanted even more lightweight solution to same problem, you might look into EtherPuppet.

On related note, if your kvm Windows XP machines stopped working with upgrade to Debian kernel 2.6.30-1-686, just upgrade to 2.6.30-1-686-bigmem (even if you don't have more memory) and everything will be o.k.

For a long time, I used combination of proxy.pac and DynamicForward in .ssh/config to enable seamless surfing over ssh tunnels. However, this time, I wanted to access university network from home DSL.

I remembered all warnings about tunneling tcp over tcp and why it is bad idea, but I really liked simplicity of reusing ssh. So, I start looking for guides to implement minimal solution like this:

client.lan                      ssh gateway             private network
eth0 192.168.1.20 <--DSL-nat--> eth0 gw.example.com
tun0 10.60.0.81 <-pointopoint-> tun0 10.60.0.80 
                                eth1 10.60.0.10 <-----> 10.60.0.0/24
It's a bit more complicated, because LAN clients have to use NAT to access private network, but that's a single line in configuration. So, let's get started...

client.lan

  1. Generate ssh key for VPN
    root@client:~# ssh-keygen -t rsa -f /root/.ssh/gw-tun0
    Generating public/private rsa key pair.
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /root/.ssh/gw-tun0.
    Your public key has been saved in /root/.ssh/gw-tun0.pub.
    
  2. Transfer it to server
    echo 'tunnel="0",command="/sbin/ifdown tun0;/sbin/ifup tun0"' `cat /root/.ssh/gw-tun0.pub` | \
        ssh dpavlin@gw.example.com 'sudo sh -c "cat >> /root/.ssh/authorized_keys"'
    
    If you didn't have authorized_keys on gw.example.com you will also have to fix group with:
    root@gw:~# chown root:root /root/.ssh/authorized_keys
    
  3. /etc/network/interfaces
    iface tun0 inet static
        pre-up ssh -i /root/.ssh/gw-tun0 -S /var/run/gw-tun0 -M -f -w 0:0 gw.example.com true
        pre-up sleep 5
        address 10.60.0.81
        netmask 255.255.0.0
        pointopoint 10.60.0.80
        up iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o tun0 -j MASQUERADE
        post-down ssh -S /var/run/gw-tun0 -O exit gw.example.com
    
    iptables are used to NAT all hosts on home network, so they can see whole private network and not just IPs on gw.example.com
    I also added following route to my home network gateway so that all traffic for private network will go through VPN:
    # netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask
    10.60.0.0       192.168.1.20    255.255.0.0
    

gw.example.com

  1. /etc/ssh/sshd_config
    #PermitRootLogin yes
    PermitTunnel point-to-point
    PermitRootLogin forced-commands-only
    
  2. /etc/network/interfaces
    iface tun0 inet static
        address 10.60.0.80
        netmask 255.255.255.255
        pointopoint 10.60.0.81
        up arp -sD 10.60.0.81 eth1 pub
    
    arp entry is used to make home IP address 10.60.0.81 visible to all hosts in private network. That, together with rinetd on client.lan enabled me to tunnel connections to other hosts on my local network by creating port redirection (again, sigh!). I could have solved that with route on ssh gateway gw.example.com, but this solution leaves configuration of incomming connections on home side of link.
  3. /etc/cron.d/gw.example.com
    # monitor tun0
    */1     * * * * root    fping -q 10.60.0.81 || ( ifdown tun0 ; ifup tun0 ) | logger -t tun0
    

So, you want to checkout source code on some server and at the same time have ability to commit or diff against local copy on your laptop? Seems like an easy task, but it does involve some unexpected steps (for me at least), so here is a quick how to...

Create ssh tunnel from target host back to your laptop (called llin in output). Edit .ssh/config and add something like:

Host server-dev.rot13.org server-dev
	Hostname server-dev.rot13.org
	RemoteForward 8022 127.0.0.1:22
You will notice that I added short name so I can type just ssh server-dev because I'm lazy.

When you login to server-dev you might think that something like svn checkout is everything that is left. However, that doesn't really work:

dpavlin@server-dev:~$ svn ls svn+ssh://localhost:8022/home/dpavlin/private/svn/SQL2XLS
ssh: localhost:8022: Name or service not known
svn: Connection closed unexpectedly
It seems that subversion doesn't like port number within hostname! So, let's make .ssh/config there also:
Host llin-svn
	Hostname localhost
	Port 8022
Let's try it out:
dpavlin@server-dev:~$ svn ls svn+ssh://llin-svn/home/dpavlin/private/svn/SQL2XLS
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password or swipe finger: 
Much better, but is asks us for password every time. We don't really like that, so we'll create ssh keys to get around this:
dpavlin@server-dev:~$ ssh-keygen -f .ssh/llin-svn
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in .ssh/llin-svn.
Your public key has been saved in .ssh/llin-svn.pub.
The key fingerprint is:
aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99 dpavlin@server-dev.rot13.org
Now we will insert generated .ssh/llin-svn.pub into .ssh/authorized_keys on laptop but allow only svnserver to be started:
command="svnserve -t" ssh-rsa AAA...rest of key...AAA== dpavlin@server-dev.rot13.org
If you want normal ssh login back to your laptop, you might leave out command="svnserve -t", but this makes me feel better. On the other hand, tunnel will be open only when we are logged into server-dev, but I usually prefer more security if possible. If you don't want to commit back to laptop, you might add -R flag to make repository read-only.

But wait, there is more! We need to tell ssh on server-dev that we are using newly generated key, so our final .ssh/config looks like this:

Host llin-svn
	Hostname localhost
	Port 8022
	IdentityFile ~/.ssh/llin-svn

We can test it now to make sure that subversion doesn't ask for password by simply checking out source code:

dpavlin@server-dev:~$ svn co svn+ssh://llin-svn/home/dpavlin/private/svn/SQL2XLS
Checked out revision 0.

Happy hacking!