mathjax

Monday, October 24, 2016

Why is ssh slow to connect?

Introduction

There are several reasons why ssh may be getting slow to connect. I'll cover here the ones that happened to me.


On the server side

  1. Make sure you have a line "UseDNS no" uncommented.
  2. Make sure you have "good" (i.e., reachable) DNS servers in /etc/resolv.conf.
  3. Make sure you have no reverse DNS lines in /etc/hosts.deny.
The server side configuration is typically in the file /etc/ssh/sshd_config. Every time you make a change to this file, remember to restart the ssh daemon ("# systemctl restart sshd").

    Item number 3 deserves some comments, since it was the hardest to get right. Denyhosts may be adding lines like  "xxx-xxx-xxx-xxx.rev.abxtelex.eu" that will trigger reverse DNS even if you have "UseDNS no" in the configuration file. You have to remove these lines, but if you have Denyhosts installed, maybe you already know how hard it is to remove entries from this file, since they keep reappearing. If you use OpenSUSE, you have a script called "/usr/sbin/dh_reenable" that will do the trick for you. If you are not using OpenSUSE or have installed Denyhosts by hand, then you have to do it manually, take a look at the Denyhosts FAQ here.

    I still don't have a good solution to problem number 3, the best thing would be that Denyhosts did not add reverse DNS entries in /etc/resolv.conf, but I did not find a way to configure it to behave like that.


    On the client side

    The global client side configuration is typically in the file /etc/ssh/ssh_config. But you can configure things on a user level by editing the file ~/.ssh/config.

    I use the local configuration file like this:

    Host *
    Compression yes
    ForwardX11=yes
    ForwardX11Trusted=yes
    GSSAPIAuthentication=no

    Host analise
    HostName 192.168.1.1

    Host home
    Hostname xxxx.noip.me


    That way I can just type "ssh analise" or "ssh home". In the "home" case, it has the advantage of making my dynamic dns name shorter.


    1. Make sure you have the line "GSSAPIAuthentication=no".

    Monday, August 1, 2016

    OpenSUSE Leap 42.1 and nvidia kernel driver

    The symptoms

    Nvidia kernel drivers were not loaded after kernel update.

    The problem

    The package nvidia-gfxG04-kmp-default-367.35_k4.1.12_1-25.1.x86_64 runs a script that regenerates the kernel drivers and creates symlinks in /lib/modules/4.1.27-27-default/weak-updates/updates. After running:

    # zypper in --force $(rpm -qa "nvidia-gfx*kmp*")

    I noticed that the links were not beeing generated:

    # zypper in --force $(rpm -qa "nvidia-gfx*kmp*")
    Retrieving repository 'network:utilities' metadata ..........................................................................................................[done]
    Building repository 'network:utilities' cache ...............................................................................................................[done]
    Loading repository data...
    Reading installed packages...
    Forcing installation of 'nvidia-gfxG04-kmp-default-367.35_k4.1.12_1-25.1.x86_64' from repository 'nVidia Graphics Drivers'.
    Resolving package dependencies...

    The following package is going to be reinstalled:
      nvidia-gfxG04-kmp-default

    1 package to reinstall.
    Overall download size: 5.8 MiB. Already cached: 0 B. No additional space will be used or freed after the operation.
    Continue? [y/n/? shows all options] (y): 
    Retrieving package nvidia-gfxG04-kmp-default-367.35_k4.1.12_1-25.1.x86_64                                                     (1/1),   5.8 MiB ( 64.6 MiB unpacked)
    Retrieving: nvidia-gfxG04-kmp-default-367.35_k4.1.12_1-25.1.x86_64.rpm ..........................................................................[done (2.2 MiB/s)]
    Checking for file conflicts: ................................................................................................................................[done]
    (1/1) Installing: nvidia-gfxG04-kmp-default-367.35_k4.1.12_1-25.1.x86_64 ....................................................................................[done]
    Additional rpm output:
    make: Entering directory '/usr/src/linux-4.1.27-27-obj/x86_64/default'
      Building modules, stage 2.
      MODPOST 0 modules
    make: Leaving directory '/usr/src/linux-4.1.27-27-obj/x86_64/default'
    /usr/src/kernel-modules/nvidia-367.35-default /
    make "CC=cc" KBUILD_OUTPUT=/usr/src/linux-obj/x86_64/default KBUILD_VERBOSE= -C /lib/modules/4.1.27-27-default/source M=/usr/src/kernel-modules/nvidia-367.35-default ARCH=x86_64 NV_KERNEL_SOURCES=/lib/modules/4.1.27-27-default/source NV_KERNEL_OUTPUT=/usr/src/linux-obj/x86_64/default NV_KERNEL_MODULES="nvidia nvidia-uvm nvidia-modeset nvidia-drm" INSTALL_MOD_DIR=kernel/drivers/video modules
    make[1]: Entering directory '/usr/src/linux-4.1.27-27'
    make[2]: Entering directory '/usr/src/linux-4.1.27-27-obj/x86_64/default'
      Building modules, stage 2.
      MODPOST 4 modules
    make[2]: Leaving directory '/usr/src/linux-4.1.27-27-obj/x86_64/default'
    make[1]: Leaving directory '/usr/src/linux-4.1.27-27'
    ld -T /lib/modules/4.1.27-27-default/source/scripts/module-common.lds -r -o nv-linux.o \
      nvidia.mod.o nvidia/nv-interface.o
    /

    Modprobe blacklist files have been created at /etc/modprobe.d to prevent Nouveau from loading. This can be reverted by deleting /etc/modprobe.d/nvidia-*.conf.

    *** Reboot your computer and verify that the NVIDIA graphics driver can be loaded. ***

    depmod: WARNING: //lib/modules/4.1.27-27-default/misc/vboxvideo.ko disagrees about version of symbol VBoxGuest_RTLogBackdoorPrintf
    depmod: WARNING: //lib/modules/4.1.27-27-default/misc/vboxvideo.ko disagrees about version of symbol VBoxGuest_RTErrConvertToErrno
    depmod: WARNING: //lib/modules/4.1.27-27-default/misc/vboxvideo.ko disagrees about version of symbol VBoxGuest_RTAssertShouldPanic
    depmod: WARNING: //lib/modules/4.1.27-27-default/misc/vboxvideo.ko disagrees about version of symbol VBoxGuest_RTAssertMsg1Weak
    depmod: WARNING: //lib/modules/4.1.27-27-default/misc/vboxvideo.ko disagrees about version of symbol VBoxGuest_RTAssertMsg2Weak
    Warning: /lib/modules/4.1.27-27-default is inconsistent
    Warning: weak-updates symlinks might not be created


    Output of nvidia-gfxG04-kmp-default-367.35_k4.1.12_1-25.1.x86_64.rpm %posttrans script:
        Creating initrd: /boot/initrd-4.1.12-1-default
        Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "nvidia   -drm" /boot/initrd-4.1.12-1-default 4.1.12-1-default
        *** Including module: bash ***
        *** Including module: warpclock ***
        *** Including module: i18n ***
        *** Including module: ifcfg ***
        *** Including module: drm ***
        *** Including module: plymouth ***
        *** Including module: kernel-modules ***
        Omitting driver i2o_scsi
        *** Including module: resume ***
        *** Including module: rootfs-block ***
        *** Including module: terminfo ***
        *** Including module: udev-rules ***
        Skipping udev rule: 91-permissions.rules
        Skipping udev rule: 80-drivers-modprobe.rules
        *** Including module: haveged ***
        *** Including module: systemd ***
        *** Including module: usrmount ***
        *** Including module: base ***
        *** Including module: fs-lib ***
        *** Including module: shutdown ***
        *** Including module: suse ***
        *** Including modules done ***
        *** Installing kernel module dependencies and firmware ***
        *** Installing kernel module dependencies and firmware done ***
        *** Resolving executable dependencies ***
        *** Resolving executable dependencies done***
        *** Hardlinking files ***
        *** Hardlinking files done ***
        *** Stripping files ***
        *** Stripping files done ***
        *** Generating early-microcode cpio image ***
        *** Constructing GenuineIntel.bin ****
        *** Store current command line parameters ***
        Stored kernel commandline:
        rd.driver.pre=nvidia
        rd.driver.pre=_drm
         resume=UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        root=UUID=xxxxxxxxxxxxxxxxxxxxxx rootflags=rw,relatime,data=ordered rootfstype=ext4
        *** Creating image file ***
        *** Creating image file done ***
        Some kernel modules could not be included
        This is not necessarily an error:
        swap
        _drm
        Update bootloader...
        Creating initrd: /boot/initrd-4.1.27-24-default
        Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "nvidia   -drm" /boot/initrd-4.1.27-24-default 4.1.27-24-default
        *** Including module: bash ***
        *** Including module: warpclock ***
        *** Including module: i18n ***
        *** Including module: ifcfg ***
        *** Including module: drm ***
        *** Including module: plymouth ***
        *** Including module: kernel-modules ***
        Omitting driver i2o_scsi
        *** Including module: resume ***
        *** Including module: rootfs-block ***
        *** Including module: terminfo ***
        *** Including module: udev-rules ***
        Skipping udev rule: 91-permissions.rules
        Skipping udev rule: 80-drivers-modprobe.rules
        *** Including module: haveged ***
        *** Including module: systemd ***
        *** Including module: usrmount ***
        *** Including module: base ***
        *** Including module: fs-lib ***
        *** Including module: shutdown ***
        *** Including module: suse ***
        *** Including modules done ***
        *** Installing kernel module dependencies and firmware ***
        *** Installing kernel module dependencies and firmware done ***
        *** Resolving executable dependencies ***
        *** Resolving executable dependencies done***
        *** Hardlinking files ***
        *** Hardlinking files done ***
        *** Stripping files ***
        *** Stripping files done ***
        *** Generating early-microcode cpio image ***
        *** Constructing GenuineIntel.bin ****
        *** Store current command line parameters ***
        Stored kernel commandline:
        rd.driver.pre=nvidia
        rd.driver.pre=_drm
         resume=UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        root=UUID=xxxxxxxxxxxxxxxxxxxxxxx rootflags=rw,relatime,data=ordered rootfstype=ext4
        *** Creating image file ***
        *** Creating image file done ***
        Some kernel modules could not be included
        This is not necessarily an error:
        swap
        _drm
        Update bootloader...

    The solution

    Uninstall the virtualbox guest packages: virtualbox-guest-kmp-default, virtualbox-guest-tools and virtualbox-guest-x11 and rerun # zypper in --force $(rpm -qa "nvidia-gfx*kmp*") as root. After that, the links are created as follows:

    /lib/modules/4.1.27-27-default/weak-updates/updates # l
    total 8
    drwxr-xr-x 2 root root 4096 Aug  1 15:41 ./
    drwxr-xr-x 4 root root 4096 Jul 21 19:03 ../
    lrwxrwxrwx 1 root root   51 Aug  1 15:41 nvidia-drm.ko -> /lib/modules/4.1.12-1-default/updates/nvidia-drm.ko
    lrwxrwxrwx 1 root root   55 Aug  1 15:41 nvidia-modeset.ko -> /lib/modules/4.1.12-1-default/updates/nvidia-modeset.ko
    lrwxrwxrwx 1 root root   51 Aug  1 15:41 nvidia-uvm.ko -> /lib/modules/4.1.12-1-default/updates/nvidia-uvm.ko
    lrwxrwxrwx 1 root root   47 Aug  1 15:41 nvidia.ko -> /lib/modules/4.1.12-1-default/updates/nvidia.ko

    Tuesday, March 8, 2016

    Arduino Shield for custom board CPLD programming and testing using pogo pins


    This post is just to show how I have used the Arduino JTAG programming hardware/software that I have discussed before.

    The idea was to have a setup where I could both program and test a CPLD based board.

    Lets see the photos:


    The photo above shows the support for the board to be programmed with the pogo pins at the center, the board itself and the top shield. I have used two identical shield boards and have spaced them to give the pogo pins the proper vertical direction.

    Some of the pads on the board to be tested are SMD, others are through hole. Of course, after it was assembled, I realized I should have left the pogos that go peek at through holes slightly higher that those that peek at SMD pads. That would have made the process of fitting the board much better.

    In the same holes, I have mounted the board support, which is a kind of "negative" of the board. It consists of two milled pcbs, with two concentric circles to give support to the board. If you look carefully to the left of the photo, you can see a small dent that is used to give the board the proper orientation.


    In this next photo, we can see the toggle clamp device used to hold the board in place in action.


    In the last photo, we can see the full stack: the Arduino at the ground, the two shield boards in the middle and the support with a board in it.

    Some details for those interested:


    Hope you like it, comments are welcome!

    Saturday, March 5, 2016

    Serial Buffer Size versus Effective Bit Rate of Arduino USB

    Introduction


    I have recently just met a few of the "gotchas" related to serial programming and memory on the Arduino, and got a few lessons. I was debugging someone else's non-working code. Non-working for, apparently, no good reason. To make a long story short, the problem was that the program used a large amount of SRAM (static RAM) memory, in the form of strings. The Arduino Uno has 32 Kib of flash, but only 2 KiB of SRAM. That is why strings in the Arduino should be kept in flash memory to save the precious SRAM. To do so, you have to use the "F()" macro, so that the compiler does that for you.

    Figuring out the problem was not easy, since using "Serial.print()"  without "F()" to debug would just make things worse in an unpredictable way. But at a certain point, I got it, and since then I tried my best to spare SRAM. That is when I started facing the problem of the size of the serial buffer.

    Serial communication on the Arduino has one big problem: there is no hardware flow control. That means that if you want reliable communication, you must implement your own flow control mechanism. Anything you come up with software, implies a greater overhead than what you would get with a hardware mechanism. But of course, using a larger reception buffer would minimize the problem. The larger the buffer, the smaller the number of times the flow control mechanism must work.

    The Arduino software has a default size of 64 bytes for the serial buffer. I wonder if that was enough, so I wrote some code to test it.

    Some theory


    Lets try to come up with a model. Linear models look interesting, for a start. Lets try the following: the time that a transfer takes \((\Delta t)\) is proportional to the number of bytes we want to transfer. If you consider a serial transmission with UART, eight data bits, one start bit and one stop bit, the time to transfer one byte is proportional to either ten times the inverse of the bit rate or some byte processing overhead \((O_{byte})\), whichever is greater. But since these bytes are transfered in blocks, we can imagine that the total transfer time also has an overhead component proportional to the number of blocks \((O_{block})\). In equations:

    \begin{equation} \Delta t = NumBytes \cdot \max \left( \frac{10}{BitRate}, O_{byte}\right) + NumBlocks \cdot O_{block} \end{equation}

    \begin{equation} \Delta t = NumBytes \cdot \max \left( \frac{10}{BitRate}, O_{byte}\right) + \frac{NumBytes}{BlockSize} \cdot O_{block} \end{equation}

    \begin{equation} \label{eqDeltaTFinal} \Delta t = NumBytes \cdot \left[ \max \left( \frac{10}{BitRate}, O_{byte}\right) + \frac{O_{block}}{BlockSize} \right] \end{equation}

    \begin{equation} \frac{\Delta t}{NumBytes} = \max \left( \frac{10}{BitRate}, O_{byte}\right) + \frac{O_{block}}{BlockSize} \end{equation}

    \begin{equation} \frac{10}{EffectiveBitRate} = \max \left( \frac{10}{BitRate}, O_{byte}\right) + \frac{O_{block}}{BlockSize} \end{equation}

    Equation \ref{eqDeltaTFinal} shows two things:

    1. We can mitigate the block overhead using a larger block size.
    2. We should try to keep the byte overhead less than 10 times the inverse of the bit rate.

    In this work, I will estimate the byte overhead and the block overhead from the measure of the effective bit rate for various block sizes.

    Arduino Bit Rates


    For the data to have some meaning, we will have to use an exact calculation of the Arduino bit rates. The formula is (for AVR's U2X bit = 1):

    \begin{equation}\label{eqnBitRate}BitRate = \frac{10 \cdot ClockFrequency}{8 \cdot (UBRR + 1)} \end{equation}

    or

    \begin{equation}\label{eqnBytePeriod}\frac{1}{ByteRate} = \frac{10}{BitRate} = \frac{8 \cdot (UBRR + 1) \cdot 1000}{ClockFrequency} \,\, ms/byte \end{equation}

    For the Arduino, \(ClockFrequency = 16 MHz \), such that \(9600\,bits/s\) is actually \(9615.4\,bits/s\) \((UBRR = 16,\,1.0406\,ms/byte)\), and \(115200\,bits/s\) is actually \(117647\,bits/s\,(UBRR = 207,\,85.144\,\mu{s}/byte)\).

    I found this nice AVR bit rate calculator, if you are curious, you can play with it.

    The Data


    Each graph consists on a log-log plot of two sets of data. The red curves are for the bit rate of 9600, and the blue curves are for the bit rate of 115200. Both curves refer to the transfer time of 32768 bytes. For each graph I have created an artificial byte overhead using a delay after receiving the byte. The log-log plot is necessary to linearize the "\(\frac{1}{x}\)" relation of total time versus block size.

    The tail of the curves can be estimated from equation \ref{eqDeltaTFinal} taking the limit when the block size is large:

    \begin{equation} \Delta t_{tail} = NumBytes \cdot \max \left( \frac{10}{BitRate}, O_{byte}\right) \end{equation}

    which is \(\left(NumBytes \cdot \frac{10}{BitRate}\right)\) or \(\left(NumBytes \cdot O_{byte}\right)\). If we assume that \( O_{byte}\leqslant \frac{10}{BitRate}\), then for a sequence of 32768 bytes we have the theoretical values of 34.078 ms and 2.785 ms for 9615.4 bps and 117647 bps respectively. Which agrees whith the measured values of 34.1 ms and 2.79 ms.

    The graphs show that for the Arduino working in 9600 bps (9615.4 bps actually), a buffer size of 17 bytes is enough to reach the minimum theoretical transfer time. For 115200 bps (117647 bps actually) a buffer size of 27 bytes will do.

    give us the estimate of \(\max \left( \frac{10}{BitRate}, O_{byte}\right)\), while the linear part gives us the estimate of \(\frac{O_{block}}{BlockSize}\).

    Reality is always full of surprises. The tail behaves as we would expect from our crude model, but for the lower values of the buffer size, we can see some unexpected things.




    Conclusion


    Depending on the byte processing overhead your algorithm has, we saw that a 63 byte buffer can have the same performance in 9600 bits/s or 115200 bits/s.

    The bumpy block overhead is something that I might analyse more carefully some day in the future.