UDP issue on Docker for WSL 2

Latest update (11/3/2021)

To simplify the deploying process, I try to use a docker-compose to initialize the docker environment by the following script:

version: "3.9"
services:
    nginx:
        image: alqutami/rtmp-hls
        container_name: "rtmp-nginx"
        ports:
            - "8080:8080"
            - "1935:1935"
        environment:
            - TZ=Asia/Shanghai
        depends_on:
            - "php"
        volumes:
            - ./nginx.conf:/etc/nginx/nginx.conf
            - ./html:/usr/local/nginx/html

    php:
        build: .
        container_name: "rtmp-php"
        ports:
            - "9000:9000"
            #- "9090:9090/udp"
        environment:
            - TZ=Asia/Shanghai
        volumes:
            - ./html:/var/www/html

But with this script, C++ program running in docker host can no longer bind to the 9090 port.

After hours of trying, I found that udp package can be sent from docker container to docker host without projecting.

Work around

While trying to send UDP request to Docker host, I find that sending UDP request to 127.0.0.1 doesn’t work at all. The program which means to process the UDP request can only run properly inside of the Docker container. After hours of searching and trying, I finally confirm that the issue is because of the WSL 2, but Docker.

To get the “localhost” of the Docker host (which means the WSL here), you have to run this command at PowerShell as administrator.

wsl -e ip -4 addr show dev eth0

Attachments

UDP server in C++

#include <stdio.h>   
#include <sys/types.h>   
#include <sys/socket.h>   
#include <netinet/in.h>   
#include <unistd.h>   
#include <errno.h>   
#include <string.h>   
#include <stdlib.h>

#define SERV_PORT   9999

int main()  
{
  int sock_fd = socket(AF_INET, SOCK_DGRAM, 0);
  if(sock_fd < 0)  
  {  
    perror("socket");
    exit(1);
  }  
  
  struct sockaddr_in addr_serv;
  int len;
  memset(&addr_serv, 0, sizeof(struct sockaddr_in));
  addr_serv.sin_family = AF_INET;
  addr_serv.sin_port = htons(SERV_PORT);
  addr_serv.sin_addr.s_addr = htonl(INADDR_ANY);
  len = sizeof(addr_serv);
 
  if(bind(sock_fd, (struct sockaddr *)&addr_serv, sizeof(addr_serv)) < 0)  
  {  
    perror("bind error:");
    exit(1);
  }  
  
  int recv_num;
  int send_num;
  char send_buf[20] = "i am server!";
  char recv_buf[20];
  struct sockaddr_in addr_client;
  
  while(1)  
  {  
    printf("server wait:\n");
      
    recv_num = recvfrom(sock_fd, recv_buf, sizeof(recv_buf), 0, (struct sockaddr *)&addr_client, (socklen_t *)&len);
      
    if(recv_num < 0)  
    {  
      perror("recvfrom error:");
      exit(1);
    }  
  
    recv_buf[recv_num] = '\0';
    printf("server receive %d bytes: %s\n", recv_num, recv_buf);
  
    send_num = sendto(sock_fd, send_buf, recv_num, 0, (struct sockaddr *)&addr_client, len);
      
    if(send_num < 0)  
    {  
      perror("sendto error:");
      exit(1);
    }  
  }  
    
  close(sock_fd);
    
  return 0;
}

UDP client in PHP

<?php
$server = '172.30.212.113';
$port = 9999;

if(!($sock = socket_create(AF_INET, SOCK_DGRAM, 0)))
{
	$errorcode = socket_last_error();
    $errormsg = socket_strerror($errorcode);
    die("Couldn't create socket: [$errorcode] $errormsg \n");
}
echo "Socket created \n";

$input = "hello!!!!!!!!!!!!!!!!!!!!!!!!!!!!!";
if( ! socket_sendto($sock, $input , strlen($input) , 0 , $server , $port))
{
	$errorcode = socket_last_error();
	$errormsg = socket_strerror($errorcode);
	die("Could not send data: [$errorcode] $errormsg \n");
}

if(socket_recv ( $sock , $reply , 2045 , MSG_WAITALL ) === FALSE)
{
	$errorcode = socket_last_error();
	$errormsg = socket_strerror($errorcode);
	
	die("Could not receive data: [$errorcode] $errormsg \n");
}
echo "Reply : $reply";

Fix Un-allowed DMA capable bus/device(s) detected

Windows 8 and later has a function named “Device encrypt”, it allows windows to enable bitlocker for local disk automatically and unlock it at startup. But if you are using a DIY rig or changed/added some hardware to your OEM machine, then this feature might be broken. So, we need to add those devices to whitelist to reenable the feature.

Whitelist the devices

Check for driver updates in windows update before doing this. Because even though the devices that without a proper driver won’t show up in the device list, but the system could detect its device ID that might not be listed in the whitelist.

Phyllali
  1. Open Regedit
  2. Navigate to Computer\HKEY_LOCAL_MACHINE\SYSTEM\
    CurrentControlSet\Control\DmaSecurity\AllowedBuses
  3. Grant yourself access to modify the registry key
    1. Right-click AllowedBuses and go to Permissions
    2. Make yourself the owner
      1. Press Advanced
      2. Next to Owner, make note of what it says (mine said SYSTEM)
      3. Next to Owner, press Change
      4. Enter your username (eg your Microsoft account email address)
      5. Press OK
    3. Grant yourself access
      1. Press Add
      2. Enter your username (eg your Microsoft account email address)
      3. Press OK
      4. Select your user
      5. Tick Full Control
      6. Press OK
  4. Under AllowedBuses, create a new String Value
  5. Run the following PS script which generates a .reg file (with all found PCI devices) in tmp directory and then imports it silently
$tmpfile = "$($env:TEMP)\AllowBuses.reg"
'Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DmaSecurity\AllowedBuses]'`
| Out-File $tmpfile
(Get-PnPDevice -InstanceId PCI* `
| Format-Table -Property FriendlyName,InstanceId -HideTableHeaders -AutoSize `
| Out-String -Width 300).trim() `
-split "`r`n" `
-replace '&SUBSYS.*', '' `
-replace '\s+PCI\\', '"="PCI\\' `
| Foreach-Object{ "{0}{1}{2}" -f '"',$_,'"' } `
| Out-File $tmpfile -Append
regedit /s $tmpfile

Restart your computer and enjoy the Device encrypt feature!

Set up LNMP environment in WSL 2 with Docker on Windows

Web development on Windows used to be annoying, since the web environment has to be deploy on a remote server or a VM. But with the release of WSL 2 and the integration of the docker backend, thing goes simple now.

Before doing that…

This post assume that you have already got a WSL 2 distro installed. If you don’t or you have a WSL 1 distro, follow the guide on Microsoft Docs to get an install or upgrade.

Phyllali

Download then install the latest Docker Desktop Installer first. After that, launch the Docker Desktop App and go to settings, to ensure that the Use the WSL 2 based engine check box has been checked.

 Use the WSL 2 based engine
Use the WSL 2 based engine

Then go to Resources -> WSL INTEGRATION tab, make sure the distro you want to use is enabled.

WSL INTEGRATION
WSL INTEGRATION

Last, open the Windows Terminal and try to run docker command in it. If it works, you can now setup the LNMP environment.

Something else


Nginx

Pull the docker image of Nginx first.

docker pull nginx

Create a work directory for Nginx then.

cd ~
mkdir test-docker && cd test-docker
mkdir www && cd www

Now create a simple index.html for testing.

nano index.html
<html>
        <head>
                <title>Docker Nginx Testing</title>
        </head>
        <body>
                <div>
                        <h3>Hello, world!</h3>
                </div>
        </body>
</html>

CTRL + O then CTRL + X to save. And then go back to create a configuration file nginx-test.conf for Nginx.

cd ..
mkdir nginx && cd nginx
mkdir conf && cd conf
nano nginx-test.conf
server {
    listen       80;
    location / {
        root   /www;
        index  index.html index.htm;
    }
}

CTRL + O then CTRL + X to save.

Note, the root parameter in the conf file is not the path on your machine (Neither WSL path above), it’s the “virtual” path in docker container. Don’t worried about this path, we are goanna map the “physical” path to it when running the docker run command.

Finally, run the docker container. As you can see, with the parameter after -v, the “physical” path “/home/user/test-docker/www” was mounted in the specified path “/www” in the container and mapped the 80 port of container to the 8080 port of local machine.

docker run --name nginx -p 8080:80 -v /home/user/test-docker/nginx/conf:/etc/nginx/conf.d -v /home/user/test-docker/www:/www nginx

Try accessing the localhost:8080 from your windows browser. You should see the hello world page now.


PHP

Unlike the Nginx, PHP version maters a lot. So, we are goanna specify the version of PHP image.

docker pull php:7.1-fpm

Then crate the work folder for PHP. The folder etc is to store the config file for PHP, named php.ini.

cd ~
cd test-docker
mkdir php && cd php
mkdir etc

To integrate MySQL to PHP later, the PHP installation must have the pdo_mysql extension installed. So, we need to write a Dockerfile to build our own PHP image. Before doing that, we need a PHP config file storing locally first. Since it’s hard to write a PHP config file manually, we are going to start the PHP container first then copy the default config file out of that.

docker run --name php -d php:7.1-fpm
docker cp php:/usr/local/etc/php/php.ini-development ./etc/php.ini

Then write the Dockerfile. The first line indicates that the new docker image to build is based on the existing image named “php:7.1-fpm”, and the last line means that pdo_mysql extension is to be installed.

nano Dockerfile
FROM php:7.1-fpm
COPY ./etc/php.ini /usr/local/etc/php/php.ini
RUN docker-php-ext-install pdo_mysql

CTRL + O then CTRL + X to save.

Now build our own PHP image, and name it as “php:my-php”.

docker build -t php:my-php .

And now we are able to start the container.

docker run --name php -p 9000:9000 -v /home/user/test-docker/www:/www php:my-php

Connect Nginx and PHP

Nginx don’t know how to handle a .php file, so we need to tell Nginx to send the .php file to PHP when finding one.

Crate a network bridge to link the two containers first. Then connect them to the same network bridge.

docker network create --driver bridge lnmp
docker network connect lnmp php
docker network connect lnmp nginx

After that, changing the Nginx config file to forward php script to php-fpm. Because we have already connected the two containers to a same “network”, so it’s ok for Nginx to find php-fpm by container name.

cd ~
cd test-docker
cd nginx
cd conf
nano nginx-test.conf
server {
    listen       80;
    location / {
        root   /www;
        index  index.php index.html index.htm;
    }
    location ~ \.php$ {
        root           /www;
        fastcgi_pass   php:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }
}

Reload the config file then.

docker exec -it nginx nginx -s reload

After reloading the config file, go to www folder to crate phpinfo.php file testing if the php-fpm working properly.

cd ~
cd test-docker
cd www
nano phpinfo.php
<?php
phpinfo();

CTRL + O then CTRL + X to save. Try accessing the localhost:8080/phpinfo.php from your windows browser. You should see the PHP Version page now.


MySQL

Pull the docker image of MySQL first, since MySQL version matters a lot, specify the version of MySQL image is necessary too.

docker pull mysql:5.7

Then crate the work folder for MySQL.

cd ~
cd test-docker
mkdir mysql && cd mysql

Crate a database file for testing.

nano test.sql
create table student(
        id int not null primary key auto_increment comment 'KEY',
        name varchar(20) not null comment 'NAME',
        age int not null comment 'AGE'
)engine=InnoDB default charset=utf8 comment 'STUDENTTABLE';
 
insert into student(name, age) values('A', 23);
insert into student(name, age) values('B', 18);

Next, we need to write a Dockerfile to config the password, database name, and copy SQL file inside.

FROM mysql:5.7
ENV MYSQL_ROOT_PASSWORD=123456 MYSQL_DATABASE=test
COPY ./test.sql /var/data/test.sql

Now build our custom MySQL image.

docker build -t mysql:my-mysql .

Then start the container, and attach it to lnmp network.

docker run --name mysql --network lnmp -p 3306:3306 mysql:my-mysql

Then enter the container, and import the test.sql to the database. Enter “123456” as password after running the second command.

docker exec -it mysql bash
mysql -u root -p
mysql> use test
mysql> source /var/data/test.sql

Then crate testmysql.php at the www folder for testing the MySQL server.

cd ~
cd test-docker
cd www
nano testmysql.php
<?php
$dsn = 'mysql:dbname=test;host=mysql';
$user = 'root';
$password = '123456';
 
try {
    $dbh = new PDO($dsn, $user, $password);
    $sql = 'SELECT * FROM student where id=?';
    $sth = $dbh->prepare($sql);
    $sth->execute([2]);
    $result = $sth->fetch(PDO::FETCH_ASSOC);
    var_dump($result);
} catch (PDOException $e) {
    echo 'Error: ' . $e->getMessage();
}

CTRL + O then CTRL + X to save. Try accessing the localhost:8080/ testmysql.php from your windows browser. You should see the SQL output now.

The lnmp environment is now fully configured.

Compiling & installing the latest OpenCV

Since apt doesn’t support 3-party components and might not install the latest version of OpenCV sometimes, installing OpenCV by building yourself might be helpful in some case.

On Ubuntu

Firstly, install all the packages that essential for building. The following packages contain libraries that OpenCV depends on, but do not include GPU acceleration and ML parts.

apt-get update && apt-get upgrade
add-apt-repository "deb http://security.ubuntu.com/ubuntu xenial-security main"
apt-get install -y build-essential cmake git unzip zip\
                pkg-config libavcodec-dev libavformat-dev libswscale-dev \
                libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev \
                libsnappy-dev libboost-all-dev libjasper-dev libdc1394-22-dev\
                python3-dev python3-numpy python3-pip ffmpeg libopenblas-dev \
                tesseract-ocr libtesseract-dev libprotobuf-dev libleveldb-dev \
                libhdf5-serial-dev protobuf-compiler libatlas-base-dev \
                libgflags-dev libgoogle-glog-dev liblmdb-dev libfaac-dev \
                gfortran libgstreamer1.0-dev libatlas-base-dev libxvidcore-dev \
                libpng-dev libopenexr-dev libwebp-dev \
                libmp3lame-dev libtheora-dev libvorbis-dev  \
                libopencore-amrwb-dev x264 v4l-utils libgdk-pixbuf2.0-dev \
                manpages-dev libopencore-amrnb-dev libgstreamer-plugins-base1.0-dev libavresample-dev
pip3 install --upgrade pip
apt-get install -f

Then, download and unzip OpenCV source code form GitHub. In order not to mess the working directory up, it is recommended to create a new folder dedicated for compilation.

Note: OpenCV may have been updated since the blog was written, so don’t just simply copy & paste the following command.

mkdir opencv_build && cd opencv_build
wget https://github.com/opencv/opencv/archive/refs/tags/4.5.3.zip
unzip 4.5.3.zip && rm 4.5.3.zip
wget https://github.com/opencv/opencv_contrib/archive/refs/tags/4.5.3.zip
unzip 4.5.3.zip && rm 4.5.3.zip
cd opencv-4.5.3

Now start building. The last line indicates the number of threads used for compilation, which should be determined according to the number of CPU cores of the machine.

mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local/ \
        -D OPENCV_ENABLE_NONFREE=ON \
        -D OPENCV_EXTRA_MODULES_PATH=~/opencv_build/opencv_contrib-4.5.3/modules \
        -D OPENCV_GENERATE_PKGCONFIG=YES \
        -D WITH_QT=OFF \
        -D WITH_OPENGL=OFF \
        -D WITH_CUDA=OFF \
        -D BUILD_EXAMPLES=ON \
        -D INSTALL_PYTHON_EXAMPLES=ON \
        -D INSTALL_C_EXAMPLES=ON ..
make -j6

If there is no error in the compilation process, then start the installation. Just running a single line of code:

make install

After all of that, the OpenCV lib files is now can be included(or imported) by CMake. But if you are trying to use an IDE like VSCode, there might be an error indicate that the lib files could not be found.

So the very last step is needed.

sh -c "echo '/usr/local/opencv4.5.2/lib' >> /etc/ld.so.conf.d/opencv.conf"
sh -c "echo 'export PKG_CONFIG_PATH=/usr/local/opencv4.5.3/lib/pkgconfig:$PKG_CONFIG_PATH' >> /etc/profile.d/pkgconfig.sh"
source /etc/profile
ln -s /usr/local/opencv4.5.3/lib/python3.8/dist-packages/cv2/python-3.8/cv2.cpython-38-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/cv2.so

To facilitate future deployment, I wrote the above process into a script, and run it in bash will automatically start the installation(Rename before using!).

mv opencv4.5.3.jpg opencv4.5.3.sh
chmod +x opencv4.5.3.sh
sudo ./opencv.4.5.3.sh

On Windows

In fact, OpenCV offers a pre-build package of the latest version, but without including opencv_contrib.

If you are using OpenCV for just studying, you can just download a copy here.

Otherwise, download the source code of OpenCV here and opencv_contrib here.