Re: qemu-nbd acting as a server or not ?

[Bob Power]

Yeah, it looks that way...

[root@fedora bob]# qemu-nbd -d /dev/nbd0  
/dev/nbd0 disconnected

All clear :

[root@fedora bob]# netstat -pnxa | head -2 | tail -1 ; netstat -pnxa | grep nbd
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
[root@fedora bob]# cat <(ss -pax | head -1 | sed -E 's/(\S+) Address:Port/\1_Address Port /g' ) <( ss -pax | grep nbd) | sed -E 's/\s+$//' | column -t
Netid  State  Recv-Q  Send-Q  Local_Address  Port  Peer_Address  Port  Process

[root@fedora bob]# qemu-nbd -c /dev/nbd0  test.qcow2

Active Unix Domain Sockets:

[root@fedora bob]# cat <(ss -pax | head -1 | sed -E 's/(\S+) Address:Port/\1_Address Port /g' ) <( ss -pax | grep nbd) | sed -E 's/\s+$//' | column -t
Netid  State   Recv-Q  Send-Q  Local_Address            Port   Peer_Address  Port   Process
u_str  LISTEN  0       1       /var/lock/qemu-nbd-nbd0  42983  *             0      users:(("qemu-nbd",pid=5211,fd=3))
u_str  ESTAB   0       0       *                        45022  *             42988  users:(("qemu-nbd",pid=5211,fd=9))
u_str  ESTAB   0       0       /var/lock/qemu-nbd-nbd0  42988  *             45022  users:(("qemu-nbd",pid=5211,fd=10))

[root@fedora fd]# netstat -pnxa | head -2  ; netstat -pnxa | grep nbd
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     STREAM     LISTENING     42983    5211/qemu-nbd        /var/lock/qemu-nbd-nbd0
unix  3      [ ]         STREAM     CONNECTED     45022    5211/qemu-nbd        
unix  3      [ ]         STREAM     CONNECTED     42988    5211/qemu-nbd        /var/lock/qemu-nbd-nbd0


SS        Netstat
Netid    =    Proto    : u_str = unix stream
Port    =    I-Node    : socketfs inode
        Flags    : ACC= Waiting for a connect request

[root@fedora lock]# lsof +E /var/lock/qemu-nbd-nbd0
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF  NODE NAME
qemu-nbd 5211 root    3u  unix 0x00000000b0829fa5      0t0 42983 /var/lock/qemu-nbd-nbd0 type=STREAM (LISTEN)
qemu-nbd 5211 root    9u  unix 0x00000000876da850      0t0 45022 type=STREAM ->INO=42988 5211,qemu-nbd,10u (CONNECTED)
qemu-nbd 5211 root   10u  unix 0x00000000482ad094      0t0 42988 /var/lock/qemu-nbd-nbd0 type=STREAM ->INO=45022 5211,qemu-nbd,9u (CONNECTED)

So it looks like 45022 is qemu-nbd acting as a client accessing qemu-nbd as a server on 42988 having requested that connection (from itself) on 42983 (?)

But now I'm more confused - isn't nbd as in "modinfo nbd" = /lib/modules/5.16.20-200.fc35.x86_64/kernel/drivers/block/nbd.ko.xz

the nbd client in the kernel interfacing with an nbd server to provide fs access via /dev/nbd0 ? or does nbd.ko.xz just talk to/need an nbd client, in this case qemu-nbd (on 45022) ?

And dunno why the netstat RefCnts are 2 and 3 ...

I *would* like to know definitively how this works from an overview/architecture perspective eg.
(qcow2 file) <data/transport> (qemu-nbd as a server) <data/transport> (qemu-nbd as a client) <data/transport> (kernel module / filesytem interface)  

..but short of an nbd/qemu-nbd developer setting the facts down (or getting into the code myself) I think I'll just have to let it go.

On Monday, 2 May 2022, 09:13:11 BST, Pascal <patatetom@gmail.com> wrote:

hi,

reading the manual (qemu-nbd 6.2.0), I would say that qemu-nbd can take on two roles: nbd server AND nbd client (not just server).

The `-c` option makes it take on both roles locally.

on my side, I see two inter-connected (eg. client/server) unix sockets :

```

# modprobe nbd

# truncate -s 256M /tmp/test.disk

# qemu-nbd -f raw -c /dev/nbd0 /tmp/test.disk

# ss -xp | head -1; ss -xp | grep nbd

Netid State Recv-Q Send-Q                 Local Address:Port  Peer Address:Port  Process
u_str ESTAB 0      0            /var/lock/qemu-nbd-nbd0 68444            * 65480 users:(("qemu-nbd",pid=5430,fd=10))
u_str ESTAB 0      0                                  * 65480            * 68444 users:(("qemu-nbd",pid=5430,fd=9))

```

this being said, maybe the qem-nbd manual could be more precise about this option...

regards, lacsaP.

Le dim. 1 mai 2022 à 19:46, Bob Power <b_power@yahoo.com> a écrit :

Hey all,

Back again...

While image mounted / in use .....

[root@fedora vd]# cat <(ps -aux | head -1) <(ps -aux | grep nbd)

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         126  0.0  0.0      0     0 ?        S<   14:17   0:00 [kworker/u9:0+nbd0-recv]
root        6541  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd0-recv]
root        6544  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd1-recv]
root        6545  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd2-recv]
root        6547  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd3-recv]
root        6550  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd4-recv]
root        6551  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd5-recv]
root        6552  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd6-recv]
root        6553  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd7-recv]
root        6554  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd8-recv]
root        6555  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd9-recv]
root        6556  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd10-recv]
root        6558  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd11-recv]
root        6560  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd12-recv]
root        6561  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd13-recv]
root        6562  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd14-recv]
root        6564  0.0  0.0      0     0 ?        I<   17:28   0:00 [nbd15-recv]
root        6575  0.0  0.0 1982620 6700 ?        Ssl  17:28   0:00 qemu-nbd -c /dev/nbd0 test.qcow2
root        6589  0.0  0.0      0     0 ?        I<   17:28   0:00 [xfs-buf/nbd0p1]
root        6591  0.0  0.0      0     0 ?        I<   17:28   0:00 [xfs-conv/nbd0p1]
root        6592  0.0  0.0      0     0 ?        I<   17:28   0:00 [xfs-reclaim/nbd]
root        6593  0.0  0.0      0     0 ?        I<   17:28   0:00 [xfs-blockgc/nbd]
root        6594  0.0  0.0      0     0 ?        I<   17:28   0:00 [xfs-inodegc/nbd]
root        6595  0.0  0.0      0     0 ?        I<   17:28   0:00 [xfs-log/nbd0p1]
root        6596  0.0  0.0      0     0 ?        I<   17:28   0:00 [xfs-cil/nbd0p1]
root        6597  0.0  0.0      0     0 ?        S    17:28   0:00 [xfsaild/nbd0p1]
root        6821  0.0  0.0 221800  2132 pts/0    S    17:33   0:00 grep --color=auto nbd

[root@fedora vd]# nmap -p 10809 localhost

Starting Nmap 7.92 ( https://nmap.org ) at 2022-05-01 18:05 BST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00010s latency).
Other addresses for localhost (not scanned): ::1

PORT      STATE  SERVICE

10809/tcp closed nbd

[root@fedora vd]# systemctl list-units --type=service | grep nbd
[root@fedora vd]# systemctl list-units --type=service | grep qemu
  qemu-guest-agent.service                                                                  loaded active running QEMU Guest Agent

[root@fedora vd]# netstat -pnltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      693/systemd-resolve
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      693/systemd-resolve
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      815/cupsd           
tcp6       0      0 ::1:631                 :::*                    LISTEN      815/cupsd           
tcp6       0      0 :::5355                 :::*                    LISTEN      693/systemd-resolve
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           719/avahi-daemon: r
udp        0      0 0.0.0.0:5355            0.0.0.0:*                           693/systemd-resolve
udp        0      0 0.0.0.0:48300           0.0.0.0:*                           719/avahi-daemon: r
udp        0      0 127.0.0.53:53           0.0.0.0:*                           693/systemd-resolve
udp        0      0 127.0.0.1:323           0.0.0.0:*                           764/chronyd         
udp6       0      0 :::5353                 :::*                                719/avahi-daemon: r
udp6       0      0 :::5355                 :::*                                693/systemd-resolve
udp6       0      0 :::47159                :::*                                719/avahi-daemon: r
udp6       0      0 ::1:323                 :::*                                764/chronyd 

So qemu-nbd is running ( has not exited ) and nothing is running as a service: no server.

I'm not saying this is a problem or a bug - it looks to me like most of the time it works and is supposed to work as a server and in this case it doesn't and isn't supposed to.

Reading the man page I just expected to see it available (via an nbd client) on port 10809....as well as /dev/nbd0 via the usual filesystem interfaces.

( So I tried with -re 2 (read-only , 2 users ) out of curiosity and stuff broke ... )

Unless I've misinterpreted what is going on, it should ideally say, for clarity, in the man page under "-c",  that in this case it does not act as a server but on a 1-to-1 basis with the host via whatever method it uses (local socket ?)

A server listening on an open port is a security consideration so it would be nice to know for certain there isn't supposed to be one if there isn't.

Bob.

On Friday, 29 April 2022, 09:01:34 BST, Pascal <patatetom@gmail.com> wrote:

`qemu-nbd -c` operate as a "local" server/client.

Le ven. 29 avr. 2022 à 09:58, Pascal <patatetom@gmail.com> a écrit :

hi,

`-t` switch for persistence (for me - `-p` is for port number).

`qemu-nbd -c` operate as a "local" server.

`qemu-nbd` still running : use `ps ax | grep nbd` and/or `ss -p | grep nbd` to see it.

regards, lacsaP.

Le ven. 29 avr. 2022 à 02:40, Connor Kuehl <cipkuehl@gmail.com> a écrit :

re-adding the list

On Thu Apr 28, 2022 at 6:51 PM CDT, Bob Power wrote:
>  Hi Connor,
> While I'm using the qcow2 image as nbd0 - ie while it is mounted and I am accessing files - so client not hung up and something still running to interface between the qcow2 image and the system, there does not appear to be any nbd service operating.

Yeah, my suspicion is that as part of creating /dev/nbd0, the kernel
gets all of the information it needs from the qemu-nbd process and is
then self-sufficient and therefore it hangs up on qemu-nbd and qemu-nbd
exits. Just a guess.

Have you tried seeing if qemu-nbd remains up to serve other clients if
you use the -p or --persistent flag?

        qemu-nbd -c /dev/nbd0 -p test.qcow2

--
Connor