Re: qemu-nbd acting as a server or not ?

[Hanna Reitz]

On 03.05.22 19:01, Bob Power wrote:
> Yeah, it looks that way...
>
> [root@fedora bob]# qemu-nbd -d /dev/nbd0
> /dev/nbd0 disconnected
>
> All clear :
>
> [root@fedora bob]# netstat -pnxa | head -2 | tail -1 ; netstat -pnxa | 
> grep nbd
> Proto RefCnt Flags       Type       State         I-Node PID/Program 
> name     Path
> [root@fedora bob]# cat <(ss -pax | head -1 | sed -E 's/(\S+) 
> Address:Port/\1_Address Port /g' ) <( ss -pax | grep nbd) | sed -E 
> 's/\s+$//' | column -t
> Netid  State  Recv-Q  Send-Q  Local_Address  Port Peer_Address  Port  
> Process
>
> [root@fedora bob]# qemu-nbd -c /dev/nbd0  test.qcow2
>
> Active Unix Domain Sockets:
>
> [root@fedora bob]# cat <(ss -pax | head -1 | sed -E 's/(\S+) 
> Address:Port/\1_Address Port /g' ) <( ss -pax | grep nbd) | sed -E 
> 's/\s+$//' | column -t
> Netid  State   Recv-Q  Send-Q  Local_Address Port   Peer_Address  
> Port   Process
> u_str  LISTEN  0       1       /var/lock/qemu-nbd-nbd0 42983  
> *             0 users:(("qemu-nbd",pid=5211,fd=3))
> u_str  ESTAB   0       0       * 45022  *             42988 
> users:(("qemu-nbd",pid=5211,fd=9))
> u_str  ESTAB   0       0       /var/lock/qemu-nbd-nbd0 42988  
> *             45022 users:(("qemu-nbd",pid=5211,fd=10))
>
> [root@fedora fd]# netstat -pnxa | head -2  ; netstat -pnxa | grep nbd
> Active UNIX domain sockets (servers and established)
> Proto RefCnt Flags       Type       State         I-Node PID/Program 
> name     Path
> unix  2      [ ACC ]     STREAM     LISTENING     42983 
> 5211/qemu-nbd        /var/lock/qemu-nbd-nbd0
> unix  3      [ ]         STREAM     CONNECTED     45022 5211/qemu-nbd
> unix  3      [ ]         STREAM     CONNECTED     42988 
> 5211/qemu-nbd        /var/lock/qemu-nbd-nbd0
>
>
> SS        Netstat
> Netid    =    Proto    : u_str = unix stream
> Port    =    I-Node    : socketfs inode
>         Flags    : ACC= Waiting for a connect request
>
> [root@fedora lock]# lsof +E /var/lock/qemu-nbd-nbd0
> lsof: WARNING: can't stat() fuse.gvfsd-fuse file system 
> /run/user/1000/gvfs
>       Output information may be incomplete.
> COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
> qemu-nbd 5211 root    3u  unix 0x00000000b0829fa5      0t0 42983 
> /var/lock/qemu-nbd-nbd0 type=STREAM (LISTEN)
> qemu-nbd 5211 root    9u  unix 0x00000000876da850      0t0 45022 
> type=STREAM ->INO=42988 5211,qemu-nbd,10u (CONNECTED)
> qemu-nbd 5211 root   10u  unix 0x00000000482ad094      0t0 42988 
> /var/lock/qemu-nbd-nbd0 type=STREAM ->INO=45022 5211,qemu-nbd,9u 
> (CONNECTED)
>
> So it looks like 45022 is qemu-nbd acting as a client accessing 
> qemu-nbd as a server on 42988 having requested that connection (from 
> itself) on 42983 (?)
>
> But now I'm more confused - isn't nbd as in "modinfo nbd" = 
> /lib/modules/5.16.20-200.fc35.x86_64/kernel/drivers/block/nbd.ko.xz
> the nbd client in the kernel interfacing with an nbd server to provide 
> fs access via /dev/nbd0 ? or does nbd.ko.xz just talk to/need an nbd 
> client, in this case qemu-nbd (on 45022) ?
>
> And dunno why the netstat RefCnts are 2 and 3 ...
>
> I *would* like to know definitively how this works from an 
> overview/architecture perspective eg.
> (qcow2 file) <data/transport> (qemu-nbd as a server) <data/transport> 
> (qemu-nbd as a client) <data/transport> (kernel module / filesytem 
> interface)
>
> ..but short of an nbd/qemu-nbd developer setting the facts down (or 
> getting into the code myself) I think I'll just have to let it go.

AFAIU (just from looking at the code), qemu-nbd always sets up a Unix 
socket if you specify `-c` (though you can apparently adjust the path if 
you want to):

https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L982
https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L387

And then connects to itself by launching a thread:

https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L1123
https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L275

nbd_init() is what seems to set up the /dev/nbd* device:

https://gitlab.com/qemu-project/qemu/-/blob/master/nbd/client.c#L1244

The most interesting bit to me there is the ioctl(NBD_SET_SOCK, 
sioc->fd) (line 1262).  So apparently qemu-nbd just hands the client 
socket over to the Linux kernel, on which the latter will then issue NBD 
requests for accesses to the /dev/nbd* device.


So with `-c` the set-up is mostly as normal, except qemu-nbd always 
creates a Unix socket; then it launches a thread to connect (as a 
client) to the NBD server it has just set up; and once negotiation is 
complete, it hands this client socket over to the kernel, so the kernel 
then becomes the NBD client.

Maybe I’m wrong, though.

Hanna