openEuler的nfs+

点击这里查看配套的教学视频。

点击跳转到nfs课程所有目录。

1 多个网卡环境

请查看《内核开发环境》

2 openeuler nfs+的使用

可以使用脚本create-enfs-patchset.sh生成完整的补丁文件，再打上我修改的补丁。切换到openEuler-1.0-LTS分支，编译前打开配置CONFIG_ENFS，可能还要关闭配置CONFIG_NET_VENDOR_NETRONOME。

挂载:

modprobe enfs # 经过我的修改已经能自动加载了
mount -t nfs -o localaddrs=192.168.53.40~192.168.53.53,remoteaddrs=192.168.53.215~192.168.53.216 192.168.53.216:/s_test /mnt/

如果没有创建/etc/enfs/config.ini，会报错failed to open file:/etc/enfs/config.ini err:-2，配置文件请参考eNFS 使用指南。只需要在nfs client端支持enfs就可以，/etc/enfs/config.ini默认配置如下:

path_detect_interval=10 # 路径连通探测周期，单位 ： 秒
path_detect_timeout=10 # 路径连通探测消息越野时间，单位 ： 秒
multipath_timeout=0 # 选择其他路径达到的文件操作的超时阈值，0表示使用 mount 命令指定的 timeo 参数，不使用 eNFS 模块的配置，单位 ： 秒。
multipath_disable=0 # 启用 eNFS 特性

除了mount命令查看之外，还可以用以下方式:

cat /proc/enfs/192.168.53.216_0/path
cat /proc/enfs/192.168.53.216_0/stat

3 我修改的nfs+补丁

我修改的补丁请查看这里。

// struct enfs_adapter_ops->owner 的引用计数参考
struct nfs_client
  struct nfs_subversion * cl_nfs_mod
    struct module *owner

// nfs_multipath_router_get 修改参考
get_nfs_version
  request_module

4 nfs+代码分析

pull request和补丁文件。

4.1 1/6 nfs: add api to support enfs registe and handle mount option

At the NFS layer, the eNFS registration function is called back when
the mount command parses parameters. The eNFS parses and saves the IP
address list entered by users.

这个补丁实现了nfs层的enfs的接口，下面的代码流程是我看代码时的笔记:

struct nfs_client_initdata
  void *enfs_option; /* struct multipath_mount_options * */

struct nfs_parsed_mount_data
  void *enfs_option; /* struct multipath_mount_options * */ 

struct nfs_client
  /* multi path private structure (struct multipath_client_info *) */
  void *cl_multipath_data;

struct enfs_adapter_ops

nfs4_create_server
  nfs4_init_server
    enfs_option = data->enfs_option
    nfs4_set_client
      .enfs_option = enfs_option,
      nfs_get_client
        nfs_match_client
          nfs_multipath_client_match
        nfs4_alloc_client
          nfs_create_multi_path_client
            nfs_multipath_router_get
              request_module("enfs")
              try_module_get(ops->owner) // 引用计数直到umount时才能释放
            nfs_multipath_client_info_init
          nfs_create_rpc_client
            .multipath_option = cl_init->enfs_option,

nfs4_free_client
  nfs_free_client
    nfs_free_multi_path_client
      nfs_multipath_router_put // 释放nfs_create_multi_path_client中一直持有的引用计数

nfs_parse_mount_options
  enfs_check_mount_parse_info
    enfs_parse_mount_options
      nfs_multipath_parse_options // parse_mount_options
        nfs_multipath_parse_ip_list
          nfs_multipath_parse_ip_list_inter

4.2 2/6 sunrpc: add api to support enfs registe and create multipath then dispatch IO

At the sunrpc layer, the eNFS registration function is called back When
the NFS uses sunrpc to create rpc_clnt, the eNFS combines the IP address
list entered for mount to generate multiple xprts. When the I/O times
out, the callback function of the eNFS is called back so that the eNFS
switches to an available link for retry.
// The high-level client handle
struct rpc_clnt
  bool cl_enfs

struct rpc_create_args
  // 这里使用了nfs层的结构体，耦合了
  void *multipath_option // struct multipath_mount_options

struct rpc_task
  unsigned long           tk_major_timeo

// RPC task flags
#define RPC_TASK_FIXED  0x0004          /* detect xprt status task */

struct rpc_multipath_ops

struct rpc_xprt
  atomic_long_t   queuelen;
  void *multipath_context;

struct rpc_xprt_switch
  unsigned int            xps_nactive;
  atomic_long_t           xps_queuelen;
  unsigned long           xps_tmp_time;

// 挂载
nfs4_alloc_client
  nfs_create_rpc_client
    rpc_create
      rpc_create_xprt
        rpc_multipath_ops_create_clnt

// 卸载
rpc_shutdown_client
  rpc_multipath_ops_releas_clnt

rpc_task_release_client / nfs4_async_handle_exception
  rpc_task_release_transport // 这里改成和主线一样
    rpc_task_release_xprt // 从主线搬运过来的

rpc_task_set_transport
  rpc_task_get_next_xprt // 从主线搬运过来的
    rpc_task_get_xprt // 从主线搬运过来的

call_reserveresult
  rpc_multipath_ops_task_need_call_start_again

call_transmit
  rpc_multipath_ops_prepare_transmit

call_timeout
  rpc_multipath_ops_failover_handle

rpc_clnt_add_xprt
  rpc_xprt_switch_set_roundrobin

rpc_init_task
  rpc_task_get_xprt // 和主线一样

4.3 3/6 nfs: add enfs module for nfs mount option

The eNFS module registers the interface for parsing the mount command.
During the mount process, the NFS invokes the eNFS interface to enable
the eNFS to parse the mounting parameters of UltraPath. The eNFS module
saves the mounting parameters to the context of nfs_client.

4.4 4/6 nfs: add enfs module for sunrpc multipatch

When the NFS invokes the SunRPC to create rpc_clnt, the eNFS interface
is called back. The eNFS creates multiple xprts based on the output IP
address list. When NFS V3 I/Os are delivered, eNFS distributes I/Os to
available links based on the link status, improving performance through
load balancing.

4.5 5/6 nfs: add enfs module for sunrpc failover and configure

When sending I/Os from the SunRPC module to the NFS server times out,
the SunRPC module calls back the eNFS module to reselect a link. The
eNFS module distributes I/Os to other available links, preventing
service interruption caused by a single link failure.

4.6 6/6 nfs, sunrpc: add enfs compile option

The eNFS compilation option and makefile are added. By default, the eNFS
compilation is performed.