TrueNAS Scale 40G 멜라녹스 이더넷 사용법
How to change port mode of ConnectX-3 NIC
Step 1. Determine the PCI ID.
You can do this using lspci
root@truenas:~# lspci | grep Mell
09:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
In the above case, 09:00.0
is the information we are looking for.
Step 2. Determine current port mode.
root@truenas:~# cat /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port1
ib
root@truenas:~# cat /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port2
eth
Using the above command, we can determine port1 is in infiniband mode, and port2 is in ethernet mode.
Step 3. How to change port mode.
There are two ways you can change the port mode. You can either…. use a simple echo command, or, you can use mstflint.
Method 1. Using echo.
root@truenas:~# echo eth > /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port1
-bash: echo: write error: Operation not supported
root@truenas:~# echo ib > /sys/bus/pci/devices/0000\:09\:00.0/mlx4_port1
# Works fine?
IF, this option does not work for you, then try using mstflint. If it does work (no error), reboot for it to take effect.
Method 2. Using mstflint.
apt-get install mstflint
Note- if you use truenas like me… you may need to Re-enable apt-get
Then use mstflint -d {your pci id} q
to query info
root@truenas:~# mstconfig -d 09:00.0 q
Device #1:
----------
Device type: ConnectX3
Device: 09:00.0
Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 8
LINK_TYPE_P1 IB(1)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 False(0)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 False(0)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)
For the LINK_TYPE
fields, IB/1 = infiniband, ETH/2 = ethernet
To change the mode, we can use mstconfig -d 09:00.0 set LINK_TYPE_P1=2
, Just make sure to replace with your PCI device id.
root@truenas:~# mstconfig -d 09:00.0 set LINK_TYPE_P1=2
Device #1:
----------
Device type: ConnectX3
Device: 09:00.0
Configurations: Next Boot New
LINK_TYPE_P1 IB(1) ETH(2)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Per the note- you will need to reboot before this takes effect.
대부분 방법1에서 해결이 완료됩니다. 윈도우에서는 자동모드여도 이더넷 네트워크로 잘 잡히는데 TrueNAS는 네트워크모드로 설정을 해주지 않으면 전혀 작동을 안하기에 위처럼 ETH 모드로 설정해줘야 합니다. 1번 방법에서 끝나면 재부팅은 필요없습니다.