Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

iSCSI Path Assignment & Reassignment Limitations – Broken or By Design Across Versions?

Pages: 1 2

matt
8 Posts
June 23, 2026, 7:19 pm

We’re currently running a production PetaSAN environment consisting of:

  • 3 Management Nodes (OSDs)
  • 2 Storage Nodes (iSCSI Gateways)

We have tested across multiple versions:

  • 3.0.0
  • 3.3.0
  • 4.0.0 (current)
  • Evaluating 4.1.0

Key Issues Observed

1. Path Assignment List Shows No Data

Under Manage iSCSI → Path Assignment List:

  • Both nodes always show 0 assigned paths
  • No visibility into actual path distribution
  • The UI only exposes a “Reassign Paths” button

This makes it impossible to audit current path placement.

2. Manual Reassign Appears Non-Functional

Under Reassign Paths → Manual:

  • Searching by:
    • Disk ID
    • Disk Name
    • IP Address

Results in no action or feedback.
It appears either broken or not implemented correctly, as it never returns any usable results.

3. Automatic Reassign Works… But Not Correctly

Under Reassign Paths → Automatic:

  • The “Path Count” option does perform changes
  • Behavior appears to:
    • Remove/add paths one at a time
    • Do so relatively gracefully (no immediate disruption observed)

However:

✅ It adjusts quantity
❌ It does NOT enforce balanced distribution

Critical Problem: No Guaranteed Path Balance

After automatic reassignment, path distribution is inconsistent:

Examples observed:

  • Some disks: 100% of paths on a single node
  • Others: 75/25 split
  • Others: 50/50 (expected)

There is no way in the WebUI to:

  • Enforce a true 50/50 distribution
  • Pin paths to specific nodes
  • Move one path at a time
  • Drain paths off a node for maintenance

Operational Impact

This creates major problems during maintenance:

  • Cannot gracefully drain iSCSI paths from a node
  • Cannot ensure safe failover before rebooting a node
  • Risk of:
    • All active paths residing on a single node
    • Storage interruption when that node goes down
  • No post-assignment way to correct imbalance via UI

What We Expected

From a storage platform, we would expect:

  • Visibility of current path placement
  • Ability to:
    • Move individual paths
    • Force rebalance (true 50/50)
    • Pin paths to gateways
  • A graceful “drain node” / maintenance mode
  • Deterministic behavior from “automatic” assignment

Questions to the Community / PetaSAN Team

  1. Is this a known limitation or a bug?
  2. Has this been addressed in 4.1.0 or newer builds?
  3. Is there a CLI or backend method to:
    • Control path placement
    • Safely drain a node before maintenance?
  4. Is “automatic” assignment expected to be non-deterministic?
#1

matt
8 Posts
June 23, 2026, 7:24 pm

I will clarify i can see path assignment when looking at the "iSCSI Disks" Page and the "Active Paths" section and there you can see how its not 50/50 and we need to be able re assign gracefully in the state it is in now. Then would be nice to gracefully fail all over to 1 node for maintenance 1 path at a time basically. I am going to start looking through the python modules and code for pages to see where these functions live.

#2

admin
3,082 Posts
June 24, 2026, 4:02 pm

Ofcourse it should not behave the way you describe.

Are you see-ing any errors ? is the cluster health OK in dashboard ?

Are you using real hardware or virtual ? do you have decent hardware ?

Can you show output of

consul members

consul kv get -recurse PetaSAN/Disks

Can you show screen shot of path assignment page showing empty list

#3

matt
8 Posts
June 28, 2026, 7:26 pm
  • Intent
    • Using as a iSCSI target portal and gateway etc.. to various hypervisors like VMware ESXi
  • No errors that I see in the webUI.
  • The cluster is OK in the dashboard and healthy.
  • We are using physical hardware and are decent.
    • Management Nodes (3 nodes)
      • OSDs are all 2.5 SATA SSDs
      • Bonded dedicated NICs
        • eth2, eth3, eth4 and eth5 (25G Link SFP Fiber)
        • Ethernet Channel Bonding Driver: v5.14.21-08-petasan

          Bonding Mode: IEEE 802.3ad Dynamic link aggregation
          Transmit Hash Policy: layer2+3 (2)
          MII Status: up
          MII Polling Interval (ms): 100
          Up Delay (ms): 0
          Down Delay (ms): 0
          Peer Notification Delay (ms): 0

    • Storage nodes (2 nodes)
      • No OSDs as these are dedicated iSCSI Gateways basically
      • Bonded dedicated NICs
        • eth2 and eth3 (25G Link SFP Fiber)
          • Ethernet Channel Bonding Driver: v6.12.0-03-petasan

            Bonding Mode: IEEE 802.3ad Dynamic link aggregation
            Transmit Hash Policy: layer2+3 (2)
            MII Status: up
            MII Polling Interval (ms): 100
            Up Delay (ms): 0
            Down Delay (ms): 0
            Peer Notification Delay (ms): 0

    • root@san02064:~# consul members
      Node Address Status Type Build Protocol DC Segment
      san02061.my.company.net 10.21.198.134:8301 alive server 1.5.2 2 petasan <all>
      san02062.my.company.net 10.21.198.135:8301 alive server 1.5.2 2 petasan <all>
      san02063.my.company.net 10.21.198.136:8301 alive server 1.5.2 2 petasan <all>
      san02064.my.company.net 10.21.198.132:8301 alive client 1.5.2 2 petasan <default>
      san02065.my.company.net 10.21.198.133:8301 alive client 1.5.2 2 petasan <default>
      root@san02064:~#
      root@san02064:~# consul kv get -recurse PetaSAN/Disks
      PetaSAN/Disks/00009:disk
      PetaSAN/Disks/00009/1:san02065
      PetaSAN/Disks/00009/2:san02064
      PetaSAN/Disks/00022:disk
      PetaSAN/Disks/00022/1:san02064.my.company.net
      PetaSAN/Disks/00022/2:san02065.my.company.net
      PetaSAN/Disks/00022/3:san02064.my.company.net
      PetaSAN/Disks/00022/4:san02064.my.company.net
      PetaSAN/Disks/00025:disk
      PetaSAN/Disks/00025/1:san02064.my.company.net
      PetaSAN/Disks/00025/2:san02065.my.company.net
      PetaSAN/Disks/00026:disk
      PetaSAN/Disks/00026/1:san02065.my.company.net
      PetaSAN/Disks/00026/2:san02065.my.company.net
      PetaSAN/Disks/00027:disk
      PetaSAN/Disks/00027/1:san02064.my.company.net
      PetaSAN/Disks/00027/2:san02065.my.company.net
      PetaSAN/Disks/00027/3:san02064.my.company.net
      PetaSAN/Disks/00027/4:san02064.my.company.net
      PetaSAN/Disks/00028:disk
      PetaSAN/Disks/00028/1:san02065.my.company.net
      PetaSAN/Disks/00028/2:san02064.my.company.net
      PetaSAN/Disks/00029:disk
      PetaSAN/Disks/00029/1:san02064.my.company.net
      PetaSAN/Disks/00029/2:san02065.my.company.net
      PetaSAN/Disks/00030:disk
      PetaSAN/Disks/00030/1:san02065.my.company.net
      PetaSAN/Disks/00030/2:san02064.my.company.net
      PetaSAN/Disks/00032:disk
      PetaSAN/Disks/00032/1:san02065.my.company.net
      PetaSAN/Disks/00032/2:san02065.my.company.net
      PetaSAN/Disks/00033:disk
      PetaSAN/Disks/00033/1:san02065.my.company.net
      PetaSAN/Disks/00033/2:san02064.my.company.net
      PetaSAN/Disks/00035:disk
      PetaSAN/Disks/00035/1:san02064.my.company.net
      PetaSAN/Disks/00035/2:san02065.my.company.net
      root@san02064:~#

 

not sure how to past or upload an image on this forum yet, but basically you see the two Storage nodes (san02064 and san02065 in this env) and they say 0 paths.

#4

matt
8 Posts
June 29, 2026, 2:44 pm

Another observation though is that originally these didn't have my.company.net appended to the hostnames when we set it up and might have to do with the webUI reading it wrong or something of conflicted as that first disk doesn't have it appended. Maybe unrelated but I do recall doing that years ago. When you do automaitc reassing it doesn't even the counts just not as expected per disk to be 50/50 as you can see so if the manual reassignment atleast worked we could do that and use that to failover 1 path at a time but other rebalance after maintenance or an event if that makes sense. I have no problem doing cmd lines either to rebalance as needed vs the webUI

#5

admin
3,082 Posts
June 29, 2026, 8:11 pm

I think the issue is with using fully qualified names (hostname + domain) instead of hostname. my understanding we do not allow using this in the iso installer. Have you manually modified the hostnames? or maybe there is a reverse DNS that resolves these names from ip addresses. you should be able to fix it from consul, hostnamectl, /etc/hosts as well as config files in /opt/petasan/config + remove any reverse DNS if you have. Best try to simulate it in a vm setup before applying on prduction.

#6

matt
8 Posts
June 30, 2026, 2:56 am

I do see a bunch of commands from a few years back where you can see the user was attempting to append the FQDN and grepping in various configs until it worked also with an open API call to below etc...

wget --method=DELETE -O /tmp/delete_test.json http://127.0.0.1:8500/v1/kv/PetaSAN/Nodes/san02062.my.company.net
wget --method=DELETE -O /tmp/delete_test.json http://127.0.0.1:8500/v1/kv/PetaSAN/Nodes/san02062

So I would agree this is likely the root cause for the webUI not working correctly on certain sections. I will recreate in a VM environment and see what the impact is as this is in production. We may offload the entire platform and adjust or reinstall as well.

 

Files I've seen modified are the following and would make sense. I used vim below but i see sed replacement statements for the files basically.

vim /etc/hosts

hostnamectl set-hostname san02062.my.company.net

systemctl stop petasan-file-sync

/opt/petasan/scripts/util/sync_file.py /etc/hosts

systemctl start petasan-file-sync

vim /opt/petasan/config/node_info.json

vim /opt/petasan/config/cluster_info.json

 

Would you be able to recommend a safe path to adjust these on a single node in order to fix it sync and then move to the next node? I'll work on setting up a VM lab in a similar senario over the next few days.

#7

admin
3,082 Posts
June 30, 2026, 1:23 pm

good you found it. If you wish, consider buying support so we can put time into this.  else make sure you replicate it in a vm test environment first.

#8

matt
8 Posts
June 30, 2026, 11:55 pm

I actually tried buy support the other week before we started doing upgrades but the popup is busted for me so I figured it was just up for the community forums. Can you check into that because we don't mind buying support for the production cluster and would like some help/support on fine tuning some things and issues we've faced with ESXi 7.X and iSCSI behavior and XCPNG on the horizon via iSCSI/NvmeOF.

#9

matt
8 Posts
July 7, 2026, 6:06 pm

Let me know if you can resolve why the popup to purchase support doesn't work or if there is another method to contact your team to get support going as i've tried other browsers or looking at the code on the page to see if i could get the link but I'd like to purchase support!

#10
Pages: 1 2