Comparison of Network Roles and Monitor IP Behavior between Ceph and PetaSAN
Bala
18 Posts
October 25, 2025, 10:38 amQuote from Bala on October 25, 2025, 10:38 amHi Team,
In Ceph cluster, we use two networks — one for Public and another for Private communication. The public network is used for front-end communication (client-to-Ceph interactions), while the private network is dedicated to internal Ceph operations, such as OSD-to-OSD data replication and heartbeat communication. When I run the ceph mon dump command, it displays the public network IPs for the monitors. All services like RBD, CephFS, S3, and Kubernetes storage provisioning operate on this same network, which is the expected behavior.
In contrast, in PetaSAN cluster, we use three or more networks — one for Management, used for cluster coordination and web management; one for Backend, which handles internal Ceph communication between OSDs; and one for iSCSI, which is responsible for communication between the PetaSAN storage nodes and the host servers where the client VMs are running. However, when I execute ceph mon dump in PetaSAN, it shows the backend network IPs for the monitors.In PetaSAN, different networks are used for different services — for example, RBD (block storage) uses the iSCSI network for front-end communication, S3 uses a separate network, NFS runs on another, and CephFS/Kubernetes storage provisioning use the backend network
Whether it is possible to use the same network for all services in PetaSAN? If so, the performance will degrade?
Hi Team,
In Ceph cluster, we use two networks — one for Public and another for Private communication. The public network is used for front-end communication (client-to-Ceph interactions), while the private network is dedicated to internal Ceph operations, such as OSD-to-OSD data replication and heartbeat communication. When I run the ceph mon dump command, it displays the public network IPs for the monitors. All services like RBD, CephFS, S3, and Kubernetes storage provisioning operate on this same network, which is the expected behavior.
In contrast, in PetaSAN cluster, we use three or more networks — one for Management, used for cluster coordination and web management; one for Backend, which handles internal Ceph communication between OSDs; and one for iSCSI, which is responsible for communication between the PetaSAN storage nodes and the host servers where the client VMs are running. However, when I execute ceph mon dump in PetaSAN, it shows the backend network IPs for the monitors.In PetaSAN, different networks are used for different services — for example, RBD (block storage) uses the iSCSI network for front-end communication, S3 uses a separate network, NFS runs on another, and CephFS/Kubernetes storage provisioning use the backend network
Whether it is possible to use the same network for all services in PetaSAN? If so, the performance will degrade?
admin
3,054 Posts
November 5, 2025, 2:34 pmQuote from admin on November 5, 2025, 2:34 pmin ceph the public network is where osds. mons, clients communicate
the optional private network is just for osd to osd replication.
most recent recommendations is to use just the public network
https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/
PetaSAN used to support configuring both newtwoks via the UI in older versions, named Backend1 (ceph public) and Backend2 (ceph private) now the UI just uses a single network by default for ceph named Backend (ceph public). However if you still need to use a separate ceph private network in PetaSAN you could configure manually in the config files.
For services, we recommend each on a separate network, they can still use the same interface.
You can however use the same network/subnet but make sure the To and From from each service do not overlap. There are some exceptions: iSCSI MPIO does require iSCSI 1 and iSCSI 2 be on separate networks.
Also Management network typically can be routed and have external access to internet for online updates, remoet emanagement and possibly ntp service, for security you do not want this network to be the same as your osds, mons..
in ceph the public network is where osds. mons, clients communicate
the optional private network is just for osd to osd replication.
most recent recommendations is to use just the public network
https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/
PetaSAN used to support configuring both newtwoks via the UI in older versions, named Backend1 (ceph public) and Backend2 (ceph private) now the UI just uses a single network by default for ceph named Backend (ceph public). However if you still need to use a separate ceph private network in PetaSAN you could configure manually in the config files.
For services, we recommend each on a separate network, they can still use the same interface.
You can however use the same network/subnet but make sure the To and From from each service do not overlap. There are some exceptions: iSCSI MPIO does require iSCSI 1 and iSCSI 2 be on separate networks.
Also Management network typically can be routed and have external access to internet for online updates, remoet emanagement and possibly ntp service, for security you do not want this network to be the same as your osds, mons..
Comparison of Network Roles and Monitor IP Behavior between Ceph and PetaSAN
Bala
18 Posts
Quote from Bala on October 25, 2025, 10:38 amHi Team,
In Ceph cluster, we use two networks — one for Public and another for Private communication. The public network is used for front-end communication (client-to-Ceph interactions), while the private network is dedicated to internal Ceph operations, such as OSD-to-OSD data replication and heartbeat communication. When I run the
ceph mon dumpcommand, it displays the public network IPs for the monitors. All services like RBD, CephFS, S3, and Kubernetes storage provisioning operate on this same network, which is the expected behavior.In contrast, in PetaSAN cluster, we use three or more networks — one for Management, used for cluster coordination and web management; one for Backend, which handles internal Ceph communication between OSDs; and one for iSCSI, which is responsible for communication between the PetaSAN storage nodes and the host servers where the client VMs are running. However, when I execute
ceph mon dumpin PetaSAN, it shows the backend network IPs for the monitors.In PetaSAN, different networks are used for different services — for example, RBD (block storage) uses the iSCSI network for front-end communication, S3 uses a separate network, NFS runs on another, and CephFS/Kubernetes storage provisioning use the backend networkWhether it is possible to use the same network for all services in PetaSAN? If so, the performance will degrade?
Hi Team,
In Ceph cluster, we use two networks — one for Public and another for Private communication. The public network is used for front-end communication (client-to-Ceph interactions), while the private network is dedicated to internal Ceph operations, such as OSD-to-OSD data replication and heartbeat communication. When I run the ceph mon dump command, it displays the public network IPs for the monitors. All services like RBD, CephFS, S3, and Kubernetes storage provisioning operate on this same network, which is the expected behavior.
In contrast, in PetaSAN cluster, we use three or more networks — one for Management, used for cluster coordination and web management; one for Backend, which handles internal Ceph communication between OSDs; and one for iSCSI, which is responsible for communication between the PetaSAN storage nodes and the host servers where the client VMs are running. However, when I execute ceph mon dump in PetaSAN, it shows the backend network IPs for the monitors.In PetaSAN, different networks are used for different services — for example, RBD (block storage) uses the iSCSI network for front-end communication, S3 uses a separate network, NFS runs on another, and CephFS/Kubernetes storage provisioning use the backend network
Whether it is possible to use the same network for all services in PetaSAN? If so, the performance will degrade?
admin
3,054 Posts
Quote from admin on November 5, 2025, 2:34 pmin ceph the public network is where osds. mons, clients communicate
the optional private network is just for osd to osd replication.
most recent recommendations is to use just the public network
https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/
PetaSAN used to support configuring both newtwoks via the UI in older versions, named Backend1 (ceph public) and Backend2 (ceph private) now the UI just uses a single network by default for ceph named Backend (ceph public). However if you still need to use a separate ceph private network in PetaSAN you could configure manually in the config files.For services, we recommend each on a separate network, they can still use the same interface.
You can however use the same network/subnet but make sure the To and From from each service do not overlap. There are some exceptions: iSCSI MPIO does require iSCSI 1 and iSCSI 2 be on separate networks.
Also Management network typically can be routed and have external access to internet for online updates, remoet emanagement and possibly ntp service, for security you do not want this network to be the same as your osds, mons..
in ceph the public network is where osds. mons, clients communicate
the optional private network is just for osd to osd replication.
most recent recommendations is to use just the public network
https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/
PetaSAN used to support configuring both newtwoks via the UI in older versions, named Backend1 (ceph public) and Backend2 (ceph private) now the UI just uses a single network by default for ceph named Backend (ceph public). However if you still need to use a separate ceph private network in PetaSAN you could configure manually in the config files.
For services, we recommend each on a separate network, they can still use the same interface.
You can however use the same network/subnet but make sure the To and From from each service do not overlap. There are some exceptions: iSCSI MPIO does require iSCSI 1 and iSCSI 2 be on separate networks.
Also Management network typically can be routed and have external access to internet for online updates, remoet emanagement and possibly ntp service, for security you do not want this network to be the same as your osds, mons..