PetaSAN v3.3.0 Kernel command line cstate settings
Pages: 1 2
bbu
7 Posts
June 5, 2025, 10:49 pmQuote from bbu on June 5, 2025, 10:49 pmHello there
PetaSAN is a really great solution! Thank you very much for all the efforts put into it!
After playing with PetaSAN in a virtual environment we moved to 3 physical servers. It works very well. The "problem" is, these are re-purposed servers that have 2 rather powerful CPU built in. They are most of the time 100% idle, since we do not have that many transactions and no EC pools. Most of the work is done in HW and it's associated offloading mechanisms.
But the environment and our electricity bill is not that happy about the CPU's running at full speed. We noticed that the CPU's run at 100% clock speed all the time. We verified the BIOS settings, tried to lower the frequency of some cores and so on to no avail. It turned out, PetaSAN provides some kernel settings preventing any power saving mechanisms to kick in:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrier
where the relevant settings are:
intel_idle.max_cstate=1
processor.max_cstate=1
Although I can understand the idea behind this choice (no latency due to frequency / efficiency scaling), energy/environment/cost wise, this is not that optimal, especially if there is a big overcommitment of CPU resources.
So my question is: what is the reason the creator of PetaSAN choose these settings? What happens if higher cstates are allowed?
For us, if the stability is not endangered, we might either run at higher cstates, configure a few cores to be on 100% frequency all the time and others to go to powersafe.
If the cstates cannot be altered, the only option would be to remove one of the physical CPU, but this then get's complex on which of the PCI slots are usable and which are not.
Thank you
Hello there
PetaSAN is a really great solution! Thank you very much for all the efforts put into it!
After playing with PetaSAN in a virtual environment we moved to 3 physical servers. It works very well. The "problem" is, these are re-purposed servers that have 2 rather powerful CPU built in. They are most of the time 100% idle, since we do not have that many transactions and no EC pools. Most of the work is done in HW and it's associated offloading mechanisms.
But the environment and our electricity bill is not that happy about the CPU's running at full speed. We noticed that the CPU's run at 100% clock speed all the time. We verified the BIOS settings, tried to lower the frequency of some cores and so on to no avail. It turned out, PetaSAN provides some kernel settings preventing any power saving mechanisms to kick in:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrier
where the relevant settings are:
intel_idle.max_cstate=1
processor.max_cstate=1
Although I can understand the idea behind this choice (no latency due to frequency / efficiency scaling), energy/environment/cost wise, this is not that optimal, especially if there is a big overcommitment of CPU resources.
So my question is: what is the reason the creator of PetaSAN choose these settings? What happens if higher cstates are allowed?
For us, if the stability is not endangered, we might either run at higher cstates, configure a few cores to be on 100% frequency all the time and others to go to powersafe.
If the cstates cannot be altered, the only option would be to remove one of the physical CPU, but this then get's complex on which of the PCI slots are usable and which are not.
Thank you
admin
3,054 Posts
June 6, 2025, 9:56 amQuote from admin on June 6, 2025, 9:56 amThank you for your nice feedback, it is a lot of work under the hood to make things work 🙂
This cpu setting does make a significant difference in latency, almost 40% better. So performance of a single thread is much improved. It is one of the top OS/kernel tuning parameter by Ceph. There are many online reference to this in the Ceph community.
Ofcourse you can change this if you really want:
In PetaSAN 4.0 you can do so via the UI under Performance Profiles...you can create your own profiles and apply them centrally ant any time. It could be one reason to upgrade.
I you do not upgrade, you can do it manually..on ALL nodes, edit:
/opt/petasan/config/tuning/current/post_deploy_script
and remove the cpu settings, then reboot ALL nodes.
Thank you for your nice feedback, it is a lot of work under the hood to make things work 🙂
This cpu setting does make a significant difference in latency, almost 40% better. So performance of a single thread is much improved. It is one of the top OS/kernel tuning parameter by Ceph. There are many online reference to this in the Ceph community.
Ofcourse you can change this if you really want:
In PetaSAN 4.0 you can do so via the UI under Performance Profiles...you can create your own profiles and apply them centrally ant any time. It could be one reason to upgrade.
I you do not upgrade, you can do it manually..on ALL nodes, edit:
/opt/petasan/config/tuning/current/post_deploy_script
and remove the cpu settings, then reboot ALL nodes.
Last edited on June 6, 2025, 9:57 am by admin · #2
bbu
7 Posts
June 9, 2025, 9:31 amQuote from bbu on June 9, 2025, 9:31 am... and constantly changing, security updates and so on. E.g. ongoing work and it is well maintained as version 4 just arrived, not only kept alive. Looking at the upgrade guide, the path from 3.3.0 to 4.0.0 appears to be straight forward.
Note: it is a bit strange to read section 7 "Upgrading from PetaSAN 4.X and above" as 4.0.0 just hit the public ...
I indeed did not research the cstate settings related to Ceph, as PetaSAN might have had it's own reasons to do this, as it is more than only Ceph. But yes, there are many references to this.
Does PetaSAN v4 implement these profiles via Tuned?
If you already planned for this in v4, I trust this is the best / long shot way to address this. So we will do the upgrade, see how much energy can be conserved and how the latency is affected (if this is easy to measure), if this does not work, we will remove one of the physical CPU.
BTW: my plan to only run a few cores at 100% and others with less, makes it too complex, as processes like OSD would need to be pinned to these CPU's, plus other services I'm not aware of. If a service is restarted, pinning would need to happen again. So I just dumped that idea ..
Thank you
... and constantly changing, security updates and so on. E.g. ongoing work and it is well maintained as version 4 just arrived, not only kept alive. Looking at the upgrade guide, the path from 3.3.0 to 4.0.0 appears to be straight forward.
Note: it is a bit strange to read section 7 "Upgrading from PetaSAN 4.X and above" as 4.0.0 just hit the public ...
I indeed did not research the cstate settings related to Ceph, as PetaSAN might have had it's own reasons to do this, as it is more than only Ceph. But yes, there are many references to this.
Does PetaSAN v4 implement these profiles via Tuned?
If you already planned for this in v4, I trust this is the best / long shot way to address this. So we will do the upgrade, see how much energy can be conserved and how the latency is affected (if this is easy to measure), if this does not work, we will remove one of the physical CPU.
BTW: my plan to only run a few cores at 100% and others with less, makes it too complex, as processes like OSD would need to be pinned to these CPU's, plus other services I'm not aware of. If a service is restarted, pinning would need to happen again. So I just dumped that idea ..
Thank you
bbu
7 Posts
June 13, 2025, 9:08 amQuote from bbu on June 13, 2025, 9:08 amWe did the update to 4.0.0, which was a smooth transition. We then tried to apply the performance profile.
Not sure what need to be done to apply the changed settings, as "Apply" does not update / rebuild grub in this case.
BUT: after the upgrade from 3.3.0 to 4.0.0, we did not restart the nodes. To see if the new settings of the performance profile were applied, we rebooted one of the nodes. This ends in the GRUB> prompt and does no longer start up the OS.
Using set prefix= and issue the "normal" command, we were able to launch the grub menu which in turn allows normal boot of the node ...
What is the "official" procedure to fix this condition?
We did the update to 4.0.0, which was a smooth transition. We then tried to apply the performance profile.
Not sure what need to be done to apply the changed settings, as "Apply" does not update / rebuild grub in this case.
BUT: after the upgrade from 3.3.0 to 4.0.0, we did not restart the nodes. To see if the new settings of the performance profile were applied, we rebooted one of the nodes. This ends in the GRUB> prompt and does no longer start up the OS.
Using set prefix= and issue the "normal" command, we were able to launch the grub menu which in turn allows normal boot of the node ...
What is the "official" procedure to fix this condition?
admin
3,054 Posts
June 13, 2025, 11:07 pmQuote from admin on June 13, 2025, 11:07 pmThe Performance Profiles run the different scripts defined in the profile. Nearly all do not require a reboot (at least in the PetaSAN default profile). The cpu speed / c-states settings do require reboots, and can be applied/changes while running (you would need to setup you own script/profile and change the parameters passed to cpupower command line). So you can have a low and high perf. profiles apply profiles at runtime without reboots.
As for the system did not start after a reboot, and you had to change the grub prefix...no that is not the official way 🙂 and ofcourse it should work without issue. Maybe you had some problems in your upgrade. If you can reproduce this using a different environment or setup that we can reproduce, please report this (preferably in the bug topic) and we will try to reproduce it.
The Performance Profiles run the different scripts defined in the profile. Nearly all do not require a reboot (at least in the PetaSAN default profile). The cpu speed / c-states settings do require reboots, and can be applied/changes while running (you would need to setup you own script/profile and change the parameters passed to cpupower command line). So you can have a low and high perf. profiles apply profiles at runtime without reboots.
As for the system did not start after a reboot, and you had to change the grub prefix...no that is not the official way 🙂 and ofcourse it should work without issue. Maybe you had some problems in your upgrade. If you can reproduce this using a different environment or setup that we can reproduce, please report this (preferably in the bug topic) and we will try to reproduce it.
Last edited on June 16, 2025, 9:50 am by admin · #5
bbu
7 Posts
June 16, 2025, 6:56 amQuote from bbu on June 16, 2025, 6:56 amGreat, but for the kernel changes to take effect, grub would need to be rebuilt, right? Would PetaSAN do that or would we need to do something to remove the cstate from the kernel?
The upgrade was successful on all the nodes:
PetaSAN version: 4.0.0
PetaSAN updated to version 4.0.0 successfully.
The console output did not mention any special errors or warnings.
All nodes experience the same issue and boot to the grub prompt.
We did not alter PetaSAN yet, we just manually put the commands to the grub prompt. How do we alter grub the PetaSAN way, e.g. make the prefix permanent?
Great, but for the kernel changes to take effect, grub would need to be rebuilt, right? Would PetaSAN do that or would we need to do something to remove the cstate from the kernel?
The upgrade was successful on all the nodes:
PetaSAN version: 4.0.0
PetaSAN updated to version 4.0.0 successfully.
The console output did not mention any special errors or warnings.
All nodes experience the same issue and boot to the grub prompt.
We did not alter PetaSAN yet, we just manually put the commands to the grub prompt. How do we alter grub the PetaSAN way, e.g. make the prefix permanent?
Last edited on June 16, 2025, 6:57 am by bbu · #6
admin
3,054 Posts
June 16, 2025, 9:49 amQuote from admin on June 16, 2025, 9:49 amThe changes are applied dynamically at runtime. no kernel rebuild/grub changes required. When you apply/change a performance profile in the UI, it will send the profile to all nodes and run the scripts associated with profile, you can change profile dynamically at runtime. The scripts have a section related to cpu tuning (it should be clear in comments), as per prev post it relies on cpupower tool as well as some other echo commands to /sys to set cpu performance governor and frequency. You need to create a new profile or edit existing one and tune these cpu parameters to reduce performance.
Regarding the grub boot issue, please see my prev post.
The changes are applied dynamically at runtime. no kernel rebuild/grub changes required. When you apply/change a performance profile in the UI, it will send the profile to all nodes and run the scripts associated with profile, you can change profile dynamically at runtime. The scripts have a section related to cpu tuning (it should be clear in comments), as per prev post it relies on cpupower tool as well as some other echo commands to /sys to set cpu performance governor and frequency. You need to create a new profile or edit existing one and tune these cpu parameters to reduce performance.
Regarding the grub boot issue, please see my prev post.
Last edited on June 16, 2025, 9:50 am by admin · #7
bbu
7 Posts
June 16, 2025, 12:05 pmQuote from bbu on June 16, 2025, 12:05 pmThat will not work, as long as the kernel parameters are like this:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrier
This is the running kernel cfg after the upgrade, and the kernel parameters intel_idle and processormax_cstate will prevent any changes to the idle states / performance profile from the OS... that's why I pointed it out.
Regarding the boot issue: no, we do not have any other environment to play with at the moment. The test environment (all virtual) was dismantled once wthe hardware based one was built.
Where should we change the prefix in PetaSAN? Simply run update-grub and grub-install any try again? Don't know where I could provide the prefix in the grub.conf ... but I'm not into grub at all ...
That will not work, as long as the kernel parameters are like this:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrier
This is the running kernel cfg after the upgrade, and the kernel parameters intel_idle and processormax_cstate will prevent any changes to the idle states / performance profile from the OS... that's why I pointed it out.
Regarding the boot issue: no, we do not have any other environment to play with at the moment. The test environment (all virtual) was dismantled once wthe hardware based one was built.
Where should we change the prefix in PetaSAN? Simply run update-grub and grub-install any try again? Don't know where I could provide the prefix in the grub.conf ... but I'm not into grub at all ...
admin
3,054 Posts
June 16, 2025, 10:05 pmQuote from admin on June 16, 2025, 10:05 pmI do not know why you had to change the grub prefix, i also do not know how to make your changes persist. It is not something we see, so unfortunately i cannot help much. You can try looking in the grub and ubuntu forums.
I do not know why you had to change the grub prefix, i also do not know how to make your changes persist. It is not something we see, so unfortunately i cannot help much. You can try looking in the grub and ubuntu forums.
bbu
7 Posts
June 17, 2025, 9:30 amQuote from bbu on June 17, 2025, 9:30 amOK, I did some digging into the grub rabbithole. The issue appears to be, that the grub config points the prefix to /EFI/ubuntu, but the PetaSAN distribution has the cfg and efi in /EFI/petasan.
As a proof of concept, we copied the petasan directory to ubuntu, which restored the boot capability of the distribution. Now, the options are:
Keep the efi and cfg in the ubuntu directory of the boot partition or alter the configuration to point to /EFI/petasan
Using a copy of the petasan directory might miss updates, as I assume the PetaSAN distro will update in /EFI/petasan. E.g. this option is less preferable.
Since I do not know how this is done in PetaSAN and you obviously changed the way it works (given the petasan directory name in the EFI directory of the boot partition), I prefer the way to fix the grub/EFI /whatever cfg to point grub prefix to look in /EFI/petasan instead of the /EFI/ubuntu
OK, I did some digging into the grub rabbithole. The issue appears to be, that the grub config points the prefix to /EFI/ubuntu, but the PetaSAN distribution has the cfg and efi in /EFI/petasan.
As a proof of concept, we copied the petasan directory to ubuntu, which restored the boot capability of the distribution. Now, the options are:
Keep the efi and cfg in the ubuntu directory of the boot partition or alter the configuration to point to /EFI/petasan
Using a copy of the petasan directory might miss updates, as I assume the PetaSAN distro will update in /EFI/petasan. E.g. this option is less preferable.
Since I do not know how this is done in PetaSAN and you obviously changed the way it works (given the petasan directory name in the EFI directory of the boot partition), I prefer the way to fix the grub/EFI /whatever cfg to point grub prefix to look in /EFI/petasan instead of the /EFI/ubuntu
Pages: 1 2
PetaSAN v3.3.0 Kernel command line cstate settings
bbu
7 Posts
Quote from bbu on June 5, 2025, 10:49 pmHello there
PetaSAN is a really great solution! Thank you very much for all the efforts put into it!
After playing with PetaSAN in a virtual environment we moved to 3 physical servers. It works very well. The "problem" is, these are re-purposed servers that have 2 rather powerful CPU built in. They are most of the time 100% idle, since we do not have that many transactions and no EC pools. Most of the work is done in HW and it's associated offloading mechanisms.
But the environment and our electricity bill is not that happy about the CPU's running at full speed. We noticed that the CPU's run at 100% clock speed all the time. We verified the BIOS settings, tried to lower the frequency of some cores and so on to no avail. It turned out, PetaSAN provides some kernel settings preventing any power saving mechanisms to kick in:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrierwhere the relevant settings are:
intel_idle.max_cstate=1
processor.max_cstate=1Although I can understand the idea behind this choice (no latency due to frequency / efficiency scaling), energy/environment/cost wise, this is not that optimal, especially if there is a big overcommitment of CPU resources.
So my question is: what is the reason the creator of PetaSAN choose these settings? What happens if higher cstates are allowed?
For us, if the stability is not endangered, we might either run at higher cstates, configure a few cores to be on 100% frequency all the time and others to go to powersafe.
If the cstates cannot be altered, the only option would be to remove one of the physical CPU, but this then get's complex on which of the PCI slots are usable and which are not.
Thank you
Hello there
PetaSAN is a really great solution! Thank you very much for all the efforts put into it!
After playing with PetaSAN in a virtual environment we moved to 3 physical servers. It works very well. The "problem" is, these are re-purposed servers that have 2 rather powerful CPU built in. They are most of the time 100% idle, since we do not have that many transactions and no EC pools. Most of the work is done in HW and it's associated offloading mechanisms.
But the environment and our electricity bill is not that happy about the CPU's running at full speed. We noticed that the CPU's run at 100% clock speed all the time. We verified the BIOS settings, tried to lower the frequency of some cores and so on to no avail. It turned out, PetaSAN provides some kernel settings preventing any power saving mechanisms to kick in:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrier
where the relevant settings are:
intel_idle.max_cstate=1
processor.max_cstate=1
Although I can understand the idea behind this choice (no latency due to frequency / efficiency scaling), energy/environment/cost wise, this is not that optimal, especially if there is a big overcommitment of CPU resources.
So my question is: what is the reason the creator of PetaSAN choose these settings? What happens if higher cstates are allowed?
For us, if the stability is not endangered, we might either run at higher cstates, configure a few cores to be on 100% frequency all the time and others to go to powersafe.
If the cstates cannot be altered, the only option would be to remove one of the physical CPU, but this then get's complex on which of the PCI slots are usable and which are not.
Thank you
admin
3,054 Posts
Quote from admin on June 6, 2025, 9:56 amThank you for your nice feedback, it is a lot of work under the hood to make things work 🙂
This cpu setting does make a significant difference in latency, almost 40% better. So performance of a single thread is much improved. It is one of the top OS/kernel tuning parameter by Ceph. There are many online reference to this in the Ceph community.
Ofcourse you can change this if you really want:
In PetaSAN 4.0 you can do so via the UI under Performance Profiles...you can create your own profiles and apply them centrally ant any time. It could be one reason to upgrade.
I you do not upgrade, you can do it manually..on ALL nodes, edit:
/opt/petasan/config/tuning/current/post_deploy_script
and remove the cpu settings, then reboot ALL nodes.
Thank you for your nice feedback, it is a lot of work under the hood to make things work 🙂
This cpu setting does make a significant difference in latency, almost 40% better. So performance of a single thread is much improved. It is one of the top OS/kernel tuning parameter by Ceph. There are many online reference to this in the Ceph community.
Ofcourse you can change this if you really want:
In PetaSAN 4.0 you can do so via the UI under Performance Profiles...you can create your own profiles and apply them centrally ant any time. It could be one reason to upgrade.
I you do not upgrade, you can do it manually..on ALL nodes, edit:
/opt/petasan/config/tuning/current/post_deploy_script
and remove the cpu settings, then reboot ALL nodes.
bbu
7 Posts
Quote from bbu on June 9, 2025, 9:31 am... and constantly changing, security updates and so on. E.g. ongoing work and it is well maintained as version 4 just arrived, not only kept alive. Looking at the upgrade guide, the path from 3.3.0 to 4.0.0 appears to be straight forward.
Note: it is a bit strange to read section 7 "Upgrading from PetaSAN 4.X and above" as 4.0.0 just hit the public ...
I indeed did not research the cstate settings related to Ceph, as PetaSAN might have had it's own reasons to do this, as it is more than only Ceph. But yes, there are many references to this.
Does PetaSAN v4 implement these profiles via Tuned?
If you already planned for this in v4, I trust this is the best / long shot way to address this. So we will do the upgrade, see how much energy can be conserved and how the latency is affected (if this is easy to measure), if this does not work, we will remove one of the physical CPU.
BTW: my plan to only run a few cores at 100% and others with less, makes it too complex, as processes like OSD would need to be pinned to these CPU's, plus other services I'm not aware of. If a service is restarted, pinning would need to happen again. So I just dumped that idea ..
Thank you
... and constantly changing, security updates and so on. E.g. ongoing work and it is well maintained as version 4 just arrived, not only kept alive. Looking at the upgrade guide, the path from 3.3.0 to 4.0.0 appears to be straight forward.
Note: it is a bit strange to read section 7 "Upgrading from PetaSAN 4.X and above" as 4.0.0 just hit the public ...
I indeed did not research the cstate settings related to Ceph, as PetaSAN might have had it's own reasons to do this, as it is more than only Ceph. But yes, there are many references to this.
Does PetaSAN v4 implement these profiles via Tuned?
If you already planned for this in v4, I trust this is the best / long shot way to address this. So we will do the upgrade, see how much energy can be conserved and how the latency is affected (if this is easy to measure), if this does not work, we will remove one of the physical CPU.
BTW: my plan to only run a few cores at 100% and others with less, makes it too complex, as processes like OSD would need to be pinned to these CPU's, plus other services I'm not aware of. If a service is restarted, pinning would need to happen again. So I just dumped that idea ..
Thank you
bbu
7 Posts
Quote from bbu on June 13, 2025, 9:08 amWe did the update to 4.0.0, which was a smooth transition. We then tried to apply the performance profile.
Not sure what need to be done to apply the changed settings, as "Apply" does not update / rebuild grub in this case.
BUT: after the upgrade from 3.3.0 to 4.0.0, we did not restart the nodes. To see if the new settings of the performance profile were applied, we rebooted one of the nodes. This ends in the GRUB> prompt and does no longer start up the OS.
Using set prefix= and issue the "normal" command, we were able to launch the grub menu which in turn allows normal boot of the node ...
What is the "official" procedure to fix this condition?
We did the update to 4.0.0, which was a smooth transition. We then tried to apply the performance profile.
Not sure what need to be done to apply the changed settings, as "Apply" does not update / rebuild grub in this case.
BUT: after the upgrade from 3.3.0 to 4.0.0, we did not restart the nodes. To see if the new settings of the performance profile were applied, we rebooted one of the nodes. This ends in the GRUB> prompt and does no longer start up the OS.
Using set prefix= and issue the "normal" command, we were able to launch the grub menu which in turn allows normal boot of the node ...
What is the "official" procedure to fix this condition?
admin
3,054 Posts
Quote from admin on June 13, 2025, 11:07 pmThe Performance Profiles run the different scripts defined in the profile. Nearly all do not require a reboot (at least in the PetaSAN default profile). The cpu speed / c-states settings do require reboots, and can be applied/changes while running (you would need to setup you own script/profile and change the parameters passed to cpupower command line). So you can have a low and high perf. profiles apply profiles at runtime without reboots.
As for the system did not start after a reboot, and you had to change the grub prefix...no that is not the official way 🙂 and ofcourse it should work without issue. Maybe you had some problems in your upgrade. If you can reproduce this using a different environment or setup that we can reproduce, please report this (preferably in the bug topic) and we will try to reproduce it.
The Performance Profiles run the different scripts defined in the profile. Nearly all do not require a reboot (at least in the PetaSAN default profile). The cpu speed / c-states settings do require reboots, and can be applied/changes while running (you would need to setup you own script/profile and change the parameters passed to cpupower command line). So you can have a low and high perf. profiles apply profiles at runtime without reboots.
As for the system did not start after a reboot, and you had to change the grub prefix...no that is not the official way 🙂 and ofcourse it should work without issue. Maybe you had some problems in your upgrade. If you can reproduce this using a different environment or setup that we can reproduce, please report this (preferably in the bug topic) and we will try to reproduce it.
bbu
7 Posts
Quote from bbu on June 16, 2025, 6:56 amGreat, but for the kernel changes to take effect, grub would need to be rebuilt, right? Would PetaSAN do that or would we need to do something to remove the cstate from the kernel?
The upgrade was successful on all the nodes:
PetaSAN version: 4.0.0
PetaSAN updated to version 4.0.0 successfully.The console output did not mention any special errors or warnings.
All nodes experience the same issue and boot to the grub prompt.
We did not alter PetaSAN yet, we just manually put the commands to the grub prompt. How do we alter grub the PetaSAN way, e.g. make the prefix permanent?
Great, but for the kernel changes to take effect, grub would need to be rebuilt, right? Would PetaSAN do that or would we need to do something to remove the cstate from the kernel?
The upgrade was successful on all the nodes:
PetaSAN version: 4.0.0
PetaSAN updated to version 4.0.0 successfully.
The console output did not mention any special errors or warnings.
All nodes experience the same issue and boot to the grub prompt.
We did not alter PetaSAN yet, we just manually put the commands to the grub prompt. How do we alter grub the PetaSAN way, e.g. make the prefix permanent?
admin
3,054 Posts
Quote from admin on June 16, 2025, 9:49 amThe changes are applied dynamically at runtime. no kernel rebuild/grub changes required. When you apply/change a performance profile in the UI, it will send the profile to all nodes and run the scripts associated with profile, you can change profile dynamically at runtime. The scripts have a section related to cpu tuning (it should be clear in comments), as per prev post it relies on cpupower tool as well as some other echo commands to /sys to set cpu performance governor and frequency. You need to create a new profile or edit existing one and tune these cpu parameters to reduce performance.
Regarding the grub boot issue, please see my prev post.
The changes are applied dynamically at runtime. no kernel rebuild/grub changes required. When you apply/change a performance profile in the UI, it will send the profile to all nodes and run the scripts associated with profile, you can change profile dynamically at runtime. The scripts have a section related to cpu tuning (it should be clear in comments), as per prev post it relies on cpupower tool as well as some other echo commands to /sys to set cpu performance governor and frequency. You need to create a new profile or edit existing one and tune these cpu parameters to reduce performance.
Regarding the grub boot issue, please see my prev post.
bbu
7 Posts
Quote from bbu on June 16, 2025, 12:05 pmThat will not work, as long as the kernel parameters are like this:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrierThis is the running kernel cfg after the upgrade, and the kernel parameters intel_idle and processormax_cstate will prevent any changes to the idle states / performance profile from the OS... that's why I pointed it out.
Regarding the boot issue: no, we do not have any other environment to play with at the moment. The test environment (all virtual) was dismantled once wthe hardware based one was built.
Where should we change the prefix in PetaSAN? Simply run update-grub and grub-install any try again? Don't know where I could provide the prefix in the grub.conf ... but I'm not into grub at all ...
That will not work, as long as the kernel parameters are like this:
BOOT_IMAGE=/boot/vmlinuz-5.14.21-08-petasan root=UUID=1a10e09e-e3cb-4886-a461-47aad6f28f54 ro quiet net.ifnames=1 intel_idle.max_cstate=1 processor.max_cstate=1 pti=off spectre_v2=off l1tf=off nospec_store_b
ypass_disable no_stf_barrier
This is the running kernel cfg after the upgrade, and the kernel parameters intel_idle and processormax_cstate will prevent any changes to the idle states / performance profile from the OS... that's why I pointed it out.
Regarding the boot issue: no, we do not have any other environment to play with at the moment. The test environment (all virtual) was dismantled once wthe hardware based one was built.
Where should we change the prefix in PetaSAN? Simply run update-grub and grub-install any try again? Don't know where I could provide the prefix in the grub.conf ... but I'm not into grub at all ...
admin
3,054 Posts
Quote from admin on June 16, 2025, 10:05 pmI do not know why you had to change the grub prefix, i also do not know how to make your changes persist. It is not something we see, so unfortunately i cannot help much. You can try looking in the grub and ubuntu forums.
I do not know why you had to change the grub prefix, i also do not know how to make your changes persist. It is not something we see, so unfortunately i cannot help much. You can try looking in the grub and ubuntu forums.
bbu
7 Posts
Quote from bbu on June 17, 2025, 9:30 amOK, I did some digging into the grub rabbithole. The issue appears to be, that the grub config points the prefix to /EFI/ubuntu, but the PetaSAN distribution has the cfg and efi in /EFI/petasan.
As a proof of concept, we copied the petasan directory to ubuntu, which restored the boot capability of the distribution. Now, the options are:
Keep the efi and cfg in the ubuntu directory of the boot partition or alter the configuration to point to /EFI/petasan
Using a copy of the petasan directory might miss updates, as I assume the PetaSAN distro will update in /EFI/petasan. E.g. this option is less preferable.
Since I do not know how this is done in PetaSAN and you obviously changed the way it works (given the petasan directory name in the EFI directory of the boot partition), I prefer the way to fix the grub/EFI /whatever cfg to point grub prefix to look in /EFI/petasan instead of the /EFI/ubuntu
OK, I did some digging into the grub rabbithole. The issue appears to be, that the grub config points the prefix to /EFI/ubuntu, but the PetaSAN distribution has the cfg and efi in /EFI/petasan.
As a proof of concept, we copied the petasan directory to ubuntu, which restored the boot capability of the distribution. Now, the options are:
Keep the efi and cfg in the ubuntu directory of the boot partition or alter the configuration to point to /EFI/petasan
Using a copy of the petasan directory might miss updates, as I assume the PetaSAN distro will update in /EFI/petasan. E.g. this option is less preferable.
Since I do not know how this is done in PetaSAN and you obviously changed the way it works (given the petasan directory name in the EFI directory of the boot partition), I prefer the way to fix the grub/EFI /whatever cfg to point grub prefix to look in /EFI/petasan instead of the /EFI/ubuntu