You need to log in to create posts and topics. Login · Register

Impressive improve performance by sysctl.conf


Mykola Perehinets
38 Posts
April 6, 2017, 3:52 pm

Hi folks,
Please verify and deploy this files on all nodes in cluster.
(example info: http://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployments).
This solutions(config files) can impressive improved ALL SYSTEM performance on ALL CEPH nodes(in your cluster).

1.Please create file /etc/sysctl.d/10-ipv6-disable.conf and add next line:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

2.Please recreate file /etc/sysctl.d/10-network-security.conf and add only next line:
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.log_martians = 1
#net.ipv4.icmp_echo_ignore_all = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.ip_forward = 0
#net.bridge.bridge-nf-call-ip6tables = 0
#net.bridge.bridge-nf-call-iptables = 0
#net.bridge.bridge-nf-call-arptables = 0
kernel.sysrq = 0

3.Please create file /etc/sysctl.d/10-performance.conf and add next line:
vm.swappiness = 0
vm.vfs_cache_pressure = 50
#vm.min_free_kbytes = 16777216
#vm.min_free_kbytes = 524288
vm.min_free_kbytes = 262144
vm.min_slab_ratio = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_writeback_centisecs = 5000
vm.dirty_expire_centisecs = 10000
vm.zone_reclaim_mode = 0
#fs.file-max = 78718144
fs.file-max = 26234859
fs.inotify.max_user_watches = 16777216
fs.inotify.max_queued_events = 65536
kernel.pid_max = 4194303
kernel.core_uses_pid = 1
kernel.panic_on_oops = 1
kernel.panic = 60

4.Please create file /etc/sysctl.d/10-net.conf and add next line:
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.core.rmem_default = 67108864
net.core.wmem_default = 67108864
net.core.optmem_max = 134217728
net.core.netdev_budget = 1200
net.core.netdev_max_backlog = 300000
#net.core.somaxconn = 65536
net.core.somaxconn = 32768
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
#net.ipv4.udp_mem = 16777216 16777216 16777216
net.ipv4.tcp_rmem = 67108864 134217728 268435456
net.ipv4.tcp_wmem = 67108864 134217728 268435456
#net.ipv4.tcp_mem = 268435456 268435456 268435456
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_max_syn_backlog = 819200
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.route.flush = 1
#net.netfilter.nf_conntrack_max = 16777216
#net.netfilter.nf_conntrack_tcp_timeout_established = 1800
kernel.nmi_watchdog = 0

5.Please increase txqueuelen up to 10000 on all network interfaces(from root):
# ifconfig ethX txqueuelen 10000

Thank you.

 

#1

slacker87
12 Posts
April 10, 2017, 1:30 pm

Mind giving a little detail on what all of these changes do to improve performance for some of us not quite yet linux experts, thanks!

#2

admin
2,858 Posts
April 11, 2017, 11:29 am

One of the main goals of PetaSAN is to make it usable by Linux experts and non-experts alike.  There is a lot of performance analysis done on Ceph to tune it for different hardware and io workloads. We plan in the future to have different config templates that the admin can choose from and/or add his own.

Currently we use almost the default config values, with the exception of changing the default Linux io scheduler from cfq to noop, increase the default os block read ahead cache from 512K to 2M and use the jemalloc library rather than tcmalloc which is faster at small (eg 4K) block sizes at the expenses of taking slightly more memory ( approx 500M more ) per OSD. The rest are all defaults.

 

 

#3

Mykola Perehinets
38 Posts
April 11, 2017, 8:36 pm

Hi,

This solution(sysctl parameters) improve performance(10-performance.conf, 10-net.conf) and security(10-network-security.conf, 10-ipv6-disable.conf) of base host Operation System. This optimization is indirectly improvement ceph performance. Additional improvements of ceph I was added in previous/other posts(in ceph.conf).

Please verify this(Operation System Solutions) and use in your production.  Thank you.

#4

ghbiz
69 Posts
June 11, 2020, 11:56 pm

Admin,

We came across this OLD 2017 topic in the forum.

Can you please let us know if these sysctl changes have been looked at. It is clear that CEPH can be configured for different work loads. Petasan clearly offers its CEPH surrounding typical iSCSI / Virtualization work load. For this reason, it is safe to believe that the OP who created this topic is using CEPH in the same manner.

 

#5

admin
2,858 Posts
June 13, 2020, 11:06 am

We do monitor any performance tuning recommendations regularly, i believe some of the settings mentioned may already be used. We have sysctl, udev, ceph, lio  tunings you select when you deploy the first node and create the cluster, you choose between different available profiles and if you open the advanced settings you can tweak them and even create a custom tuning profile. We also have a tuning service that sets up some cpu turnings required after boot ( intel specific ).

Generally i would not recommend you change settings yourself unless you study them, also many of the tunings you may find online could benefit under specific benchmark conditions but are generally bad in other cases,  of course if you find any enhancements we would love to know 😉

#6