You need to log in to create posts and topics. Login · Register

dm-writecache not flushing after upgrade 3.0.1 -> 3.2.1?


jeenode
27 Posts
April 12, 2024, 3:07 am

I have run an upgrade from Petasan 3.0.1 to 3.2.1 on one of the nodes of 4 node cluster and after that it seems like dm-writecache is not flushing as per high_watermark/low_watermark?

Here is my writecache target settings:

 

ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.48-main: 0 1952415744 writecache s 254:7 254:6 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.49-main: 0 1952415744 writecache s 254:10 254:9 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.50-main: 0 1952415744 writecache s 254:13 254:12 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.51-main: 0 1952415744 writecache s 254:16 254:15 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.52-main: 0 1952415744 writecache s 254:19 254:18 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.53-main: 0 1952415744 writecache s 254:22 254:21 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.54-main: 0 1952415744 writecache s 254:25 254:24 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.55-main: 0 1952415744 writecache s 254:28 254:27 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.56-main: 0 1952415744 writecache s 254:31 254:30 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.57-main: 0 1952415744 writecache s 254:34 254:33 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.58-main: 0 1952415744 writecache s 254:37 254:36 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.59-main: 0 1952415744 writecache s 254:40 254:39 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.60-main: 0 1952415744 writecache s 254:43 254:42 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.61-main: 0 1952415744 writecache s 254:46 254:45 4096 start_sector 909384 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.62-main: 0 1952415744 writecache s 254:49 254:48 4096 start_sector 909416 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.63-main: 0 1952415744 writecache s 254:52 254:51 4096 start_sector 909416 high_watermark 50 low_watermark 49 writeback_jobs 512 autocommit_blocks 65536 autocommit_time 0 nofua pause_writeback 0 max_age 0

 

And dmsetup status shows no outstanding flushing blocks for any dm-writecache targets that are over high_watermark (see below) and I dont see any write IO on spinners.

ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.48-main: 0 1952415744 writecache 0 29074514 22868185 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.49-main: 0 1952415744 writecache 0 29074514 20750181 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.50-main: 0 1952415744 writecache 0 29074514 18155134 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.51-main: 0 1952415744 writecache 0 29074514 14550452 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.52-main: 0 1952415744 writecache 0 29074514 12236710 0 <- this should be flushing
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.53-main: 0 1952415744 writecache 0 29074514 19107015 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.54-main: 0 1952415744 writecache 0 29074514 19910628 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.55-main: 0 1952415744 writecache 0 29074514 14011301 0 <- this should be flushing
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.56-main: 0 1952415744 writecache 0 29074514 22732387 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.57-main: 0 1952415744 writecache 0 29074514 16664940 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.58-main: 0 1952415744 writecache 0 29074514 10755495 0 <- this should be flushing
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.59-main: 0 1952415744 writecache 0 29074514 17384693 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.60-main: 0 1952415744 writecache 0 29074514 21874422 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.61-main: 0 1952415744 writecache 0 29074514 21589126 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.62-main: 0 1952415744 writecache 0 29075530 18117581 0
ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.63-main: 0 1952415744 writecache 0 29075530 19628249 0

 

Maybe I am missing something?

Thanks!

#1

jeenode
27 Posts
April 12, 2024, 3:08 am

Forgot to say that I can trigger flush "manually" like below and it works ok:

dmsetup message ps--b4156a3e--39e0--481d--8ef7--0f0457f8197d--wc--osd.62-main 0 flush

 

#2

admin
2,858 Posts
April 12, 2024, 11:04 am

you probably hit

https://www.petasan.org/forums/?view=thread&id=1305

it is not something we see in out test, but likely happens if the cache fills ups and not empty quick enough.

to fix, on all nodes run:

vgs | grep wc-osd | cut -d ' ' -f3 | xargs -I{} dmsetup message {}/main 0 pause_writeback 0

it will be solved in 3.3 out later this month.

 

#3

robindewolf
8 Posts
April 22, 2024, 11:59 am

Hello,

We had the same issue on our PetaSAN cluster, executing the command on all nodes in the cluster solved the issue.

We are looking forward to the 3.3 release later this month.

Can you provide the release notes of this version in order to prepare the update?

Thanks!

#4