I can't offer you any solutions unfortunately but I have been fighting a similar issue for nearly two years now. I was starting to think it was my cassette as I have a sunrace on there and thought maybe not going shimano was causing my issues with the shifting not being entirely perfect.

I have the same derailleur but paired with the GRX shifter up front.

My largest two cogs (11-42) are fine like you. Then my third cog is the worst, often when going from large to small down the gears it doesnt want to go into third and will sit clicking on the second. Other times it'll just jump the third entirely and go straight to forth. Even once it is in the third cog it still clicks as if it is trying to shift up to the second cog and isnt entirely aligned properly on the third.

4the and 5the are also a little sketchy but will usually go into them a lot more reliably but they never fully sound / feel 100% aligned. Sometimes if I put a lot of power through the cranks in these gears it will skip a tooth aggressively which worries me that I will end up snapping a tooth off of one of them.

I am constantly tweaking my barrel adjuster trying to get it better but it always tends to just move the problem around slightly on the cassette and never solves it.

I have changed the cable out as well just in case but that didnt help. I have also changed the crankset in this time which had a very slightly different offset resulting in mm changes in my chain line (I went from an FSA crankset to a GRX one) which i hoped maybe would inadvertently smooth out those slight kinks. It did make it marginally better but overall the improvement was negligible.

So I'm sorry I can't offer any insights but you aren't alone and reading this post makes me think maybe the issue is the derailleur all along for me as that would be the common denominator between us.

I'll pop back to this thread to see if anyone can offer any insights but I have had professional mechanics look at mine (as I usually do all my own work) but that offered no improvement.