Amazon Seller Policies pushing me towards Walmart

I have been selling on Amazon for almost two years. It was great at first and a nice source of revenue, however a month or so ago I had two orders have products damaged in shipping and this threw my order defect rate over the threshhold and I lost my buy box. I could have handled it better, as one of the shipments did not appear damaged but instead of fighting the customer I just refunded them. Amazon told me I should not have done that since they automatically then treat it as a defect. Now I am forced to wait for a 60 day period to pass so those defects filter out. Due to this I started looking into Walmart and have started listing products there. My question is do you think Amazon is taking this into consideration regarding their punitive measures against sellers who have a couple late shipments or a defect. I think two damages that are clearly the shipping company’s fault, one in which I let Amazon know was likely just the customer lying about a damage, should not result in such a severe punishment. I think Amazon is pushing sellers to consider competitive platforms when they act so callously. I would likely never have looked into selling on Walmart had they not put me in a position where my revenue would be severely diminished for up to 3 months because of things more or less out of my control.

TL;DR I think Amazon is too severe with sellers and when they delist products or take away the buy box from sellers are they pushing these sellers to consider other selling platforms ultimately hurting Amazon in the long run and helping their competitors.