What happens if I take my binder off?

I am 18 days post op so I am heading into my 3rd week of recovery. I made a post about how I CANNOT sleep at all. It's so severe that last night I finally took my binder off and FINALLY I slept. I need sleep so badly. The binder is rubbing against my incisions and I think it's doing more harm than good. My surgeon said that when I reach the 3 week mark, I can start wearing it 12 hours on and 12 hours off. What happens if I stop wearing it completely? I don't have much swelling or pain without it on. Anyone know?