I do this, along with pamusb, so I don’t need a password for sudo etc. (which allows a longer more secure password than I might otherwise use). Depending on the threat model, I think it’s fine to just leave it in. One day I’ll get it to shutdown --now on ripping out the key drive without dismounting first, sort of a break glass in emergency thing. Same thing can be achieved with pulling the key and holding the power button though (even if it’s a bit rude to the filesystem)