I would like to play around with encryption, and it is strange that you cannot experiment with it but if it's not thousands of dollars to get a frequency I think it would be a fun little thing to play with.
Two options:
FCC Experimental Licensing:
This is what manufacturers, researchers, etc. use for setting up a license to do experimental stuff. It's an option for you, but in reality, what you are doing isn't true experimentation. You just want to try something you can't do on ham radio.
Other option:
Actually 2...
1. License an itinerant frequency and list out all the common analog/digital emission designators. Encryption isn't an emission type, so just do narrow FM, P25, DMR, NXDN, etc.
Drawback: Itinerant frequencies are SHARED, which means you may have other users on the frequency.
2. If you don't want to share, then you need to work with a frequency coordinator to identify an available frequency or frequency pair that is available and suitable for what you are doing. A coordinated frequency will give you some protection from interference from other users.
Drawback: You'll have to pay for the frequency coordination (few hundred bucks per frequency/location) + the FCC licensing fees.
You'll also need to come up with some sort of legit usage. "Playing around with it" isn't going to get you an FCC license. Time to develop a small business….
Just remember, if you go this route, you can't use non-type certified equipment. You'll need suitable Part 90 equipment.