You don't understand how digital authentication algorithms work. The authentication key is never disclosed publicly. There are 2 parts to the key, a public part and a private part. The public key is derived from the private key, but in such a way that it is mathematically impossible to calculate the private key from the public key.
When authentication is required, the entity requesting access is sent a random bit string. They use the private key to perform a signature operation on the random bit string, and then send it back to the authenticator. The authenticator uses the public key to reverse the signature operation on the signed random bit string to see if it resolves back to the original random bit string. If it does, the authentication is successful and access is granted. If it doesn't, access is denied.
The private key is held by one individual, and the public key can be distributed to anyone. The private key cannot be discovered by packet sniffing, because it is never sent over a public channel. In many systems, the user generates their own public and private keys, and submit the public key when configuring their user account. That way, there is no central database containing private keys that could be compromised. Each user is responsible for the security of their own private key.
I understand public key encryption quite well, actually. I use RSA (along with the block cipher AES) extensively in my data processing operations. What you described is incomplete as a authentication solution, and the element that is missing would obviate the need for a PKE such as RSA in this situation. Here's why:
Public key encryption (PKE) such as RSA indeed allows you to publicly distribute half of a keyset (one part public, one part private) where the public key can be used to encrypt one side of a message, and the private key can be used to decrypt it. Simply connecting to a server, furnishing and having the server encrypt a random string, sending it back and then having the client decrypt it, only determines that the client has the matching private key to the public key. It does nothing else. In order for authenticate to occur, as you say, some sort of unique
signature is required. Where does the signature come from? Either both sides must
already posses the valid signature, or else the party interested in authentication must acquire the signature (generally called a
certificate)
from a trusted third party with a known IP address. This happens every time a web browser connects to a web server via HTTPS.
All of the above is unnecessary if both parties know
in advance a
shared secret.. This is why your WIFI connections don't use third party based authentication. Both WIFI server and client know the shared secret in advance. Browser connections to websites do not have shared secrets in advance. This is why PKE based third-party certificate based authentication is required by the browser to make sure the website is who they say they are.
Now then, in the case of something like DMR security, PKE based authentication would be total overkill and completely unnecessary, and an undo burden on the server. If both sides know the shared secret in advance, no encryption beyond a hash of the shared secret + salt (random string) is necessary for the server to know that the client is who he says he is. This is how mail servers used to work before TLS became prevalent. If we both know the shared secret in advance, when the client connects, the server can send a random string which the client appends to the shared secret, makes a hash, and sends it back to the server. Since the server knows the shared secret it can validate the hash, and now knows that the client possesses the shared secret, and is thusly authenticated. No PKE necessary. So yes, if a shared secret was simply sent to the server without this hashing, it could easily be sniffed, which is why in my "stream of thought" rambling corrected myself by saying it could work by hashing it with a random salt string.
As for using a PKE based system to authenticate, RSA requires a fair amount of horse power, and I believe would put an undo burden on the servers. Aside from that, unless the server knows the shared secret in advance, then a trusted third party would be required. So now this has to be properly managed, not to mention the extra overhead involved at the server. I don't see this happened for something like Brandmeister which has thousands of of connections at a time and all kinds of TCP/IP connection traffic constantly. Plus, it's flat out unnecessarily if the server knows the shared secrets in advance, which seems reasonable.
Of course, none of this applies to local repeaters. They already have encryption for the non-amateur world, and they are not likely to do back flips to accommodate hams. Not the mention that all the radios out there would not be compatible with any such authentication scheme. Fuggedaboudit.
My two cents, anyway.
They already have encryption available for non-amateur use, which is what anyone other than hams are using to solve this problem. And if they did, it would render useless all the DMR radios already out there. This is Fantasyland stuff.
