Over simplified, TCAS equiped airplanes will broadcast indiscriminately into the abyss and say "who are you". The other airplane's transponder will reply back with their squawk code and altitude. The TCAS system will observe the direction the reply came from and how long it took, to make a map of where nearby transponder equipped planes are relative to it.
When the TCAS senses a potentially dangerous close approach between it and other planes it will first issue the pilots a "traffic advisory" to the pilots.
When most transport category jets are equipped with TCAS II which provides a "resolution advisory" when needed. The two planes automatically negotiate with each other to give opposite RAs. I.e. one plane will climb and the other will descend.
Also remember that TCAS pre-dates ADSB and works with Mode S transponders talking to planes with mode C or S transponders.
