[Nero_3D]

There isn't any accurate function, they simple calculate the shortest distance between a line (where the player is looking at) and a point (the other player), if it is smaller than x (lets say 5.0) than the player is looking at the other one

Also you should specify if you mean the camera or the player character itself

the formula for the distance is d = |front vector x (player2 pos - player1 pos)| / |front vector|
The front vector is from player1 who is supposed to look at player2