You’re correct the payload may be getting set as an active payload, but when you run your program to query with get_target_payload() it will pull from the installation file.
I know that the primary and secondary client stream the TCP offsets, so you can parse through the stream as shown here: How to process package from robot controller - #4 by ajp
Unfortunately there currently isn’t any built in method for reading the active payload as you described.
You can also try setting the payload in a thread on polyscope instead, the thread could get updated from a variables passed in using a RTDE interface, or other external sources. When the payload is set in the same program that is querying the target payload, you will read the active payload.