Ahhh I get you! I figure Hiroe's family is probably from a neighboring kingdom not!Japan to go with not!China assigned there for diplomacy/trade administration. It would mean they couldn't be just unilaterally banished overnight (at least not without creating an embarrassing diplomatic incident), so Hiroe and Shisui could have had at least some time to form a friendship before Hiroe's family got reassigned to work with someone else. Shisui asking Hiroe to join her years later makes sense, tho! They even could have exchanged letters in secret during that time, so there's already some history of Hiroe being her secret confidant.

And because making choices is my greatest weakness, what if Shisui doesn't know Hiroe's secret but suspects something is up with her after a maybe-supernatural event like that ghost(?) incident? Especially if there's an actually supernatural part and another thing that someone tries to pass off as supernatural, and being close as they are, Shisui picks up on how differently Hiroe reacts to the two outwardly-similar situations (and maybe how protective Zenko is toward her).