Make sure we're looking at the right location. The frame could have
received a motion event from a pointer from a different wl_seat, but
under X it looks like our core pointer moved. Move the frame pointer
to the button press location before deciding what to do.