Magnets induce voltages when moved, and otherwise only affect active parts of the circuit (i.e. which have a current flowing through).
Can you check if the wake-up happens only when jiggling the magnet around or also when standing still?