DA14531 SWD only connects after programming firmware

I have a custom board with an STM32 programming the DA14531 on bootup through SPI (DA14531 as SPI slave, bootloader step 1).

I'm running into some issues as detailed in this post, so attempted to connect through a debugger. The strange thing is, according to the bootloader graph (BootROM sequence, figure 5, AN-B-072) it should enable "JTAG" after all boot steps fail, but the only way I'm able to connect my debugger over SWD is if I first let the STM32 flash firmware into the DA14531 RAM and _then_ connect the debug probe.

P0_5 should already be the default SWDIO pin, but in the firmware I call this to make sure it is selected (copied from the examples):
FPGA_HELPER(FPGA_GPIO_MAP_2, SWD_DATA_AT_P0_5);
GPIO_Disable_HW_Reset();

Why would the SWD not connect if the BootROM fails as shown in the BootROM sequence?

Kind regards,
Kas