connecting two hosts on different subnets but the same physical network

I have two hosts that are on different subnets but on the same physical network. Host 1 broadcasts a UDP message that includes it's IP address. Host 2 gets this message and attempts to establish a TCP connection the Host 1. The correct port is known to both hosts. I see the traffic with a sniffer. Host 2 does send a request to host 1's address, with its address as the source. Host 1 never responds to the initial connection request from host 2.

This all works as long as both hosts are on the same subnet. For instance if both are on 192.168.0.0/24 all works. If one is on 192.168.5.0/24 and the other on 192.168.0.0/24 it fails.

If it makes any difference, both hosts are Linux on a Raspberry Pi.

Is it possible to make this work?