UFW Config to only allow 1 IP address

Having looked at several sources, I'm stumped as to how to do this. I want the Ubuntu computer to only be able to communicate with one other computer, at IP address 1.2.3.4 from across the internet.

My attempt (with a new Ubuntu Server VM) as root:

ufw disable

ufw default deny incoming
ufw default deny outgoing

ufw allow in on 1.2.3.4
ufw allow out on 1.2.3.4

ufw allow from 192.168.1.0/24
ufw allow to 192.168.1.0/24

ufw enable

ssh user@1.2.3.4

The output of this attempt is a ssh connection timeout. I've also tried these in place of lines 4-5:

ufw allow from 1.2.3.4
ufw allow to 1.2.3.4

I can't get it to work. Basically, I want the system's firewall to block all internet communication except to that server. I need it totally secure.