Connection rate limit per IP with nftables

I'm attempting to limit connections per IP to port 25 using nftables.

Snippet of my rules:

Code:#!/usr/sbin/nft -f

flush ruleset

table ip filter {
set ban {
type ipv4_addr
size 65535
flags dynamic,timeout
timeout 5m
}
chain input {
type filter hook input priority filter; policy drop;
iif "lo" accept
ct state invalid drop
ct state established,related accept
ct state new tcp dport 25 limit rate over 1/second burst 1 packets add @ban { ip saddr }
ip saddr @ban counter drop
# other rules inserted here
}
}My understanding is this will allow one new connection every 1 second. Exceeding that rate will add the IP to the ban set for 5 minutes. Tested and seems to work. However, if the rate is currently exceeded by some IP address and a different IP tries to connect just once, it also gets added to the ban set. So it appears this is not a per IP rule, but rather a simple rate limit for the port. Is there a way to implement this per IP?

Thanks!