If caddy is acting as a proxy for anything, you should not need to forward that port externally. Local host firewalls allowing traffic on your local network is sufficient.
Depending on your physical host layout you may be looking at an issue with nat reflection.
You have not given us enough about your topology to assist in troubleshooting.
Inference is dirt cheap in comparison. Hundreds to thousands of concurrent users can be served by hardware costing in the high-thousands to low-ten-thousands.
Training those same foundational models is weeks to months of time on tens to hundreds of millions worth of hardware.