You're allowed to bring 2 bottles back to the US. Buying duty free at the airport is more expensive than buying at Walmart, Costco or La Europea in downtown Cancun. If you have a non-stop, direct flight from Cancun back to your US city and buy duty free, the airline will hold it for you in storage on the plane. The duty free shop puts it in a plastic bag with the receipt and your name attached. If you have to change planes, duty free isn't a good option because the bottles must be placed in checked baggage. I always buy at one of the above-mentioned stores, take bubble wrap with me and wrap the bottles in that and a beach towel, and place in the middle of my checked baggage. Have done this for several years and, so far, no breakage issues to deal with.
I've never tried to bring back more than 3 bottles, but never had any problem when declaring them. If I remember right, I declared them, and the CDP Officer asked me how much they cost, not how big they were. That sound right?
Just curious, if you have already gone through security, why wouldn't you be able to purchase duty free bottles to take with you? Do you have to go through security again or is this purchased before security? I haven't been to Cancun in a few years. We are going next week....