Using machine learning method to search the anomalous quartic gauge couplings via tri-photon production at future hadron colliders

Examining the self-couplings of gauge bosons, as dictated by the non-Abelian gauge symmetry in the Standard Model (SM), provides invaluable insights into the SM's gauge structure. Any departure from SM's predictions regarding gauge boson self-coupling could hint at the presence of new physics beyond the SM. This involves adjusting self-interactions through an effective field theory approach. This study aims to investigate anomalous quartic gauge dimension-8 couplings related to γγγγ and γγγZ vertices via tri-photon production in future hadron colliders, namely the HL-LHC (√s=14 TeV) and the FCC-hh (√s=100 TeV) with integrated luminosities of 3 ab^(-1) and 30 ab^(-1). Generated events, considering parton showering and detector effects, are analyzed using Monte Carlo methods and also using machine learning methods which are called TMVA with Boosted Decision Trees. Limits of anomalous quartic gauge couplings f_T8/Λ^4 and f_T9/Λ^4, obtained at a 95% confidence level, with and without systematic errors for both future hadron colliders. Results for the FCC-hh collider indicate that constraints on anomalous quartic gauge couplings are about three orders of magnitude stronger than the best experimental limits by the ATLAS collaboration at the LHC. Factoring in a realistic 10% systematic uncertainty, the limits degrade by about one order of magnitude but still surpass those reported by ATLAS by two orders of magnitude.