The present study reports heat-transfer performance, exergy analysis, entropy generation, and pressure drop of shell and helically coiled heat exchanger (SHCHE) with Al₂O₃-CuO/water hybrid nanofluid (HYNF) as a working fluid. Helical coil is made of copper material with 54 turns and pitch ratio is 31.35 mm. Hot oil streams at the shell with 75° C, and the working fluid streams at the helical coil with 30° C. The volume fraction of the nanoparticles is considered as 0.1 vol.%. Reynolds number of the oil is fixed as 900 and the Reynolds number of the working fluid varies from 6000 to 15,000. The numerical code is validated with the earlier experimental work. Highest thermal performance is obtained by using 0.1 vol.% HYNF than nanofluids and base fluid. Role of mass flow rate, and Reynolds number on heat-transfer rate, effectiveness, total entropy generation, exergetic efficiency, exergy loss, and dimensionless exergy loss are investigated. An ~ 20% increase in Nusselt number and ~ 48% increment in exergetic efficiency are noted with the usage of HYNF. Entropy generation of SHCHE is lower by adding nanoparticles. This study enables the readers to understand the irreversibility of heat transfer in shell and helically coiled heat exchanger.