Setup the private nameservers on WHM -
1. Login to WHM.
2. Go to DNS Functions >> Edit DNS Zone.
3. Select the domain name from drop-down list >> Click on Edit.
4. Enter the SOA record and email address for your private nameservers.
5. Now you will need to create the A record for ns1 and ns2 nameservers. You will need to enter the IP address in A record field.
6. Save the changes.
It will take 24 to 72 hours for DNS propagation.
Note -
You will need to create the nameservers at your domain registrar's end and will need to enter those nameservers in ns1 and ns2 field in WHM.