I wanted to provide an update on the problem which affected all APNIC IPv4 reverse DNS zones.
APNIC was notified via the public DNS operations mailing lists on Wednesday, 16/03 at 00:30 UTC+10 of a DNSSEC problem which affected all APNIC IPv4 reverse DNS zones.
During the reverse zone updates, which happened on Tuesday, 15/03 at 09:15 UTC+10, incorrect DS records were used. Once we became aware of the problem, the APNIC network operations team began working on resolving the issue immediately and all zones were corrected by Wednesday, 16/03 at 02:40 UTC+10. Due to DNS cache effects, the old cached data may take time to expire in the public DNS and be replaced with the new information.
APNIC do have testing scripts to check for this kind of operational problem; however, in this instance the checks did not detect the issue. The team is currently investigating the root cause of this monitoring failure.
We would like to thank the operations community for reporting the issue to us. We recognise this is a significant service outage.
A more detailed service announcement is being compiled and I'll email the list again with that information once it is completed. Thanks again for your patience and apologies for the inconvenience this outage caused.
Technical Services Director