Max Re Capital's earnings for 2001 to 2005 may be reduced by up to $50m.
The Securities and Exchange Commission is reviewing three finite retrocessional contracts written by the reinsurer.
Keith Hynes, executive vice-president and chief financial officer at Max Re, indicated the company informed the SEC after discovering the problem itself.
Max Re's audit and risk management committee is also conducting a review of the transactions.
The company may have to reduce its earnings by $50m, depending on whether the transactions are classified as financial reinsurance and as deposit accounting.
Hynes could not give a firm deadline on the SEC's investigation, although he hopes that it is concluded swiftly.