The following is an explanation of the 2 types of Authentication. This is well documented on the web!
Classic Mode: This is nothing but Windows Authentication. If any web application is created with Classic Mode Authentication then it cannot applicable to configure forms based authentication. That is basically it.
Claims Based: In SharePoint 2010 for a web application we can enable both windows AND forms authentication. In earlier implementation to do this, we have to create two web applications which has different zones and different authentication. But, with the new claims based authentication a single application can have the capability to configure both windows and forms under single URL. All this is possible because of the authentication framework is built on Microsoft Identify Foundation, and it uses “Geneva” framework to handle this authentication.
Convert from Classic Windows to Claims Based Authentication: To configure a Windows Authentication application to use Forms Based Authentication then you can convert from Classic Mode to Claims Based. But, there is no UI exist for doing this conversion. The only way around is through PowerShell.
WARNING! This cannot be UNDONE!
At the PS Prompt Type the following commands and press enter after each one.
$app = get-spwebapplication "http://<your site's URL>"
$app.useclaimsauthentication = "True"
$app.Update() Don't forget the brackets!
IMPORTANT: If an application is created using Claims Based Authentication and if you want to convert to Classic Mode then it is not possible either through the UI or through PowerShell.