To go to this dialog, you just need to click on the "Settings" icon in the main tool bar:

Then on the "Compatibility" tab.
Two new options were added in v17:
If you wish to take advanted of object notation (see section Using object notation), this option allows the use of the characters ".", "[" And "]" as object notation symbols, to access tokenized properties - in variable, table, field, and project method names.
Enabling this option is mandatory in a database created in a version prior to 4D v17 since the ".", "[" And "]" characters were previously allowed in the names of these elements and are now discouraged.
Warning:
- this option requires the database to be in unicode;
- when enabling this option, it is recommended to check the conformity of your existing code via the MSC (see Verifying the application). To help developers bring their applications into compliance with this rule, the Check Application action automatically searches for the presence of these unwanted characters in variable, table, field, and method names. If these characters are detected, errors are reported by the MSC:

In this case, you must rename these elements in your application.
In an object, dates can now be defined as dates and not as strings, making the use of dates in objects easier and more intuitive. This allows the OB SET and OB Get commands to be used without the Is date constant, and object notation can be applied to them just like any other object. To use dates in objects in a converted database, simply check the "Use date type instead of ISO date format in objects" option. See Compatibility page.
The "Use 4DVAR Comments instead of Brackets" option was removed from the Compatibility page of this dialog box in 4D v16.
The older your version of 4D, the more options will be displayed:

For more information, see Compatibility page.