Skip to main content
Delphix

Multibyte Database Character Sets May Cause Errors on Insert or Update (KBA5589)

 

KB

KBA# 5589

 

Issue

When masking a database, if the database uses a multibyte character set (for example, AL32UTF8 on Oracle) there may be problems with algorithms that auto-truncate their values whereby we miscalculate the acceptable length of the string.

An example of this is the secure lookup algorithm. The secure lookup algorithm uses a file of lookup values when masking. It will "lookup" the value in the column to find a replacement value in the lookup file. Once it has a value that it intends to use as a replacement, it will truncate the value (if necessary) to fit the column length.

A database with a multibyte character set can cause the engine to miscalculate the length (in characters) of the column which can result in the engine attempting to INSERT or UPDATE values that are too large for the column.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases
6.0 6.0.0.0

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Resolution

You can work-around this issue by using a lookup file for which all values are smaller (in characters) than the size of the column. There is also an open issue to address this and a fix should be available in a future release of the Masking Engine.


Troubleshooting

The errors you might see are database and (sometimes) version dependent. Some examples of these errors are:

Oracle

ORA-12899: value too large for column

MS SQL Server

 Msg 8152, Level 16, State 14, Line 2
 String or binary data would be truncated.

DB2

DB2 SQL Error: SQLCODE=-302, SQLSTATE=22001, SQLERRMC=1