Allow UTF-8 with bom to override CCSID attribute for IFS files Merged

There are three RFEs that have similar requests 143259, 135926, and 143226 related to the fact that the CCSID attribute of a file does not reflect the actual contents, which has a BOM that is expected to identify the encoding.

These files in general are created on a platform such as a PC that understands only ASCII and/or ASCII-like CCSIDs such as UT-16 or UTF-8. It is much easier for that platform and the applications to make determinations about the encoding based on content of the file without the need for a CCSID attribute. The IBM i does not have that same environment and the content of any file could be EBCDIC or ASCII or ASCII-like so the CCSID attribute is extremely important when an application reads/writes data out of/into the file in text mode. The data could be any string of bits and bytes and we certainly rely on the user/application to inform us of the encoding of that data. What is a BOM in 1208, is something entirely different in 1200 not to mention that 1208 is not the only CCSID that has BOM defined.

It is important to note that when the CCSID of the file is set correctly there are no problems. As has been noted in at least one of the RFEs, the TYPE command in FTP, the Change Attribute (CHGATR) command, the Qp0lSetAttr()???Set Attributes API, or setccsid Qshell utility are options that can be used to set the CCSID for a file.

There are different suggestions in these requests.
- Create a new directory attribute to direct new files created and linked to be assigned the CCSID based on the BOM in the data or inherited from the parent.
- Determine the CCSID when the file is opened based on the data.
- Have an option on the CHGATR command to set the CCSID based on the *CONTENT.

Since all of these RFEs have basically the same goal, 143259 and 135926 are being marked as duplicates and will set 143226 as Under Consideration. Any further commentary should be put under 143226.

The file system cannot be made to guess at the content nor can we use a CCSID because the data is ???probably' UTF8, etc. Only the users know the content of the files. The file system will need to be extremely careful to not change the current behavior for a solution to work. This means that any solution would most certainly require the users to take some steps to have their desired results.

The file system team will consider this request for future development.

Due to processing by IBM, this request was reassigned to have the following updated attributes:
Brand - Servers and Systems Software
Product family - Power Systems
Product - IBM i
Component - IFS (Integrated File System) and Servers
Operating system - IBM i
Source - COMMON

For recording keeping, the previous attributes were:
Brand - Servers and Systems Software
Product family - Power Systems
Product - IBM i
Component - Db2 for i
Operating system - IBM i
Source - COMMON

IBM has modified some of the information in this request. Please contact us if you have any questions or additional changes.

In particular the Headline was updated. Previous Headline was: "Allow UTF-8 with bom in include SQL statement ( PL/SQL, Stored procedures, UDTF and compound statement)". Please let us know if the Description should also have any updates to match the new Headline.

Also, this line was added to the end of the Description: "In summary, this request is to allow the UTF-8 BOM in IFS files at the system level."

The COMMON Europe Advisory Council (CEAC) has reviewed this requirement and recommends that IBM view this as a MEDIUM priority requirement that should be addressed.

Background: The CEAC members have a broad range of experience in working with small and medium-sized IBM i customers. CEAC has a crucial role in working with IBM i development to help assess the value and impact of individual RFEs on the broader IBM i community and has therefore reviewed your RFE.

To find out how CEAC help to shape the future of IBM i, see CEAC @ ibm.biz/BdYSYj and the article "The Five Hottest IBM i RFEs Of The Quarter" at ibm.biz/BdYSZT

Therese Eaton – CEAC Program Manager, IBM

The CAAC has reviewed this requirement and recommends that IBM view this as a high priority requirement that is important to be addressed. The bom of UTF-8 files is optional and should be ignored, if present.

Background: The COMMON Americas Advisory Council (CAAC) members have a broad range of experience in working with small and medium-sized IBM i customers. CAAC has a key role in working with IBM i development to help assess the value and impact of individual RFEs on the broader IBM i community, and has therefore reviewed your RFE.

For more information about CAAC, see www.common.org/caac

For more details about CAAC's role with RFEs, see http://www.ibmsystemsmag.com/Blogs/i-Can/May-2017/COMMON-Americas-Advisory-Council-%28CAAC%29-and-RFEs/

Nancy Uthke-Schmucki - CAAC Program Manager

Attachment (Description): The error