Background and Developments in EDI
Electronic Data Interchange (EDI) was developed in the 1980’s to enable the computer systems of buyers and sellers to exchange pre-defined data over a secure, reliable network. This provided fast and accurate data that was otherwise subject to processing and handling delays inherent in surface mail, faxes and data entry.
EDI works by encoding data in a very specific, standard format. EDI communication consists of one or more messages, each conforming to a subset of the standard format. Within each message, delimiters define a series of segments, the first three characters of which contains the code for the type of the segment. Within each segment, a series of codes and numbers (separated by another delimiter) define the data being transmitted. Due to the extensive use of very compact codes and concise data structures, EDI messages are almost impossible to humanly create and interpret without extensive documentation.
The EDI file is then sent from the sender to the receiver through either a private network connection, a Value-Added Network (VAN) or Internet in a secure environment. Costs for using a VAN are based on frequency and volume, and the VAN supplies additional security and tracking functions.
The use of EDI is possible only because of the agreement (otherwise known as an interchange agreement) between sender and receiver on the use of codes imbedded in each message. These agreements are captured in a transaction set. Initially these formats and codes were developed ad hoc, with standards being introduced by national (e.g., ASCI. X12 in the USA) and international bodies (e.g., UN/EDIFACT) beginning in the 1980s. These bodies have continued to maintain existing and develop new transaction sets as needed.
Extensible Markup Language (XML) works by allowing users to define a hierarchical set of tags that are embedded into a file that contains the information being communicated. It’s difficult to define XML in one sentence because of its dual function as a scripting language and a file format. But, it is a technology that has evolved from HTML, which is the language used by Internet web pages. Both HTML and XML are file formats that allow interaction with their user. This feature that allows “interaction”, separates XML from other EDI file formats like X12 and UN/EDIFACT. Because EDI is defined as the exchange of data without any human intermediary, the XML format with its interactive feature does not fit into the definition of EDI without exception; otherwise the XML format (in standard format) would have been considered to be just like any other EDI standard formats.
Although, there are some differences between both technologies, they both have a similar purpose. Both EDI and XML formats are structured to describe the data they contain. The EDI structure has a record-field-like layout of data segments and elements, which makes the EDI file shorter, but not easily understandable; while the XML format has tags, which is more easily understood, but having larger file sizes. Comparatively, XML EDI is 100 fold larger in byte data transfer requirements to transmit than traditional EDI. XML formed EDI transactions are much more complex to parse and manipulate; although they are much easier to format for web presentation. Below is a visual comparison between a simple EDI and XML message:
The primary difference between traditional EDI documents and the new versions of XML based EDI documents is their end use. Traditional EDI is designed for computer to computer interface. The XML EDI format (presentation of the basic information) is designed for human use or consumption, excellent for Web presentation and internet site usage.
Both traditional EDI and XML-EDI can and should co-exist. The functional usage of each type is separate and designed for different consumption. The replacement of computer to computer documents by human readable presentation formats makes for poor business efficiency. The vast majority of electronic transactions can (and should) be computer to computer; maximizing the efficiencies inherent in this form of communication and business transaction processing. XML-EDI can be utilized to capture web based or human interface interactive transactions; which are much fewer in volume and usually less significant in business financial impact.
Selecting which business transactions should be formatted in traditional EDI or XML-EDI depends on the destination of each transaction. If the recipient is another computer, then the use of traditional EDI formats is favourable. If the intended recipient is a human operator that would manually interpret the information and possibly modify or enhance its content, then XML-EDI is the solution.
WCO Data Model
Customs data harmonization and standardization began with the G-7. The finance ministers of the world’s seven largest economies initiated Customs data standardization and made significant progress, but more had to be done. Because of the complexity of the issue, the scope of the G7 effort was limited to basic Customs processes. The WCO participated in the G7 effort and recognized the need for a more comprehensive data set. As the international forum for Customs’ administrations, the WCO was the ideal organization to continue this standardization. The WCO gave all member Customs’ administrations, regardless of the size of their economy, the opportunity to participate and opened the standardization process to the business and trade organizations.
Version 1 of the WCO Customs Data Model was issued in 2002. The WCO Secretariat formed the Data Model Project Team (DMPT) under the Information Management Sub-Committee (IMSC) to continue the standardization work. WCO Members participating in the DMPT were asked to analyze their Customs’ requirements and identify additional information that had to be added to the more robust WCO Customs Data Model. In 2003 the WCO published Version 1.1 of the WCO Customs Data Model in order to cover the Supply Chain Security requirements, which include the list of 27 data elements required to identify high risk consignments.
In 2005, Version 2 of the WCO Customs Data Model was released. This version included Customs and transportation data for release of goods at the border. As the WCO was working on Version 2, the “whole-of-government-crossborder-single window” concept was gaining momentum. Such a single window is the government single entry point for the submission of international standardized data and messages for import, export, and transit of goods, conveyances, equipment, and crew. The WCO recognized the significance of the single window and realized that there was no forum for developing a whole-of-government set of data. WCO Members decided to fill this gap and agreed to include single window requirements in the Data Model. Thus the WCO Customs Data Model became the WCO Data Model.
At the WCO Council’s June 2008 Sessions, the three basic components of the WCO Data Model Version 3 were approved. These basic components; the Business Process Models, the Data Sets and the Overall Information model, are the foundation for the further development of Version 3 (phase 2) and to result in Version 3 of the WCO Data Model. The WCO Data Model Project Team plans to release the fully fledged Version 3 in December 2009. Version 3 will make it feasible to implement and operate a Single Window environment.
This version includes requirements for Customs, Agriculture, Food Safety, Marine Safety, Statistics, Immigration (crew) and Environment Protection (Basel Convention). Version 3 is, it must be stressed, not the Customs interpretation of trade agencies’ requirements. For Version 3, DMPT members were asked to consult with trade agencies in their countries and identify the additional requirements that needed to be added to create the required single window data standard. In addition to Members’ input, representatives from transport, agriculture, environment and marine safety were asked to contribute and review the contents of Version 3.
The WCO concluded that a whole-of-government data set needed a corresponding whole-of-government message called the WCO Government Cross- Border Regulatory Message (GOVCBR). The concept and development of the GOVCBR message has been approved by international standardization bodies and publish as a UN/EDIFACT message. In addition to approval of the GOVCBR message, the contents of version 3 are consistent with the international standards of the United Nations Trade Data Elements Directory (UNTDED).
The GOVCBR message will cater for the combined export/import declaration referred to as a Compound message. It also caters for a very small collection of data requirements for, say, a simplified import declaration plus AEO authorization and the use of the UCR. This smaller collection of data requirements is referred to as a “Mini Message”. The message will also cater for the response from the Cross-Border Regulatory Agencies to the Declarant.
There are many in government and trade who believe that the WCO Data Model will result in the submission of more data. In the WCO’s view this is not the case. All of the elements in version 3 have been vetted against Member countries’ legal requirements. Many of the elements are not routinely part of the Customs declaration. While they may not be part of the Customs declaration, it is information that is sent in some way, in some format, or on some form in support of the declaration process. A careful analysis will reveal that the elimination of multiple submissions of redundant data will actually result in a reduced amount of data needing to be submitted. From a practical Customs application point of view this is a rather simplistic statement. None of the supply chains in the world are nearly mature enough to include both commercial and government regulatory information in a single UN/EDIFACT submission. However, what it does provide is an aspiration which in time will yield a result. The US 24-hour Advance Manifest Rule was in itself a contentious issue in 2002, yet proved almost a non-event with the world’s exporters grudgingly accepting the challenge.
The WCO SAFE Framework of Standards uses a subset of the Data Model. The realisation of the Customs-to-Customs pillar of the Framework will be possible with the Data Model. The Revised Kyoto Convention (RKC) encourages signatories to adopt international standards and develop single window-style processing. The Customs Enforcement Network (CEN) and e-ATA will adopt the international standards of the WCO Data Model too. Considerable time, effort, knowledge, and talent have gone into the development of version 3 of the Data Model which will provide stability and predictability for business-to- government and government-to-government exchange of data. There is nothing in the “standards” world that can match the contents and comprehensiveness of version 3. The WCO encourages its members to use the WCO Data Model as the standard for Customs processing and the single window.
Where is SARS in relation to the WCO Data Model?
SARS procured a DEC Alpha server, and DECEDI software management application which would manage the set up of EDI trader profiles; the translation of trader EDI messages into Customs in-house format; and, convert Customs response messages into UN/EDIFACT syntax for transmission to the trader’s EDI service provider. Telkom X.400 serves as Customs-2-Buisness communication protocol. The DEC Alpha server ran OpenVMS as the operating system. At that point in time, Customs’ sole application system was the Customs Automated Processing of Entries (CAPE) system. EDI was successfully piloted at ORTIA in September 1998, and was later rolled out nationally in March 1999.
From 1999 to date several significant developments occurred in the Customs EDI space. Various projects under the SARS’ SIYAKHA programme (2001) saw the introduction of the Exports, Common Customs Area (CCA), and Manifest Acquittal (MAS) systems. Together with CAPE, these are the primary business applications which are now supported by an EDI frontend. In the last few years, developments on other applications have also been released into production, as introduced in the paragraphs below.
In 2004, SARS introduced internet-based EDI, secured by Public Key Infrastructure (PKI). This has enabled many more traders to participate electronically with Customs. A PKI is an automated system that manages the generation, maintenance, and delivery of encryption and digital signature keys. Both key types, encryption and digital signature, have two related components: a public-key component that is accessible to all users, and a private-key component that must be secured from access by others. The public key and other identification information are stored in a digital certificate that is digitally signed by a Certification Authority (CA). The digital signature of the CA on the digital certificate binds the identity of the end-entity with its public-key. It also guarantees that the public key has not been tampered with.
The EDI Internet (EDIINT) standards were developed by the Internet Engineering Task Force (IETF) to address issues concerning secure communication techniques for EDI messaging over the internet using AS1/AS2. Applicability Statement 1 (AS1) and Applicability Statement 2 (AS2) are security standards defined by the IETF that allow business transactions to move securely over the Internet. The AS1 standard secures file attachments over e-mail. However, leading e-Commerce players are moving to AS2 which is based on Hyper Text Transfer Protocol and its secure form (HTTP and HTTP/S) and the secure form of Multipurpose Internet Mail Extensions (S/MIME). PKI security measures address privacy of the message, its authenticity integrity, and non-repudiation.
Collaboration with certain Other Government Agencies (OGAs) over the last 3 to 4 years has provided an inroad into the single window environment – SARS receives permit control and quota information from the International Trade Administration Commission (ITAC), and provides status response information to ITAC. Similarly, SARS established an EDI interface arrangement with the South African Reserve Bank (SARB) for the exchange of customs import/export information to facilitate forex reconciliation. SARB fixed exchange rates are also distributed by SARS to the trade to enable importers for currency conversion and duty calculation purposes.
The Customs and Excise Act, Section 101A provides for electronic communication of customs information via EDI. A specific requirement is that all electronic traders must enter into an interchange agreement with SARS; and, successfully meet SARS’ message interchange test requirements, before being permitted to file transactions electronically. Simultaneously, such traders register as approved electronic traders with Customs. Most traders operate EDI through 3rd party computer bureau service providers. These offer an entire supply chain documentation management, warehousing, costing, track and trace, and import / export planning functionality, with UN/EDIFACT translation capability and communication interface with Customs.
SARS EDI requirements are published in the SARS EDI Users Manual. This document specifies all requirements in relation to the interchange agreement, registration as an electronic trader, and provides detailed technical specifications in regard to all EDI interchange messages (mapping, branching diagrams, standardised codes lists, etc.)
An EDI Helpdesk is available to service providers should any EDI related queries arise. Computer bureaus/Service providers have access via the Internet to SARS’ Remedy call logging facility where calls may be logged, tracked and traced. The calls are automatically routed to the SARS EDI Administration team for speedy resolution ensuring improved client service.
The SARS EDI Gateway is identified as a mission critical system. Due to this status, a high availability environment is implemented by including redundancy within this infrastructure. A second EDI Gateway is deployed within the Disaster Recovery (DR) site, to ensure maximum uptime. Should one of the systems become inoperable, the other will take over with the minimum of disruption to the trade.
The existing EDI architecture has proven stable for almost a decade, providing support for almost 20 million declaration, manifest and response messages per year. With these transaction volumes set to double in a short period of time, several questions are posed relating to maintenance, capacity planning, and possible migration to a more sophisticated architecture in support of Customs new generation integrated technology solution.
South African Customs – a Decade of EDI
By comparison to many developed countries, South African Custom’s uptake in the use of EDI has been quite remarkable. Unlike many other countries, South Africa chose not to offer ‘Direct Trader Input’ as a communication option to the customs industry. Its initial electronic interface came in the form of a diskette system which enabled traders (via their service providers) to present customs clearances on magnetic media for ptocessing at Customs. Third Party Service providers (computer bureaus as they are known) to the logistics and freight industry have been the catalyst here and are responsible nowadays for more than just the provision of IT systems and support. These entities have been around long before the introduction of EDI in 2000, and a few of these companies have a sound knowledge of the customs business, outside of the IT domain. Yet, credit for the initial drive in getting the SA Customs’ EDI programme off the ground is due to Tertius Joubert, an e-commerce expert, who prior to his involvement with Customs, had several years experience and success with the implementation of EDI in the private sector. Having a detailed knowledge and understanding of the logistics industry, and over the past 10 years a significant grasp and insight on customs matters, Tertius has provided SARS invaluable support in putting it on the e-commerce map. This expertise coupled with ongoing developments in the use of EDI and XML EDI areas means that the local Customs Adminsitration is well positioned to meet the challenges of the future.
Current Approach and Achievements 2009/2010
The main objective of the e-initiatives in SARS is to increase the use of Electronic Data Interchange (EDI) for both Cargo and Goods declarations and thereby creating an environment where the submission of paper manifests, declarations and supporting documents can be limited to the bare minimum and thereby taking SARS Customs as well as the Clearing and Forwarding fraternity as close as possible to the “paperless environment”. In a move to create a near paperless processing environment SARS Customs has established several mechanisms for electronic submission (via EDI) of information:
- Electronic submission of declarations;
- Electronic submission of manifest information; and
- Electronic receipt of e-release notifications
Given the benefits associated with improved levels of automation, document processing, sorting, filing and storage it is in SARS Customs interest to improve the adoption of these electronic submission mechanisms over the manual submission alternatives still in use today.
In order to provide the required legal backing to reach the above-mentioned goal the following amendments to the Customs and Excise rules were published:
- Government Notice R 699 dated 29 June 2009. This Government Notice provides for the use of electronic messages for the release/detention of consignments.
- Government Notice R .814 dated 31 July 2009. This notice provides for the mandatory submission of cargo and goods declarations using Electronic Data Interchange.
- Government Notice R.1178 dated 11 December 2009. This notice provides for the requirements in respect of the provision of supporting documents.
The import and export systems have been enhanced to comply to and support the above legislation. Emphasis has however been placed on goods declarations:
- A mini call centre was established in October 2009 at head office, engaging traders regarding the above mentioned government notice R.814; final date on manual submission of more than 20 declarations per month.
These efforts increased the electronic international import declaration submission figures from 92% to 98% between June 2009 and August 2010, international export the figures increased from 64% to 96% in the same period.
For the BLNS trade, BLNS import EDI uptake the figures increase from 1% to 27% between June 2009 and August 2010, BLNS exports increased from 20% to 89% in the same period.
- Insight behind the WCO Data Model (mpoverello.wordpress.com)
- WCO – 2012 is the year of Connectivity (mpoverello.wordpress.com)
- New Zealand releases trade implementation guidelines for GOVCBR (mpoverello.com)