NIEM
NIEM Q&A – When to Use, Why, and Tips to Get Started
Dec 18th
Every month or so I receive questions about using NIEM in projects. The questions typically have the same theme, so I thought I would write up a quick post summarizing the questions I get most often and some answers to get you started.
Should I use NIEM in my project?
Do you need a single, canonical data model for exchanging data between systems? Do you have control of the systems you are integrating? Do you have an existing data model used within your systems? Is NIEM a requirement for your project/grant/contract?
If you answer ‘Yes’ to any of these questions then you may consider using NIEM. NIEM is best used where it will have the most benefit – describing exchanges of information between organizations & systems, harmonizing the language we use to talk to each other (i.e., Organization instead of Agency, Sur Name instead of Last Name), and defining requirements at the data exchange level. NIEM is not a one-size-fits-all tool and you will not get maximum ROI unless you have a strong use case for it and implement it properly.
Should I use NIEM for my database schema?
The short answer is probably not.
NIEM is an Xml data model and it’s best use is describing the data elements exchanged between systems. If you need to send data from System X to System Y then NIEM is a good candidate for doing this. You can define the elements you need to exchange and then build a NIEM IEPD to define the exchange. Using NIEM as a database schema does not always yield the best results. In most cases you are better off storing the data in an optimized way for your database platform and using adapters to translate data from the DB into NIEM.
How do I add new fields to NIEM?
In response to this question I would pose a different one – why do you need to add the fields? Can you derive them from other fields in the exchange? In most cases the 80% NIEM has built in will get you what you want. Fields like row-level timestamps or system specific ones may not be necessary and I would tactfully question those. If you find that you must add a field the best route to take is to define an extension schema in a custom namespace and add the fields there. Try to derive from base NIEM fields. For example, if you need to add a field to OrganizationType then derive your new type from OrganizationType and add the necessary elements.
Avoid adding the field and not specifying it in the schema properly. The pitfalls of doing so may not happen immediately but eventually one will and you’ll thank me for warning you.
I would question why you need to add fields as long as possible without delaying your implementation and deployment. In most cases you really don’t need the fields.
Can I use [Name_of_IEPD] for my project?
It depends. Does it have what you need? Are you ok reusing someone else’s work (one of the benefits of NIEM!), then yes. Sometimes existing IEPDs will be a good fit and get you the 80% or more. Other times the perspective may be vastly different from yours. You can certainly propose the use of an existing IEPD and have your business analysts and other technical staff review to see if it is a good fit.
The key to success with NIEM is that it brings together the business and technical teams involved in the exchange. If a particular IEPD is not able to bridge those gaps you may need to create your own. As NIEM adoption grows, more IEPDs are becoming available and many of them are coming from national groups that are looking at how things are done across the country to find the best 80% that fits most cases.
Getting Started with NIEM – Quick Tips
- Build a dictionary of the data you have or would like to have
- Search IEPDs to find one that matches your needs
- Map your data elements to the elements in the IEPD
- Store the data in a manner that makes sense for your database platform
- Translate your data from its source into NIEM and vice-versa
- Shoot for the 80% because in most cases that is what is really needed
More information about NIEM
For those of you working in the .NET Space check out my previous articles on NIEM + .Net
Visual Studio/.NET/WCF/LINQ and NIEM Tips
Nov 10th
Over the years while working with .NET and NIEM I’ve run into a few kinks that require a setting tweak or an extra step to get it to work.
.NET General
Choices for working with NIEM Xml (From the .NET Base class library. There are third-party tools out there to assist as well):
- Serialization / Deserialization – building objects from Schemas using utilities like Xsd.exe, Wsdl.exe, SvcUtil.exe or an open source tool such as WSCF/WSCF.Blue.
- System.Xml – .NET Classes to work directly with Xml Structures. Includes X-Path, Xsd, etc.
- Linq to Xml – LINQ queries designed and optimized for Xml. VB.Net supports Xml Literals within class files and intellisense in Linq to Xml queries.
Linq
- System.Xml.Linq.XDocument, XElement, et. al. are not Serializable. You cannot accept them as an input or return them as an output from services (Web/Sockets/Queues/etc) or any place that will attempt serialize them.
- With Linq to Xml using XDocument or XElement changes the axis for your query. If you parse a string to an XDocument your query starts at the root node. If you parse a string into XElement your query starts from the first element in the Xml and not at the root. Typically this isn’t a major hassle but the problem is that you may not notice it until you actually try a query at run time and do not get the results you expect. (See also Unit Testing!)
WCF
- MaxReceivedMessageSize and MaxStringContentLength will be your enemies. Every service you write, you will want to change those values from the default 65536 otherwise callers to your service will get a nice Fault Exception when they send you a message larger than that.
- MaxItemsInObjectGraph will need to be adjusted if you are going to be serializing large objects using direct serialization or you try to return a very large list or array of objects.
- MaxArrayLength should be adjusted in conjunction with MaxItemsInObjectGraph. Typically you’ll need to adjust both. You’ll use arrays more if you do any interop work between .NET and other technology stacks as Arrays are typically safer for Xml serialization.
- For interop scenarios your best bets are to stay with simple data types supported by Xml Schema (Xsd): String, Date, Integer, and arrays thereof. Adding specific .NET types like GUID, List<T>, etc will add additional schemas to your service contract and may cause issues with other technology stacks.
- Data Contract Serialization (the default) gives you less control over your Xml messages. Xml Serialization gives you fine grain control and is usually a better fit for working with NIEM in WCF.
Visual Studio
- Visual Studio solutions only allow an Xml Namespace to be defined once. If you try to import two XSDs that define the same namespace you will not be able to leverage the Xml Intellisense and document validations provided by the Xml parser in Visual studio. This really comes into play if you attempt to import LEXS digest and payload schemas in the same solution.
- Typically I would create a project in a VS Solution just for Xml Schemas. You can use a class library and it is not necessary to reference it from other projects. Schemas are used at the solution level, even when the schemas are in a project.
- While navigating an Xml document if you type an opening bracket, select an Xml element, and press tab twice studio will create an Xml fragment with all the required elements.
These are certainly not all the tips, but its a start. Feel free to contact me if you have some to share or have any questions!
Tweets Preserved: NIEM NTE Tweetbook
Nov 7th
Yesterday afternoon @NIEMNTE posted: “#NIEM 2009 National Training Event Tweetbook is posted on-line! http://bit.ly/3kOE90”
This is the first I’ve seen of a Tweetbook and the concept is really a neat one. Weighing in at 84 pages the NIEM NTE Tweetbook collects the tweets from 52 individuals who were actively tweeting during the conference using the #niemnte hashtag. A great way to get a glimpse of what happened during those three days in Baltimore.
If you’ve never seen a Tweetbook or are curious as to what value tweeting could bring to a conference, give it a read. There are lots of great tweets highlighted that show how much you can say in 140 characters.
LINQ to XML: Generating National Information Exchange Model XML Documents
Dec 18th
There are many ways to build XML documents but the main approach I have used in the past was to build an object model and leverage serialization. This approach was effective but costly in terms of serialization time and complexity of building the objects. I have previously blogged about using LINQ to XML to transform and parse NIEM XML documents but have not written about generating NIEM XML with LINQ.
For a new project at REJIS we were updating an existing interface to use NIEM XML. After some discussion with Sudhir Umarji I chose to use a pre-built N-Dex Booking IEPD. This provided the base with all the elements we would need to exchange. One of our senior developers began working with me to create a new service based solution.
Because we generally have a "vertical" market for our services we tend to build the service and the client. The approach we use is that the service returns the format that makes the most sense, which is always one of our domain object structures; a very simple object model for all the different documents and data elements we work with. The client uses a factory/adapter pattern where the adapters implement an interface and then through configuration we specify the concrete adapter to use at runtime. In code we create the instance using System.Activator. This gives us the most flexibility with outputting to different formats, databases, etc. One of our services using this pattern has three adapters in production and all are called from the same client program.
For this new service endeavor we would be using a similar approach but this time we would be using an adapter to transform records into the N-Dex Booking IEPD format. For this adapter I wanted to try generating the XML with LINQ. Since we work in VB.NET we have the advantage of using XML literals which turned out to be a pretty useful way for us to generate NIEM XML. It allows us to visually see the document we would be creating and break it into manageable chunks to control the output.
The problems encountered mostly revolved around managing the IDs and creating the associations as well as guarding against null elements in our object model and preventing them from outputting malformed XML. We also had to look a little into how to handle "for each" loops. This can be accomplished by using an embedded expression such as "<%= From person in MyRecord.People Select BuildPersonElement(person)%>" (Imagine here that BuildPersonElement takes a person object and outputs the appropriate XML).
Some tips:
- Identify the static parts of your document and break those into separate methods.
- A lot of structures in NIEM are very repeatable, identify those, and provide methods to assist in building them.
- Use good defensive programming in your methods to protect yourself from nulls. If your methods do not have all the required data elements to build the XML structure simply return nothing (unless it makes sense to throw an exception).
- Identify all your element IDs that will need to be created and manage those with lists.
- Use the VB.Net Import xmlns statements to import all your namespaces into your code file. This will help LINQ keep the namespaces prefixed and reduce the size of your XML.
- If you bring the schemas into one of the studio projects you can use that to get intellisense. For N-Dex based schemas this may not work because separate schemas that reference the same namespaces are used for the digest and structured content. We included the digest schemas and used studio to validate our XML instances.
