Jan 092014
 

Colectica Repository and Portal have many settings that can be adjusted by the system administrator. These include settings such as language sort orders, synonyms, item types to display, and appears within configuration. A developer creating Colectica Addins recently asked how this API could be used for storing settings in custom extensions.

The Repository settings consist of a key associated with an optional string and long integer values. There are four web service calls that can interact with the Repository Settings store.

// Removes the repository setting.
void RemoveRepositorySetting(string settingName);

// Sets the repository setting.
void SetRepositorySetting(RepositorySetting setting);

// Gets the repository setting.
RepositorySetting GetRepositorySetting(string settingName);

// Gets all of the repository settings.
Collection<RepositorySetting> GetRepositorySettings();

The Repository Setting class looks like this

public class RepositorySetting
{
    public string SettingName { get; set; }
    public string Value { get; set; }
    public long? LongValue { get; set; }
}

Our built in settings use this key value store, normally with a json representation of our settings stored as the value. Here is an example of how we could store synonyms for DDI item types using the settings API.

// A key defined somewhere in the class
string SingularItemTypeNamesKey = "Colectica:Portal:SingularItemTypeNames";

// Add some synonyms
Dictionary<Guid,string> names = new Dictionary<Guid,string>();
names.Add(DdiItemType.DdiInstance, "Project");

// Create the repository setting to store
RepositorySetting setting = new RepositorySetting();
setting.SettingName = SingularItemTypeNamesKey;
setting.Value = JsonConvert.SerializeObject(names);

// Create the web services client and set the setting
WcfRepositoryClient client = GetClient();
client.SetRepositorySetting(setting);

Similarly, here is an example of how you could retrieve the singular synonyms.

// Create the web services client and retrieve the setting
WcfRepositoryClient client = GetClient();
var setting = service.GetRepositorySetting(SingularItemTypeNamesKey);

if (setting == null || setting.Value == null)
{
    return new Dictionary<Guid, string>();
}

// If the setting exists, serialize it
var results = JsonConvert.DeserializeObject<Dictionary<Guid, string>>(setting.Value);

Best Practices

Use a unique key name. We suggest a colon separated key in the form Organization:Product:Description. For example, MyOrg:MyCustomProject:MySettingName.

Store many settings using a single key. For example, if you are storing translations you could store all of your translations in one key with all the data stored in json or xml in the value. Our example used a dictionary. You could also use lists, a custom class, etc. This allows you to minimize the number of web service calls over the network.

Cache your settings once they are retrieved. If the settings will be used multiple times, cache them in your program so you do not have to call the web services repeatedly.

Sep 182013
 

In an enterprise, there are often numerous data management systems that contain specific data for different domains. These data silos are often difficult to integrate when creating a holistic view of the data life cycle. This post will detail how to create a web services layer over existing databases that will expose DDI metadata. DDI is an open standard for documenting the data lifecycle. Using DDI, multiple data sources can be combined to create the ‘big picture’ view.

The Read Only View

The simplest way to expose DDI from an existing system is to create a Web Services facade. This facade will implement several functions that are needed to expose a data source as an ISO 11179 repository, a standard on which DDI is based. One option is to allow the existing system to perform all updates and management of its own data, while providing a read only view to other systems for integration. To accomplish this with DDI and Colectica, the following abilities should be present in the web services facade.

Viewing an Item

The most basic function of a repository is to retrieve an item. In Colectica, this will most likely be an item serialized as DDI 3. Given an ISO 11179 international registration data identifier (IRDI), the web service calls GetItem and GetItems will return a RepositoryItem object containing information about an Administered Item and its XML serialization.

Versioning

An ISO 11179 repository manages multiple versions of Administered Items. The web service call GetVersionHistory can list all versions of an item in a repository.

Relationships and Search

Searching for relationships between items is needed to efficiently browse items in a hierarchy. To enable a read only view, the web services facade should implement GetRelationshipBySubject and GetTypedRelationships, and GetSet to enable relationship searching. To enable text based searching, the web services facade should implement Search and SearchTypedSet.

Optimizations

Often when browsing, only basic information about an item is needed for display. This often includes the item type, its identity, and a basic label. Implementing GetRepositoryItemDescriptions to provide this basic information can speed up user interactions with the web services layer.

Summary

These 9 abilities encompass all that is needed to create a read only view on top of an existing data management system. These functions also enable creation of local checkouts of the items.

  • If the system already manages items using the DDI standard this is very straight forward.
  • If the system manages data in the DDI content model but not in a DDI serialization of versioning system, a translation layer may be required for the serialization and identification beneath the web service facade.
  • If the data managed by the system is not part of the DDI content model, the data should most likely not be put behind a web services facade. It should instead be documented using the DDI standard. This includes describing variables, datasets, and concepts that describe the data.
Mar 102012
 

I received a followup question to my post about registering 11179 items in the Colectica Repository. This question involves working with the Colectica SDK and its DDI model in conjunction with the Repository.

How do I connect to the Repository and retrieve a DdiInstance, such as the YourDdiInstance() method in your previous post?

First we will create the repository client. In this example we will use the built in Active Directory authentication and send the credentials of the user running the program (The user who asked the question uses the Active Directory authentication and roles). Notice the username and password are not specified as they were in the example from my previous post.

// Create the web services client
var client = new WcfRepositoryClient("localhost", 19893);

If we know the item’s identification, we can retrieve the item. If not, we can perform a search on the repository. The basic GetItem has many variations with different processing options, retrieving item lists, and sets of relationships. The simple GetItem and GetLatestItem are shown below.

// Get an item by 11179 identifier
IVersionable item = client.GetItem(id, agency, version);

// Or get the latest version
item = client.GetLatestItem(id, agency);

DdiInstance instance = item as DdiInstance;

To make it extremely easy to work with DDI items in the Colectica Repository, we will wrap this client with additional methods using the DdiClient. This also avoids the type checking and casting if you want to access properties of the DdiInstance not present on IVersionable. There are also similar methods for each DDI item type as the one shown below!

// Wrap the web services client
DdiClient ddiClient = new DdiClient(client);

// Get the Ddi Instance
DdiInstance instance = ddiClient.GetDdiInstance(
  id, agency, version, ChildReferenceProcessing.Instantiate)

The client calls allow controlling how child items are populated. If we have an unpopulated DdiInstance, we can use a similar method call to fill it with data and find its children.

// an unpopulated item with its identification. Children items 
// may come back  from the client as unpopulated depending on 
// the child processing that is selected. Here is an example of 
// how to populate such an item with the client

DdiInstance instance = new DdiInstance() 
{ 
  Identifier = id, 
  AgencyId = agency, 
  Version = version,
  IsPopulated = false 
};   

// Populate the Ddi Instance
ddiClient.PopulateDdiInstance(instance, 
  false, ChildReferenceProcessing.Instantiate);


// Or as shown in Update 1 of my other post, populate the entire 
// item hierarchy from the Repository
GraphPopulator populator = new GraphPopulator(client);
instance.Accept(populator);

// Do something with the instance
foreach(StudyUnit study in instance.StudyUnits)
{
  //.....
}

Cheers!
Dan

May 222011
 

Colectica Repository can store any metadata items that conform to the ISO 11179 naming scheme for registered items. The DDI 3 Addin for Colectica Repository additionally allows for indexing of contextual and relationship information. Here is a brief code example showing how DDI 3 items can be registered in the Colectica Repository.

First we will create some DDI 3 based metadata using the Colectica SDK. If you don’t have the SDK you can create the DDI 3 by hand or using your favorite XML library.

// Create a DDI 3 Concept using the Colectica SDK
Concept concept = new Concept() { AgencyId = "example.org" };
concept.ItemName["en-US"] = "Given Name";
concept.Description["en-US"] = @"A character-string (e.g. `Billy' and `Peter')
        given to people as a first name (or, in most Western countries, as a
        middle name), usually shortly after birth.";

// Create a DDI 3 Question using the Colectica SDK
Question q1 = new Question() { AgencyId = "example.org" };
q1.QuestionText["en-US"] = "What is your first name?";
TextDomain domain = new TextDomain();
domain.Label["en-US"] = "First Name";
q1.ResponseDomains.Add(domain);

// Link the question and concept
q1.Concepts.Add(concept);

Then we will create the repository client, using the supplied credentials.

// Create the web services client
var client = new WcfRepositoryClient(
    "username", "password", "localhost", 19893);

We can Register any object made by the Colectica SDK using the built in mappings.

// Register a 11179 administered item using
// the Repository Client helper functions
client.RegisterItem(concept, new CommitOptions());

Alternatively, we can access the web services layer and construct the proper SOAP payload.

// Register a 11179 administered item using the Web Services directly
Collection<Note> notes = new Collection<Note>();
string serialization = q1.GetXmlRepresentation(notes).OuterXml;
RepositoryItem ri = new RepositoryItem()
{
    CompositeId = q1.CompositeId,   // agency, id, and version
    Item = serialization,           // item's serialization
    ItemType = q1.ItemType,         // model defined item type identifier
    IsDepricated = false,
    IsPublished = q1.IsPublished,
    IsProvisional = false,          // only used in the local repository
    Notes = notes,                  // notes about the item being registered
    VersionDate = q1.VersionDate,
    VersionRationale = q1.VersionRationale,
    VersionResponsibility = q1.VersionResponsibility
};
client.RegisterItem(ri, new CommitOptions());

As you can see, we added both the DDI concept and the DDI question item to the repository. The Colectica SDK has methods to gather all items that are linked and create sets of items to be registered. It also has the ability to detect changed items automatically, so a program can quickly determine which items should have new versions registered after a user action.

Update 1: Registering items in a DDI instance

Here is more information about registering items in the Repository based on some followup questions.

When adding a concept to a question, the hierarchical relationship is established. Is this inline, or by reference at the XML level?.

The DDI 3 standard allows for either including items inline or by references in many locations. Colectica Repository will process and store items using either format. If it is a DDI 3 item, the Repository will additionally index the text and relationship information about the item using the DDI 3 Addin. Note that only the item being registered and its relationship are processed, each item must still be registered individually or in a batch operation.

Colectica Designer will always use the referencing mechanisms in DDI when interacting with the Repository. This is for speed of processing and to allow the easiest sharing and harmonization of items across multiple Studies and Instances. You can learn more about how Designer determines item boundaries by reading about Concise Bounded Descriptions.

How can I register all items in a DDI Instance? How can I update only the changed items in a DDI instance?

If you are using DDI with an XML library, you can use the following xpath queries to find all the items in your Instance to register.

//*[@isVersionable]
//*[@isMaintainable]

You can then loop over the XML nodes returned by the XPath and register the results. If you are using the Colectica SDK, you can find all items in an Instance as follows:

// obtain your DDI Instance in some fashion
DdiInstance instance = YourDdiInstance();

// Find all items
ItemGatherer gatherer = new ItemGatherer();
instance.Accept(gatherer);
Collection<IVersionable> allItems = gatherer.Items;

// You can also find only the modified items
DirtyItemGatherer gatherer = new DirtyItemGatherer();
instance.Accept(gatherer);
Collection<IVersionable> changedItems = gatherer.DirtyItems;

How do I export a whole instance to DDI3?

There are several ways to export a DDI instance from the Repository. One way is to use the Repositoy’s command line tools and to write an XML document. Another is to programmatically export a DDI instance using the SDK:

// obtain your DDI Instance in some fashion.
// Only the identification is needed since we will populate the instance
DdiInstance instance = YourDdiInstance();

// populate the entire class hierarchy from the Repository
GraphPopulator populator = new GraphPopulator(client);
instance.Accept(populator);

// Create the XML document for the DDI Instance
DDIWorkflowSerializer serializer = new DDIWorkflowSerializer();
XmlDocument doc = serializer.Serialize(instance);

A third option is to use an XML library and construct the DDI Instance.