I have been developing software for 23 years now (perhaps for a lack of being able/allowed to do anything else marketable). Things have changed so fundamentally in that fairly short (at least to me) number of years:

• Access to information that helps us practitioners in our work, as it is being developed and as it changes. (The WWW, Google, etc.)
• Development of new technologies and techniques to simplify the mechanics of software development and make it accessible to people who were not traditionally part of the IT “priesthood”. (PC’s, Object-Oriented Programming, new development methodologies, etc.)

However, we become so concerned with the sizzle sometimes that we forget about the steak, and when that happens, we have a recipe for a poor project outcome. We like to show our clients how easy it is to do something in SharePoint, like create a custom list, add a column, change the name that appears in the column, add calculated fields and so forth. Why not? It is one of the very strengths of the platform.

Rewind back a few years to the days of the IT “priesthood”. The least little change required filling out forms in triplicate and sending them to various departments for review (hopefully, the department head wasn’t on vacation or ill!) This grew from IT’s responsibility to ensure system integrity; therefore, they were given the authority to put controls in place to help ensure that integrity.  We want to let users do things for themselves that they had to rely on us to do before, because it takes time away from our “important” work, and the users tend to have an inflated sense of where their problem fits in the grand scheme of things.  The users begin “doing [some of] the jobs we don’t want to do”, to borrow a catchphrase from the previous administration.

Don’t get me wrong, the very reason IT exists is to make our clients more productive and to help them make better decisions. I wouldn’t propose to limit our users’ authorized activities to “free speech zones,” but to simply not allow them to carry weapons nearby that could kill the SharePoint beast and bring us all into that dark place with wailing and gnashing of teeth (not quite the boardroom scene with Donald Trump, but close enough.)

My modest proposal involves taking a small step backward: moving to the place where users cannot, intentionally or otherwise, damage the normal operation of the system.

Perhaps if a user proposes a change to some field in a list somewhere, or a new workflow, or a new calculation, some process should take place to determine:

• Why the change is called for
• Alternatives to the change
• Impacts to other portions of the system
• How to test the change
• Contingency plans in case the change creates problems.

I know, Software Development 101 stuff, right? Some will say, “But we’re not writing code here; it should just work.” To clear the misconception, code is being written on your behalf when you do these things, and all code has bugs. (The last release of IBM’s OS/360 had over 13,000 known bugs.) As far as “just working” is concerned, even that requires work.

As we begin developing SharePoint solutions for mission-critical applications (I would not comfortably recommend this to any client), we do our clients a disservice when we fail to introduce rigorous controls into the process. We would do this for any other application platform; SharePoint should never get a free pass just because it’s new(er).

In our example, I would call the class WarehouseLocation; include properties Location, BinNumber, and Id (to correspond to the ID of the list item); and then include a GetWarehouseLocationById method to retrieve a single warehouse location by the specified list item ID. If the warehouse location needs to be updated, a method called UpdateWarehouseLocation could update the warehouse location list item according to the contents of the WarehouseLocation object.

To simplify maintainability, I also like to create string constants that identify internal field names for the columns in the list as part of the object definition. In this way, if I do have to change an internal field name that I am referencing, I will only have to make the change in one place. I would also make these string constants public so they can be referenced from related classes.

To take the “n-tier” concept further, much of the custom code that would be initiated by an event receiver, workflow, etc. would actually act on these business objects that are abstractions of SharePoint list items.

The problem is a familiar one for those who work in the document management world and need to get the last approved version of the list item. The SharePoint Object model gets us to the red zone, but doesn’t bring us all the way to the goal line. To Microsoft’s credit, they understand it’s a problem and give us a workaround.

I wrote this function, based on Microsoft’s example, to return all of the items in a custom list that have at least one approved version. In that case, for each item, it will grab the last approved item and stuff it into a dataset. The data table will have the same name as the display name of the list.

And yes…it uses LINQ.

        private static DataSet GetListItemsWithLastApprovedVersions(SPWeb aWeb, string aListName)
        {
            SPList myList = aWeb.Lists[aListName];
            DataSet ds = new DataSet();

            DataTable dt = myList.Items.GetDataTable();
            dt.Rows.Clear();
            ds.Tables.Add(dt);

            DataTable aDataTable = ds.Tables[0];
            myList.Items.Cast<SPListItem>().Where(
                t => t.Versions.Cast<SPListItemVersion>().Any(
                    v => v.Level == SPFileLevel.Published)).ToList().
                    ForEach(t =>
                    {
                        int aListItemId = t.ID;
                        SPListItemVersion lastApprovedVersion =
                            t.Versions.Cast<SPListItemVersion>().
                                Where(v => v.Level == SPFileLevel.Published).
                                OrderByDescending(v => v.VersionId).
                                First();
                        DataRow aRow = ds.Tables[0].NewRow();
                        ds.Tables[0].Rows.Add(aRow);
                        foreach (SPField aField in lastApprovedVersion.Fields)
                        {
                            if (aDataTable.Columns.Contains(aField.StaticName))
                                aRow[aField.StaticName] = lastApprovedVersion[aField.StaticName];
                        }
                        aRow.AcceptChanges();
                    }
            );

            return ds;
        }

After some Googling, I noticed Rob Fisch’s page, that describes how to query the UserData view. This view contains all of the content for all of the lists in the SharePoint content database, melded into a single table.

The Holy Grail for me was to create SQL Server views in the content database that would represent the data in my custom lists. In the SharePoint API, each list exposes a property called “SchemaXml”. Each <Field> element in in the SchemaXml property includes two attributes–one called “ColName” (the column name in UserData) and “DisplayName” (the user-specified column name in the SharePoint list.)

Below is the code I wrote to generate the views. I called this from the site provisioning code of my site definition:

Given the use of LINQ in these functions, I should point out that you need .NET 3.5 to make this work.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint;
using System.Data.SqlClient;
using System.Xml;

namespace CMSSiteDefinition
{
    class DBUtil
    {
        public static void GetViewDDL(string listName, SPWeb theWeb, SqlConnection aConn)
        {
            SPList theList = theWeb.Lists[listName];
            XmlDocument aDoc = new XmlDocument();
            aDoc.LoadXml(theList.SchemaXml);
            Dictionary<String, String> fieldMap = new Dictionary<string, string>();
            IQueryable<XmlElement> fieldElements =
                Queryable.AsQueryable<XmlElement>(aDoc.GetElementsByTagName("Field").Cast<XmlElement>())
                .Where(e => e.Attributes["ColName"] != null && e.Attributes["DisplayName"].Value.Length > 0);
            int longestColName = fieldElements.Max(e => e.Attributes["ColName"].Value.Length);
            fieldElements.ToList().ForEach(e =>
            {
                fieldMap[e.Attributes["DisplayName"].Value] =
                    e.Attributes["ColName"].Value;
            });

            StringBuilder sql = new StringBuilder();
            string viewName = String.Format("_rpt_{0}_vw", listName.Replace(" ", String.Empty));
            sql.Append(string.Format("IF OBJECT_ID(N'dbo.{0}', N'V') IS NOT NULL\n", viewName));
            sql.Append(string.Format("DROP VIEW dbo.{0}", viewName));
            SqlCommand dropCmd = new SqlCommand(sql.ToString(), aConn);
            dropCmd.ExecuteNonQuery();
            sql.Length = 0;
            sql.Append(String.Format("CREATE VIEW dbo.{0} AS SELECT", viewName));
            bool firstField = true;
            foreach (string aKey in fieldMap.Keys)
            {
                sql.Append(String.Format("\n{0}\tUD.{1}\t[{2}]", !firstField ? "," : String.Empty,
                    fieldMap[aKey].PadRight(longestColName), aKey));
                firstField = false;
            }
            sql.Append("\nFROM dbo.Lists LT INNER JOIN dbo.UserData UD ON LT.tp_ID = UD.tp_ListId");
            sql.Append(String.Format("\nWHERE LT.tp_Title = '{0}'", listName));
            SqlCommand aCommand = new SqlCommand(sql.ToString(), aConn);
            aCommand.ExecuteNonQuery();
        }

        public static void GenerateReportViews(SPWeb web, SPSite site)
        {
            // Create views to be used in reports
            SqlConnection aConn = new SqlConnection(site.ContentDatabase.DatabaseConnectionString);
            aConn.Open();

            // Select generic lists to create views
            IQueryable<SPList> genericLists = Queryable.AsQueryable<SPList>(web.Lists.Cast<SPList>())
                .Where(l => l.BaseTemplate == SPListTemplateType.GenericList);

            // Call GetViewDDL to create view for each list
            genericLists.ToList().ForEach(l => { GetViewDDL(l.Title, web, aConn); });

            aConn.Close();
        }

    }
}

As an example, if your site had a custom list called “My Custom List”, a view would be created in the content database called _rpt_MyCustomList_vw.

Note: If you have versioning enabled for the custom list, this resulting view will return all versions. If you only want the current version, you can modify the WHERE clause to include tp_IsCurrentVersion = 1.

Here’s a little bit of funkiness that can bite you if you aren’t looking for it: When you refer to a list item (SPListItem) in a custom list (SPList), both the SharePoint Object Model and web service expect you to specify the “internal name” for the field, which is not the same as the “display name”, which is the name SharePoint shows us in its web interface. SharePoint also lets you change the “display name” as much as you want, but leaves the “internal name” the same as it was when it was created originally.

I experienced this recently on a project where the SharePoint site was set up with a column with a display name of “Time in Days” when it was originally created. The customer decided to change the semantics of the column to mean hours instead of days, so the display name was changed to “Duration” but the internal name remained “Time_x0020_in_x0020_Days”.

In terms of preventing the application from throwing exceptions, it’s a good thing because the event receiver code will continue to refer to the same field. However, it creates an issue for anyone who has to maintain the receiver later on if (s)he isn’t aware of the meaning of the data in that field.

There are two “must-have” tools that allow you to see internal names for the fields and even test your SPQuery XML for correctness:

• Stramit SharePoint 2007 Caml Viewer
• U2U CAML Query Builder

You can’t beat the price on these tools (Free) and they will save you time and sanity.

Server Application Unavailable

The web application you are attempting to access on this web server is currently unavailable.  Please hit the “Refresh” button in your web browser to retry your request.

Administrator Note: An error message detailing the cause of this specific request failure can be found in the application event log of the web server. Please review this log entry to discover what caused this error to occur.

The application event log had a message with this description:

Description:
Failed to execute request because the App-Domain could not be created. Error: 0×80070005 Access is denied.

I had to give read/write permissions to the local IIS_WPG group in order to make this problem go away.

Needless to say, I’m talking about a development environment here, and not production.

Thanks, Maxime!