The User Experience Challenge

Just a decade ago iframes were frowned upon on the web. Not all browsers supported them and some organizations even applied browser policies to block iframes. Fast-forward to today and iframes are a staple of web development and the cornerstone of both the Facebook and SharePoint 2013 app architecture. But that doesn't mean iframes don't come with some unique challenges. One challenge I've been particularly concerned about is the user experience (UX) of apps for SharePoint and how apps from the Office Marketplace appear in existing SharePoint sites.

Thankfully, MSDN has fantastic resources and guidance on UX design for apps in SharePoint 2013, which should be considered the definitive guide for designing SharePoint app UX. In this post I will explore some of these resources and provide samples and lessons learned for optimizing the user experience of apps for SharePoint 2013.

SharePoint Apps 101

To understand the UX challenge, it helps to have some basic understanding of SharePoint’s app architecture. Apps for SharePoint deliver additional capabilities, but run outside of the host SharePoint site. These apps can be SharePoint-hosted as HTML/client-side script OR Cloud-hosted anywhere in the world using any web technology (ex: PHP from a Private Cloud web server on-premise, ASP.NET MVC from Azure, Ruby hosted with a 3rd party ISV, etc). The important point is that in all cases, the app’s code runs outside of the SharePoint site collection that is consuming it. Yes, this is even the case with SharePoint-hosted apps, which get hosted in a different domain in the farm. For more information on app hosting options, see Hosting options for apps for SharePoint.

Apps for SharePoint can be displayed in two basic ways.  The first option is to display app pages in full-screen (full content area of a browser), which is the only required experience an app must provide. In this configuration, apps are launched from links or custom actions in SharePoint, which redirect to the remote app host along with some contextual information requested by the app developer. When this redirection occurs, users are taken completely out of the site and to the app (as the browser URL will indicate).  The second delivery option is to display the app pages in an iframe via SharePoint dialog or the new app part.  SharePoint dialogs are similar to 2010 and can be leveraged in an app through a custom action.  App parts are very similar to Page Viewer web parts, but have a mechanism to dynamically pass contextual information and app part properties to the page they display. With SharePoint dialogs and app parts, users stay in their SharePoint site, but have iframe windows into app pages.

Regular SharePoint Page	App Page Displayed in Full-Screen
App Page Displayed in a SharePoint Dialog	App Pages Displayed in App Parts

Now back to the UX challenge. Imagine a SharePoint 2013 site with a number of app parts. Without meticulous design, each of these app parts could appear like completely separated experiences. With the introduction of the Office Marketplace, app developers will be developing apps for thousands of tenants, each with their own unique style/branding. So how does an app developer write apps that take on the appearance of a tenant's environment? The answer is a little different for each display option:

Pages Displayed in Full-Screen

For app pages displayed in full-screen, SharePoint redirects users away from the SharePoint site and to the location where the app is hosted (along with some contextual information passed through URL parameters and/or POST messages). When created in Visual Studio, SharePoint-hosted pages reference the host site styles by default, while cloud-hosted pages start relatively un-styled.  Without the developer taking an action, it will seem that SharePoint-hosted pages automatically fit in with the host site while cloud-hosted pages have nothing from the host site to style them. Luckily, app developers for cloud-hosted apps can inherit the appearance of a specified SharePoint site by wrapping app pages in the SharePoint Client Chrome Control. The Chrome Control adds header html (title, breadcrumb, icon, etc.) and injects styles from the referenced SharePoint site for styling other html elements. Here are the high-level steps to implement the Chrome Control:

Step 1: Configure the query string in the AppManifest.xml to load the correct default app page and pass the appropriate contextual information through the URL

Step 2: Add a div placeholder element to the top of all app pages for hosting the SharePoint Client Chrome Control

Step 3: Include script on app pages to reference the SP.UI.Controls.js script from the host site and use it to load the Chrome Control into the placeholder from step #2

Pages Displayed in iframe Elements

App parts or "client web parts" display app pages in iframes on the host SharePoint site. Although similar to a Page Viewer web part, app parts have the added ability for custom web part properties.  These properties get passed to the app part's page via URL parameters (along with additional contextual information the developer requests). Since app parts and pages displayed in the SharePoint dialog do not display full-screen, they should NOT leverage the SharePoint Client Chrome Control like pages displayed in full-screen. Instead, SharePoint styles can be incorporated by referencing style resources from the host SharePoint site. Here are the high-level steps to implement SharePoint style resources in an app part page:

Step 1: Add a Client Web Part to your SharePoint 2013 app project

Step 2: Configure the client web part's Content Source in the Element.xml to correct page and pass the appropriate contextual information through the URL

Step 3: Include script on app part pages to inject a new style sheet link element into the head of the page referencing the /_layouts/15/defaultcss.ashx resource from the host SharePoint site

Here are the high-level steps to implement SharePoint style resources in a page displayed in the SharePoint dialog:

Step 1: Add a UI Custom Action (Host Web) to your SharePoint 2013 app project

Step 2: Configure the UrlAction in the Element.xml to correct page and pass the appropriate contextual information through the URL and set HostWebDialog="true" on the Custom Action (HostWebDialogHeight and HostWebDialogWidth should also be set)

Step 3: Include script on the dialog pages to inject a new style sheet link element into the head of the page referencing the /_layouts/15/defaultcss.ashx resource from the host SharePoint site

So I've Referenced SharePoint Styles…Now What?

Importing SharePoint style resources from the referenced site is only half the battle. App developers need to be aware of SharePoint style classes in order to take on the appearance of the SharePoint site. The MSDN documentation on Apps for SharePoint design guidelines has several comprehensive tables detailing common SharePoint style classes. However, a good bowser developer tool (such as Internet Explorer's F12 Developer Tools) can help identify additional classes.

Resizing app parts can be another challenge, particularly in highly dynamic app pages. Resizing is controlled by the app part's height and width web part properties, which are ultimately applied to the height and width of the iframe that gets rendered. These properties can be configured to a default size by the developer in the Elements.xml, but can be changed by a site designer when placed on a page. Unfortunately, app parts do not auto-size as the content contained in them grows. Additionally, an app page cannot walk the DOM hierarchy outside the page to adjust itself (this is blocked across domains). However, SharePoint 2013 implements the HTML5/javascript postMessage framework to achieve resizing from an app part page. window.postMessage enables safe cross-domain communication between a parent page and iframe page, provided they are listening for messages.  Here is SharePoint's implementation of postMessage for app part pages:

IMPORTANT NOTE: At the time of this post, postMessage resizing had a bug in the SharePoint 2013 Preview.  This is a known bug that will be fixed in the final release.  Later in the article, I will provide additional details of postMessage communication and a sample solution.  Here are some general recommendations for resizing app part pages:

• Microsoft recommends sizing app parts in increments of 30px.  If all app developers follow this guideline, app parts will more smoothly dovetail into a tenant's page (there is nothing worse than two web parts displayed side-by-side, but 2-3px off...increments of 30px greatly help this).
• Width is typically more challenging to deal with as height, since inadequate width will usually wrap elements and thus increase the height required to display all page content. Height is also challenged by dynamic elements such as a Repeaters and GridViews that grow vertically. As such, I recommend avoiding overuse of no-wrap styling.
• Regardless of resizing capabilities, you should try designing to a predictable size. This isn't a foolproof approach, since styles can have a huge impact on rendering height (ex: padding and font size). However, careful design planning can get you close. I recommend constraining dynamic content such as implementing paging on GridView elements. This combined with the ClientWebPart configuration for DefaultHeight and DefaultWidth properties can deliver a great result in most cases.
• Try to minimize scrolling containers. SharePoint minimally uses these and you should too. They make fixed-size design easy but can quickly make a page look like an iframe mess.

Getting Started

If you haven’t already figured it out, apps get most contextual information about the host SharePoint site through URL parameters (user information is also available through other methods).  MSDN has some great documentation on URL string and tokens in apps for SharePoint that details the dynamic URL parameter options.  In addition to these parameters, developers can pass their own.  In my case, I passed a DisplayType parameter so my script could be reused and have a method to determine if the page was being displayed in full-screen or in an iframe.  This could be especially helpful if your app contained a page that is leveraged across both display options.  All the samples below were developed using Autohosted apps.  Autohosted apps are deployed in Azure, which emphasizes the separation between SharePoint and my app (it also enables me to write server-side code).

Step 1: Provision a Developer Site for debugging your app (this must be site collection using the “Developer Site” template).  I’m debugging my app in Office 365, so I provisioned my Developer Site in the Office 365 Admin Portal:

Step 2: Create new project in Visual Studio 2012 using the App for SharePoint 2013 template:

Step 3: Select the site you created in step 1 for debugging and leave the default hosting option of Autohosted and click Finish.  You may be prompted to authenticate to the SharePoint site:

If you are new to apps for SharePoint 2013, you will notice the solution contains two projects.  The first project is the actual SharePoint app, which will contain nothing but xml manifests/elements to define the contents of the app.  The second project is a web application to host all the markup and code for the app.  When debugging Autohosted apps, SharePoint will direct app requests to this web application running on localhost in IIS Express.  If you deploy the app, it will be provisioned inside Azure.

Solution Explorer of a New App for SharePoint 2013 Project

Creating an App with Full-Screen Display

By default, new SharePoint 2013 app projects already include a page to handle the launch experience, which is always a full page.  This page can be found in the web application project under Pages\Default.aspx.  The new project could immediately be deployed, but let's go a little deeper by adding some addition information and the Chrome Control:

Step 1 – The Manifest: Locate the AppManifest.xml in the SharePoint app project and open it.  Visual Studio provides a great AppManifest editor that launches by default.  Locate the Start page field...this tells SharePoint what the default page is for the app.  The Start page should already point to Default.aspx and have a special {StandardTokens} parameter in the Query string field.  SharePoint will use this to pass contextual information to the app's default page (ex: Site URL, Language, etc.).  Update the Query string field with additional parameters for SPHostTitle={HostTitle}, SPHostLogo={HostLogoUrl}, and DisplayType=FullScreen as seen below:

Step 2 – The Chrome Placeholder: Open the Default.aspx markup page and add a div placeholder for the SharePoint Client Chrome Control.  This should be the first element in the body/form of the html page as seen below:

Step 3 – The Script: Script to load the SharePoint Client Chrome Control can be added directly to the page.  However, I created a script file so I could re-leverage the script on multiple pages.  I also wanted the script to leverage my custom DisplayType URL parameter for reuse across full-screen and iframe page displays.  The script has three basic components:

1. Document loaded event to check the DisplayType and initialize the appropriate script (/_layouts/15/SP.UI.Controls.js in the case of full-screen pages):
   

   $(document).ready(     function () {         //Get the DisplayType from url parameters         var displayType = decodeURIComponent(getQueryStringParameter('DisplayType'));         //Get the URI decoded SharePoint site url from the SPHostUrl parameter.         var spHostUrl = decodeURIComponent(getQueryStringParameter('SPHostUrl'));         //Build absolute path to the layouts root with the spHostUrl         var layoutsRoot = spHostUrl + '/_layouts/15/';         //Execute the correct script based on the displayType         if (displayType == 'FullScreen') {             //Load the SP.UI.Controls.js file to render the App Chrome            $.getScript(layoutsRoot + 'SP.UI.Controls.js', renderSPChrome);         }         else if (displayType == 'iframe') {             //Create a Link element for the defaultcss.ashx resource             var linkElement = document.createElement('link');             linkElement.setAttribute('rel', 'stylesheet');             linkElement.setAttribute('href', layoutsRoot + 'defaultcss.ashx');             //Add the linkElement as a child to the head section of the html             var headElement = document.getElementsByTagName('head');             headElement[0].appendChild(linkElement);         }     });

2. Callback function for loading the chrome after the appropriate scripts have loaded:
   

   function renderSPChrome() {     //Get the host site logo url from the SPHostLogoUrl parameter     var hostlogourl = decodeURIComponent(getQueryStringParameter('SPHostLogoUrl'));     //Set the chrome options for launching Help, Account, and Contact pages     var options = {         'appIconUrl': hostlogourl,         'appTitle': document.title,         'appHelpPageUrl': 'Help.html?' + document.URL.split('?')[1],         'settingsLinks': [             {                 'linkUrl': 'Account.html?' + document.URL.split('?')[1],                 'displayName': 'Account settings'             },             {                 'linkUrl': 'Contact.html?' + document.URL.split('?')[1],                 'displayName': 'Contact us'             }         ]     };         //Load the Chrome Control in the divSPChrome element of the page     var chromeNavigation = new SP.UI.Controls.Navigation('divSPChrome', options);     chromeNavigation.setVisible(true); }

3. A utility function to pull named parameters from the URL:
   

   function getQueryStringParameter(urlParameterKey) {     var params = document.URL.split('?')[1].split('&');     var strParams = '';     for (var i = 0; i < params.length; i = i + 1) {         var singleParam = params[i].split('=');         if (singleParam[0] == urlParameterKey)             return singleParam[1];     } }

After referencing the custom UX script, Microsoft AJAX, and jQuery, the head of the markup page should look like similar to the following:

Step 4 – Testing:  I added text and controls to my page referencing known CSS classes from SharePoint (ex: ms-accentText, ms-listviewtable, etc.).  I then deployed and tested my full-screen app page in three scenarios…with the custom chrome script removed, with the chrome script and the default composed look, and the chrome script with several alternate composed looks.  Here are the results:

Without Chrome Control

Chrome Control with Default Composed Look

Chrome Control with "Sea Monster" Composed Look

Chrome Control with "Nature" Composed Look

Creating an App with App Part Page Display

Step 1 – Adding the App Part: Right-click the SharePoint app project and Add> New Item to bring up the new item dialog.  Select Client Web Part (Host Web) from the list, name it, and click the Add button:

You should notice that the new client web part added an Elements.xml file to the project but no markup or code.  Remember, the SharePoint app project will contain nothing but xml manifests/elements to define the contents of the app.  All markup and code will live in the web application project that is deployed to the web server (Azure in the case of our Autohosted app).

Step 2 – The Element.xml: Open the client web part’s Elements.xml file.  This file contains the app part’s configuration for the markup page to render, the default size, and any custom web part properties. I added a Property for useSiteStyles so I can toggle the SharePoint style resource on and off inside my app part.  Next, set the Content Src to the appropriate page from the web application project and including the contextual URL parameters for the useSiteTypes property, {StandardTokens}, and DisplayType=iframe as seen below:

Step 3 – The Script: If you looked carefully at the full-screen script, you may have noticed a check for DisplayType of “iframe”.  That’s right; the script for importing SharePoint style resources is already in place to support app parts and SharePoint dialogs:

However, our app part will conditionally use this script based on the useSiteStyles web part property value.  I decided to implement this server-side using the RegisterClientScriptInclude method in my page loaded event:

Step 4 – Testing: Like the full-screen app page, I added text and controls to my app part page referencing known CSS classes from SharePoint (ex: ms-accentText, ms-listviewtable, etc.).  Next, I created a two column wiki page and placed my app part in each column side-by-side. The left app part had useSiteStyles set to false and the right app part had this property set to true for a nice side-by-side comparison.  Finally, I tested the app parts in three scenarios…without paging on the GridView, with paging on the GridView, and with several alternate composed looks.  Here are the results:

App Parts without GridView Paging

App Parts with GridView Paging

App Parts with "Sea Monster" Composed Look

App Parts with "Nature" Composed Look

App Part in Older Browser**

**One thing I learned developing this post is that IE now supports transparent windowed elements such as iframes...even cross-domain.  I was startled to see the site's background images showing through the app parts.  To do this, I explicitly set the background-color of the app part pages to transparent.  Older and non-IE browsers may not have the same results (I tested IE and Chrome and both rendered transparent).  The picture above illustrates the look without a transparent body on the app part pages and what I would expect in older browsers.

Creating an App with SharePoint Dialog Page Display

App pages can also be launched inside of the SharePoint dialog framework using custom actions.  From a UX perspective, the SharePoint dialog is implemented almost identically to an app part.  Both should typically be displayed with SharePoint styles, but without the chrome control.  Functionally, pages launched in the SharePoint dialog tend to differ from app parts in the contextual information passed to the page (ex: selected list or items), how the page is launched (ribbon button, contextual menu, etc.), and how it displays on the screen (in the dialog instead of an element on a SharePoint page).  Here are the steps to deliver a page in the SharePoint dialog: 

Step 1 – Adding the Custom Action: Right-click the SharePoint app project and Add> New Item to bring up the new item dialog.  Select Custom Action (Host Web) from the list, name it, and click the Add button:

You should notice that the new custom action added an Elements.xml file to the project but no markup or code.  Remember, the SharePoint app project will contain nothing but xml manifests/elements to define the contents of the app.  All markup and code will live in the web application project that is deployed to the web server (Azure in the case of our Autohosted app).

Step 2 – The Element.xml: Open the custom action's Elements.xml file.  This xml is very similar to traditional SharePoint custom actions, but takes advantage of special app URL parameters and has properties for launching the page in the SharePoint dialog (HostWebDialog, HostWebDialogWidth, HostWebDialogHeight).  Also notice my use of DisplayType=iframe so I can reuse my chrome/style script.  If you are new to custom actions, the following adds a "Launch Page" button to the "Manage" group of the "Files" tab in a Document Library:

Step 3 – The Script: Identical to app parts...just make sure you include the script on the app page.

Step 4 – Testing: As mention above, the custom action adds in this solution adds a button to the "Files" tab for Document Libraries.  The app page it launches will display all the selected files in the Document Library.  This is a little different from other parts of the app, in that the app needs read permissions to list(s) in the site.  The pictures below illustrate this permission setting in the AppManifest, the app permission screen in SharePoint where the app is trusted to read a specific list, and the launched app page in the SharePoint dialog (with consistent styling):

Permissions Settings in AppManifest

App Permissions in SharePoint

App Page Displayed in SharePoint Dialog
 

postMessage Resizing

As mentioned earlier, the final release of SharePoint 2013 will support app part page resizing through postMessage client-side communication.  In this model, the parent page and iframe page can communicate to each other, provided they are listening for messages.  SharePoint 2013 will listen for resize messages from app part pages and adjust the height and width of the app part based on the size specified in the message.  The MSDN documentation for this will not work in the SharePoint 2013 Preview, but will be available in the final SharePoint 2013 release.  Since postMessage is a relatively new to most developers, I have provided a code sample that illustrates this communication in a simple web application (outside of SharePoint).  When SharePoint 2013 is released, look for an update or new post outlining postMessage resizing in SharePoint.  You can download the postMessage solution here:  postMessage.zip.

Conclusion

The UX Challenge with apps for SharePoint 2013 will exist.  App developers need to be aware of SharePoint style classes and the methods discussed in this article for achieving the best UX.  Consider the MSDN documentation on UX design for apps in SharePoint 2013 as the definitive source for UX design decisions. With these resources and good design planning an app developer can achieve excellent results for almost any tenant!

You can download my sample project here:  OptimizingAppUX.zip

Hope this help!

In this two part series, I will explore developing real-world solutions via apps for SharePoint 2013.  Part 1 will focus on the basic app delivery without complex tenancy considerations (the solution will store all information in a list on the tenant).  In part 2, we will explore tenancy considerations in the Office Marketplace for solutions leveraging cloud storage in SQL Azure and other advanced scenarios.

The Solution

Microsoft uses a popular employee recognition application called Kudos.  I can send a kudos to anyone in the organization to recognize them for exceptional work.  When a kudos is submitted, an email is sent to the recipient, the submitter, and both of their managers (if applicable).  Additionally, the kudos is added to social newsfeeds for a wider audience to see.  Statistics are available to see historic kudos activity for employees.  I felt like this was a perfect candidate for a SharePoint 2013 app, given the rich integration with social, reporting, and workflow.  The video below outlines the final solution of Part 1, which is detailed throughout this post (including the provided code):

(Please visit the site to view this video)

Getting Started

The solutions in this series will be developed as Autohosted solutions, which are ultimately deployed to Azure for hosting.  I should note that the solution in Part 1 could have been developed as a SharePoint hosted solution, but I chose Autohosted given the direction we will take in Part 2.  Since Austohosted solutions provide the flexibility of server-side code, Part 1 has a really nice mix of CSOM and SSOM.  It also does a good job of illustrating the differences in the three web locations…the host web (consumes the app), and app web (hosts SharePoint-specific components of the solution), and the remote app web (hosts the app logic).  An app will always have a host web but not always a remote web (SharePoint-hosted apps) or app web (optional in Autohosted and Provider hosted apps).  The Part 1 solution leverages all three as outlined in the diagram below:

Building the Solution

I started by creating an App for SharePoint 2013 project in Visual Studio leveraging my Office 365 Preview tenant for debugging and configured as Autohosted.  The app will read profiles, write to lists/newsfeed, and use SharePoint utilities for sending email, all of which require specific app permissions.  App permissions are set in the AppManifest.xml.  Here are the permissions I set for the Kudos app:

With app permissions in place, I turned my attention to storage of kudos.  For Part 1, I decided to use a SharePoint list for storage.  In the future I will modify the solution to use SQL Azure for storage, which will be better for capacity/growth and much more efficient for advanced analytics and reporting (not to mention cleaner to work with compared to CAML).  Lists in SharePoint apps get provisioned in the app web, which is on a separate domain from the consuming site.  Below is the simple list structure I came up with for storing kudos.  Notice the manager fields are optional, since not all employees have a manager or have it set on their profile:

Next, I turned my attention to the markup and script that users would interact with to submit kudos.  Apps for SharePoint are required to provide a full-screen user experience, but I also wanted the app to be surfaced in existing SharePoint pages.  To do this, I added a Client Web Part to the SharePoint project and will deliver the same kudos page in both full-screen and in the app part.  For details on designing/styling for both experiences, see my previous post on Optimizing User Experience of Apps for SharePoint 2013.  The Kudos app has two views.  The first view has contains an employee look-up form and displays kudos statistics to the user such as the number of kudos sent/received over time and the users balance (we will limit the user to 4 submissions/week).  The second view will display details of the kudos recipient (Name, Profile Picture, Title, etc) and provide the form for the kudos text.  Below are pictures of these two views that we will dissect later in this post:

Kudos Stats and Lookup Form	Kudos Entry Form

When the Kudos app is first loaded, it will display kudos statistics for the user (ex: number of kudos sent and received over time).  This is achieved through SSOM CAML queries on page load (note: this could also be done client-side):

Most of the heavy lifting prior to submitting a kudos will be handled client-side (employee lookups, profile queries, etc).  This requires referencing several script file from the host site including SP.js, SP.Runtime.js, init.js, SP.UserProfiles.js, and SP.RequestExecutor.js.  The last two of these are probably the least familiar but particularly important.  SP.UserProfiles.js provides client-side access to profiles and SP.RequestExecutor.js wraps all of our client-side requests with the appropriate OAuth details.  Here is how I referenced them dynamically from the host site:

The People Pickers in SharePoint 2013 have a great auto-fill capability I wanted to replicate in my employee lookup.  After some hunting and script debugging, I found a fantastic and largely undocumented client-side function for doing users lookups.  SP.UI.ApplicationPages.ClientPeoplePickerWebServiceInterface.clientPeoplePickerSearchUser takes the client context and a ClientPeoplePickerQueryParameters object to find partial matches on a query.  I wired this into the keyup event of my lookup textbox as follows:

Once a user is selected, script will query for detailed profile information on submitter and recipient and toggle to the kudos entry view.  No be surprises here, but a decent amount of CSOM using the new UserProfile scripts:

When the user submits a kudos, the kudos form will execute it's one and only postback.  In reality, everything the postback does with SSOM could be achieved client-side with CSOM.  However, I'm looking to do some advanced things in Part 2 that I think will be easier server-side (ex: impersonate the social post as a "Kudos" service account).  The postback does three basic things…adds a kudos record to the kudos list on the app web, creates a post on the social feed with a recipient mention, and emails the kudos to the submitter, recipient, and their managers (if applicable).  Here is the postback code:

That's about it!  Here are some screenshots of the Kudos app in action:

Kudos App in Existing WikiPage (during lookup)

Kudos Form with Selected Recipient

Newsfeed with Kudos Activity

Kudos Email with Managers CC'ed

Final Thoughts

Kudos was a really fun app to develop.  It's an app that almost any organization could leverage to further capitalize on the power of social in SharePoint 2013.  Best of all, it runs in Office 365!  Look for Part 2 in the next few weeks, where I will expand upon the solution to leverage SQL Azure storage, workflow, and address multi-tenancy for the marketplace.  I hope this was helpful and get you excited about building fantastic apps for SharePoint 2013!

Kudos Part 1 Code: KudosPart1.zip

Several months ago I wrote a post about building real-world apps for SharePoint 2013.  In that post, I walked through the creation of an employee recognition app for SharePoint 2013 called Kudos.  The original Kudos app leveraged SharePoint lists for storage and wasn’t very configurable.  In this post, we’ll evolve the Kudos app to address several of these limitations, add some additional capabilities, and discuss the tenancy considerations for autohosted apps.

The Solution

The new Kudos app will leverage SQL Azure for data storage, which is much more efficient, scalable, and easier to query in our solution.  The new app will also differentiate between app visitors and app administrators by allowing administrators to configurable app-specific settings.  Gamification features will be added to allow configurable “badges” to be included in Kudos submissions.  Finally, the app will be updated to handle direct navigation where context tokens are null.  All of these features are illustrated in the following video:

(Please visit the site to view this video)

Autohosted App Tenancy

Before jumping into the intricacies of the new app, I want to discuss the tenancy considerations of autohosted apps.  When an autohosted app is installed into a SharePoint site, all the app components (web/database/workflow) are auto-provisioned in a Microsoft owned Azure account (o365apps.net).  Not only is provisioning automatic, high-availability and disaster recovery is automatic (hence the name “autohosted”).  This happens each time the app is installed in a SharePoint site (even within the same Office 365 tenancy).  For example, if I installed my autohosted app into 8 team sites, it would result in 8 separate web sites and databases in Azure.  The diagram below illustrates the tenancy of autohosted apps.  If the isolated tenancy of autohosted apps doesn’t work for you, provider-hosted apps offers more flexibility.  However, with flexibility comes complexity as things are less “automatic”.

Storage with SQL Azure

When I first learned about the ability to provision databases on-demand for autohosted apps, it sounded like some sort of voodoo magic.  How would a newly provisioned app know what connection string to use?  It turns out that Microsoft came up with a very elegant solution to the problem.  In fact, managed CSOM provides APIs to retrieve either a SqlConnection object or the raw connection string using the client context.  Before I jump into these APIs, let me explain the setup to leverage databases in autohosted app solutions.

The new Kudos app will leverage the ADO.NET Entity Framework to talk with SQL Azure. The Entity Framework supports two models…entities first or database first.  Although it is a matter of preference, I tend to use the database first approach except for in MVC apps.  To do this, we’ll start with a database that we will later use to generate our entity model.  Visual Studio 2012 includes LocalDB which is essentially an improved on-demand version of SQL Express.  I like to create isolated instances in LocalDB for my apps, which can be done with the SqlLocalDB.exe command line:

Next, we need to add a database project to our app solution.  Right-click the solution and select Add > New Project.  In the Add New Project dialog, find the SQL Server Database Project template under Other Languages > SQL Server.  Give the project a name and add it to the solution:

We need to make the app project aware of this new database project (almost like adding a project reference).  To create this reference, set the SQL Database property on the app project to our new database project.  When we set this property, Visual Studio will warn us that the target for the database project will be changed to SQL Azure:

Next, we can add all the appropriate tables and procedures to the database project.  For Kudos, I added tables for Kudos and AppSettings.  I also added a GetKudosHistory stored procedure for pulling historical statistics on a user.  A stored procedure will be more efficient than directly querying the entity model with numerous where clauses.  Finally, I added a post-deployment script to seed the AppSettings table with some default settings:

Before we can generate an entity model, we need to publish the database to the LocalDB instance we created earlier.  Remember, this is automatic once we deploy the autohosted app, but here we need to publish manually so we can generate the initial entity model.  The Publish Database wizard can be launch be right-clicking the database project and selecting publish:

With the database published in LocalDB we can turn our attention to the entity model, which will be added to the web project.  The ADO.NET Entity Data Model selection can be found under Visual C# > Data in the Add New Item dialog:

Using the Entity Data Model Wizard We, we will select “Generate from Database”, provide the connection to our database on LocalDB, and select all the database objects we want represented in our entity model:

Side note: I prefer my entity model to contain a constructor that accepts a connection string.  This constructor will get generated automatically if I change the “Code Generation Strategy” property on the entity model from None to Default.  However, I also need to delete the two .tt files that are nested under the model in solution explorer.  I’ve done it this way, but it is all a matter of preference:

As mentioned earlier, CSOM includes new APIs for working with SQL Azure databases in autohosted apps.  AppInstance.TryGetAppDatabaseConnectionDirect can be used to retrieve a SqlConnection object for the database in SQL Azure.  With this approach, you should configure the connection string in your web.config with the name LocalDBInstanceForDebugging and the API will dynamically use this connection when debugging:

Although powerful for many development scenarios, a SqlConnection object is not the most appropriate for use with the Entity Framework.  Instead, we will leverage AppInstance.RetrieveAppDatabaseConnectionString to retrieve the raw connection string in the Kudos app.  This API will NOT return the raw connection string when debugging, so the code will check for a null connection string and directly leverage the LocalDBInstanceForDebugging connection if appropriate:

Autohosted app deployment automatically handles all the connection string wire-up magic with SQL Azure.  However, it won’t generate an Entity Framework connection string, which requires additional metadata.  Instead, we can manually convert the connection string using the EntityConnectionStringBuilder class.  I did this in a ConnectionUtil class as seen below:

Gamification/Badges

The new employee recognition apps includes the ability to send badges to the recipient of a Kudos.  A document library in the app web is the perfect storage location for these badges.  This will allow an administrator to add/remove badges as is necessary.  Most of the work in developing the badge funcationality is in the BadgePicker control for selecting a badge in the Kudos form:

We also want to include the badge in the social post to the newsfeed:

Distinguishing App Roles

As I started to develop real-world solutions using the app model, I quickly identified the need to distinguish an app administrator from a normal visitor.  For Kudos, the administrator needs the ability to configure quotas, quota durations, notification settings, and badges for the application. 

I see two distinct approaches to application roles…inherit the permissions from the app web or build a complete custom permission model in the apps storage (ex: SQL Azure).  A custom model might be necessary for complex permission needs.  For Kudos, I just need to know if a user is an administrator and the app web can easily provide that.  You might ask why I’m going to the app web and not the host web for checking permissions.  Checking permissions on the host web would require my app to have full control on the host web, which it would otherwise not require.  It is bad practice for your app to request more permissions than it needs.  This is especially true in our case, since the app is free to check permissions on its own app web (which are inherited from the host web).

I decided to surface my admin screens through the settings menu in the chrome control.  This posed another challenge, since this menu is rendered using client-side script.  I found that checking the effective permissions of the user took too many API calls to do client-side (one call to get the current user and one to get effective permissions for the user).  Instead, I decided to check effective permissions in managed code and conditionally output the appropriate script.  I delivered this through a base page class that all of my pages inherited from:

Deciding on client-side or managed CSOM is an important consideration to weigh for all API calls.  Each will likely take the same number of calls, but managed code should have less latency when multiple API calls are required.  This is especially true for autohosted apps, where the app and the SharePoint Online farm might be hosted from the same Microsoft data center.  At worst, the app and the farm would communicate over Microsoft’s massive communication backbone between data centers.

Direct Navigation

It possible (and likely) that some users will try navigating directly to the app instead of through SharePoint.  This “direct navigation” is a special use case to develop around, especially when the app leverages OAuth for communication back to SharePoint (which will be 100% of the time in Office 365).  When a user launches an app from SharePoint, SharePoint posts a context token to the app.  Through this context token, the app can talk to Azure ACS to get an access token it can use for calling SharePoint APIs.  Without a context token, the app will be denied access to the SharePoint APIs.  Luckily, SharePoint 2013 provides an app redirect page specifically to refresh context tokens (/_layouts/15/appredirect.aspx).  TokenHelper.cs includes built-in functions to build the correct redirect url.  All apps for SharePoint should be written to take advantage of this.  In my case, I built a ContextUtil class to handle the app redirect and cache my tokens for an appropriate length of time.

Style "Toggles"

In Part 1, I developed a reusable script to add the chrome control or host web styles based on a DisplayType url parameter.  I still think this is a great practice, as it allows the same app pages to be displayed in full-screen, app parts, and dialogs.  However, I’ve grown tired of the delayed style "toggle" that occurs when the chrome control or host styles are finally loaded into the app page (ugly page…wait for it…YAY, pretty page).  During preview, this sometimes took 4-8 seconds.  In the new Kudos app, I’ve defaulted the body display style to none on all pages and used callbacks to display the body once the chrome control or host styles have completely loaded.  This provides a much better user experience for the user:

Final Thoughts

The new Kudos employee recognition app is a great reference for autohosted apps.  It makes good use of new SharePoint APIs, maximizes different UX delivery methods, and follows several patterns that I would consider best practices for developing apps for SharePoint.  Best of all, I’m providing 100% of the code to the community.  So pull down the code, take note of the patterns/intricacies, and go build some great apps for SharePoint! 

Kudos Part 2 Code: KudosAdvanced.zip

Feature Stapling is a popular development practice for adding functionality to a specific type of site in SharePoint.  Although it has become the preferred approach over custom site definitions, it still requires the deployment of a farm solution.  Unfortunately, SharePoint Online doesn’t support features stapling or custom site definitions.  So how can an organization deploy capabilities to a group of existing sites or new sites that match specific criteria?  The answer is app deployment through an app catalog…something I like to call “app stapling”.  Here is a video that illustrates the concepts outlined in this blog post:

(Please visit the site to view this video)

The App Catalog

The App Catalog is a special site collection aimed at storing, managing, and delivering Office/SharePoint apps to the enterprise.  Administrators of an App Catalog can upload apps and make them available to site owners (they can also flip a kill switch to disable the apps).  However, administrators can also use the catalog to push apps to specific site collections, managed paths, and site templates.  To get started, you should provision an app catalog in your tenancy (on premise users will need to do additional steps outlined in Configure an environment for apps for SharePoint).  This can be done from SharePoint Admin Center in Office 365 by clicking on the Apps link in the side navigation and then selecting App Catalog:

Once the App Catalog has been provisioned, it should look similar to this:

App Deployment

In order to "push" and app out to specific sites, we first need to upload it to the App Catalog.  Navigate to the App Catalog and select the Apps for SharePoint link in the side navigation.  You can simply drag/drop an .app file into this library to make it available to the organization:

Next, we need to install the app into the App Catalog (not to be confused with uploading to the App Catalog).  This seems a little odd, but it is the only way to push it specific sites in the tenancy.  You can install the app into the App Catalog just like you would from any other SharePoint site.  Select Site Content > add an app > From Your Organization > and select the app to install:

Now that the app is installed into the App Catalog, we can manage the deployment to other sites.  Select Site Content from the App Catalog and find the installed app.  Selecting the ellipse should display a "Deployment" option does doesn't display in other sites:

This Manage App Deployment screen allows an administrator deploy/retract an app to/from specific site collections, managed paths, and site templates.  Performing this "app staple" will work on all existing sites and any new sites that meet the criteria.

A few interesting notes about how this functions:

• Similar to deployment, an App Catalog administrator can retract apps from specific sites (or flip the kill switch to disable it from all sites)
• Because the app is pushed by an administrator, site owners will not be able to remove the app from a site that meets the deployment criteria.  Not even a site collection administrator can remove the app.
• This centralized deployment also share the same centralized app resources (App Web and Remote Web).  Essentially, the app is deployed, but not installed in the sites.  All sites will leverage the App Web and Remote Web from the instance installed in the App Catalog.  This significantly changes tenancy considerations for SharePoint-hosted and Autohosted apps, which typically get their own dedicate app resources.
• Because of centralized deployment, remote events such as “Handle App Installed”, “Handle App Uninstalled”, and “Handle App Upgrade” will only fire once (when the app is installed in the App Catalog).  Because of this, I think it will be hard to leverage “App Stapling” for branding purposes.  I’ve successful developed branding apps that deploy/set a masterpage from the App Web, but the remote events were a key part of that solution.

Final Thoughts

App deployment through the App Catalog provides a great deal of flexibility and governance for SharePoint deployments on-premise and in Office 365.  I can see “App Stapling” being used for a number of scenarios where feature-stapling has been used in the past.

One of the interesting capabilities of the new SharePoint/Office app models is combining them to deliver complete real-world solutions.  In fact, an app for SharePoint can contain/embed an App for Office.  Combining apps can lead to some powerful scenarios. 

Imagine an expense report app for Office that interacted with Word to assemble a complete expense report.  Now think about that delivered as the default document of a document library in SharePoint (perhaps even with powerful workflow capabilities).  Hopefully you can start to see this as a power way to package capabilities together.  This post will focus on embedding an app for Office in an app for SharePoint and exposing it through a document library.  The video below illustrates the solution outlined in this post:

(Please visit the site to view this video)

To get started, we need an app for SharePoint project.  Any hosting type will work, but I went with SharePoint-hosted for simplicity (use provider-hosted or autohosted if you want to leverage server-side code).  Once the project has been provisioned in Visual Studio 2012, we need to add an app for Office to the project, which should be an option in the Add New Item dialog:

Visual Studio needs to know what type of app for Office we are developing.  The options include content apps for Excel or task pane apps that target Word, Excel, and/or PowerPoint.  I selected a task pane app for Microsoft Word:

Next, we have an option to select an existing Office document to associate the app with.  This is great when an existing document template already exists.  The default option is to start with a blank Office document, which is what I selected:

After completing the wizard, a number of new assets will be added to the project, including presentation elements for delivering the UI of the app for Office (styles, script, and html).  It also added the Office document (via module) and the app manifest for the Office app (seen below as ExpenseReportApp.xml):

The Word document (ExpenseReportApp.docx) should already be associated with our app for Office.  Other than that, it is an empty Word document (remember we could have selected an existing document).  When you open it, don't be concerned that the app for Office doesn't load...we haven't deployed the html page yet:

Next, we need to add a document library to store expense reports and associate with our Office document.  Launch the Add New Item dialog to add a List to the project:

Base the new list on a customizable Document Library and click Next.  Be careful not to click Finish, or we will miss an important step:

The next screen allows us to select an Office document to associate as the document template for the new library.  This is the key step in wiring our app for Office to the new document library.  Had we skipped this screen, the library would use the default blank Word document:

Associating the new library with a document template will automatically generate a custom content type for library and list template.  To complete the solution, we just need to update the entry point of the SharePoint app to the URL of our document library.  You can look at the ListInstance to find this URL and then set it in the AppManifest.xml of the SharePoint app project:

When we debug or deploy the solution, the app should take us directly to the document library.  The document library should already have the content type association we need to launch our app for Office:

And here is the final result of the app for Office launched from our document library:

I hope this post helped illustrate the interesting capabilities that can be delivered by combining apps for SharePoint and apps for Office.  To make it even more interesting, an app for Office could call back into SharePoint APIs.  I’d love to hear about the creative combinations you come up with, so feel free to post them below!

One of the creative ways Apps for SharePoint can be exposed in a site is through the SharePoint dialog.  In this way, apps can deliver contextual capabilities to a SharePoint site.  Dialogs will typically be launched from custom actions in the app.  In this post I will discuss how to launch an app through the SharePoint dialog, pass information into the app, and close the dialog from within the app.

Launching the Dialog

Launching an app page in the SharePoint dialog requires the use of a custom action.  Apps for SharePoint support Menu Item Custom Actions and Ribbon Custom Actions.  Both of these can be deployed to the Host Web or the App Web.  The wizard for adding these custom actions makes it very easy to scope custom actions appropriately as is shown in the table below:

As you can see in the table above, Menu Item Custom Actions have additional flexibility to scope against specific Content Types or File Extensions.  Ribbon Custom Actions are unique because they scope to a specific ribbon tab and group, which a wizard will prompt you for when added to an app:

In additional to location, Ribbon Custom Actions typically have icons associated with them.  Ribbon icons are especially challenging when deploying the custom action to the host web.  An App for SharePoint can only deploy images to the app web (via module) or remote web, neither of which will render nicely in the ribbon of the host web.  The solution is to specify the image source as a Base64 encoded image as seen below:

I created a quick tool to Base64 encode images, which is provided at the end of this post.  Here is the UI of that tool:

A custom action can be converted to leverage the SharePoint dialog by adding the HostWebDialog, HostWebDialogHeight, and HostWebDialogWidth attributes to the CustomAction element in the Elements.xml:

Passing Context to the Dialog

App will generally get contextual information from URL parameters or information posted from SharePoint.  Custom actions have a number of dynamic URL parameters they can leverage for contextual information:

Closing the Dialog

Autohosted and Provider-hosted Apps for SharePoint are hosted outside of SharePoint, and as such can’t leverage typical SharePoint scripts to close the dialog (ex: SP.UI.ModalDialog.commonModalDialogClose()).  Luckily, I have hunted through many lines of core.js to find you the solution!  Similar to resizing app parts (client web parts), we can use the HTML5 postMessage API to close the dialog from the app (and optionally refresh the parent page).  This shouldn’t come as a surprise, since the postMessage API was meant to provide cross-domain communication and is used in other app scenarios.  CloseCustomActionDialogRefresh and CloseCustomActionDialogNoRefresh are the two messages SharePoint is “listening” for to close the dialog.  I wrapped these in a simple function I can releverage across all my apps:

Final Thoughts

Leveraging custom actions with the SharePoint dialog makes an app feel even more integrated than with app parts.  They provide contextual integration and are the only mechanism for pushing a change in UI on the host web (remember an app part is optionally added by a user).  Below are links to a code sample and the Image Encoder tool mentioned above:

App for SharePoint with dialogs:  SharePointDialogs.zip

Image Encoder Tool: ImageEncoder.zip

Want to deliver an internal/corporate “YouTube” for your organization using SharePoint?  Looking to maximize your SharePoint deployment by incorporating video/media delivery?  Worried about the storage/bandwidth implications of allowing anyone in the enterprise to contribute video/media?  Then then post if for you!  I will outline a solution that addresses many of the limitations to native media delivery in SharePoint 2013.  The solution outlined in this post is also illustrated in the following video:

(Please visit the site to view this video)

Steve Fox wrote a great post a few months back on SharePoint 2013 and Windows Azure Media Services.  In it, Steve illustrated the use of Azure Media Services to deliver media within an app for SharePoint.  Steve introduced some powerful concepts that I want to take to the next level.  I envision a complete solution that allows users to contribute any media format, from any asset library in SharePoint, and send it through Azure Media Services for encoding and hosting (possibly around the globe using Azure’s Content Delivery Network).  But why Azure Media Services?  To understand why, it’s important to consider the video capabilities native to SharePoint 2013, their limitations, and past case studies on SharePoint as a media platform.

Video and SharePoint

Out of the box, SharePoint provides very basic video delivery capabilities.  Asset libraries allow users to upload videos, which are stored as SQL BLOBs in content databases (just like any other file in SharePoint).  Videos tend to be larger in size, which are both inefficient for BLOB storage and quickly grow content databases to maximum recommended capacity (200GB).  Videos are also subject to the maximum file upload size in SharePoint, which is 50MB by default with a hard limit of 2GB.  Videos that are uploaded into SharePoint can be consumed by “progressive download” in the exact same format and quality as the contributor uploaded.  This means that a 1080P .wmv uploaded to SharePoint will only be consumable as a 1080P .wmv file.  A progressive download also lacks intelligent fast-forward capabilities.  If a user only cares about the end of a long video, they have to download the entire video to get to the end.  Many enterprises with SharePoint (including Microsoft) have implemented solutions aimed addressing these challenges, including Remote Blob Storage (RBS), Blob Caching, Branch Caching, Bitrate Throttling, Encoding/Transcoding, DRM, and many more.

Academy is Microsoft’s internal social video platform built on SharePoint.  Academy sets a high standard for media delivery maturity, with customized media uploads, geographically distributed streaming, adjustable quality by device/connection, and advanced encoding workflows to accept almost any media format.  Academy is impressive, but represents years of effort, refinement, and lessons learned from digital asset and web content management.  Most organizations would find it very challenging to close the gap between media delivery native to SharePoint and what Microsoft has built internally with Academy…until now!

Academy (Microsoft's internal social video platform built on SharePoint)

The landscape has changed significantly since Academy was first deployed at Microsoft several years ago.  Now, Windows Azure can provide the platform for media storage (Blob Storage), encoding/streaming (Media Services), and global distribution (Azure CDN) with very little CAPEX.  Additionally, the new SharePoint app model can help deliver these Azure capabilities, providing a highly customized media delivery experience to any SharePoint farm…even in SharePoint Online/Office 365.

The Solution

Our app will exploit the best of both SharePoint 2013 and Windows Azure for media contribution, consumption, and management.  These capabilities will be delivered through a number of pages in our app and ribbon buttons in the host site.  I have detailed each of these solution components below.  The solution also leverages a SQL Azure database to keep track of videos sent through the app and app preferences/settings.

Upload.aspx

Upload.aspx (Empty)

Upload.aspx (Processing)

We want media contributors to have a familiar experience contributing videos through our app.  Since videos are traditionally uploaded through the ribbon on asset libraries, our solution will leverage the same ribbon to launch the upload page for our app.  The asset library ribbon button will launch the upload page inside the SharePoint dialog.  You can see this ribbon button and the upload dialog in the two screenshots above and the custom action xml to make it happen below:

The upload page is where most of the magic happens, allowing users to send videos to Azure Media Services for encoding, thumbnail generation, and blob storage.  Uploading large videos, sending them to the cloud, and encoding could take significant time.  Most users won’t want to sit around waiting for this to complete.  For the proof of concept (POC), many of these processes are executed on a separate thread so the upload page respond once all the data posts.  Ideally, we would leverage an Azure Worker Process for this long running process (which is similar to a Windows Service).  However, our solution is implemented as an autohosted app for SharePoint for simplicity in deployment/tenancy.  Threading was easier to implement in that model for this POC.

The solution leverages a ProcessMediaUtil class to upload media to Azure Blob Storage, encode the videos to a common format, publish the encoded media, and generate thumbnails.

The ProcessMediaUtil will also create the "video set" in SharePoint once all the Azure jobs are complete.  The new video set will reference the Azure-hosted video via an embed code (which is stored in the hidden VideoSetEmbedCode column of the video set).

Our solution will provide the user with status updates as the Azure job(s) process.  To do this, our app will host a REST/JSON status service.  This service is secured by the same context token our app leverages and can be called periodically from client-side script on our page.

Player.aspx

Player.aspx embedded in SharePoint video set page

In SharePoint 2013, videos can be contributed as a file, URL, or embed code (IFRAME to video).  You might be curious why the solution needs a player page instead of just providing SharePoint with a URL to the video hosted in Azure Media Services.  Unfortunately, SharePoint will not accept a parameterized video URL (ex: MyVideo.mp4?contextToken=xzy123) such as the one Azure Media Services will provide.  Even if it did, our solution might want to leverage a different media player that supports smooth streaming or advanced video analytics (ex: how long the user watched the video).  Instead, we will leverage a video embed code, which is ultimately an IFRAME pointing to a page hosting the video.  This is same way we would reference a YouTube video in a SharePoint asset library.  Azure Media Services does not provide a player page, so our app will deliver it.  In the screenshot above, you see the typical video set page displayed in SharePoint...the Player.aspx app page is being displayed in an IFRAME to display the video (the IFRAME is the same size as the video).  To make the player page dynamic, our embed codes will always pass a video id to the player page via URL parameter.  Our player page will use this video id to lookup the streaming URL stored in the app’s SQL Azure database.  We will also pass the host web URL in case the player page needs to get a context token from SharePoint.

The result is a video set in SharePoint that looks exactly like any other.  With so many moving parts, I’ve provided a diagram to clear any confusion:

Diagram of the player page and embed code logic

Default.aspx

Default.aspx page to display media processed by app

With videos being stored in Azure and the video set living in SharePoint, there exists the potential for orphaned items (“video sets” without corresponding videos and videos without corresponding “video sets”).  The default.aspx page allows users to identify potential orphan items or errors that may have occurred during processing.  It is the default page when a user enters the full-screen view of the app.  Nothing fancy here…just a GridView that displays information from our app database.

Settings.aspx

Settings.aspx for configuring app settings

In order for our app to integrate with Azure, it will need an account name and access key for Azure Blob Storage and Azure Media Services.  The settings page will allow an app administrator to configure these account settings, which will be stored in the app’s database.  The settings page will also allow an app administrator to specify additional app administrators and select the output format(s) for encoding.  Nothing interesting here, other than the multi-user people picker control.

Multi-user people picker for setting app admins

The settings page is security trimmed to app administrators.  Because we need at least one app administrator, the app will make use of the app installed remote event to capture the user that installs the app.

Handle App Installed in app project properties

Final Thoughts

This solution might seem like a lot of work.  However, it addresses most of the serious limitations in native SharePoint video delivery that will only become more obvious as video contribution increases in a farm.  I have provided the code for the solution in the link below.  This is NOT a production ready solution.  However, it could be the start for taking media to the next level in your organization!

App for SharePoint Solution: AzureMediaManager.zip

Cross-site publishing is one of the powerful new capabilities in SharePoint 2013.  It enables the separation of data entry from display and breaks down the container barriers that have traditionally existed in SharePoint (ex: rolling up information across site collections).  Cross-site publishing is delivered through search and a number of new features, including list/library catalogs, catalog connections, and the content search web part.  Unfortunately, SharePoint Online/Office 365 doesn’t currently support these features.  Until they are added to the service (possibly in a quarterly update), customers will be looking for alternatives to close the gap.  In this post, I will outline several alternatives for delivering cross-site and search-driven content in SharePoint Online and how to template these views for reuse.  Here is a video that outlines the solution:

(Please visit the site to view this video)

Apps for SharePoint

The new SharePoint app model is fully supported in SharePoint Online and can be used to deliver customizations to SharePoint using any web technology.  New SharePoint APIs can be used with the app model to deliver an experience similar to cross-site publishing.  In fact, the content search web part could be re-written for delivery through the app model as an “App Part” for SharePoint Online. 
Although the app model provides great flexibility and reuse, it does come with some drawbacks.  Because an app part is delivered through a glorified IFRAME, it would be challenging to navigate to a new page from within the app part.  A link within the app would only navigate within the IFRAME (not the parent of the IFRAME).  Secondly, there isn’t a great mechanism for templating a site to automatically leverage an app part on its page(s).  Apps do not work with site templates, so a site that contains an app cannot be saved as a template.  Apps can be “stapled” to sites, but the app installed event (which would be needed to add the app part to a page) only fires when the app is installed into the app catalog.

REST APIs and Script Editor

The script editor web part is a powerful new tool that can help deliver flexible customization into SharePoint Online.  The script editor web part allows a block of client-side script to be added to any wiki or web part page in a site.  Combined with the new SharePoint REST APIs, the script editor web part can deliver mash-ups very similar to cross-site publishing and the content search web part.  Unlike apps for SharePoint, the script editor isn’t constrained by IFRAME containers, app permissions, or templating limitations.  In fact, a well-configured script editor web part could be exported and re-imported into the web part gallery for reuse.

Cross-site publishing leverages “catalogs” for precise querying of specific content.  Any List/Library can be designated as a catalog.  By making this designation, SharePoint will automatically create managed properties for columns of the List/Library and ultimately generate a search result source in sites that consume the catalog.  Although SharePoint Online doesn’t support catalogs, it support the building blocks such as managed properties and result sources.  These can be manually configured to provide the same precise querying in SharePoint Online and exploited in the script editor web part for display.

An easier approach might be to directly reference a list/library in the REST call of our client-side script.  This wouldn’t require manual search configuration and would provide real-time publishing (no waiting for new items to get indexed).  You could think of this approach similar to a content by query web part across site collections (possibly even farms) and the REST API makes it all possible!

The content search web part uses display templates to render search results in different arrangements (ex: list with images, image carousel, etc).  There are two types of display templates the content search web part leverages…the control template, which renders the container around the items, and the item template, which renders each individual item in the search results.  This is very similar to the way a Repeater control works in ASP.NET.  Display templates are authored using HTML, but are converted to client-side script automatically by SharePoint for rendering.  I mention this because our approach is very similar…we will leverage a container and then loop through and render items in script.  In fact, all the examples in this post were converted from display templates in a public site I’m working on. 

Even one of the more complex carousel views from my site took less than 30min to convert to the script editor approach.

End-result of script editors in SharePoint Online

Separate authoring site collection

Final Thoughts

I hope this post helped illustrate ways to display content across traditional SharePoint boundaries without cross-site publishing and how to template those displays for reuse.  SharePoint Online might eventually get cross-site publishing feature, but that doesn’t mean you have to wait to achieve the same result.  In fact, the script approach is so similar to display templates, it should be an easy transition to cross-site publishing in the future.  I want to give a shout out to my colleague Nathan Miller for has assist in this vision.

I’ve always been a big fan of self-service site provisioning in SharePoint.  That process is even better in SharePoint 2013 (including SharePoint Online), with the ability to specify a custom site creation form.  Owning the provisioning process is incredibly powerful for farm/tenant administrators to configure the options and outputs of site provisioning.  By leveraging the new app model, we can customize the provision process in SharePoint Online to activate specific features, capabilities, and branding to our sites.  This post will explore the process for delivering custom site creation using an app for SharePoint.  The video below demonstrates the solution outlined in this post.

(Please visit the site to view this video)

Self-Service Sites in SharePoint 2013

Self-service provisioning is improved in SharePoint 2013 by allowing an administrator to specify a custom creation form.  This form can take over the provisioning process and will be the cornerstone of the solution outlined below.  Without implementing a custom form, the default form only allows a user to specify a title.  The default output will be a Team site created as a sub-site at a specified location.  Again, if we want to capture additional details or provide different options, we must introduce a custom creation form.

Default Create a Site form in SharePoint 2013

The Solution

Because the app needs the ability to create sub-sites and site collections anywhere in the tenancy, it will need FullControl permission on the entire tenancy.  The app will also need to make app-only calls to SharePoint, so it can work with tenant objects or sites outside the context.  Both these settings can be configured in the Permissions tab of the AppManifest.xml.

AppManifest Permissions for out App

Our app will enable administrators to dynamically configure site creation settings using a Library in the app web.  I’m using a library instead of a list, so each option can be displayed in the site creation form with an icon.  Each row in the Library will represent a site provisioning option for an end-user.  The columns to support this include the following:

• Title– the title of the configuration option (ex: Small Team Site)
• Site Template– the WebTemplate name to be used in provisioning (ex: STS#0 for Team Site)
• Base Path– the absolute URL where this option will get provisioned (ex: https://tenant/teams/)
• Site Type– choice of “Subsite” or “Site Collection”
• MasterPage URL– url of the masterpage file to apply (leave blank for no branding)
• Storage Maximum Limit– the storage quota in MB (only applicable for site collections)
• UserCode Maximum Limit– the user code quota in points (only applicable for site collections)

Library Configuration in App Project

I leveraged a module in the solution to pre-load a few provisioning options, but an administrator could easily edit these options or add new options through the library.

Our app will deliver a site creation form that will query the library to display site creation options.

Once the user submits the site creation form, the app will provision differently based on site type (Subsite or Site Collection).  One thing that is common between the two methods is the need to execute app-only calls, since we will likely be provisioning with a different context from where the form is hosted (ex: site collection will require the context of the tenant administration site).  The TokenHelper contains a GetAppOnlyAccessToken method to get the access token for a specific site that is different from the context of the form.

To provision a sub-site, we need to establish context with the site that will host our new sub-site (ie – the parent site).  To apply a brand to a sub-site, I’m requiring the master page to exist in the Master Page Gallery of the root web in the site collection.  That way, I can just set the MasterUrl and CustomMasterUrl after the new sub-site is provisioned.

Provisioning a site collection is a little more complex.  First we need to establish context with tenant administration site and use a Tenant object for creation.  The Tenant object can be found in the Microsoft.Online.SharePoint.Client.Tenant assembly which comes with the SharePoint Online Management Shell download.  Provisioning the site collection can take a while to complete, so this occurs asynchronously using the SpoOperation class.  We can leverage this to wait until the creation operation is complete before trying to apply the master page.  Applying a master page to a site collection is a little more complex.  The master page needs to live within the site collection, so our solution will download the master page from the location referenced in the configuration and upload it into the Master Page Gallery of the new site collection before setting MasterUrl and CustomMasterUrl on the site.  This requires that scripts and styles in the master page are referenced using absolute paths.  Pulling down and applying an entire design package would likely be a more elegant, but was overkill for this proof of concept.

Because our site creation form will be launch in the “Start a Site” dialog, our app needs to be able to communicate back with the SharePoint page that hosts the dialog.  This communication is necessary to make the dialog page visible, resize the dialog, close the dialog, and navigate away from the dialog.  This cross-domain communication is achieved using the HTML5 postMessage API, where SharePoint “listens” for MakePageVisible, Resize, CloseDialog, and NavigateParent messages from the page displayed within the dialog IFRAME.  Below are the scripts to implement this messaging, which I hunted down through thousands of line of javascript.

In order for our site creation form to launch from the “add site” link on the “sites” page, we need to configure it as the “Start a Site” form in the tenant administration site.  This field is limited to 255 characters, so we will likely need to remove some of the standard tokens that are passed to our app.  At minimum, the URL MUST include SPHostUrl and SPAppWebUrl parameters.  Our app will leverage these to get context and access tokens from SharePoint.

"Start a Site" configuration in SharePoint admin center

Our app should now be ready for primetime!  Here is a screenshot of our custom app launched from the “Start a Site” dialog and in full-screen mode.

Our custom provisioning app launched from the "Create a Site" link

The fullscreen version of the app

Final Thoughts

I hope this post helped illustrate the power of delivering self-service site creation and how the app model can take it to the next level.  I see this and “App Stapling” as the primary two patterns for pushing capability/features into sites and site collections in SharePoint Online.  Please note that the provided code is not production ready and also references on of my tenants.

Solution code: http://sdrv.ms/XSBX7n

Microsoft re-engineered the search experience in SharePoint 2013 to take advantage of the best capabilities from FAST plus many new capabilities built from the ground up.  Although much has been said about the query side changes of search (result sources, query rules, content by search web part, display templates, etc), the feed side of search got similar love from Redmond.  In this post I’ll discuss a concept carried over from FAST that allows crawled content to be manually massaged before getting added to the search index.  Several basic examples of this capability exist, so I’ll throw some advanced solution challenges at it.  The solution adds a sentiment analysis score to indexed social activity as is outlined in the video below.

(Please visit the site to view this video)

The content enrichment web service (CEWS) callout is a component of the content processing pipeline that enables organizations to augment content before it is added to the search index.  CEWS can be any external SOAP-based web service that implements the IContentProcessingEnrichmentService interface.  SharePoint can be configured to call CEWS with specific managed properties and (optionally) the raw binary file.  CEWS can update existing managed property values and/or add completely new managed properties.  The outputs of this enrichment service get merged into content before it is added to the search index.  The CEWS callout can be used for numerous data cleansing, entity extraction, classification, and tagging scenarios such as:

• Perform sentiment analysis on social activity and augment activity with a sentiment score
• Translate a field or folder structure to a taxonomy term in the managed metadata service
• Derive an item property based on one or more other properties
• Perform lookups against line of business data and tag items with that data
• Parse the raw binary file for more advanced entity extraction

The content enrichment web service is a synchronous callout in the content processing pipeline.  As such, complex operations in CEWS can have a big impact on crawl durations/performance.  An additional challenge exists in the enrichment of content that hasn’t changed (and thus doesn’t get crawled).  An item only goes through the content processing pipeline during full crawls or incremental/continuous crawls after the item is updated/marked dirty.  When only the enriched properties need to change, a full crawl is the only out of the box approach to content enrichment.

The solution outlined in this post addresses both of these challenges.  It will deliver an asynchronous CEWS callout and a process for marking an indexed item as dirty so it can be re-crawled without touching/updating the actual item.  The entire solution has three primary components…a content enrichment web service, a custom SharePoint timer job for marking content in the crawl log for re-crawl, and a database to queue asynchronous results that other components can reference.

High-level Architecture of Async CEWS Solution

Enrichment Queue (SQL Database)

Because of the asynchronous nature of the solution, operations will be running on different threads, some of which could be long running.  In order to persist information between threads, I leveraged a single-table SQL database to queue asynchronously processed items.  Here is the schema and description of that database table.

Content Enrichment Web Service

As mentioned at the beginning of the post, the content enrichment web service callout is implemented by creating a web service that references the IContentProcessingEnrichmentService interface.  There are a number of great example of this online, including MSDN. Instead, this post will focus on calling asynchronous operations from this callout.  The main objective of making the CEWS callout asynchronous is to prevent the negative impact a long running process could have on crawling content.  The best way to do this in CEWS is to collect all the information we need in the callout, pass the information to a long running process queue, update any items that have values ready from the queue, and then release the callout thread (before the long running process completes).

Process Diagram of Async CEWS

Below is the callout code in its entirety.  Note that I leveraged the entity framework for connecting to my enrichment queue database (ContentEnrichmentEntities class below):

The content enrichment web service is associated with a search service application using Windows PowerShell.  The configuration of this service has a lot of flexibility around the managed properties going in and out of CEWS and the criteria for triggering the callout.  In my example the trigger is empty, indicating all items going through CEWS:

Timer Job (Force Re-Crawl)

The biggest challenge with an asynchronous enrichment approach is updating the index after the CEWS thread is released.  No API exists to directly update items in the search index, so CEWS is the last opportunity to augment an item before it becomes available to users executing queries.  The best we can do is kick-off an asynchronous thread that can queue enrichment data for the next crawl.  Marking individual items for re-crawl is a critical component to the solution, because “the next crawl” will only crawl items if a full crawl occurs or if the search connector believes the source items have updated (which could be never).  The crawl log in Central Administration provides a mechanism to mark individual indexed items for re-crawl

CrawlLogURLExplorer.aspx option to recrawl

I decompiled the CrawlLogURLExplorer.aspx page and was pleased to find it leveraged a Microsoft.Office.Server.Search.Administration.CrawlLog class with a public RecrawlDocument method to re-crawl items by path.  This API will basically update an item in the crawl log so it looks like an error to the crawler, and thus picked up in the next incremental/continuous crawl.

So why a custom SharePoint timer job?  An item may not yet be represented in the crawl log when it completes our asynchronous thread (especially for new items).  Calling RecrawlDocument on a path that does not exist in the crawl log would do nothing.  The timer job allows us to mark items for re-crawl only if the most recent crawl is complete or has a start date after the crawl timestamp of the item.  In short, it will take a minimum of two incremental crawls for a new item to get enrichment data with this asynchronous approach.

With these three solution components in place, we get the following before/after experience in search

Before	After

Final Thoughts

Content enrichment is a very mature but powerful search customization.  I hope this post helped illustrate a creative use of content enrichment that can take your search experience to the next level.

Code for solution: ContentEnrichmentServices.zip

I have spent the last year traveling the world and authoring numerous posts/solutions to evangelize SharePoint development using apps for SharePoint.  Apps for SharePoint are incredibly powerful, but many SharePoint developers still gravitate to farm solutions based on familiarity and misconceptions on the "limitations" of the app model.  I'm confident that almost ANY SharePoint solution can be delivered in the app model with good understanding and a little creativity.  To support my case, I decided to assemble a comprehensive list of common SharePoint customizations, evaluate their support in the new app model, and provide alternate solutions where feasible.  Although it is clear that gaps exist between apps and farm solutions, alternatives are available in my cases.  Feel free to send me a note if you feel I'm missing something from this list.

Customization	App Capability	Comments/Limitations/Alternatives
Alternate CSS (in hive)		Apps cannot deploy files to SharePoint's root files/hive. Instead, css can be deployed to SharePoint using a module or app installed event.
Alternate CSS (in module)		Apps can use modules to deploy css to SharePoint, but only to the app web…not the host web. Referencing css from an app web can have inconsistent results in a host web across depending on the users browser and browser settings. As an alternative, a app installed event can be used to deploy css to a host web.
Alternate CSS (apply)		Apps can leverage an app installed event to apply alternate css to a SharePoint site. This is similar to using feature receivers in farm solutions for the same results.
App for Office		Deploying an app for Office inside SharePoint solution is completely new in 2013 and fully supported using apps for SharePoint. This can be a very powerful components to include in solutions, as is outlined in this post
Application Page		Apps cannot deploy files to SharePoint's root files/hive, which includes the layouts folder. As an alternative, web part pages can be deployed through a module or app installed event receiver.
Column		Declarative columns can only be deployed to the app web…not the host web. If you need to deploy columns into the host web (ex: a content syndication hub), you can leverage an app installed event to provision columns programmatically in the host web.
Content Type		Declarative content types can only be deployed to the app web…not the host web. If you need to deploy content types into the host web (ex: a content syndication hub), you can leverage an app installed event to provision content types programmatically in the host web.
Custom Action		Apps support only specific types of custom actions, including ribbon commands and context menus in both the app web and host web. These can be very powerful when combined with dialog boxes, but do not cover the full spectrum of custom actions available to farm solutions (ex: Site Actions menu).
Delegate Control		Apps for SharePoint do not support "Control" elements that can be used with delegate controls in a master page. Delegate controls are a common mechanism for swapping out functionality of a site using a farm solutions (particularly useful with the AdditionalPageHead delegate). As an alternative, the same result can be achieved through the design in a custom master page (ex: place specific html or server controls in the master page).
Feature Receiver		Apps have remote events for "App Installed", "App Uninstalling", and "App Upgraded". All of these can be used in a similar way to feature receivers in farm solutions. It should be noted that app remote events do not implement reliable messaging.
Feature Stapling		Apps can be "stapled" through the app catalog.  This has more flexible rules than traditional stapling, allowing specific sites, managed paths, or site templates to get apps pushed to them. The app catalog even provides a mechanism for app retraction. However, apps installed through app stapling share a single app web (from the app catalog) and app remote events only fire during installation into the catalog. For more information, see my post a few months back on "app stapling".
Field Type		Custom field types require files to be deployed into SharePoint's root files/hive, which is off limits to the app model. As an alternative, apps can easily deploy custom forms that can leverage almost any web/html controls imaginable. Custom field types have been discouraged, so this approach is even better than farm solutions IMO.
Image (in hive)		Apps cannot deploy files to SharePoint's root files/hive. Instead, images can be deployed to SharePoint using a module or app installed event.
List Definition		Apps can deploy custom list definitions, but only to the app web and not the host web.
List Event Receiver		Apps for SharePoint can deploy remote event receivers that provide similar functionality to list event receivers in farm solutions. However, they can only be deployed to lists/libraries in the app web, not the host web.
List Instance		Declarative list instances can only be deployed to the app web and not the host web. If you need a list provisioned in the host web, you can alternatively look at provisioning the list programmatically into the host web using an app installed event.
Master Page (in hive)		Apps cannot deploy files to SharePoint's root files/hive. Instead, master pages can be deployed to SharePoint using a module or app installed event.
Master Page (module)		Apps can use modules to deploy master pages to SharePoint, but only to the app web…not the host web. Referencing a master page from an app web can have inconsistent results in a host web across depending on the users browser and browser settings. As an alternative, a app installed event can be used to deploy a master page to a host web.
Master Page (apply)		Apps can leverage an app installed remote event to apply a master page to a SharePoint site. This is similar to using feature receivers in farm solutions for the same results.
Module		Apps can leverage modules to declaratively deploy files into SharePoint. However, they can only deploy to the app web. As an alternative, a app installed event can be used to deploy files to a host web.
Script (in hive)		Apps cannot deploy files to SharePoint's root files/hive. Instead, script can be deployed to SharePoint using a module or app installed event.
Script (module)		Apps can use modules to deploy script to SharePoint, but only to the app web…not the host web. Referencing script from an app web can have inconsistent results in a host web across depending on the users browser and browser settings. This also poses a cross-domain issue for the script. As an alternative, a app installed event can be used to deploy script to a host web.
Search Configuration		The query side of search is largely re-architected in SharePoint 2013 and supports the export/import of search configuration. Apps for SharePoint can be used to consistently and efficiently deploy a search configuration to a number of sites!
Service Application		Cloud-hosted apps for SharePoint (Provider and Autohosted) can deliver similar functionality to service applications. Apps can have tenancy permissions and configured to make app-only API calls (ex: outside the context of a user). It all depends on the desired integration and capabilities to determine if an app is an acceptable alternative.
Site Definition		Site definitions deploy files into SharePoint's root files/hive, which is not a capability of apps for SharePoint. As an alternative, consider web templates, "app stapling", or controlling the provisioning process.
Theme		Although I would question the sanity of someone looking to create a theme (we seem to change themes in each release of SharePoint), they can be deployed through an app module (to app web) or app installed event (to host web). This includes the new "composed looks" in SharePoint 2013 that include several files (master pages, theme color palette, fonts, images, etc).
Timer Job		An external process (ex: worker role) can deliver similar functionality to a timer job given the app has tenancy permissions and the ability to perform app-only calls. It all depends on the desired integration and capabilities to determine if an app is an acceptable alternative. I did this for media encoding in this solution and for site collection provisioning in this solution.
User Control (controltemplates)		Apps cannot deploy files to SharePoint's root files/hive, which includes the controltemplates folder. As an alternative, user controls can live in the remote web of an app and used in some similar ways to a user control in controltemplates, but not as delegate controls or page directives in SharePoint pages.
Web Service (ISAPI)		ISAPI web services (under the _vti_bin url) are deployed to the root files/hive of SharePoint, which is off limits to apps. As an alternative, "Cloud-hosted" apps for SharePoint (Provider and Autohosted) can host custom web service in their remote web
Web Part		Apps can deploy "App Parts" (also called Client Web Parts) to SharePoint, but these can only have certain types of custom properties (string, int, bool, enum) and cannot use native web part connections. As an alternative, App Parts can leverage postMessage APIs to communicate to SharePoint (and possibly to each other).
Web Part Page		A web part page is simply an .aspx file deployed through a module. Apps support modules, but only to the app web and not the host web. An alternative is to use the app installed event to add the web part page to the host web. I give this a 50% because it would be impossible to add web parts declaratively to the web part page using this approach (would have to be programmatically or through the UI)
Web Template		Web templates live in the solutions catalog of the host web's site collection root. An app installed event can be used to deploy a web template .wsp to the solution catalog, but cannot activate it.
Workflow		Workflow is re-architected in 2013 to be declarative and is completely supported in the new app model.
Workflow Activity		Workflow is re-architected in 2013 to be declarative and is completely supported in the new app model.

Important Note

Apps for SharePoint are designed to leverage the app web for all SharePoint things it needs.  The app web is a hidden sub-site under the host web that provides isolation for the app.  This is why ALL the declarative elements of an app are deployed to the app web and not host web (ex: modules, columns, content types, etc).  Apps are architected this way so they leave no trace of themselves on the host web when they are uninstalled.  The list above mentions the use of the app installed event to deploy elements to the host web.  This is NOT an appropriate practice for general app development, especially apps that are designed for the Office Store.  The approach is more appropriate for internal development, including customers moving to SharePoint Online or with extensive farm solutions.

Getting Creative

I have two creative tips that have helped me close the gap between apps and farm solutions:

1. Control the provisioning process...by doing this, you can tack on just about anything you can imagine using CSOM before you hand the site over to a user.  This includes, branding, specialized permissions, feature activation, etc.  I recently authored a solution that shows how to do this with the app model (even provisioning site collections).  You can find that solution here.
2. Add App-only Permission to your app...doing this allows your app to call SharePoint APIs outside the context of a user.  This is enormously helpful for long running processes/jobs or from other apps.  The solution referenced in tip #1 also uses this approach.

Closing Words

Beyond the list above, it is important to recognize the numerous benefits of the new app model.  Apps for SharePoint do not run server code on the SharePoint farm, making them much safer to run (including in SharePoint Online) and don't impact upgrades/service packs.  They are cloud-ready and cater to the numerous HTML/JS developers in the world, opening the door to numerous solution possibilities and cost efficient developer resources.  One of the biggest advantages I've realized is in my development environment.  In the past, I've carried a huge laptop with multiple cores and 32GB or memory.  This was required because SharePoint assemblies weren't remoteable, requiring a full SharePoint server to be an effective developer.  Today, I can use almost anything with a browser and a developer site collection (add Visual Studio for cloud-hosted apps).  The picture below is my new SharePoint development machine and my old development machine...ahhh life is good!

With the introduction of apps for SharePoint, many have speculated that sandbox solutions are dead/deprecated.  This is accurate for solutions containing assemblies running on the Sandboxed Code Service (aka - SPUCHostService.exe).  However, declarative solutions are very much still in play and widely used internally by SharePoint (ex: Web Templates and Design Manager).  A declarative .wsp package (one containing no assemblies) is a powerful way to provision elements into the host web.  So don't be afraid to leverage the solutions gallery as long as your .wsp packages don't contain code.

If your solution contains code and a farm solution isn't an option (read: SharePoint Online), you might consider an approach I outlined in my recent post on App Approaches for Common SharePoint Customizations.  In it, I discussed the use of app installed remote events to provision elements in the host web.  This pattern is generally discouraged, as apps for SharePoint should uninstall cleanly (hence the reason for an app web).  However, it is appropriate for internal apps (not marketplace apps) containing programming logic and host web dependencies.  The app model should not be confused as a deployment mechanism, so if your solution doesn't have code...consider the solution gallery.

I have found that almost every app for SharePoint can benefit from leveraging a social platform.  It might be as simple as using a social platform for basic profile information/pictures.  The most obvious choice for many app developers is to leverage SharePoint as the social platform.  After all, the app manifest allows developers to request permissions to the social features of SharePoint.  However, Microsoft offers a more feature-rich social platform in Yammer.  Microsoft’s future social investments will be heavily concentrated on Yammer and the Yammer/SharePoint integration.  A glimpse into these investments are detailed in Jared Spataro’s post on the enterprise social roadmap.  By targeting both social platforms, SharePoint app developers greatly increase their customer reach and better position their apps for the direction of the SharePoint platform.  In this post, I will outline some patterns and lessons learned from developing Social Nucleus, an app for SharePoint that can leverage SharePoint or Yammer as its social platform.

(Please visit the site to view this video)

App Catalogs

Both SharePoint and Yammer have public and private catalogs for apps.  In SharePoint, internal/private apps can be published in an app catalog site collection and made available within an organization.  Public SharePoint apps can be found in the SharePoint Store, a public marketplace for solutions that have been tested and validated by Microsoft.  In Yammer, the only difference between a public and private app is a “Global” flag.  Apps without this global flag are only available to the home network of the app developer.  Apps that are deployed to the Global App Directory go through a vetting process with Yammer, and if approved, are marked "Global" and listed in directory.  App developers can build one app and have it listed in two app catalogs...very cool!  This is the pattern I took with Social Nucleus, which is listed FREE in both the SharePoint Store and Yammer’s Global App Directory.

SharePoint Store	Yammer Global Catalog

Yammer Development

If you are interested in developing apps against Yammer, I highly recommend reviewing their API Documentation.  You will find it thorough, very similar to SharePoint, and easy to use.

Authentication

Both SharePoint and Yammer app models leverage OAuth to allow an app to access platform resources on behalf of the user.  When you register a new Yammer app, it looks VERY similar to the appregnew.aspx page used to register apps for SharePoint:

SharePoint App Registration	Yammer App Registration

Yammer provides both client-side and server-side authentication approaches, but perhaps the easiest is to leverage the Yammer Login Button in the JavaScript SDK.  Referencing this script (with an app client id) will handle the entire OAuth process against Yammer (including API calls).

Building a hybrid social app with SharePoint/Yammer brings some interesting considerations when it comes to authentication.  This is especially true when an app has two entry points…entry from SharePoint and entry from Yammer.  SharePoint OAuth will require us to know the specific things about how the app is installed in SharePoint, such as the host web URL and app web URL.  These values are typically passed to the app as URL parameters from SharePoint.  Without these parameters, our app will automatically assume the user navigated from Yammer (or entered the app URL directly) and default to Yammer as the social platform.

When the app is accessed from SharePoint, we know the user has already authenticated to SharePoint.  However, we don’t know if the user is already authenticated to Yammer (regardless of single sign-on configuration).  As such, our app will check the login status (yam.getLoginStatus) of the user in Yammer and display Yammer’s Login Button accordingly.

APIs

Yammer’s REST APIs are very similar to the REST APIs available in SharePoint.  Both platforms have basic APIs for getting profile information, following/followers, posts, etc.  To minimize the platform-specific logic, I added a layer of abstraction between the social platforms and the user interface using platform independent functions and objects.  The key was to determine the properties my app needed for each entity and how those properties mapped to the json returned from the APIs.  Below is a sample of how I did this for a user.

As you can see in the sample above, both platforms have their own APIs for making cross-domain REST calls.  In SharePoint, we use the executeAsync method of the RequestExecutor class (found in the SP.RequestEcecutor.js library).  In Yammer, we use the yam.request method.  Both of these methods do more than just handle cross-domain calls, they also handle all the OAuth/token stuff so that it is transparent to the developer.

Pitfalls and Lessons Learned

Although most of my development on this hybrid app was smooth sailing, there were a few pitfalls I ran into that you can learn from:

Cross-domain Conflicts
It turns out that SharePoint’s RequestExecutor and Yammer’s JavaScript SDK conflict with each other on event listeners used in processing the response on cross-domain REST requests.  I spent hours trying to unregister the conflicts, but ultimately found an easier way to address the issue.  I decided to load the Yammer scripts by default (so I can check for Yammer login status) and redirect the user to the same page (but without the Yammer scripts) if they select SharePoint as the social platform.  This means that only one platform’s libraries are registered at any given time.

Throttling
Yammer institutes rate limits on their REST APIs.  These limits vary, but are throttled at 10 request in 10 seconds for the profile and following APIs I leveraged in Social Nucleus.  This comes into play for APIs that are paged.  For example, popular users or groups could have thousands of followers in Yammer.  The API for followers brings back blocks of 50 at a time.  I originally looked to load all followers recursively, but quickly ran into these throttled limits.  Instead, I notify the user that that the first 50 are loaded and allow them to load more.

Global Catalog
Before an app for Yammer is flagged as “Global”, it will only function against the home network of the developer.  Many organizations might have multiple networks (ex: divisions) but still want the app private.  Yammer has the ability to mark apps with a hidden “Global” flag so the app can function against multiple networks, but not display in the global app catalog.  If you have this scenario, I would suggest posting a message to the Yammer Developer Network.  Another related roadblock exists with app submissions to the SharePoint Store.  During the validation process for the SharePoint Store, testers will need to validate the app against a test/demo network in Yammer in order to approve the app.  For this to occur, you will likely need Yammer app validation prior to SharePoint app validation

Redirects
Since Yammer uses a static URL redirect during OAuth, I originally found it difficult to manage development and production environments.  My solution was to register two Yammer apps…a development app that redirects to localhost and a production app that redirects my product web application running in Windows Azure.  I used a simple DEBUG check to spit out the correct include.  If you look at the code below, you might find it odd that I used LiteralControl objects instead of .NET's ScriptManager.  Yammer requires an additional attribute (data-app-id) on the script include that required me to do it this way.

IE Security Zones
IT and InfoSec Pros might love Internet Explorer’s security zones, but I despise them as an app developer.  When SharePoint/Yammer and your app live in different security zones, users can experience inconsistent results (ex: pictures fail to load since they live in another zone).  Even worse…when your app defaults to the “Internet” zone, the entire Yammer OAuth process breaks (popups and cross-window communication is blocked).  The only client-side solution I have found is to put the app in the “trusted sites” security zone.  Chrome and Firefox don't have this security "feature" and thus don't have the issue.  I plan on writing a post on these security zone hardships with apps in the future, but be warned…

Final Thoughts

I hope this post helped illustrate the ease of developing apps for SharePoint AND Yammer.  Yammer is a significant investment to SharePoint and will drive Microsoft’s social direction in the platform.  If you develop apps for SharePoint, consider Yammer in those solutions if they have a social play.  Providing Yammer hooks will give the app greater visibility/reach and better position it for the future.

One of the exciting announcements at the 2014 SharePoint Conference, was the general availability the Office App Model Samples (OfficeAMS) on Codeplex.  OfficeAMS is grass-roots effort by volunteer Microsoft developers to provide code samples using the Office/SharePoint app model that address common customization scenarios.  The samples in OfficeAMS include many scenarios that the SharePoint community felt was impossible or challenging to deliver without full-trust code.  It includes almost 30 complex scenarios such as site provisioning, branding, templating, profile sync, UX controls, and many more. Vesa Juvonen has a more comprehensive write-up of OfficeAMS on his blog (Vesa was also a significant contributor to the effort).

One of the scenarios that I contributed in OfficeAMS is a Taxonomy Picker control for provider-hosted apps. I used this control in several SPC sessions and was surprised by the number of people interested in using it in their apps. In this post, I will detail this Taxonomy Picker control and how to use it in your apps.  I’ve also included a video that outlines the use of the control.

(Please visit the site to view this video)

Setup

The taxonomy picker connects to TermSets in the Managed Metadata Service in SharePoint. To do this from an app, the app needs a minimum of Read permissions to the Taxonomy scope. The app needs Write permissions to the Taxonomy scope if you plan to use “Open” TermSets that users can create Terms in (aka – “Folksonomies”).

Taxonomy scoped Permissions in AppManifest.xml

The app project also needs to include several script and style files from the OfficeAMS download.  These are available in the Contoso.Components.TaxonomyPicker project.  The important files are as follows:

• Scripts\taxonomypickercontrol.js
• Scripts\taxonomypickercontrol_resources.en.js
• Styles\taxonomypickercontrol.css
• Styles\Images (entire folder of images)

The taxonomy picker is a client-side control, which uses the client-side object model (CSOM) to query the Managed Metadata Service.  To do this, the app needs to reference a number of JavaScript files from SharePoint, such as sp.runtime.js, sp.js, sp.requestexecutor.js, init.js, and sp.taxonomy.js.  The sp.taxonomy.js file allows the app to make calls into the Managed Metadata Service. I like to use Microsoft’s AJAX library to load these scripts dynamically using the getScript extension.  This library is automatically added to ASP.NET pages with a ScriptManager control. Below is the complete block of script I use to load the appropriate files for both the taxonomy picker and the SharePoint client chrome control: 

Binding the Taxonomy Picker

After the setup steps, the taxonomy picker is extremely easy to implement. All you need is a hidden input control in html and one line of JavaScript to convert the input into a taxonomy picker:

The taxpicker extension takes two required parameters...the options for the taxonomy picker and the SharePoint client context object (SP.ClientContext).  The taxonomypicker.js version I’ve attached also has a third optional parameter for a value changed callback (not in OfficeAMS yet).  Below is a table that details the options you can pass into the taxpicker extension and some examples of bindings:

Parameters

You can find the ID of a TermSet by selecting the TermSet in the Term Store Manager.  The Term Store Manager is exposed to site collection administrators within s site collection or from Tenant/Central Administration.

Getting TermSet ID from Term Store Manager

Reading and Writing Values

If you want to work with the taxonomy picker in .NET, you need to follow the optional steps in the setup section above to include the Microsoft.SharePoint.Client.Taxonomy reference and extension class.  You should also change the hidden fields to include the attribute runat=”server”.  The taxonomy picker control can be set by initializing the value of the hidden field with JSON in the following format [{"Id":TermID, "Name": TermLabel}] (ex: [{"Id":"a8ff0f61-2c10-4add-8307-cb1712703887", "Name": "Exchange"}]).  The TaxonomyPickerExtensions.cs extension class includes extension methods for converting TaxonomyFieldValue and TaxonomyFieldValueCollection objects into this JSON format:

The taxonomy picker will store the selected terms in the hidden field using JSON string format.  These values can be accessed by other client-side scripts or server-side following a post.  The JSON will include the Term Name, Id, and PathOfTerm (ex: World;North America;United States).  JSON.parse can be used client-side to convert the hidden input’s value to a typed object and any number of server-side libraries can be used (ex: JSON.net).

Conclusion

Building in the SharePoint app model does NOT mean you have to compromise on user experience. I hope this taxonomy picker helps illustrate that.  You can get the taxonomy picker and many other code samples in the Office App Model Samples on Codeplex.

Code sample from blog: TaxonomyPickerSample.zip

The Office App Model Samples (Office AMS) contain some great examples of writing console applications using CSOM. A console application is the foundation for building timer jobs against SharePoint Online and offers a safer alternative to full-trust timer jobs written for on-premises SharePoint. In this post, I will show you how incredibly easy it is to build a timer job as a console application, package it, deploy it as a Web Job in Windows Azure, and schedule it to periodically run against SharePoint. Kirk Evans authored a fantastic post last month on Building a SharePoint App as a Timer Job that I highly recommend you read to understand the logistics of running a console application using OAuth and the app model.

(Please visit the site to view this video)

OAuth or Service Account

Kirk’s sample leveraged OAuth and App Only permissions to perform operations on the SharePoint tenant. The alternative is to leverage a service account and explicitly authenticate as this account to perform operations.  This can be done using normal Windows Authentication or using the SharePointOnlineCredentials class with SharePoint Online:

I prefer the OAuth approach used by Kirk as it is easier to provide tenant-wide permissions and you don’t have to deal with periodic password changes a service account might encounter.  The OAuth approach requires you to deploy a provider-hosted app and then leverage that app’s client id and client secret in a console application. Kirk’s post covers this in great detail.

The example below is a timer job written to leverage CSOM and OAuth in a console application. This specific job copies OneDrive Usage Guidelines to every OneDrive for Business site (aka – every my site). This is a very common ask from customers and represents a perfect scenario for app model timer jobs.

Packaging

After testing the console application locally, we can look to packaging it for deployment into Windows Azure. The first step in packaging is to ensure all the referenced assemblies will be available in the cloud. By default, Windows Azure won’t have CSOM assemblies (ex: Microsoft.SharePoint.Client, Microsoft.SharePoint.Client.Runtime). Any assembly reference in question should be configured with “Copy Local” set to True.

After configuring these references, rebuild the project to create a new output. A traditional provider-hosted app packages with an .app extension, which is deployed into an app catalog. The console application will package as a .zip of the applications output directory (ex: <project folder>\bin\Debug or <project folder>\bin\Release). It does not matter what you name this zip file. Notice below that I zipped up the Debug output folder that contains the console application executable and referenced assemblies:

Deploying and Scheduling

Azure Websites have a new feature called Web Jobs that can host background operations (ex: console applications) to run on demand, continuously, or on a schedule.  The first step is provisioning an Azure Websites in the Azure Management Portal (https://manage.windowsazure.com). Select “Web Sites” from the side navigation and use the NEW button to create a new web site. Here is an example of using Quick Create to create a Web Site in the North Central Azure data center:

Once the Web Site finishes provisioning (5-10 seconds), open it by clicking it in the Web Site listing. You will notice a new tab across the top for WEBJOBS (currently in Preview). Click on this link to display the list of Web Jobs (show be empty):

Next, click on the “ADD A JOB” link to launch the New Job dialog. Specify a name for the job, browse to the zip file we created of the console application’s output directory, and set the Web Job to “Run on a schedule” before clicking the next button:

On the next screen, you can specify the reoccurrence and schedule for the Web Job. The schedule can easily be changed later and the Web Job can be run on demand (very similar to a typical SharePoint Timer Job):

After configuring and accepting the schedule, the Web Job will deploy into Windows Azure and begin running on the specified reoccurrence. The Web Jobs listing will show the status of the job, the time and outcome of the last run, and links to change the schedule and view the logs. This screen also provides the ability to run Web Jobs on demand by selecting a job and clicking on the RUN ONCE button in the footer:

Clicking on the schedule link will take you to the schedule dashboard, which provides some high-level metrics of jobs and links the tweak the schedule:

Final Thoughts

Kirk Evans and Office AMS have outlined some great patterns for performing operations against SharePoint using console applications and CSOM. If the operation a timer job is performing can be refactored to use CSOM, Windows Azure Web Jobs can help duplicate the SharePoint Timer Service and scheduling/executing interface. This pattern works both for SharePoint Online and SharePoint On-Premises and will help better prepare you for upgrades and the cloud.

Download the OneDrive Usage Guidelines Timer Job used in this post: http://1drv.ms/1jox3Hq

Congratulations, your organization has rolled out Yammer, the best darn enterprise social platform on the planet! You probably already have some great adoption momentum, exciting new communities of knowledge, and employees/customers collaborating across organization boundaries like never before. But now it’s time to start analyzing the information contained within Yammer, identify key trends/insights, and use those trends/insights to become a more responsive organization. You might even have your boss (or their boss) on your back to start measuring ROI from the Yammer investment. Where to start…

Sure, Yammer provides high-level metrics, exports, and APIs that together, contains most of the raw data you would use to perform social mining on the enterprise. However, exports and API make most Yammer Administrators feel like the information is still locked far inside Yammer. They need simple and flexible reporting tools that are familiar and easy to use. Fortunately, the Microsoft BI stack with Microsoft Excel and Power BI are here to the rescue!

In this post, I will outline the step to take standard data exports from Yammer and convert them into detailed reporting models with rich data visualizations. Other than a few data enhancement utilities (that I'll provide for free), we'll achieve everything using Microsoft Excel and Power BI. The steps outlined in this post are also illustrated in the video below and in a related session I delivered at the 2014 SharePoint Conference titled Yammer mining - dig in and "listen" to what your big *social* data is saying.

(Please visit the site to view this video)

Collecting Raw Social Data

We will use a combination of Yammer data exports and APIs to collect the data for our reporting model. Yammer Network Administrators can collect data exports from Yammer’s Network Admin portal.  The data export interface only has a few parameters such as the export start date and checkbox options for attachments and external networks. Anything more granular will need to be achieved through post-export filtering.

1. Login to Yammer as a network administrator (only available to network admins)
2. Navigate to the Network Admin portal within Yammer
3. Select “Export Data” from the “Content and security” section of the side navigation
4. Select a start date for the export (read: all additional filter must be completed after the export)
5. Optionally include attachment and external networks