Design considerations in building a Large lists/Document Library in SharePoint 2010

Here are some of the check points that you would want to consider while designing large lists/libraries in your application. You can go through this nice article (http://download.microsoft.com/download/7/F/0/7F069D0B-B6BD-4692-868B-E8555BB72445/DesigningLargeListsMaximizingListPerformance.docx) if you would like to know more details and analysis about some of these items.

1.      Create indexes. Improves search and reads. But, add and update slows down. Ensure an Index is created for every view filter.

2.       Reduce the number of lookups as much as possible. De-normalize where ever it is possible. No view should have more than configurable 8 lookups.

3.       Reduce the lookups which bring in more than 2000 records.

4.       Create views which has filtering on the indexed column. This way you are keeping your data well below the threshold limit on views of 5000

5.       Reduce the number of the columns in a large list. SharePoint stores in multiple rows when multiple columns (see table 1 below) of the same data type is used. Also called as Row Wrapping.

6.       If you have multiple content types defined in a large lists then separate them into multiple lists. This way you are reducing the list size and potential performance problems.

7.       List thresholds also impact document libraries. So ensure that you put the documents in several folders. The threshold applies only on the containers. Folder is a container that is valid even in a list as well.

8.       Use SharePoint Search for searching. You can scope the results to a particular List/Library. However, this has a limitation that the results will be limited to the last crawl only.

a.       You can use Search Query Object model. There are tools to aid in building the query. (http://sharepointsearchserv.codeplex.com/ and  http://mosssearchcoder.codeplex.com/)

9.       You can use Developer dashboards to view load times and queries that are affecting the Views.

10.   Use Content Iterator Developer API to work against large lists in an iterative manner whiling working on smaller sets in each iteration.

11.   Try and use Caching (Output, Object and BLOB cache) at various levels to prevent hitting the SQL Server.

12.   Try setting Throttling limits as a last resort. You can set them in General Settings on a web application.

13.   You may also want to set the Allow Object Model override, if you wish to capture more records programmatically.

a.       Minimize the number of unique permissions you give (i.e. list item permissions degrades the performances on large lists). Maximum of 50K unique permissions is allowed (Recommended is to keep it below 5K). Even, permission breaking at Folder is considered as one unique permission. Each time permissions inheritance is broken a new scope ID is created. Each time you query on a view, you join against the scopes table and when a query is performed, each unique access control list (ACL) must be parsed and processed.

14.   For designing Large Document libraries, you may want to consider the Content DB Size as well depending on the number of documents, versions and average size of each document.

a.       You may also want to consider using Remote BLOB storage for these documents.

b.      You may also want to consider third party solutions for archiving (ex: Symantec Enterprise Vault) while maintaining the search ability to these documents.

c.       Think about organizing the files into folders and provide tagging feature so that metadata based navigation & Search can be used to reach to the document in a consistent manner. This also helps in configuring Retention and Permissions for administration.

d.      Think about using Content Organizer feature that can automatically move files based on the Content Types and Metadata without any intervention by users. Create new folders once the limit is reached.

e.      Open with Explorer will not work if the number of documents in a folder is more than the limit.

15.   Minimize the number of Multi valued columns. This decreases the performance.

16.   Queries on the Managed metadata columns are faster than the Choice type fields. Managed metadata columns support multi value as well. 

17.    For all you may know, storing the data in a custom database would make more sense in certain scenarios. You could consider using BCS to perform CRUD operations on this data.

       Table 1

Column Type	Number of Columns per Table Row
Single line of text	64
Choice and Multiple lines of text	32
Date and Time	8
Yes/No	16
Number and Currency	12
Calculated	8
Int, Single Value Lookup, People and Group, Managed Metadata	16
Unique Identifier	1

Before you implement a large list consider the business case and requirements. Requirements such as service level agreement (SLA), time to backup and restore, size of content, amount of content (number of items), and access times are all important to consider. Depending on the size and demand of the application, you must make important choices at multiple levels, including hardware, content storage, and SharePoint information architecture. A large application with millions of items and hundreds of concurrent users might require standalone hardware for the specific project, although a document repository with tens of concurrent users and tens of thousands of documents may work fine with existing shared hardware and a single document library in an existing site.

The end results of planning should be a list of column types (names, data type, and usage), indexes, folder structure, usage of pages and links for navigation, planned structure of permissions, estimated number of items and total data size. Details should also include information about the types of queries that will be performed and how data from the list will be accessed, created, and updated. 

Must have tools for SharePoint 2010 developer

Developer Environment Setup

• Install Windows Server 2008 R2
• Install SQL Server 2008 R2 Dev
• Install Visual Studio 2010 Professional - Customize all remove all the unnecessary ones like the SQL Express.
• Install SharePoint 2010 Enterprise.
• (Optional) Install the latest Cumulative Update. June 2012 as of now.  

Note: Ensure to run the windows updates after every major installation.



Install all the required VS extensions:

• NuGet - http://visualstudiogallery.msdn.microsoft.com/27077b70-9dad-4c64-adcf-c7cf6bc9970c?SRC=Home
• CKS Development Tools (Server) - http://visualstudiogallery.msdn.microsoft.com/ee876627-962c-4c35-a4a6-a4d89bfb61dc
• SharePoint Power Tools - http://visualstudiogallery.msdn.microsoft.com/8e602a8c-6714-4549-9e95-f3700344b0d9
• JSLint - http://visualstudiogallery.msdn.microsoft.com/961e6734-cd3a-4afb-a121-4541742b912e . Configure Tools > JSLint options in VS.
• (Optional) TFS Power Tools - http://visualstudiogallery.msdn.microsoft.com/c255a1e4-04ba-4f68-8f4e-cd473d6b971f

Third party tools:

• Install ULS Viewer - http://code.msdn.microsoft.com/ULSViewer/
• Install SharePoint Manager 2010 - http://spm.codeplex.com/
• Install SharePoint Designer 2010 - http://www.microsoft.com/en-us/download/details.aspx?id=24309
• Install Beyond Compare - http://www.scootersoftware.com/ . Configure your VS 2010 - TFS. Follow the article http://www.scootersoftware.com/support.php?zz=kb_vcs#tfs
• SPDisposeCheck - http://archive.msdn.microsoft.com/SPDisposeCheck
• Install CAML Query Builder -  http://www.u2u.be/res/tools/camlquerybuilder.aspx (2007 version works well for 2010 for most of the basic scenarios)
• Install FxCop 10.0 - http://www.microsoft.com/en-us/download/details.aspx?id=6544
• Install Search Everything tool for quick search of your files by name. http://www.voidtools.com/

Introduction to SharePoint 2013 Apps

What is an app for SharePoint?
In simple description, an app for SharePoint is a web application that is registered with SharePoint using an app manifest. The app manifest defines the properties of the app like where is the app hosted, what is the start location of the app, what permissions/scopes are requested etc. The following code is a very simple app manifest that basically tells SharePoint to register the app and invoke a remote page when the app starts.

<?xml version="1.0" encoding="utf-8" ?>
<App xmlns="http://schemas.microsoft.com/sharepoint/2012/app/manifest (http://schemas.microsoft.com/sharepoint/2012/app/manifest)"
    ProductID="{90A1A0AE-4B83-4DD6-874A-76C1AA23C1FD}"
    Version="1.0.0.0"
    SharePointMinVersion="15.0.0.0"
    Name="BookStore">
  <Properties>
    <Title>Share Books</Title>
       <StartPage>https://localhost:8082/bookshome.aspx/?{StandardTokens}</StartPage>
  </Properties>
  <AppPrincipal>
    <RemoteWebApplication ClientId="A01DA985-DFB9-42F9-B20D-013FA60C587B" />
  </AppPrincipal>
</App>

What are the different modes of Application deployments? 
SharePoint-hosted apps: Here the app is hosted in SharePoint itself. The only way it can communicate with the SharePoint is through CSOM/REST (Javascript). No server side code is allowed. When you install this kind of app, SharePoint creates a new website called the app web. 

1. Provider-hosted apps: Here the app is hosted by a provider. Optionally, it can provision a new website called the app web in SharePoint. Provider is responsibile for the isolation of the tenants. CSOM/REST can be used to communicate with the SharePoint apps. 
2. Auto-hosted apps: Here the Web and SQL Azure components required for the app are provisioned by Windows Azure. Optionally, it can provision a new website called the app web in SharePoint. Here multitanancy is supported by Windows Azure itself. CSOM/REST can be used to communicate with the SharePoint apps. 

Authentication options in apps for SharePoint
When the app is running within the SharePoint, then the app is intrisically authenticated. For the apps on the cloud, to integrate with SharePoint there are two ways:
* Using client-side code along with the cross-domain library.
* Using server-side code along with OAuth.

App permissions in SharePoint 2013
An app for SharePoint requests the permissions that it needs during installation from the user who is installing it. The developer of an app must request, through the app manifest file, the permissions that the particular app needs to be able to run. The user who installs should have the required permissions first, then app must either be granted all the requested permissions or should not be granted any permissions.


Application Permissions
* The permission requests specify both the rights that an app needs and the scope at which it needs the rights. These permissions are requested as part of the app definition through the node .
* Permission request scopes indicate the location in the SharePoint hierarchy where a permission request applies.
* An app for SharePoint has its own identity and is associated with a security principal, called an app principal. Like users and groups, an app principal has certain permissions and rights. The app principal has full control rights to the app web so it only needs to request permissions to SharePoint resources in the host web or other locations outside the app web.

If an app is granted permission to one of the scopes, the permission applies to all children of the scope. Some of the basic scopes are as follows:
Scope URI  :  Description
* http://sharepoint/content/sitecollection  :  The permission request scope URI to the site collection where the app is installed. 
* http://sharepoint/content/sitecollection/web  :  The permission request scope URI to the website where the app is installed. 
* http://sharepoint/content/sitecollection/web/list  :  The permission request scope URI to the list where the app is installed. 
* http://sharepoint/content/tenant  :  The permission request scope URI to the tenancy where the app is installed.

For each scope, an app can have the following rights:
Read, Write, Manage, FullControl
These corresponds to the default permission set of SharePoint Reader, Contribute, Designer & Full Control. Unlike SharePoint, these app permissions cannot be customized.

You could filter the lists on which the permission set is valid, by using the BaseTemplateId property.

<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web/list" Right="Write">
    <Property Name="BaseTemplateId" Value="101"/>
</AppPermissionRequest>

In SharePoint 2013, end users have four different opportunities to manage app permissions:
* During app installation
* By using explicit permissions management
* By using an end-user consent UI
* During app removal  

App authorization policies
This defines the target (User/App) on which the permission set is validated on for the requested operation. 
1. User and app policy - Content DB authorization checks are made on both User and the App for sufficient permissions to perform the operation. 
2. App-only policy - Only the App is validated for sufficient permissions.  
3. User-only policy - Only the User is validated for sufficient permissions.  


References: 
SharePoint 2013 Apps Overview: http://msdn.microsoft.com/en-us/library/sharepoint/fp179930(v=office.15)
Developing Apps for SharePoint 2013 : http://msdn.microsoft.com/en-us/library/office/apps/jj220038(v=office.15).aspx

Setting Site Permissions during the Site Provisioning

Setting custom permissions during site provisioning can be a confusing affair. With some investigation, I was able to get the concepts clear. 

Below is the design we used for Groups and permissions in one of our projects during the site provisioning. Hope it helps you in your projects as well.

Note: All groups are created at the Site collection level. They will be visible at all the webs that will be created under it.

SiteCollection Feature Activation

1. Create custom RoleDefinitions (Permission Set) {ViewEditRD, ViewAddEditRD} at the Site Collections. This will get inherited at all the webs. Don’t get confused with the Permission breaking at the Web level.

2. Create an Admin group (SiteAdmins) at the SiteGroups collection. This will be used to provide admin activities on the Sub webs or Lists that will undergo inheritance breaking.

New Project Provisioning

1. Create the Project with Inheritance of the Permission broken.

2. Create the Custom Groups {PMGroup, TeamMebersGroup} for every Web created.

3. Associate the Custom Groups created at the Site Collection with Read permissions.

4. Associate the Custom Groups created at the Web Level with appropriate RoleDefinitions {ViewEditRD/ViewAddEditRD}. SPRoleDefinition defines the set of permissions permitted on SharePoint objects. 

5. Associate the SiteAdmins group at the Administrator role at the Web level.

6. Break the Permissions at the List level and apply the Required RoleAssignments based on the RoleDefinition and Groups. SPRoleAssignment class is used to bind together a Group and RoleDefinition with a SharePoint Object (web, list or a document library). 

Working with SharePoint Online using NodeJS

SharePoint Online (SPO) data can be accessed using Server Object Model (Restricted API Set as it is for Sandboxed solution), but, only when the application is hosted within the SPO site collection. For rest of the scenarios you will need to use Client Side Object Model (CSOM). 


There are three approaches to accessing SharePoint/SPO data from client applications: 
CSOM (JavaScript, Silverlight, .Net Managed)  
Web Services
REST Interfaces



Accessing remote SPO data from hosted applications using Silverlight/JavaScript is not possible as browser clearly rejects such requests treating them as cross-site scripting (also, known as XSS Attack). The only way to access such data is to either use desktop based applications or write server side code on your web server using .Net Managed, Web Services or REST interfaces. Before you can access the data you need to Remotely get authenticated to SPO site. 

The flow to remotely get authenticated is to, 

1. Send using SAML 1.1, the credentials (over an https) to the SPO STS endpoint https://login.microsoftonline.com/extSTS.srf. If the request is successful, then STS returns a Token.
2. Pass this token to SPO and fetch two cookies (called FedAuth and rtFa). 
3. You need to pass these two cookies on every request made to SPO.


Note: FedAuth cookies are written with an HTTPOnly flag. This means that client side browsers are instructed to not allow any scripts to read cookies and thereby preventing a cross-site scripting (XSS) attack.


Demonstrating with an example, I will use REST interfaces and for the sake of learning some new platform, I have chosen NodeJS (A platform built on Chrome's JavaScript runtime for easily building fast, scalable network/web applications.) as the server technology.


Steps: 
1. Create an Office 365 free trial using http://www.microsoft.com/en-us/office365/free-office365-trial.aspx?WT.z_O365_ca=Try_free-office365-trial_header_en-us . Say, your domain is mydomain.sharepoint.com
2. Navigate to the newly created SPO site http://mydomain.sharepoint.com (http is for P-plans and https is for E-plans).
3. Create a Custom List 'Contacts' and add a couple of list items.
4. Install NodeJS from http://nodejs.org/#download
5. Copy the cotents of the https://github.com/lstak/node-sharepoint.git into a folder.
6. Through a console, Navigate to the folder and execute the command $npm install sharepoint If that doesn’t work then try the following $npm install sharepoint@0.0.5
7. Create a new file under the same directory and name it as spo.js and put the code snippet provided below.
8. Execute the command $node spo.js
9. While the server is waiting on for requests make a request to http://127.0.0.1:1337/


Output: You will see 'Hello World' on the browser, but behind the scenes the server has made a request to your SPO team site and fetched the data. Verify this by checking the  logged information on the Server console. 

 var http = require('http');
http.createServer(function (req, res) {
 res.writeHead(200, {'Content-Type': 'text/plain'});

 var SP = require('./sharepoint');

 // use the domain name which you have access to.
 var spo = new SP.RestService("http://mydomain.sharepoint.com/teamsite/");
 spo.signin('prashanthkn@mydomain.onmicrosoft.com', 'password', function(err, data) {
  // check for errors during login, e.g. invalid credentials and handle accordingly. 
  if (err) {
  console.log("Error found: ", err);
  return;
  }

  // start to do authenticated requests here....
  var oList = spo.list('Contacts');

  oList.get(function(err, data) {
   data.results.forEach(function(item) { 
   console.log(item.Id); 
   console.log(item.Title); 
   });
  });  
 });

 res.end('Hello World\n');
}).listen(1337, '127.0.0.1');

console.log('Server running at http://127.0.0.1:1337/');

Once you have the data, you can decide how you would want to share it with your clients.

Go through the NPM (Node Package Manager) documentation @ http://npmjs.org/doc/install.html
Good read about SPO Authentication under the hood. http://www.wictorwilen.se/Post/How-to-do-active-authentication-to-Office-365-and-SharePoint-Online.aspx


Hopefully, this gets you started on evaluating NodeJS and SharePoint Online integration.

Some best practices of developing SharePoint 2010 applications

Came across some nice videos on best practices from Ted Pattison. Here is the gist of what he suggests.

Development

• Use Site template instead of the site definition. Hell during the upgrades from one version of SP to another.
• Deploy the site templates as sandboxed solution in the staging/test environment to check out the bottlenecks and depending on the needs deploy them in production as sandboxed or farm based.
• Use Feature Stapling. element to staple a feature to a site definition.
• Version Features and anticipate for feature upgrades.
• Hide the Features that have been stapled as hidden. Use the Visual studio properties windows to hide them. This ensures that they are not seen by the site admins.
• Keep in mind about various cycles of Activation and Deactivation. Deactivation leaves behind the list instances and next time activations will break during the deployment. Ensure to write appropriate code to either delete or backup previous versions before deleting them.

Testing and Deployment.

• Use the SPDisposeCheck utility sparingly.
• Ensure to use the keyword using() especially to SPSite object to let the .net framework take care of the disposing. Do not dispose the SPSite object created by others that is received through the event receivers or anything like that. Ex: properties.parent.site
• Test your environment without any SDKs installed. Probably a VM or something would be good. Keep your QA environment with at least two boxes. One for the Web/App and another for the DB Server.
• Test your QA machines on debug mode to catch any unknown errors.
• If you are deploying a farm based solutions prefer to use a script like PowerShell to install them.
• Updates to web.config needs to be automated. For safe control entries you can add data to the manifest file. However for other entries use the SP Object model. See some methods of updating web.config http://msdn.microsoft.com/en-us/library/ms439965.aspx  / http://msdn.microsoft.com/en-us/library/bb861909.aspx

 For more information check out the videos @ http://msdn.microsoft.com/en-us/sp2010devtrainingcourse_bestpractices_unit

Extending the debug session in SharePoint 2010

Error Message
The Web server process that was being debugged has been terminated by Internet Information Services (IIS). This can be avoided by configuring Application Pool ping settings in IIS. See help for further details.


Resolution
By default, the IIS application pool waits 90 seconds for an application to respond before it closes the application. This process is known as "pinging" the application. To resolve this issue, you can either increase the wait time or disable application pinging entirely.


To access the IIS app pool settings

1. Open IIS Manager.
2. In the Connections pane, expand the SharePoint server node and click Application Pools.
3. On the Application Pools page, select the SharePoint application pool (typically "SharePoint - 80") and then, in the Actions pane, click Advanced Settings.
4. To increase the wait time before IIS timeout, change the value of Ping Maximum Response Time (seconds) to a value larger than 90 seconds.
5. To disable IIS pinging, set Ping Enabled to False.

for more information on the troubleshooting in SharePoint 2010 
http://msdn.microsoft.com/en-us/library/ee231594.aspx

SharePoint 2010 Scrollbar is not showing up

There are cases where your SharePoint pages renders without a vertical scroll bar or it is disabled. However, if you hold your left button down and drag downwards you will be able to see the rest of the page.
The chances of this scenario showing up are higher on non-IE browsers. Mostly occurs because of your custom master page. 


We have tried different articles out there on the internet. Here is a simple hack that did the trick. 


//Just put the below snippet in the footer control so that it will be called on all the pages.

 function adjustDivSize() {  
     var winHeight = $(window).height();  
     $('#s4-workspace').css("height", winHeight);  
   }  
 $(document).ready(adjustDivSize);  
 $(window).resize(adjustDivSize);  

Cache implementation for your Web & WCF resource

There are different ways of implementing the caching.

1.       At the server, where you store data in-memory using the ASP.Net Cache / Application classes.

2.       In the client (Browser). This definitely gives more advantage in terms of performance.

This article describes in detail about the second type of caching.

What are the resources that can be cached?

Static resources like html, scripts - js, style sheets – css, images – gif, jpg etc
Ex: http://www.domain.com/images/banner.jpg

Dynamic resources which are REST based urls  - if you think that these resources don’t change their data very frequently then they can be a good candidate for caching.

Ex:

http://www.domain.com/service/mobile.svc/partnerslist

http://www.domain.com/service/mobile.svc/partnersdetails/1

                                                               

What do we intend to save by caching on the browser?

Loading the server to fetch the data and pass it across

Even making an http request to the server. There are other ways to save the number of requests to server by using minification process. Where we can merge several js/css files into single file. You can read my previous articles to know more about it.

Implementing Caching

This is simple by adding Expires Header (when you know exactly when to expire) to the IIS. http://technet.microsoft.com/en-us/library/cc770661(WS.10).aspx

You can also use the Max Age header when you are not sure of the number days you need to cache the resource. The above link describes for both.

Challenges

Finding the exact Expires/MaxAge becomes hard as we will not be able to predict our bug fix cycles due to hot fixes etc.. specially in case of smaller projects.

Solution

Two common solutions both deal with manipulating the URL:

1.       Manipulate the calls to the resource by adding a query string. This requires to do a find a replace to all the locations where the resource is being referenced. Ex: http://www.domain.com/app/images/banner.jpg?rev=2345 . This becomes a painful process to manipulate it every time a resource is changed. There are very high chances that we may miss out modifying a revision query string. This also has some minor drawbacks that some proxy servers don’t cache urls which contain query string.

For REST based WCF Service, add this piece of code before the response.

OutgoingWebResponseContext ctx = WebOperationContext.Current.OutgoingResponse;

ctx.Headers.Add(HttpResponseHeader.CacheControl, String.Format("public, max-age={0}", maxAge)); //in seconds.

2.       The second more robust solution “Fingerprinting” is to inject some kind of hash into the url. Ex: http://www.domain.com/CBUSTZqTMzNV8qFU/0.jpg

Here the part of the url before the file name is the fingerprint or a hash generated based on the contents of the resource being accessed. This requires that we use some mechanism like a URL Rewrite module to do a Rewrite of requests. Ex: Search for a pattern (.*)/CBUST[A-F0-9]{32}(/.*) and do a rewrite to {R:1}{R:2} For more information on URL Rewrite http://www.iis.net/download/urlrewrite

In your aspx files, you will reference your resources like <script type="text/javascript" src="<%=Utilities.CacheBusterUrl("/_layouts/scripts/jquery-1.4.3.min.js"%>">script>

This will end up creating a url something like http://www.domain.com/_layouts/scripts/CBUSTZqTMzNV8qFU/jquery-1.4.3.min.js

Add this piece of code in your Page_Load if it is a page.

if (!this.Page.IsPostBack)

{

this.DataBind();

}

If it is a user/web control then ensure that the page that is adding this user/web control has the above piece of code.

The Fingerprinting method has its own limitation that any resource referenced in .css file or being rendered from SharePoint library cannot be cached like this.

Implementation of the CacheBuster utility is as follows.

    public static class Utilities

    {

        private static Hashtable Md5Map = Hashtable.Synchronized(new Hashtable());

        public static string CacheBusterUrl(string sourceUrl)

        {

            try

            {

                string filesystemPath = HttpContext.Current.Server.MapPath(sourceUrl);

                string fileMD5;

                lock (Md5Map.SyncRoot)

                {

                    if (Md5Map.ContainsKey(filesystemPath))

                    {

                        fileMD5 = Md5Map[filesystemPath].ToString();

                    }

                    else

                    {

                        fileMD5 = CreateHash(filesystemPath);

                        Md5Map.Add(filesystemPath, fileMD5);

                    }

                }

                return BuildCacheBusterURL(sourceUrl, fileMD5);

            }

            catch (Exception ex)

            {

                return sourceUrl;

            }

        }

    }

Security Warning

Be aware that if your Web Application stores cookies then caching resources at the proxy servers can be a security threat. So make sure to use https whenever you decide to store cookies.

Not able to debug SharePoint solution

There are cases when you would want to debug your SharePoint solution, but in vain, that debug pointer never gets caught. Reason, your symbols are not matching the binaries it was built with. The following are some of the common tasks we try during these times.

• Delete the entire bin & obj folders.
• Ensure that you are on Debug mode.
• Do a Clean & Rebuild of the solution.
• Put an System.Diagnostics.Debug.Assert(false); statement. During runtime, the process will throw an exception and then attach the w3wp process. Sometimes it is confusing which w3wp process to attach because there will always be more than one.

Inspite of the above, I have seen it doesn’t attach to the debugger some times.

What has worked consistently, until now is you can force the symbols (pdbs) to be loaded from the bin directory.

Tools > Options > Debugging > Symbols > Click on the Folder icon and specify the path to your bin/pdb directory.

If your build machine and the server machines are different then you can use this to cache the symbol files in the server or debugging machine.

The above steps would hold good for any .net projects. Hope this setting helps you in debugging next time.