Solr Cloud Setup with Zookeeper for Sitecore

Introduction

Apache Solr includes the ability to set up a cluster of Solr servers that combines fault tolerance and high availability. Called SolrCloud, these capabilities provide distributed indexing and search capabilities, supporting the following features:

• Central configuration for the entire cluster
• Automatic load balancing and fail-over for queries
• ZooKeeper integration for cluster coordination and configuration.

Solr Installation

1. Install JREin the system
2. Add a system environment variable called JAVA_HOME that points to your java installation:
   • Variable Name: JAVA_HOME
   • Variable Value (set as necessary): C:\Program Files (x86)\Java\jre1.8.0_144

3. Download Solr and Unzip it in to C:/Solr (you can also choose the directory based on your requirement)
   • NOTE: Remove any intermediary folders to avoid a folder structure like C:/solr/solr-6.6.0/server
   • This should now be a valid path on your file system: C:/solr/server
   • Copy the conf folder from C:/solr/server/solr/configsets/sample_techproducts_configs/conf to C:/solr folder (can rename conf to config) and check schema.xml and solrconfig.xml is present

4. Repeat step 1 to 3 into one more different computer which will be required to setup two solr instances.

Generating a XML Schema for Solr

To know how to generate Schema in Sitecore Click here

This updates the schema.xml and adds all the necessary fields that Sitecore needs. When the tool has finished, copy the file to the conf directory in the solr\ conf\ folder

Installing External Zookeepers

1. Download and install 7-zip from  http://www.7-zip.org/download.html
2. Download and extract Zookeeper using 7-zip from http://zookeeper.apache.org/releases.html
3. open the conf folder from zookeeper directory and rename the “zoo_sample.cfg” to “zoo.cfg”
4. Open zoo.cfg in any text editor (notepad++) and change the below configuration based on your requirement:

• The directory where the snapshot is stored

    dataDir=/var/lib/zookeeperdata/1

• The port at which the clients will connect

    clientPort=2181

• The number of ticks that the initial synchronization phase can take

    initLimit=5

• The number of ticks that can pass between sending a request and getting an acknowledgement

    syncLimit=2

• Add server entries that list all servers that run Zookeeper instances. For example:

server.1=zookeeperServer1:2888:3888
server.2= zookeeperServer2:2889:3889
server.3= zookeeperServer3:2890:3890

• Create myid file (no extension) in the dataDir folder (/var/lib/zookeeperdata/1) and set server id number in that file. For example, on zookeeperServer1 the file should have 1. On zookeeperServer2 it should have 2 and so on.

5. Open Command prompt and navigate to “Bin” folder of zookeeper. Run the zkserver command which will start the zookeeper server

6. Repeat steps 1-5 on all machines that run Zookeeper instances.

Start Solr in Cloud Mode

1. Open command prompt in administrator mode and navigate to “bin” folder Solr instance
2. Run the following command to start the solr

solr start -cloud -p 8983 -z “zookeeperServer1:2181, zookeeperServer2:2181, zookeeperServer3:2181”

3. Navigate to browser and Verify that you can see the SOLR Dashboard at http://localhost:8983/solr/#/
4. Repeat the steps 1-3 on all machines that run solr instances.

Pushing Config file to Zookeeper-

1. Modify solrconfig.xml to replace the schemaFactory class.
   • Remove any ManagedIndexSchemaFactory definition if it exists.
   • Add a  ClassicIndexSchemaFactory definition as shown below

     <schemaFactory class="ClassicIndexSchemaFactory"/>

2. Run the following command in command prompt (Administrator mode) from any solr server:

server\scripts\cloud-scripts\zkcli.bat -zkhost "zookeeperServer1:2181, zookeeperServer2:2181, zookeeperServer3:2181" -cmd upconfig -confname solrconfig -confdir C:\solr \conf\SolrConfig

3. For any change in the schema file run the above command

                                                        Pic: Centralized Configuration files in Solr could

Creating new Collections and Alias –

For creating the collection and alias, we need to use Solr collection API Urls and change the parameters based on the project requirement. To understand more about Solr collection API please click here

Browse the Below URLs to create the collection in Solr. These URLs need to be browsed only from one solr instance which will automatically create replicas in all the servers.

“Make sure there should not be any blank space in solr API URLs while creating the collections. if there are any space then it will add %20 in URL which may create some unknown issue.”

Create collection:
http://localhost:8983/solr/admin/collections?action=CREATE&name=sitecore_core_index&numShards=2&replicationFactor=2&maxShardsPerNode=2&createNodeSet=IPAddressSolr1:8983_solr, IPAddressSolr2:8983_solr&collection.configName=solrconfig
http://localhost:8983/solr/admin/collections?action=CREATE&name=sitecore_master_index&numShards=2&replicationFactor=2&maxShardsPerNode=2&createNodeSet=IPAddressSolr1:8983_solr, IPAddressSolr2:8983_solr&collection.configName=solrconfig
http://localhost:8983/solr/admin/collections?action=CREATE&name=sitecore_web_index&numShards=2&replicationFactor=2&maxShardsPerNode=2&createNodeSet=IPAddressSolr1:8983_solr, IPAddressSolr2:8983_solr&collection.configName=solrconfig

Create Rebuild collection:
http://localhost:8983/solr/admin/collections?action=CREATE&name=sitecore_core_index _rebuild&numShards=2&replicationFactor=2&maxShardsPerNode=2&createNodeSet=IPAddressSolr1:8983_solr, IPAddressSolr2:8983_solr&collection.configName=solrconfig
http://localhost:8983/solr/admin/collections?action=CREATE&name=sitecore_master_index _rebuild&numShards=2&replicationFactor=2&maxShardsPerNode=2&createNodeSet=IPAddressSolr1:8983_solr, IPAddressSolr2:8983_solr&collection.configName=solrconfig
http://localhost:8983/solr/admin/collections?action=CREATE&name=sitecore_web_index _rebuild&numShards=2&replicationFactor=2&maxShardsPerNode=2&createNodeSet=IPAddressSolr1:8983_solr, IPAddressSolr2:8983_solr&collection.configName=solrconfig

Create Alias:

http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=sitecore_core_indexMainAlias&collections=sitecore_core_index
http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=sitecore_core_indexRebuildAlias&collections=sitecore_core_index_rebuild
http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=sitecore_master_indexMainAlias&collections=sitecore_master_index
http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=sitecore_master_indexRebuildAlias&collections=sitecore_master_index_rebuild
http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=sitecore_web_indexMainAlias&collections=sitecore_web_index
http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=sitecore_web_indexRebuildAlias&collections=sitecore_web_index_rebuild

Similarly we can use the above command to create the multiples of collection for the sitecore locale sites.

Configure the Sitecore to point to Solr as a ServiceBaseAddress and make the necessary DLLs and Config changes to support the solr. For more information click here

Important Note:

1. Always keep the Solr config file into Zookeeper server
2. Don’t stop the same solr instance which is mentioned as a ServiceBaseAddress in sitecore. However if Load balancer IP/Hostname is used as ServiceBaseAddress then it will not cause any issue.
3. Minimum 3 Zookeeper servers must be setup.It should always follow the 2*N+1 rule where N is the possible number of system failure. For Example, 3 Zookeeper server cluster can handle 1 server failure and 5 Zookeeper cluster can handle 2 failure.
4. Any custom fields should be added in to Schema.xml
5. If multiple sites are pointing to the same Solr instance then we must need to create the separate indexes and collections for the respective sites.
6. We need to disable HTTP Caching if load balancer is not used.
7. Make sure there should not be any blank space in solr API URLs while creating the collections. if there are any space then it will add %20 in URL which may create some unknown issue.