Dropping all couchbase indexes with bash script

Indexes are more important in Couchbase than usual relational databases and they have to be created and updated more often. Dropping an index can be done using a N1ql query, but when you have many indexes and you are updating them often it is taking time to update all queries for dropping indexes.  The script below is finding all of indexes in the system and will drop them in all buckets.

It is supposed that the couchbase has been installed on docker and there is no password for buckets. There are three different queries as main parts of script. The first one is finding all indexes and all buckets. The second one is removing primary index for a bucket and the last one removing indexes in a bucket one by one. The first for loop is looping over buckets and the second one over all indexes. The script is trying to remove an index from bucket regardless if it exists or not (If it doesn’t exist an error response will be the result of query).

The script can be improved but it gets the job done.

COMMAND="docker exec -i \${CONTAINER_NAME} sh -c 'cbq -q --engine='"

indexes=$(cat <<EOF | eval ${COMMAND}
  SELECT name, keyspace_id FROM system:indexes;

bucketNames=$(echo "$indexes" | grep -Po '(?<="keyspace_id": ")[^"]*' | tr ' ' '\n' | sort -u | tr '\n' ' ')
indexNames=$(echo "$indexes" | grep -Po '(?<="name": ")[^"]*')
for bucketName in $bucketNames; do 
echo "removing indexes for " $bucketName
   echo DROP PRIMARY INDEX ON ${bucketName};
   cat <<EOF | eval ${COMMAND} 
      DROP PRIMARY INDEX ON ${bucketName};  

  for indexName in $indexNames; do 

   echo "removing index with name: " $indexName
   echo DROP INDEX ${bucketName}.${indexName};
   cat <<EOF | eval ${COMMAND} 
      DROP INDEX ${bucketName}.${indexName};  


Finding all clearcase checked in files between two dates (Windows)

For finding checked in files in main stream fter special date in ClearTool command tool go to the root of the view you want to see the checked in files for and then type the command below:

find . -name '*' -ver 'version(\main\LATEST) && created_since(03-November-2016)' -print

For finding checked in files between two dates, specific period, write the command below:


find . -name '*' -ver 'version(...\main_1\LATEST) && created_since(12-October-2016) && !created_since(01-November-2016)' -print


Split strings in XSLT 1

More than 15 years after introducing XSLT 2 and more than 8 years after introducing XSLT 3, XSLT 1 is still being used frequently by many users. One of the difficulties that I experienced with XSLT 1 is simple String operations. In this post I will write the way for splitting strings in XSLT 1 which can be used in higher version as well.

The task is to split a string containing periods and for every substring if it contains whitespaces in the middle of it (not in beginning or end of substring) then surrounding it with single quotation marks. For example for the input “C.Program Files (x86).Internet Explorer “, the output should be “C.’Program Files (x86)’.’Internet Explorer’“.

<!--=============Function definitions  ========================-->
<xsl:template name="recursiveAddQuoteMark">
  <xsl:param name="string"/>
  <xsl:param name="separator" select="'.'"/>
      <xsl:when test="not(contains($string,  $separator))">
          <xsl:call-template name="addQuotMark">
              <xsl:with-param name="string" select="normalize-space($string)"/>
          <xsl:variable name="prefix">
              <xsl:call-template name="addQuotMark">
                  <xsl:with-param name="string" select="normalize-space(substring-before($string, $separator))"/>
          <xsl:variable name="suffix">
              <xsl:call-template name="recursiveAddQuoteMark">
                  <xsl:with-param name="string" select="substring-after($string, $separator)"/>
          <xsl:value-of select="concat($prefix, $separator, $suffix)"/>

<xsl:template name="addQuotMark">
  <xsl:param name="string"/>
  <xsl:param name="space" select="' '"/>
      <xsl:when test="contains($string,  $space)">
          <xsl:value-of select='concat("&apos;",$string , "&apos;")'/>
          <xsl:value-of select="$string"/>


Configuring TortoiseSVN in Intellij

Why using source control plugins in IDE instead of stand alone client application?

Usually source version control systems are offering plugins for different integrated development environment (IDE). Using these plugins will save time and will make developers more effective. They have not to leave the IDE for different version control operations like commit, checkout, update, revert and more. They can get revision history and compare different revisions or get some specific changes in one file inside the IDE.

Another benefit of using these plugins are ability to integrate them with issue tracking systems. It will be possible to create change lists based on a some specific issue and connect source control and issue tracking to each other.

Configuring TortoiseSVN in Intellij

The SVN plugin installed in Intellij is not working with the new version of SVN (while writing this post TortoiseSVN 1.8.5 and Intellij IDEA 13.5). So it is not possible to use stand alone application and the SVN plugin in Intellij on the same folder. With other words you should either use stand alone application or Intellij plugin.

To solve this you should download and install SVN command line client. In this page you can find some implementation of command line client. I chose VisualSVN and downloaded its command line client. After downloading zip file and unzipping it copy the content of ”bin” directory to   your TortoiseSVN  bin directory (probably this path: C:\Program Files\TortoiseSVN\bin).

Now you should configure Intellij so it uses the new downloaded client.  In Settings->Version Control->Subversion check ”Use command line client” and then browse to svn.exe file in the above path. See the figure below:

Use SVN command line client

Use SVN command line client


The next step will be to checkout files from your repository inside Intellij. In the Intellij menu choose VCS->Checkout from version control->subversion.

In the new window add your repository and click on checkout. In next window select the checkout destination directory. In next one you can choose some checkout options like folder and the revision. See figure below.



The next window is asking about the working copy format. You should choose the same format as your ToroiseSVN version number in this case 1.8.

Answer to the warning about creating project no and then you are finished. You can see the files in Intellij.

The next step is to introduce the folder or whole project to Intellij as version controlled by subversion.  By doing this you can see the subversion alternative while right clicking on version controlled files and folders in the Intellij Project view. For this purpose in Settings->Version Control choose that your project is version controlled by subversion or add the folder that you want to version control it. See the figure below:


After this it is possible to see the Subversion on the context menu while right clicking on a file or folder and do usual version control operations. See the figure below.


The ”changes” view in Intellij will help you to see the last changes and manage your change list. If it is not activated you can activate it by pressing Alt+9 short commando or by choosing the menu option View->Tool Windows->Changes.

With right clicking on your version controlled folder and choosing ”show in explorer” option you can open the folder in the windows explorer.  You can see that the folder even is version controlled by the ToroiseSVN stand alone application and you can use it for version control your files while having the ability to do that inside Intellij.

Upgrading the third-party libraries under Perforce source control

Some times it is easier to keep track of the third-party libraries in your project with help of the source control system. If the library has too many files it can be little complex to upgrade it. You are going to get some new files, you have to delete some old files and you should replace some existing files with the new one. Doing this while using Perforce as version control system is something that I am forgetting every time. So I am going to register it here.

For example if you have the recommended ”jre” for running your application under Perforce,  you should upgrade it as below.

  1. Open the ”jre” folder under Perforce in Windows File explorer.
  2. Right click on it and uncheck the read-only check box. Press OK.
  3. Delete all files and folders under the ”jre” folder.
  4. Copy files and folders from the new ”jre” folder to the empty ”jre” folder under Perforce.
  5. Open Perforce (P4V) with your workspace. Run refresh on the ”jre” folder.
  6. right click on the ”jre” folder in Perforce and choose ”reconcile offline work”.
  7. You should see a Windows like below. Click on reconcile and submit your changelist.


Eclipse as a common IDE for developers team

Every efficient software development team should have a common easy deployable IDE which  can be installed and reinstalled in a simple way. There are several good reasons for this. Some of them:

  1. Minimizing the configuration cost: Configuring an IDE for an average or big project is a time and resource consuming process. By creating a deployable common IDE  it is possible to minimize the cost by reinstalling the IDE when something is going wrong after a while. With other words: don’t lose time to fix it, start it over instead.
  2. The new developers will be able to gain insight into the code directly and they don’t need to face the configuration problems while preparing an IDE.
  3. Using the same version of IDE, libraries and the programming language leads to  better standards and code quality. All developers will be familiar with common problems caused by version incompatibility and can help each other. They will see the same set of warnings and errors by using the same compiler version and can have the same policy toward them.
  4. It is possible to define the same set of preferences for IDE and same code formatting’s policy.  It will help developers to keep standards high.

Rest of this post will describe a process for preparing eclipse as a common shared IDE shared by a developing team.

  1. Choose a path which all developers will install Eclipse and JDK to it. It will be better to keep JDK and Eclipse at same folder.  “C:\Programs\companyName\DevEnv” will be used as example here.
  2. Deploy the chosen JDK into above folder path. It should be a JDK that you can compile your code with it.
  3. Download the preferred  version of Eclipse. While writing this post the download site was  http://www.eclipse.org/downloads/.
  4. Unzip the downloaded file to the chosen folder path. Don’t try to start Eclipse! Starting eclipse for first time will create some local configurations that other developers don’t need it. Name the unzipped folder to NoneStartedEclipse_XX which XX is the eclipse version.
    1. It is a good practice to name the version of Eclipe and JDK in respective folder name. In the below structure XX respective YY are Eclipse and JDK versions.
         – Programs
                                               – DevEnv
                                                             – NoneStartedEclipse_XX
  5. The next step is to refer to JDK in the “eclipse.ini” file. Open the “eclipse.ini” file under  “C:\Programs\companyName\DevEnv\NoneStartedEclipse_XX” and add these two rows:
  6. Other arguments related to JVM can be defined in “eclipse.ini” file too. If developers have enough memory on their machines and you are using eclipse 64 bits version then you should specify the minimum and maximum size of the memory allocation pool (-Xms, -Xmx) to use the abilities for eclipse 64 bits version. It is good practice to set the both Xms and Xmx to the same value. In this case Java heap memory size error will be found at start of Eclipse if there is no enough memory on the machine.

    Another JVM-related argument is the required java version. You can redefine this argument to force developers to use a higher version of Java than the default version. Before you can see an example of these arguments.


8.  Next step is the configuration of the proxy settings. If your company are using a proxy server you should configure it in Eclipse for connecting to Eclipse Marketplace or downloading extra plugins. For this purpose you should create a file named “org.eclipse.core.net.prefs” under this path: “C:/Programs/companyName/DevEnv/NoneStartedEclipse_XX/configuration/.settings”. The file will be created automatically first time starting eclipse but because other information created by first start of eclipse are unnecessary for all developers you can create this file manually. You should create “.settings” folder manually too.  If you are using windows it will not allow you to change the name of the new folder too “.settings” because of the dot (“.”) in the beginning. You can copy the folder from an old eclipse version and empty it or you can use Windows commando and MD-command to create the folder. An example of the content of  “org.eclipse.core.net.prefs”:


NonproxiedHosts at above are exceptions from proxy server and will be accessed directly. A good example is issue-tracking systems like Jira.  The most important arguments in the file are the name of proxy server (proxyData/HTTP/host) and the proxy port (proxyData/HTTP/port).

9. After configuring proxy can you install needed plugins.The below steps can be taken for this purpose.

I.  Copy NonStartedEclipse_XX folder to a new folder named StartedEclipse_XX. Run “eclipse.exe” under this folder. Use an empty workspace for starting eclipse. Close eclipse and compare these two folders (NonStartedEclipse_XX and StartedEclipse_XX). You are going to find out what information are added to eclipse folder by starting eclipse for first time.

Some of this new added information are local and should not be added to a common IDE using by whole team. But removing some of these information will cause in damaging eclipse. Later you will see a list of the folders which you can remove after installing plugins to prevent other developers to get local information when they are installing the configured IDE.

II. Create a readme text file under “C:/Programs/companyName/DevEnv/” and write in it the version of eclipse and JDK. You should document every plugin you are installing and its version in this file too. Here is a example of this file’s content:

The zip-file contains:
–    Eclipse Kepler                     4.3 (64 bit)
–    JDK 1.7.0_2                        (64 bit)
–    Eclipse Standard/SDK                        
–    Eclipse SDK                                                  4.3.0.I20130605-20000
–    Memory Analyzer                               
–    Memory Analyzer (Charts)                 
–    Graphical Editing Framework GEF    
–    Graphical Editing Framework GEF SDK
–    Eclipse XSL Developer Tools                      1.3.300.v201302282018

 III.  Some plugins like version control system are chosen by your companies policy. Other can be chosen by your applications need. T.ex. if you are developing an Eclipce plugin; maybe you are using GEF for drawing, then you should install GEF plugin. The other plugins can be some useful plugins for developing like code analyze, Memory analyzer and so on.

There are different ways for installing plugins. Some of them can be installed via “Install new software” option in eclipse help menu. You can find a bunch of useful plugins under eclipse standard update site. The picture below shows installing memory analyzer under “General Purpose Tool”-category of Eclipse kepler’s  standard update site.


Some of plugins in eclipse standard update site are uncategorized and if you want to find them in the update site you should remove the check mark from the check box “Group item by category” on the above window.

Some of plugins require to be add using a new update site. T.ex. Graphical Editing Framework can be installed by adding this update site to the eclipse update sites. (Adding an update site can be done using the “Add…” button in the above window.

Some of the plugins can be installed by Eclipse Marketplace. You can find Marketplace under Eclipse help menu. In some eclipse editions Marketplace plugin are not installed by default.  Then you can install it via eclipse standard update site. The below picture shows installing findbugs plugin via Eclipse Marketplace.

installingFindBugsOne of the problem with marketplace is that that it takes time until the last version of a plugin is available via the marketplace. It can happen that a new version of a plugin is represented on the plugin’s update site but the marketplace is yet referring to an old version of plugin which maybe not be compatible with last Eclipse version. The other problem is that that some plugins cannot be installed via the marketplace (there is no Install button for them). Therefore it is a good practice to check the last version of plugin via the plugin’s  homepage before updating it via Eclipse marketplace. For doing this you can click on the “more info” link for the plugin and then choosing “learn more”. You will be redirected to the plugin’s webpage on the marketplace. As shown on the below figure.marketplaceOn the plugin’s webpage on the marketplace you can choose home icon to go to the plugin’s official webpage and check the last version of plugin there. You can check if the update site has been given in the official plugin’s webpage is the same as the update site has been given in the plugin’s homepage on the Eclipse marketplace. The update site on the plugin’s homepage on the Eclipse marketplace can be found by clicking on the green down arrow. See the below picture.


IV.  You can see a list of useful plugins below.

      • You should control if you need to install a plugin for your source control system.
      • Memory Analyzer and   Memory Analyzer (Charts)  for checking the heap. Both of them can be got from Eclipse standard update site.
      • Find bugs for  static analysis to look for bugs in Java code. Can be downloaded from eclipse marketplace.
      •     Eclipse XML Editors and Tools and    Eclipse XSL Developer Tools. Working with XML and XSL nowadays is a part of a developers everyday job. Eclipse plugins for XML and XSL are useful tools while working with xml files in Eclipse. Both can be downloaded via Eclipse standard update site.
      • Atlassian connector for Eclipse. If it has support for your issue tracking system.
      • EclEmma Java Code Coverage is a good help for writing better unit tests.
      • SWTBot for UI testing in Eclipse and SWT. If you are writing eclipse plugins.
      • Eclipse Core Tools  (http://www.eclipse.org/eclipse/platform-core/downloads.php#updates) even they are not updated for a while it is possible to find dead code in Java with its help.
      • CodePro Analytix  (https://developers.google.com/java-dev-tools/download-codepro)  Googles plugin for code analyze.

10.  After you are finished with installing plugins you can remove unnecessary local configuration From StartedEclipse folder as mentioned in 9.I in this post. You can remove the below files and folders:

    • C:\Programs\companyName\DevEnv\StartedEclipse_XX\configuration\.sttings\org.eclipse.ui.ide.prefs
    • C:\Programs\companyName\DevEnv\StartedEclipse_XX\configuration\ org.eclipse.core.runtime is used to refer to local workspace and should be removed.
    • C:\Programs\companyName\DevEnv\StartedEclipse_XX\configuration\ org.eclipse.e4.ui.css.swt.theme
    • C:\Programs\companyName\DevEnv\StartedEclipse_XX\configuration\ org.eclipse.equinox.app
    • C:\Programs\companyName\DevEnv\StartedEclipse_XX\configuration\ org.eclipse.osgi
    • C:\Programs\companyName\DevEnv\StartedEclipse_XX\configuration\ org.eclipse.ui.intro.universal

11.  You should test your development environment before the other developers are going to use it. Copy the StartedEclipse_XX to the same path and change the name to TestingEclipse_XX. Start the copied eclipse toward an empty workspace.

    • Import your projects to the new workspace.
    • If you need to run some scripts for preparing your code to be compiled (T.ex. scripts for generating code from xml files) do this. You should not have any compilation error.
    • Connect your projects to the source control system and if you are using Atlassian connector, connect it to your issue tracking system. Check if you can do usual operations as checkout, activating issues and so on.
    • Check if all of the needed GUI-features are existing. T.ex. Search, build and so on. Check the list of perspective and views too.
    • Build your application and run all types of your test. You should run all of them without problem.
    • Deploy or install the built application and check if it works.
    • If your team have other scripts and operations that are being used by developers frequently you should test them too.
    • Check the plugins you are installed to control if they are working properly.

12.   If everything was OK on the before step you can define a set of common preferences for eclipse to be shared between developers. For this purpose you can define a plugin_customazation.ini file under your eclipse installation (StartedEclipse_XX folder).  You should add a line to your “eclipse.ini” file referring to the plugin customization file. T.ex. the below line:


 Below can you see some examples of prefrences:

Activating most of Java warnings is a useful and increases code quality. You can see some java related preferences.


You  can activate  ant warnings too:



Another good preferences is showing line numbers in editors.


A way to find out preferences is to start your testing eclipse with an empty workspace. Then close the eclipse and save the workspace somewhere else. Start eclipse again and in the windows menu select the preferences and change them to values you want. Close the eclipse. Compare changed workspace with the saved workspace. Most of the time you can find the new preferences under below path under your workspace.  “metadata\.plugins\org.eclipse.core.runtime\.settings\”.

You can just copy and paste preferences to  “plugin_customazation.ini” file. As you see on the examples above you should write the name of respective plugin before the preferences key. The name of plugin is the same as the file name under “.metadata\.plugins\org.eclipse.core.runtime\.settings\” folder under workspace without the “.prefs”-extension.

13.  Rename StartedEclipse_XX to Eclipse_XX and zip it together with readme.txt and JDK  in the same file. Put the file somewhere on the companies intranet. Write an script that copies the zip file from the intranet and unzips it to “C:\Programs\companyName\DevEnv”. The developing environment is ready to be used by the team.

It is all about details

For some years ago when I was reading Andrew S. Tanenbaum’s ”Modern Operating Systems” this sentence caught my attention:  “Abstraction is the key to managing complexity”. I remember that I printed it out and pasted it behind my desktop with some other quotations! What is interesting about this sentence is the answer to another question: why are we using abstraction? Why are we using Object Oriented, Modeling and other abstraction-related techniques? The answer was obvious in Tanenbaums’s sentence: To avoid complexity! The reason for avoiding complexity: Simplicity is giving us confidence and saves our time and other resources (money, energy, etc.)! As Ray Ozzie (CTO, Microsoft Corporation) says:

“Complexity kills. It sucks the life out of developers, it makes products difficult to plan, build and test.”

I believe details are one of the major sources of complexity and complexity is an exponential function of details. More details are resulting in much more complexity! Details are everywhere in our code. Isn’t possible to say that every if-clause is indicating a try to manage a detail and every programming’s loop is a try to reduce details by applying some common rules on objects!?

To understand how abstraction is helping to manage details we can use Uncle Bob (Robert C.Martin)’s description about Dependency Inversion Principle (DIP) in clean code:

 “In essence, the DIP says that our classes should depend upon abstraction, not on concerete details.”

Details are not just a problem in programming. They are an important issue in all of science fields and even everyday life! There are many techniques and methods about how to manage them and to remember them. Maybe our preconception about human races, people coming from a city or a province and men or women are because to get rid of all individual details about every different person!

In other words categorization like abstraction is another way to manage details. An important part of many sciences is categorization. To put phenomena and objects under same category is giving us the possibility to find details about them easier and to know how to act about/on them!

I think generalization (https://en.wikipedia.org/wiki/Generalization) is another known method to minimize details. Look at the definition of generalization from Wiktionary (http://en.wiktionary.org/wiki/generalisation):

  “The formulation of general concepts from specific instances by abstracting common properties.”

Actually maybe categorization and generalization are just two abstracting methods and it is not necessary to put them alongside abstraction as detail management techniques.

Unfortunately our techniques for eliminating details are very limited and our knowledge is being more and more complex by adding more details for every second. It is not possible to generalize every time because there are always exceptions.  One of the named advantages of functional programming is reducing side effects and I think these side effects are caused by details. But the functional programming never have been popular (At least the pure one) and I think the reason is that that is impossible to remove all side effects while programming. The reality is much different from pure abstraction. I like these sentences of Robert C.Martin in Clean Code:

“One might argue that a book about code is somehow behind the times—that code is no longer the issue; that we should be concerned about models and requirements instead. Indeed some have suggested that we are close to the end of code. That soon all code will be generated instead of written. That programmers simply won’t be needed because business people will generate programs from specifications. Nonsense! We will never be rid of code, because code represents the details of the requirements. At some level those details cannot be ignored or abstracted; they have to be specified. And specifying requirements in such detail that a machine can execute them is programming. Such a specification is code.”

I deeply agree with him. Programming? It is all about details.