how to shuffle sort mapreduce

Shuffling is the process by which intermediate data from mappers are transferred to 0,1 or more reducers. Each reducer receives 1 or more keys and its associated values depending on the number of reducers (for a balanced load). Further the values associated with each key are locally sorted.

http://stackoverflow.com/questions/22141631/what-is-the-purpose-of-shuffling-and-sorting-phase-in-the-reducer-in-map-reduce

how to Capybara

• sudo apt-get update

sudo apt-get install xvfb
sudo apt-get install x11-xkb-utils
sudo apt-get install xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic
sudo Xvfb :10 -ac

java -version
sudo apt-get install default-jre


The answer is simple, use nohup command line-utility which allows to run command/process or shell script that can continue running in the background after you log out from a shell:
The syntax is as follows
nohup command-name &


testing your Ruby code with Rspec,
We’re going to look at how to can use Rspec to do TDD in Ruby.
Rspec is one of the best frameworks for testing in Ruby,


Selenium is a automated web testing framework
Selenium IDE which is just record and replay macro
 It is very limited in functionality and does not scale well if you want to deploy in multiple servers.
 Selenium WebDriver which is flexible and let you run selenium headless in servers with no display.

 If you want to create some robust process automation which needs to run 24X7 and you need reliability, then your only choice is to have Selenium in a server. But in order to run, Selenium needs to launch a browser. If there are no display to the machine, the browsers are not launched. So in order to use selenium, you need to fake a display and let selenium and the browser thinks they are running in a machine with a display.

 configuring and running selenium headless in Ubuntu using Mozilla Firefox as our primary browser


 JavaScript Testing with Selenium & Capybara-Webkit
 By default Capybara uses Rack::Test which is a headless browser emulator
 If you need to test JS as part of your integration suite, then you need to use another driver.

 you’ll need to set up xvfb in order to use either Selenium or Capybara-Webkit
 Now selunium and capybara-webkit will use xvfb when launching a browser.

 There are alternative x-servers and alternative ways to use the x-server from the specs (headless gem)
 It’s possible to use Chrome or another WebKit-based browser

 http://tutorials.jumpstartlab.com/topics/capybara/capybara_with_selenium_and_webkit.html

Behaviour Driven Development for Ruby

Behaviour Driven Development for Ruby
http://rspec.info/

Rspec

It’s pretty easy to install Rspec. Pop open that command line and run this:
gem install rspec
http://code.tutsplus.com/tutorials/ruby-for-newbies-testing-with-rspec--net-21297

RVM - Ruby enVironment Manager

• Without RVM, it’s pretty difficult to have more than one version of Ruby on your computer.

RVM stands for Ruby enVironment Manager.

http://code.tutsplus.com/tutorials/why-you-should-use-rvm--net-19529

•  Postgres uses information from the operating system to determine the language and encoding of databases

 sudo /usr/sbin/update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

 PostgreSQL is the database of choice in the Ruby community
sudo apt-get install postgresql libpq-dev

 Creating the Database Instance & Adding a User

  2  ls /usr/lib/postgresql/9.3/bin/initdb -D /usr/local/pgsql/data
    3  createuser vagrant

sudo mkdir -p /usr/local/pgsql/data
sudo chown postgres:postgres /usr/local/pgsql/data
sudo su postgres
/usr/lib/postgresql/9.3/bin/initdb -D /usr/local/pgsql/data
createuser vagrant

There are several options for managing Ruby versions, but we’ll use RVM with the standard "single user" method.
From your SSH session, we first need to install the curl tool for fetching files, then can use a script provided by the RVM team for easy setup:

sudo apt-get install curl
\curl -sSL https://get.rvm.io | bash
source /home/ubuntu/.rvm/scripts/rvm

The RVM tool has an awesome tool for installing all the various compilers and packages you’ll need to build Ruby and common libraries

Then install both Ruby 1.9.3 and 2.1
rvm install 1.9.3
rvm install 2.1

You can set either as your default Ruby. For 2.1
rvm use 2.1 --default
rvm use 1.9.3 --default

verify it:
which ruby
ruby -v

Rails’ Asset Pipeline needs a JavaScript runtime. There are several options, but let’s install NodeJS:
sudo apt-get install nodejs

http://tutorials.jumpstartlab.com/topics/vagrant_setup.html

how to Gemfile

• let’s create a project folder and throw this in a Gemfile:

source :rubygems
gem "sinatra"
gem "shotgun"
gem "cucumber"
gem "capybara"
gem "rspec"

bundle install

Optionally—if you’re using RVM—you could install these gems for this project only by running rvm --rvmrc --create 1.9.2@cucumber_example;
run this before bundle install)

http://code.tutsplus.com/tutorials/ruby-for-newbies-testing-web-apps-with-capybara-and-cucumber--net-21446

• you can just google for whatever functionality you’re looking for. Once you find the gem, install it like this

gem install GEM_NAME

 If you’d like to upgrade, run
 gem update --system
 gem -v

 sudo gem install maruku
 sudo gem install aws-s3

 There are two ways you can use gems.
 Some are stand-alone ruby programs that you’ll run (most often from the command line) to do something.
 Then, there are gems that you’ll only use from inside projects
  Ruby doesn’t load everything by default, so you can use require to load extra libraries you want to use
 
  Once you build a project, you might want to share it, or use it on another computer. However, anyone else who runs it will need to have all the right gems installed
 
  Bundler is a gem itself; you can install it by running
  sudo gem install bundler.
 
  in the root of your project, create a file named Gemfile. This will declare what gems you need for this project.
  The first line(s) of your Gemfile will tell Bundler where to get your gems. Gems live in online repositories,


http://code.tutsplus.com/tutorials/ruby-for-newbies-working-with-gems--net-18977

tools

• Gherkin is the language that Cucumber understands. It is a Business Readable, Domain Specific Language that lets you describe software’s behaviour without detailing how that behaviour is implemented.

Gherkin serves two purposes — documentation and automated tests
https://github.com/cucumber/cucumber/wiki/Gherkin

• Rspec is a story runner tool, Cucumber is a BDD feature flow explanation

and Capybara is the automation library where all the elements and
pre-defined objects are found. All together Capybara, Rspec and Cucumber
allows an user to automate the scripts and run them.
http://code.tutsplus.com/tutorials/ruby-for-newbies-testing-web-apps-with-capybara-and-cucumber--net-21446

how to create standalone maven project from command prompt

Windows
Set the environment variable
JAVA_HOME to C:\Program Files\Java\jdk1.6.0_21

Windows
Append the string ;%JAVA_HOME%\bin to the end of the system variable, Path.


Windows
Set the environment variables using system properties.
M2_HOME=C:\Program Files\Apache Software Foundation\apache-maven-2.2.1
M2=%M2_HOME%\bin
MAVEN_OPTS=-Xms256m -Xmx512m

Windows Append the string ;%M2% to the end of the system variable, Path.

C:\Users\jupiter>java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) Client VM (build 25.31-b07, mixed mode)

C:\Users\jupiter>mvn -version
Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 2015-03-13T22:10:2
7+02:00)
Maven home: C:\Downloads\apache-maven
Java version: 1.8.0_31, vendor: Oracle Corporation
Java home: C:\Program Files (x86)\Java\jdk1.8.0_31\jre
Default locale: fr_FR, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "x86", family: "dos"

mvn archetype:generate
306
com.mytutos.maven.examples
test11

must-have jenkins plugins

• Git 
• Git Client
• Git Server
• Disk Usage
• Build Flow
• Build Pipeline
• Parameterized Trigger
• Build Monitor
• Build Name Setter
• Cucumber Test Result
• Cobertura
• Checkstyle
• PMD
• Priority Sorter
• Join
• Build Trigger BadgeCopy Artifact
• Radiator View
• Dashboard View
• Warnings
• JDK Parameter
• Multiple SCMs
• JaCoCo
• Performance
• Thinbackup
• Timestamper
• Chuck Norris
• JobConfigHistory
• SafeRestart

how to install jenkins update1

previous platform

• windows 7- 64 bit
• java version "1.7.0_40"
• apache-maven 3.1.0
• eclipse Kepler 4.3.1
• m2e - Maven Integration for Eclipse-      http://www.eclipse.org/m2e/download/
• artifactory-3.0.3
• jenkins 1.535

current platform

• windows 7- 64 bit
• java version "1.8.0_45"
• Apache Maven 3.3.1
• Apache Ant(TM) version 1.9.4
• Eclipse Java EE IDE for Web Developers. Version: Luna Service Release 1a (4.4.1)
• Artifactory version 3.5.3
• Apache Tomcat/8.0.20
• Jenkins ver. 1.611

Problem: when upgraded to jdk 1.8.x it is not possible to upgrade jenkins by putting war file.Therefore platform should be built from scratch.


to verify installations on command prompt

maven
mvn --version
system variable
M2=C:\Downloads\apache-maven\bin
M2_HOME=C:\Downloads\apache-maven
MAVEN_OPTS=-Xms256m -Xmx512m


jdk
java -version
system variable
JAVA_HOME=C:\Program Files (x86)\Java\jdk1.8.0_45
path variable
%JAVA_HOME%\bin;

ant
ant -version
system variable
ANT_HOME=C:\Downloads\apache-ant
path variable
%ANT_HOME%\bin

how to install hadoop ubuntu

determine hadoop version if exists on ubuntu

how to mongodb win7

configure port
mongod --dbpath C:\Users\eyup\Downloads\mongodata --port 27017

configure log file for database
mongod --dbpath C:\Users\eyup\Downloads\mongodata --port 27017 --logpath C:\Users\eyup\Downloads\mongodata\logs\mongodb.log


run mongodb with configuration file.syntax problems with command prompt experienced

mongod -f C:\Users\eyup\Downloads\mongodata\config\mongodb.cfg

mongod -f "C:\Users\eyup\Downloads\mongodata\config\mongodb.cfg"

mongod --config "C:\Users\eyup\Downloads\mongodata\config\mongodb.cfg"


mongod --config C:\Users\eyup\Downloads\mongodata\config

mongod.exe --config C:\Users\eyup\Downloads\mongodata\config\mongodb.conf

mongod.exe -f C:\Users\eyup\Downloads\mongodata\config\mongodb.conf

how to install mongodb ubuntu

First import public key of 10gen repository in our system

 sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10

For Ubuntu:

echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list

sudo apt-get update
sudo apt-get install mongodb-org

install robomongo app

sudo apt-get install robomongo-0.8.4-i386$
tar xvfz robomongo-0.8.4-i386.tar.gz
 robomongo-0.8.4-i386/bin$ ./robomongo

ubuntu@ubuntu:~$ sudo service mongod stop
mongod stop/waiting
ubuntu@ubuntu:~$ sudo service mongod start
mongod start/running, process 19201
ubuntu@ubuntu:~$ mongo --version
MongoDB shell version: 2.6.4

how to install mongodb windows8

platform

Windows Server 2008 R2 Standart 64 bit

download

mongodb-win32-x86_64-2008plus-2.6.4

extract

run mongod

 

 

 

 

 

 

 

 

 

 

connect test

run with config file?

Run as service?

Mongodb gui for windows

http://robomongo.org/

Robomongo-0.8.4-i386

how to proxy with apache maven

Maven local settings
C:\Users\jupiter\.m2

Maven Global settings
C:\apache-maven\conf


Copy  C:\apache-maven\conf\settings into C:\Users\jupiter\.m2

Edit and add in settings.xml

  <!-- proxies
   | This is a list of proxies which can be used on this machine to connect to the network.
   | Unless otherwise specified (by system property or command-line switch), the first proxy
   | specification in this list marked as active will be used.
   |-->
  <proxies>
    <proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>x.x.x.x</host>
      <port>8080</port>
   </proxy>
  
  </proxies>

working with properties in pom.xml

 pom.xml file

<!-- organizing properties for grouping purposes -->
    <properties>
        <spring.security.version>3.2.0.RELEASE</spring.security.version>
    </properties>


<dependency>
            <groupId>org.springframework.security</groupId>
            <artifactId>spring-security-taglibs</artifactId>
            <version>${spring.security.version}</version>
        </dependency>

how to Versions Maven Plugin

Versions Maven Plugin
The Versions Plugin is used when you want to manage the versions of artifacts in a project's POM


pom.xml, right click
Run As -> Maven build...
Goals:versions:display-plugin-updates


versions:display-plugin-updates
Displays all plugins that have newer versions available.

[INFO] Require Maven 2.0.1 to use the following plugin updates:
[INFO]   maven-war-plugin .............................................. 2.0.2
[INFO]
[INFO] Require Maven 2.0.2 to use the following plugin updates:
[INFO]   maven-site-plugin ........................................ 2.0-beta-7
[INFO]
[INFO] Require Maven 2.0.6 to use the following plugin updates:
[INFO]   maven-clean-plugin .............................................. 2.5
[INFO]   maven-deploy-plugin ........................................... 2.8.1
[INFO]   maven-install-plugin .......................................... 2.5.1
[INFO]   maven-resources-plugin .......................................... 2.6
[INFO]   maven-site-plugin ............................................. 2.0.1
[INFO]   maven-surefire-plugin ......................................... 2.4.3
[INFO]   maven-war-plugin ................................................ 2.4
[INFO]
[INFO] Require Maven 2.0.9 to use the following plugin updates:
[INFO]   maven-compiler-plugin ........................................... 3.1
[INFO]   maven-surefire-plugin .......................................... 2.17
[INFO]
[INFO] Require Maven 2.1.0 to use the following plugin updates:
[INFO]   maven-site-plugin ............................................. 2.1.1
[INFO]
[INFO] Require Maven 2.2.0 to use the following plugin updates:
[INFO]   maven-site-plugin ............................................... 3.0
[INFO]
[INFO] Require Maven 2.2.1 to use the following plugin updates:
[INFO]   maven-clean-plugin ............................................ 2.6.1
[INFO]   maven-compiler-plugin ........................................... 3.3
[INFO]   maven-deploy-plugin ........................................... 2.8.2
[INFO]   maven-install-plugin .......................................... 2.5.2
[INFO]   maven-resources-plugin .......................................... 2.7
[INFO]   maven-site-plugin ............................................... 3.4
[INFO]   maven-surefire-plugin ........................................ 2.18.1
[INFO]   maven-war-plugin ................................................ 2.6
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:11 min
[INFO] Finished at: 2015-04-10T21:01:09+02:00
[INFO] Final Memory: 9M/21M
[INFO] ------------------------------------------------------------------------


versions:display-dependency-updates
scans a project's dependencies and produces a report of those dependencies which have newer versions available.

versions:use-latest-releases
searches the pom for all non-SNAPSHOT versions which have been a newer release and replaces them with the latest release version.

pom.xml, right click
Run As -> Maven build
choose run options

http://mojo.codehaus.org/versions-maven-plugin/index.html

how to update apache maven

installed maven version 3.1.0
update maven version is 3.3.1
maven.apache.org

rename apache-maven.bck
C:\Downloads\apache-maven

extract apache-maven-3.3.1-bin.zip
under
C:\Downloads\apache-maven

eclipse luna configuration
Windows > Preferences > Installations > Add
Installation home :C:\Downloads\apache-maven

Tick new maven config
Apply > OK

how to log4j configuration for spring

web.xml file

<!-- log4j config for spring -->

<context-param>

<param-name>log4jConfigLocation</param-name>

<param-value>file:/C:/Users/jupiter/workspace/webapp4_custformdbbcryptcond2/src/main/resources/welcome/log4j.properties</param-value>

</context-param>

<listener>

<listener-class>org.springframework.web.util.Log4jConfigListener</listener-class>

</listener>

log4j.properties file

# Root logger option

log4j.rootLogger=debug, file, stdout

#log4j.rootLogger=debug, stdout

# Direct log messages to a log file

log4j.appender.file=org.apache.log4j.RollingFileAppender

log4j.appender.file.File=C:/logs/log.log

log4j.appender.file.MaxFileSize=1MB

log4j.appender.file.MaxBackupIndex=1

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss} ----%5p---- # 

http://stackoverflow.com/questions/13153004/how-to-configure-logging-in-spring-mvc-web-application

 

how to define minimum maven version in pom.xml

<properties>
....

</properties>

 <prerequisites>
        <maven>2.2.1</maven>
    </prerequisites>

how to install j2ee

jdk-8u20-windows-x64

java_ee_sdk-7-windows-ml

eclipse-jee-luna-R-win32-x86_64

run command prompt

do not forget parantheses for jdk installation path

java_ee_sdk-7-windows-ml -j "C:\Program Files\Java\jdk1.8.0_20"

how to maven configuration eclipse luna

This tutorial aims to configure eclipse luna with local maven repository.This configuration assumes no proxy setup for internet connection.

Window -> Preferences -> Maven -> Archetypes -> Add Local Catalog

Locate your local catalog file

how to restore artifactory

Three directories, extract backups

C:\Users\jupiter\.artifactory
C:\Users\jupiter\.eclipse

(local repository)
C:\Users\jupiter\.m2 


Login artifactory
Admin -> Import & Export -> Repositories ->Import Repository from Path

 run artifactory
check on the numbers of artifacts

how to install artifactory

PLATFORM (previous)
windows 7 family edition 64-bit
jdk 1.7.0
tomcat 7.0.26 (64-bit edition)

I want to install new version of tomcat, jdk and artifactory.My aim is to put artifactory backups back on this new platform.This tutorial is about running artifactory war file on tomcat.

PLATFORM (new)
windows 7 family edition 64-bit
jdk1.8.0_31
apache-tomcat-8.0.20-windows-x86 


C:\Users\jupiter>java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) Client VM (build 25.31-b07, mixed mode)

extract apache-tomcat-8.0.20-windows-x86.zip
tomcat folder should look like this
C:\Downloads\apache-tomcat

"Server Status" and "Manager App" interfaces require manager-gui role and "Host Manager" requires admin-gui role.

add these lines and restart tomcat if it works.

C:\Downloads\apache-tomcat\conf \tomcat-users.xml


<role rolename="manager-gui,admin-gui"/>
<user username="tomcat" password="tomcat" roles="manager-gui,admin-gui"/>


configure tomcat port.
C:\Downloads\apache-tomcat\conf\server.xml

 <Connector port="5555" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />


This file sets java_home manually and runs tomcat on the folder mentioned.
contents of "tomcat start.cmd" file.

@echo off
SET JAVA_HOME=C:\Program Files (x86)\Java\jdk1.8.0_31
SET CATALINA_HOME=C:\Downloads\apache-tomcat
C:\Downloads\apache-tomcat\bin\startup.bat

run "tomcat start.cmd" on folder
C:\Downloads\ciroot

test tomcat whether it runs
tomcat runs on http://localhost:8080/


extract artifactory-3.5.3.zip
copy C:\Downloads\artifactory-3.5.3\webapps\artifactory.war file
into C:\Downloads\apache-tomcat\webapps folder.

restart tomcat.

test if artifactory runs
http://localhost:5555artifactory

artifactory first time login.
user/password  admin/admin

inittab in ubuntu

check on
/etc/init/rc-sysinit.conf

how to configure jdk (eclipse luna)

Window > Preferences > Java > Installed JREs > Standart VM > JRE home

choose
C:\Program Files (x86)\Java\jdk1.8.0_31

C:\Users\jupiter>java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) Client VM (build 25.31-b07, mixed mode)