Saturday, June 30, 2012

Install hadoop on ubuntu 10.04 LTS

Install hadoop on ubuntu 10.04 LTS server installion.
Install ubuntu l0.04 LTS,
Update,upgrade and install ssh
#apt-get update
#apt-get upgrade
#apt-get install ssh

Install sun-6-java jdk
# See https://launchpad.net/~ferramroberto/

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:ferramroberto/java

# Update the source list
$ sudo apt-get update

# Install Sun Java 6 JDK
$ sudo apt-get install sun-java6-jdk

$ sudo update-java-alternatives -s java-6-sun

Create new user hadoop and group
$ sudo addgroup hadoop 
$ sudo adduser --ingroup hadoop hduser

Generate ssh key to auto login to manager
#su - hduser
#ssh-keygen -t rsa -P ""
#cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys 

Disable ipv6 and reboot 
#vim /etc/modprobe.d/blacklist
add new line in it
blacklist ipv6  
Or
#disable ipv6 
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
You have to reboot your machine in order to make the changes take effect.
You can check whether IPv6 is enabled on your machine
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
 0 means IPv6 is enabled, a value of 1 means 
disabled (that’s what we want).

Hadoop Distributed File System (HDFS)
Image copy by javacodegeeks.com
















Download hadoop from mirror site
$ cd /usr/local 
$ sudo tar xzf hadoop-1.0.3.tar.gz 
$ sudo mv hadoop-1.0.3 hadoop 
$ sudo chown -R hduser:hadoop hadoop

Confirm java home folder
#ls -l `whereis javac`

Modify hadoop home folder hadoop-env.sh 
#vim hadoop/conf/hadoop-env.sh
uncomment export JAVA_HOME and modify it
export JAVA_HOME = /usr/lib/jvm/java-6-openjdk/

Config hadoop config file 
#vim hadoop/conf/core-site.xml
add these line into it
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/hadoop/tmp/dir/hadoop-hadoop</value>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost</value>
</property>

Config hadoop file with hdfs-site.xml
#vim hadoop/conf/hdfs-site.xml
add these line into it
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

Config hadoop file with mapred-site.xml
#vim hadoop/conf/mapred-site.xml
add these line into it
<property>
    <name>mapred.job.tracker</name>
    <value>localhost:54311</value>
</property>

Formating the namenode
#hadoop/bin/hadoop namenode -format

Start cluster
#hadoop/bin/start-all.sh

Check hadoop process
#hadoop/bin/jps

Use netstat to check all service running status
#netstat -plten | grep java

Stop cluster
#hadoop/bin/stop-all.sh

Start cluster
#hadoop/bin/start-all.sh

Mkdir a folder for gutenberg and touch three files with contents
#mkdir /tmp/gutenberg
#cd /tmp/gutenberg
#vim 1.txt
#vim 2.txt
#vim 3.txt

Use hadoop fs copyFromLocal copy files to hdfs folder
#hadoop/bin/hadoop fs -copyFromLocal /tmp/gutenberg gutenber

Check hdfs folder content
#hadoop/bin/hadoop fs -ls 
#hadoop/bin/hadoop fs -ls gutenberg

Use java wordcount to calculate the words number
#hadoop/bin/hadoop jar hadoop-mapred-examples-0.21.0.jar 
wordcount gutenberg gutenberg-output


Hadoop Web Interfaces

    http://localhost:50070/ – web UI of the NameNode daemon
    http://localhost:50030/ – web UI of the JobTracker daemon
    http://localhost:50060/ – web UI of the TaskTracker daemon



Vishal Vyas

Thursday, June 21, 2012

Apache Hadoop

What Is Apache Hadoop?

The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.







Hadoop is a framework written in Java for running applications on large clusters of commodity hardware and incorporates features similar to those of the Google File System and of MapReduce. HDFS is a highly fault-tolerant distributed file system and like Hadoop designed to be deployed on low-cost hardware. It provides high throughput access to application data and is suitable for applications that have large data sets.  

 The Hadoop Distributed File System: Architecture and Design:
The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. HDFS relaxes a few POSIX requirements to enable streaming access to file system data. HDFS was originally built as infrastructure for the Apache Nutch web search engine project. HDFS is part of the Apache Hadoop project, which is part of the Apache Lucene project. 










Vishal vyas 

Wednesday, June 6, 2012

Today is Internet Expansion Day


Today Internet technology will take one big step forward by moving to a new web protocol, i.e. IPv6 that will not only be a safe approach to interhttp://www.blogger.com/blogger.g?blogID=6417358664313700940#editor/target=post;postID=2457728847953887105net but also help to develop trillions of new IP addresses.
“The Internet we’ve relied on so far has space for 2^32 addresses – about 4.3 billion,” Google says in an IPv6 Day blog post signed by its internet evangelist, Vint Cerf. “The new, larger IPv6 expands the limit to 2^128 addresses – more than 340 trillion, trillion, trillion!”

It has also been said that this development will help to make internet secure by protecting us from internet crime, warfare and terrorism. There is a security code referred to as IPSEC that would do away with anonymity on the web as reported by Prof. Alan Woodward, a professor at the University of Surrey in the UK. This code would help to catch cyber criminals.

“If IPv6 isn’t implemented you’d soon have to share a single address with multiple people or even a whole neighborhood. This tangled, constrained Internet would be unsafe and unsustainable,” added Cerf.
“Complete transition will take time. Some users may need to upgrade their home routers or possibly download updated operating system software to enable IPv6 in parallel with IPv4. If you’re interested in when you’ll get IPv6 connectivity (if you don’t have it already), we encourage you to reach out to your ISP and ask.”

Via:- Google Official Blog 

Thanks,
Vishal Vyas

How To Install CVS Server on Redhat Linux










yum install cvs 

Create a CVS user and group.
useradd cvs
passwd cvs
Create, setup and own cvsroot directory
mkdir /usr/local/cvsroot
cvs -d /usr/local/cvsroot init
chown cvs:cvs /usr/local/cvsroot
Set default CVSROOT
vi /etc/profile
Append following line:
export CVSROOT=/usr/local/cvsroot
And load it into the shell
./etc/profile
Add /etc/xinetd.d/cvspserver and make it look as follows:
vi /etc/xinetd.d/cvspserver
 
service cvspserver
{
        disable = no
        socket_type = stream
        wait = no
        user = cvs
        group = cvs
        log_type = FILE /var/log/cvspserver
        protocol = tcp
        env = '$HOME=/usr/local/cvspserver'
        log_on_failure += USERID
        port = 2401
        server = /usr/bin/cvs
        server_args = -f --allow-root=/usr/local/cvsroot pserver
}
Restart xinetd:
service xinetd restart
You can check /var/log/messages file for relevant error messages.
Add users to this group:
adduser <username> -g cvs
Change their password
passwd username


Thanks,
Vishal Vyas

Tuesday, June 5, 2012

Install Postfix and Dovecot on Redhat


Download and Install:

Check if both Postfix and Dovecot is not yet installed in your system.

# rpm –q postfix
# rpm –q dovecot

If you installed your box as Minimal, for sure both postfix and dovecot not yet installed. Let’s download and install both Postfix and Dovecot.

# yum install postfix
# yum install dovecot

Sendmail and system-switch-mail
By default, sendmail is installed to your system and serves as the default MTA. Since we want to use postfix instead of sendmail, let’s get rid of it! You can use system-switch-mail to do that easily.

Check if it is installed

# rpm –q system-switch-mail

If not installed, install it

# yum install system-switch-mail

And run system-switch-mail

# system-switch-mail

And select Postfix as your default MTA.

Goodbye Sendmail
To finally get rid of sendmail in to your system, go and completely erase it!

# rpm –e --nodeps sendmail

Setting up Postfix
Enough for the intro; let’s do now Postfix setup.

1. Go to /etc/postfix directory

# cd /etc/postfix

2. Copy the original main.cf to another file

# cp main.cf main.cf-bak-orig

We copy the original main.cf file to a new file named main.cf-bak-orig in case we want to go back to the previous configuration. This is always my practice, that any modification I’m doing; I always do a backup first. Very handy in case something goes wrong.

3. Clear the existing main.cf file

# cat /dev/null > main.cf

This command will quickly clear the content of you main.cf file.

4. Edit and add the following lines to main.cf

myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
mydestination = $myhostname localhost.$mydomain localhost $mydomain
relay_domains =
home_mailbox = Maildir/

Here I assume Internet domain name is “example.com”. Users are configured to send mail as “user@example.com”, we call this machine by the name of “mail.example.com”, and “home_mailbox = Maildir/" for qmail-style delivery.

5. Check Postfix

# postfix check

6. Reload Postfix

# postfix reload

Setting up Dovecot :-
Dovecot is very easy to setup and easy to configure. In fact, the last time I setup Dovecot in our local mail server I just simply make sure that under protocols, imap and pop3 are there (protocols = imap pop3). Surprisingly nothing more I did and it works out-of-the-box. Only one thing you may need to get the authentication works – but hey if your users are in /etc/passwd there’s hardly anything you have to do.

Don’t forget to restart dovecot after you edited and saved /etc/dovecot.conf

# /etc/init.d/dovecot restart

Firewall Configuration :-
Make sure you open the SMPT (tcp/25), POP3 (tcp/110), and IMAP (tcp/143) ports in your iptables.

# vi /etc/sysconfig/iptables

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 110 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 143 -j ACCEPT

Restart iptables

# /etc/init.d/iptables restart

That’s all folks! With your choice of mail clients, your users can now send and receive emails.

Optionally, you can install SquirrelMail for your web-based email.

Just download SquirrelMail

# yum intall squirrelmail

Apache and PHP will be automatically downloaded and installed as well. Why not add MySQL so that you can do more things?Here’s how to do it, in case you want.

Edit some settings in SquirrelMail

# /usr/share/squirrelmail/config/conf.pl
SquirrelMail Configuration : Read: config.php
---------------------------------------------------------
Main Menu --
1. Organization Preferences
2. Server Settings
3. Folder Defaults
4. General Options
5. Themes
6. Address Books
7. Message of the Day (MOTD)
8. Plugins
9. Database
10. Languages

D. Set pre-defined settings for specific IMAP servers

C Turn color off
S Save data
Q Quit


Start Apache

# /etc/init.d/httpd start

Conclusion
This is a general very quick guide that will guarantee you a working Mail Server Setup in a latest Red Hat environment. Other optimization and enhancement can be done as your specific requirements arise.
Open Postfix mail server

http://192.168.1.1/webmail


Thaks,
Vishal Vyas

Monday, June 4, 2012

mysql replication

MySQL replication allows you to have an exact copy of a database from a master server on another server (slave), and all updates to the database on the master server are immediately replicated to the database on the slave server so that both databases are in sync. This is not a backup policy because an accidentally issued DELETE command will also be carried out on the slave; but replication can help protect against hardware failures though. 







Create Replication User 
Create an account on the master server that the slave server can use to connect. This account must be given the 


REPLICATION SLAVE privilege: 
#mysql -uroot -p
#CREATE USER 'user_vishal'@'localhost' IDENTIFIED BY 'pass_123';
#GRANT REPLICATION SLAVE ON *.* TO 'user_vishal'@'Slave Ip Addd' IDENTIFIED BY 'pass_123';
#mysql>FLUSH PRIVILEGES;
#show master status\G;

In master server:-
vim /etc/my.cnf
log-bin=mysql-bin
server-id=1
innodb_flush_log_at_trx_commit=1
sync_binlog=1

Restart MySQl


On Slave Server:-
vim /etc/my.cnf
server-id=2
master_host=(master ip)
master_user=vyas
master_password=123

Restart MySQl
#mysql -uroot -p
CHANGE MASTER TO MASTER_HOST='Master Host',
MASTER_HOST=’X.X.X.X’,
MASTER_USER=’user’,
MASTER_PASSWORD=’password’,
MASTER_PORT=3306,
MASTER_LOG_FILE=’mysql-bin.000001?,
MASTER_LOG_POS=98,
MASTER_CONNECT_RETRY=10;

START SLAVE;
SHOW SLAVE STATUS\G;

Thanks,
Vishal Vyas

Saturday, June 2, 2012

Install Subversion [SVN] on Linux Server

how to install SVN (Subversion) server on Fedora , Red Hat (RHEL).
SVN .. Vishal vyas






Install Packages:-
# yum install mod_dav_svn subversion

 

Modify Subversion config file /etc/httpd/conf.d/subversion.conf

Add following config to /etc/httpd/conf.d/subversion.conf file:-
LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so

<Location /svn>
   DAV svn
   SVNParentPath /var/www/svn
   AuthType Basic
   AuthName "Subversion repositories"
   AuthUserFile /etc/svn-auth-users
   Require valid-user
</Location>

How to Add SVN  users:-

Use  command:
## Create user vishal ##
htpasswd -cm /etc/svn-auth-users vishal
New password:
Re-type new password:
Adding password for user vishal

## Create testuser2 ##
htpasswd -m /etc/svn-auth-users vyas
New password:
Re-type new password:
Adding password for user vyas

 

How to Create and configure SVN repository:-

mkdir /var/www/svn
cd /var/www/svn

svnadmin create reponame
chown -R apache.apache reponame

chcon -R -t httpd_sys_content_t /var/www/svn/reponame

## Following enables commits over http ##
chcon -R -t httpd_sys_rw_content_t /var/www/svn/reponame 
Goto http://localhost/svn/reponame address and you should see 
something like following, write username and password:  
 

 Create trunk, branches and tags structure under testrepo:-

Create “template” directories with following command:
mkdir -p /opt/vishal{trunk,branches,tags}
Then import template to project repository using “svn import” command:
svn import -m 'Initial import' /opt/vishal/ http://localhost/svn/repo
Adding         /tmp/svn-structure-template/trunk
Adding         /tmp/svn-structure-template/branches
Adding         /tmp/svn-structure-template/tags

Committed revision 1.
 
Check results on browser and see testrepo revision 1:- 
 
Thanks,
Vishal Vyas  
 

How to Install docker-compose CentOS7

 Install Docker-compose on CentOs7 Linux. yum install -y epel-release yum install -y python-pip pip install --upgrade pip pip install ...