Environment setup for big data analytics

This articles covers basic tools and technologies to use when conducting the first steps on big data analysis.

Linux as the base OS
- Debian
- Ubuntu
- RedHat
- CentOS
For basic data processing:
- Bash shell: environment for running multiple command-line Linux tools for data manipulation
  - Comes pre-installed with Linux; check Reference manual for usage
  - Bash might be not the default Linux shell, see how to switch to it
  - Learn Bash by examples
  - Most important Linux Commands and phenomenons to master for data manipulation:
    - cat – output/concetanate a stream / file
    - head, tail – show the head or the tail of a stream / file
    - grep – to filter streams / lines of a file
    - sed – to manipulate streams / lines of a file
    - Regular expressions – used in a wide set of Linux tools
      - Explanation and tutorials
- AWK – simple data reformatter with compact coding features
  - The GNU Awk User’s Guide
  - AWK Tutorials
- Python – easy to learn, effective programming language with a huge amount of libraries available for various tasks. Great for data manipulation used from the command line.
  - Python Download
  - Python Documentation
  - Python Beginners Guide
  - Python Tutorials
And the big data analysis framework chosen based on the type of data analyzed. For the first step tutorials our suggestion would be:
- Hadoop, single cluster setup (can be downloaded pre-installed to a virtual appliance)
- Java based MapReduce programs
- Pig MapReduce query language