Linux_august29_2017

Linux Tutorial - ScINET

August 29, 2017

Jonathan Shao, Computational Biologist, NEA

Howto get on

SciNet

● Windows http://www.putty.org/

Linux - Terminal

● Mac - Terminal -

Method 1 Opening with Finder

1. Open a new Finder window by clicking the Finder

icon to the far left of your Dock.

2. From the sidebar, choose Applications, and then go

to Utilities. Alternatively, you can just press ⌘

Command + ⇧ Shift + U .

3. Double click on the Terminal.

SSH to SciNet

ssh [email protected] - Terminal

Putty.ext

ssh -o TCPkeepAlive=yes -o ServerAliveInterval=20 -o ServerAliveCountMax=100 mary.smith@scinet-login.bioteam.net -XA

Note: when you ssh in for the first time - hit “yes” to accept.

SSH to SciNet

Basic Linux Commands - Directory Tree

linux commands are in lowercase

http://www.shril-sy.info/page/l/linux-directory-tree.html

https://technet.microsoft.com/en-us/library/bb497051.09sybm04_big(l=en-us).gif

Basic Linux Commands - Listing files “ls”

(all linux commands in lowercase)

Example:

>ls

This command lists the contents of the current directory.

>ls helloworld

This lists the name of a specific file

>ls -lah

This command lists all the contents in the current directory in great detail

Basic Linux Commands - Listing files ls

Notice a few things and the difference between an ls and a ls -lah

In blue is is a directory called “test_dir” and in white is the name of a file “helloworld”

drwxrwxr_x

d stand for directory

r - read

w -write

x - execute

Jonathan.shao jonathan.shao - who owns the file (user and group ownership)

4.0K is the size of the file

Aug 7 is the Aug date of this year.

helloworld/test_dir are the names of the files and directory

Basic Linux Commands Changing Directories cd

>cd <path to where you are going>

cd means change directory

Example:

>cd /test_dir

>cd /scinet01/gov/usda/ars/scinet/project/

cd space “and the name of a directory” moves you down or forwards

>cd ..

cd dot dot means move up a directory or backwards

Basic Linux Commands - Directory Tree -SciNet

Scinet

Note: (execute the commands in order for the comments to make sense)

>cd /scinet01/gov/usda/ars/scinet/project/

This changes directory to the project directory. This is the hard-path to the

project directory.

>cd ..

This will move you up or backwards to the scinet directory

>cd project

This will move you forwards to the project directory from the scinet folder

>cd ../../../../../../

This will move you all the way up to the root / directory

cd /usr

Changes directory to the

usr directory

Basic Linux Commands, Help, Where am I, Changing Owners

>man <name of linux command>

man cp

man pwd

man chmod

man is the help file for each command

>pwd

pwd shows the path of working directory

>chown john.smith:john.smith helloworld

changes the owner of the files of helloworld

>chmod 755 helloworld

changes rwx of helloworld and makes it executable

chgrp -R proj-nea_bioinformatics /project/nea_bioinformatics

Basic Linux Commands, Moving and Renaming files, History

>mv <original file> <file to be renamed or moved>

>mv helloworld helloworld2

This renames the file from helloworld to helloworld2

>mv helloworld test_dir

This moves the file helloworld to test_dir

>history > my_history_August8_2017.txt

The command history gives your saved commands and the “>”

redirects the output from history into a file.

Basic Linux Commands – Coping Files

>ls

helloworld

>cp <original file to be copied> <copy of file>

>cp helloworld helloworld.txt

Here we have copied the helloworld file to helloworld.txt

>ls

helloworld helloworld.txt

>cp <path><original file to be copied> <path><copy of file>

>cp helloworld /home/shaoj/test_dir

>cd /home/shaoj/test_dir

>ls

helloworld

Basic Linux Commands – Unzip and Untar Files

>gunzip test.gz

gunzip unzips a .gz file

>tar -xvf test.tar

tar untars a tar file

Linux Commands useful for SciNet

>module load bowtie2

>which bowtie2

>module unload bowtie2

>srun --pty -p short -t 48:00:00 -n 20 -N1 /bin/bash -l

>lfs quota -gh project_folder /scinet01

>squeue

>sinfo

>chgrp -R proj-nea_bioinformatics /project/nea_bioinformatics

Linux Commands- Getting Your Files Off the SciNet

scp <file> <folder location>

Putting a file on the scinet:

>scp file.txt ma[email protected]ioteam.net:/scinet01/gov/usda/ara/scinet/project/work_dir

Getting a file to your home directory from the scinet:

>scp mary.sm[email protected]:/scinet01/gov/usda/ara/scinet/project/work_dir/file.txt /home/smithm

Linux Commands – Running a Program

>blastn -query test.fas -db nr -out test.blastn

The red dashes are switches/parameters that are passed into

the blastn executable.

The first entry blastn in green is usually the executable

program.

Here I am passing the query name test.fas using the query

switch, the database nr using the db switch and outputting the

file test.blastn using the out switch to the executable blastn.

How Do I Remove My Mess? Practice Good Hygiene

rm <file>

Note that the the rm command is unforgiving. It will delete the

file permanently.

>ls

helloworld helloworld.txt

>rm helloworld

The file helloworld will be deleted

>ls

helloworld.txt

Here you will only see helloworld.txt, since helloworld was

deleted

Yikes! My Linux Program is Stuck or I Made a Mistake

>control+C

>control+Z

Halts commands

Hit the X button and close your terminal.

>squeue

Look at jobs in the queue and find your rogue id

133432 medium at-258 john.s R 12:09:23 1 sn-cn-13-1

>scancel <job id>

Example: scancel 133432

I Can’t Find My Files, Please Help

This will find file1 on your hard-drive

find file1 /home/smithmary

This will find file1 on /home/smithmary

find file1 /

This will find file1 on whole computer starting with root

Helpful Hints in Linux

You can cheat and hit the tab button to auto-complete

Example:

>ls

file_this_is_along_file_name.txt

>ls file_this_(hit tab button)

Auto-complete will

>ls file_this_is_along_file_name.txt

Notice that I used underscores and all lowercase for my file

names. This can make life easier, since Linux is case sensitive!

How Do I View a File?

>nano file1

control X – hit y to save

Note that vi is the other editor. It is more powerful, but has a much harder

learning curve.

gedit file1

If you have gedit installed, it works much like wordpad in windows

>head file.txt

shows the top of the file

>tail file.txt

shows the bottom of the file

head and tail are useful for viewing large files

More Helpful Hints in Linux

You can cheat and use the wildcard “*”

The use of * is a regular expression.

Example:

>ls

file_this_is_along_file_name.txt

apple.txt

pear.txt

file2.txt

>ls file*

file_this_is_along_file_name.txt

file2.txt

More Helpful Hints in Linux

Qiime meta data tabulate \

--m-input-file taxonomy.qza \

--o-visualization taxonomy.qzv

These lines are equivalent

Qiime meta data tabulate --m-input-file taxonomy.qza –o-visualization taxonomy.qzv

Linux Cheat sheet

>cd - change directory

>cd .. - moves up a directory

>cd /project/microbiome_workshop/amplicon/data/ - moves you

to this directory

>ls - lists the contents of a directory

>pwd - path of working directory

>cp file1 file2 – copies file1 and creates a duplicate file2

>mkdir dir – create new directory dir

>rm file1 – removes file1

>chmod 777 file – 4 read, 2, write , 1 execute – adds up to 7

Read write execute for owner, group world

ssh user@host

>gunzip file.gz – decompress file

>tar –xvf – untar file

>control+C, or control+Z – halts command

>mv file1 renamedfile – renames file

>mv file1 test_dir – moves file1 to test_dir

Thank you

Next - Adam Rivers - Amplicon Analysis with

QIIME2