>> $ tentakel -g compute /bin/hostname
>> $ tentakel -g power /bin/hostname
Thursday, October 7, 2010
Saturday, October 2, 2010
Tuesday, September 28, 2010
UCSC genome
User: 'browser', password: 'genome' - full database access permissions
User: 'readonly', password: 'access' - read only access for CGI binaries.
User: 'readwrite', password: 'update' - readwrite access for hgcentral DB
User: 'readonly', password: 'access' - read only access for CGI binaries.
User: 'readwrite', password: 'update' - readwrite access for hgcentral DB
Friday, September 24, 2010
Interpreting samtools flagstat output
I get the following output from running running the flagstat command in samtools for one particular bam file:
22283920 in total
0 QC failure
0 duplicates
20536595 mapped (92.16%)
22283920 paired in sequencing
11141960 read1
11141960 read2
17996862 properly paired (80.76%)
19980715 with itself and mate mapped
555880 singletons (2.49%)
852545 with mate mapped to a different chr
92547 with mate mapped to a different chr (mapQ>=5)
I am not sure what the 8th and 9th lines mean:
line 8: 17996862 properly paired (80.76%)
line 9: 19980715 with itself and mate mapped
I would have thought (looking at line 8 alone), that this means that 17996862 of the 20536595 mapped tags are "properly" (on the same chr, within the limits of the allowed insert size and same(?) orientation) paired with their mate. But I have no idea what line 9 means. Any help would be greatly appreciated!
Thanks
samtools flagstat output interpretation description
asked Jun 17 at 15:31
sasseq
16●3
One Answer: oldestnewestmost voted
2
At the risk of being overly technical, line 8, "properly paired", simply means the properly paired flag is set - it is entirely up to the program writing it to define "proper". Generally, this is set by an aligner to indicate what you describe, same chr, opposite orientation, and a within few deviations from the expected insert size. But this varies depending on the aligner used.
Line 9, means that both the forward and reverse are mapped -- somewhere, anywhere. Properly paired is a subset of these reads. You'll also find that the total of line 9 and the read pairs in which only one tag is successfully mapped (singletons) is equal to the total mapped reads. 19980715 (both mapped) + 555880 (singletons) = 20536595 (total mapped reads).
Lastly, since the sum of "proper pairs" and "itself and mate mapped to different chr" is less than "itself and mate mapped", it appears your aligner does require additional directionality or distance constraints for a "proper pair".
~J
answered Jun 22 at 10:24
jmanning2k
171●11
22283920 in total
0 QC failure
0 duplicates
20536595 mapped (92.16%)
22283920 paired in sequencing
11141960 read1
11141960 read2
17996862 properly paired (80.76%)
19980715 with itself and mate mapped
555880 singletons (2.49%)
852545 with mate mapped to a different chr
92547 with mate mapped to a different chr (mapQ>=5)
I am not sure what the 8th and 9th lines mean:
line 8: 17996862 properly paired (80.76%)
line 9: 19980715 with itself and mate mapped
I would have thought (looking at line 8 alone), that this means that 17996862 of the 20536595 mapped tags are "properly" (on the same chr, within the limits of the allowed insert size and same(?) orientation) paired with their mate. But I have no idea what line 9 means. Any help would be greatly appreciated!
Thanks
samtools flagstat output interpretation description
asked Jun 17 at 15:31
sasseq
16●3
One Answer: oldestnewestmost voted
2
At the risk of being overly technical, line 8, "properly paired", simply means the properly paired flag is set - it is entirely up to the program writing it to define "proper". Generally, this is set by an aligner to indicate what you describe, same chr, opposite orientation, and a within few deviations from the expected insert size. But this varies depending on the aligner used.
Line 9, means that both the forward and reverse are mapped -- somewhere, anywhere. Properly paired is a subset of these reads. You'll also find that the total of line 9 and the read pairs in which only one tag is successfully mapped (singletons) is equal to the total mapped reads. 19980715 (both mapped) + 555880 (singletons) = 20536595 (total mapped reads).
Lastly, since the sum of "proper pairs" and "itself and mate mapped to different chr" is less than "itself and mate mapped", it appears your aligner does require additional directionality or distance constraints for a "proper pair".
~J
answered Jun 22 at 10:24
jmanning2k
171●11
Thursday, September 23, 2010
trackDb
UCSC genome browser
Kent source tree
Get it here: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip
Or view it online: http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/
Important README files: kent/src/product/README.xxx
System design
The genome browser is composed by following parts:
For service
server-side CGI binaries and HTML files
hg.conf configuration file
For administration
utility tools
.hg.conf file
data
stored in Mysql
stored in external files (optional, such as bigBed, bigWig, ...)
Design principle:
Each species will occupy a Mysql database. Inside each database:
each track correspond to one table, and each track belongs to a group
one trackDb table exists for one species, used to describe tracks
each record of the data in trackDb defines one track
one grp table to hold group info
one hgFindSpec table
Apart from databases for individual species, special databases exist:
hgFixed
hgcentral
Constructing new database
Download kent source and compile everything. See src/product/README.trackDb on how to compile everything.
Create hg.conf file at /cgi-bin/ directory. Sample file can be found at: http://genome-test.cse.ucsc.edu/~kent/src/unzipped/product/ex.hg.conf
Create .hg.conf file at your home directory:
$ cat > ~/.hg.conf
db.host=127.0.0.1
db.user=hguser
db.password=hguser
db.trackDb=trackDb
db.grp=grp
central.db=hgcentral
central.host=127.0.0.1
central.user=hguser
central.password=hguser
$ chmod 600 ~/.hg.conf
Create mysql database, make sure all folders at mysql directory are with user/grp of mysql.mysql. If not, will throw "errno: 13" when trying to modify it!!
To add new tracks:
Refer to general guide: kent/src/product/README.trackDb
generate track data in appropriate format (http://genome.ucsc.edu/FAQ/FAQformat.html)
Create a table to hold the track data. Need to identify the format of the track data file, and use corresponding loader program to load it. Following example is for bed format:
$ hgLoadBed dbName trackTableName file.bed
Realistic example: $ hgLoadBed -noBin -bedGraph=4 hg19 track_name data.bedGraph
Loader program source locates at: kent/src/hg/makeDb/
Create/update the trackDb:
Compose a new trackDb.ra file (by editing the old one) with configuration section for the new track
Example *.ra files: src/hg/makeDb/trackDb/[organism name]/[genome version]
Information on trackDb options: src/hg/makeDb/trackDb/README. Also see next section.
Compose a makefile at the place where trackDb.ra file resides. Could be like:
trackDbSql=/home/cgs/twlab/xzhou/kent/src/hg/lib/trackDb.sql
DB=hg19
all::
~/latest/hgTrackDb . ${DB} trackDb ${trackDbSql} .
Run $ make all to update the trackDb table.
Also tracks could be in bigBed or bigWig formats. See: http://www.mail-archive.com/genome@lists.soe.ucsc.edu/msg00924.html
hgsql hg19 -e 'drop table if exists myLocalBigWig; create table myLocalBigWig (fileName varchar(255) not null); insert into myLocalBigWig values ("/gbdb/hg19/bbi/myLocalBigWig.bw");'
About trackDb.ra files:
Example files for human hg19 can be found at: kent/src/hg/makeDb/trackDb/human/hg19/
use blank lines to separate tracks
each line begins with an attribute name and value, separated by space
Configurations
field: track
track trackName [override]
field: type
a lot of types there...
Track height: use maxHeightPixels 128:32:16
Track color: color r,g,b
Sub groups: subGroup1 sampleType Sample_Type fetalK=fetalK CD34=CD34 ....
Kent source tree
Get it here: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip
Or view it online: http://hgwdev.cse.ucsc.edu/~kent/src/unzipped/
Important README files: kent/src/product/README.xxx
System design
The genome browser is composed by following parts:
For service
server-side CGI binaries and HTML files
hg.conf configuration file
For administration
utility tools
.hg.conf file
data
stored in Mysql
stored in external files (optional, such as bigBed, bigWig, ...)
Design principle:
Each species will occupy a Mysql database. Inside each database:
each track correspond to one table, and each track belongs to a group
one trackDb table exists for one species, used to describe tracks
each record of the data in trackDb defines one track
one grp table to hold group info
one hgFindSpec table
Apart from databases for individual species, special databases exist:
hgFixed
hgcentral
Constructing new database
Download kent source and compile everything. See src/product/README.trackDb on how to compile everything.
Create hg.conf file at /cgi-bin/ directory. Sample file can be found at: http://genome-test.cse.ucsc.edu/~kent/src/unzipped/product/ex.hg.conf
Create .hg.conf file at your home directory:
$ cat > ~/.hg.conf
db.host=127.0.0.1
db.user=hguser
db.password=hguser
db.trackDb=trackDb
db.grp=grp
central.db=hgcentral
central.host=127.0.0.1
central.user=hguser
central.password=hguser
$ chmod 600 ~/.hg.conf
Create mysql database, make sure all folders at mysql directory are with user/grp of mysql.mysql. If not, will throw "errno: 13" when trying to modify it!!
To add new tracks:
Refer to general guide: kent/src/product/README.trackDb
generate track data in appropriate format (http://genome.ucsc.edu/FAQ/FAQformat.html)
Create a table to hold the track data. Need to identify the format of the track data file, and use corresponding loader program to load it. Following example is for bed format:
$ hgLoadBed dbName trackTableName file.bed
Realistic example: $ hgLoadBed -noBin -bedGraph=4 hg19 track_name data.bedGraph
Loader program source locates at: kent/src/hg/makeDb/
Create/update the trackDb:
Compose a new trackDb.ra file (by editing the old one) with configuration section for the new track
Example *.ra files: src/hg/makeDb/trackDb/[organism name]/[genome version]
Information on trackDb options: src/hg/makeDb/trackDb/README. Also see next section.
Compose a makefile at the place where trackDb.ra file resides. Could be like:
trackDbSql=/home/cgs/twlab/xzhou/kent/src/hg/lib/trackDb.sql
DB=hg19
all::
~/latest/hgTrackDb . ${DB} trackDb ${trackDbSql} .
Run $ make all to update the trackDb table.
Also tracks could be in bigBed or bigWig formats. See: http://www.mail-archive.com/genome@lists.soe.ucsc.edu/msg00924.html
hgsql hg19 -e 'drop table if exists myLocalBigWig; create table myLocalBigWig (fileName varchar(255) not null); insert into myLocalBigWig values ("/gbdb/hg19/bbi/myLocalBigWig.bw");'
About trackDb.ra files:
Example files for human hg19 can be found at: kent/src/hg/makeDb/trackDb/human/hg19/
use blank lines to separate tracks
each line begins with an attribute name and value, separated by space
Configurations
field: track
track trackName [override]
field: type
a lot of types there...
Track height: use maxHeightPixels 128:32:16
Track color: color r,g,b
Sub groups: subGroup1 sampleType Sample_Type fetalK=fetalK CD34=CD34 ....
Thursday, September 16, 2010
session for UCSC genomebrowser
i've enabled wiki track with sessions by these steps:
1.
Download and install mediawiki , better in a root subfolder of your mirror
2.
As written in kent/src/hg/wikiMod/README
$IP = MEDIAWIKIINSTDIR
cd kent/src/hg/wikiMod/
cp SpecialUserloginUCSC.php $IP/includes/specials/
cp SpecialUserlogouUCSC.php $IP/includes/specials/
cp configuration.SpecialUserloginUCSC.php $IP/SpecialUserloginUCSC.php
cp configuration.SpecialUserlogoutUCSC.php $IP/SpecialUserlogoutUCSC.php
cp UserloginUCSC.php $IP/includes/templates/
at the end of $IP/LocalSettings.php add these rows :
require_once( "$IP/extensions/SpecialUserloginUCSC.php" );
require_once( "$IP/extensions/SpecialUserlogoutUCSC.php" );
require_once( "$IP/includes/templates/UserloginUCSC.php" )
3.
Check if db "hgcentral" has this table: namedSessionDb.
If not use this sql to create it : CREATE TABLE namedSessionDb (
userName varchar(64) not null, # User name (from genomewiki).
sessionName varchar(255) not null, # Name that user assigns to
this session
contents longblob not null, # CGI string of var=val&... settings.
shared tinyint not null, # 1 if this session may be shared with
other users.
firstUse datetime not null, # Session creation date.
lastUse datetime not null, # Session most-recent-usage date.
useCount int not null, # Number of times this session has been
used.
settings longblob not null, # .ra-formatted metadata
#Indices
PRIMARY KEY(userName,sessionName)
);
3a.
rsync your local goldenPath/html dir with remote
htdocs/goldenPath/html/ dir.
4.
I've installed mediawiki in a subfolder of a root folder. I called it w.
Modify hg.conf adding these rows:
wiki.host=SERVER/w # w is MY installation dir of mediawiki.
wiki.userNameCookie=wikidbUserName ######### use the firebug to get the right name for wikidbUserName wikidbUserID and wikidb_session
wiki.loggedInCookie=wikidbUserID
wiki.sessionCookie=wikidb_session
wikiTrack.URL=http://SERVER/w
wikiTrack.browser=SERVER
wikiTrack.dbList=hg18,mm9,hg19
Note. If you have problem with genome session login, please check with
firefox live_http_header plugin what data is passed after login from
mediawiki. And modify wiki.userNameCookie or wiki.loggedInCookie or
wiki.sessionCookie values.
#LDAP AUTH To use ldap authentication with mediawiki download
LdapAuthentication extension from here:
cd MEDIAWIKIINSTDIR/extensions/
wget -v
http://upload.wikimedia.org/ext-dist/LdapAuthentication-trunk-r65285.tar.gz
tar xvzf LdapAuthentication-trunk-r65285.tar.gz
cd MEDIAWIKIINSTDIR
modify LocalSettings.php adding these rows (at the bottom! ) :
#enable LDAP require_once(
"$IP/extensions/LdapAuthentication/LdapAuthentication.php" );
$wgAuth = new LdapAuthenticationPlugin();
#$wgLDAPDebug = 3; //to enable ldap debug
$wgLDAPDomainNames = array( "CAMPUS" );
$wgLDAPServerNames = array( "CAMPUS" => "LDAPSERVER");
$wgLDAPSearchAttributes = array('CAMPUS' => 'uid');
$wgLDAPBaseDNs = array( "CAMPUS"=>"YOUR BASE DN" );
$wgLDAPEncryptionType = array( "CAMPUS" => "tsl" );
Add this row :
$this->mName=strtolower($this->mName);
after row 763 of SpecialUserloginUCSC.php in MEDIAWIKIDIR/includes/special/
#END LDAP AUTH
1.
Download and install mediawiki , better in a root subfolder of your mirror
2.
As written in kent/src/hg/wikiMod/README
$IP = MEDIAWIKIINSTDIR
cd kent/src/hg/wikiMod/
cp SpecialUserloginUCSC.php $IP/includes/specials/
cp SpecialUserlogouUCSC.php $IP/includes/specials/
cp configuration.SpecialUserloginUCSC.php $IP/SpecialUserloginUCSC.php
cp configuration.SpecialUserlogoutUCSC.php $IP/SpecialUserlogoutUCSC.php
cp UserloginUCSC.php $IP/includes/templates/
at the end of $IP/LocalSettings.php add these rows :
require_once( "$IP/extensions/SpecialUserloginUCSC.php" );
require_once( "$IP/extensions/SpecialUserlogoutUCSC.php" );
require_once( "$IP/includes/templates/UserloginUCSC.php" )
3.
Check if db "hgcentral" has this table: namedSessionDb.
If not use this sql to create it : CREATE TABLE namedSessionDb (
userName varchar(64) not null, # User name (from genomewiki).
sessionName varchar(255) not null, # Name that user assigns to
this session
contents longblob not null, # CGI string of var=val&... settings.
shared tinyint not null, # 1 if this session may be shared with
other users.
firstUse datetime not null, # Session creation date.
lastUse datetime not null, # Session most-recent-usage date.
useCount int not null, # Number of times this session has been
used.
settings longblob not null, # .ra-formatted metadata
#Indices
PRIMARY KEY(userName,sessionName)
);
3a.
rsync your local goldenPath/html dir with remote
htdocs/goldenPath/html/ dir.
4.
I've installed mediawiki in a subfolder of a root folder. I called it w.
Modify hg.conf adding these rows:
wiki.host=SERVER/w # w is MY installation dir of mediawiki.
wiki.userNameCookie=wikidbUserName ######### use the firebug to get the right name for wikidbUserName wikidbUserID and wikidb_session
wiki.loggedInCookie=wikidbUserID
wiki.sessionCookie=wikidb_session
wikiTrack.URL=http://SERVER/w
wikiTrack.browser=SERVER
wikiTrack.dbList=hg18,mm9,hg19
Note. If you have problem with genome session login, please check with
firefox live_http_header plugin what data is passed after login from
mediawiki. And modify wiki.userNameCookie or wiki.loggedInCookie or
wiki.sessionCookie values.
#LDAP AUTH To use ldap authentication with mediawiki download
LdapAuthentication extension from here:
cd MEDIAWIKIINSTDIR/extensions/
wget -v
http://upload.wikimedia.org/ext-dist/LdapAuthentication-trunk-r65285.tar.gz
tar xvzf LdapAuthentication-trunk-r65285.tar.gz
cd MEDIAWIKIINSTDIR
modify LocalSettings.php adding these rows (at the bottom! ) :
#enable LDAP require_once(
"$IP/extensions/LdapAuthentication/LdapAuthentication.php" );
$wgAuth = new LdapAuthenticationPlugin();
#$wgLDAPDebug = 3; //to enable ldap debug
$wgLDAPDomainNames = array( "CAMPUS" );
$wgLDAPServerNames = array( "CAMPUS" => "LDAPSERVER");
$wgLDAPSearchAttributes = array('CAMPUS' => 'uid');
$wgLDAPBaseDNs = array( "CAMPUS"=>"YOUR BASE DN" );
$wgLDAPEncryptionType = array( "CAMPUS" => "tsl" );
Add this row :
$this->mName=strtolower($this->mName);
after row 763 of SpecialUserloginUCSC.php in MEDIAWIKIDIR/includes/special/
#END LDAP AUTH
Subscribe to:
Posts (Atom)