The CCP4 Package.

CCP4 Workshop ACA, Texas 2002

presentation by Alun Ashton, [email protected]

1

General Overview..

The CCP4 Package.



2

On What Systems Will It Work?SGi: Irix (o32, n32 and 64bit)Linux Systems (Redhat, SuSE…..)Compaq OSF1 etcSun (Sun OS)HP (All apart from Xwindows)IBM (AIX)Mac OSXMicrosoft Windows+



3

Where It Can Be Downloaded CCP4 ftp site:

ftp://ftp.ccp4.ac.ukftp ftp.ccp4.ac.ukuser name: ftppw [email protected] pub/ccp4/current/binary

CCP4 mirror sites:Japan

ftp://ftp.protein.osaka-u.ac.jp/mirror/ccp4 USA

ftp://ftp.sdsc.edu/pub/sdsc/xtal/CCP4



4

How BIG Is the Download?

unpacked size =160Mb tar.gz size = 36 Mb tar.Z size = 56 MbBinary size = 25 – 107 Mb



5

InstallationBinaries or source code?UnpackingDirectory structureccp4.setupConfigureSource Code MakeBinary MakeWindows Installing on multiple systems



6

Installation IIYou do not need root privileges on UNIX systems

Just enough space!You DO need Admin privileges on Windows NT/2K

This is used to set up environment variables etc.

What else do you need to install CCP4?Source code installation:

Compilers C and FORTRAN.Make (UNIX)

Gzip/Uncompress/tar (UNIX)ccp4i requires tcl/tk/BLTAstexViewer requires a Java VM.



7

Binaries or Source Code?

Binary Source

Install (inc ease) speed Download speed Developers (Customisation) Range of systems/portability Compiler problems/price Ease of support



8

Binaries or Source II?

SupportedBinaries?

yes

no

Developer?FastNet?

SourceCode andCompile

yes

no

ReliableCompiler?

yes

no

yes

Binary+

Source code

no

(Currently supported: irix, linux, OSF1, SunOS, Windows)Binaries

or source?



9

Unpacking

Download the file into an area with enough space to download and unpack it.To uncompress a .gz file use gunzipTo uncompress a .Z file use uncompressA tar file is a collation of many files: To extract the files from a tar file use:

tar –xvf file.tar



10

Directory Structure

ccp4_version.tar xtal/ccp4_version/

ccp4i etc include lib src unsupported x-windows

doc, examples, help, htm

l, man, m

anual

CC

P4 interface, Interface

documentation, roadm

aps

CC

P4 utility scripts

ccp4.setup, environ.def and default.def

Library src and data

Program

Source C

ode

bin

Older program

s that are being phased out

Program

s with

graphical user interfacethat require x-w

indows



11

ccp4.setup

All CCP4 programs require ccp4 environment variables set-up to enable them to run.

ccp4.setup files found in include sub directoryccp4.setup-dist and ccp4.setup-bashIt is essential that the following environment variables are

correctly assigned:CCP4_MASTER (where the ccp4_vers.tar was unpacked)CCP4_SCR (CCP4 scratch directory), CCP4I_TCLTK (where ccp4i finds bltwish)

Most other environment variables use these variables.source ccp4.setup before configure and running CCP4 programsAutomatically source ccp4.setup in login files (.cshrc or .login)

(UNIX variants only)



12

ccp4.setup II

BINSORT_MEMSize of memory to use for sorting before suing disk

CCP4I_HELP Usually $CCP4/ccp4i/help

BINSORT_SCR Disk area to use for sorting CCP4I_TCLTK Which bltwish to use

CBIN Where ccp4 binaries will live CCP4I_TOP Usually $CCP4/ccp4i

CCP4 Top directory of ccp4 CHTML $CCP4/html

CCP4_BIN $CBIN CINCL $CCP4/include

CCP4_LIB Where ccp4 libraries will live CLIBD $CCP4/lib/data

CCP4_MASTER Usually $CCP4/../ CLIBS $CCP4/lib/src

CCP4_OPENOverwrite files or not.

UNKNOWN = overwriteCPROG

$CCP4/src home of the ccp4 program code

CCP4_SCRWhere ccp4 programs will write temporary files.

PUBLIC_FONT84$CCP4_LIB/font84.dat a font data file.

Environment variables

There is no need to learn these (and there are more) unless things go wrong!



13

configure

Configuring allows system specific flags to be assignedConfigure gives you flexibility, and a quick and easy

way to change all the important make files Configure settings can be overridden by environment

variablesconfigure + Makefile.in Makefile + config.status

Once the system has been configured you can either make the source code or install the binary files.

A log of the configure settings are in config.status .

More than one configure is run by $CCP4/configure

(UNIX variants only, when making from source code)



14

configure (II)configure [--flags] [operating system]

Operating system

(choice of 17!)

--flags

(choice of 15!)meaning

irix, irix64 SGi --with-xCompile the X

programslinux

/_compaq_compilersLinux --{*}dir src/bin/lib/tmp

hpuxHewlett Packard

--with-shared-lib Compile shared libs

osf1 Compaq --only-libs Only compile libs

Darwin Mac OSX --with-warnings One for the developers!

sunosSun

Microsystems



15

Source Code Make(installing from source code only)

Once you’ve downloaded the package, unpacked, set the environment variables and configured the package you are now ready to make the suite!

In $CCP4 type make.You might want to keep a log with make > make.log

Make will firstly make the library components before the programs.

Once complete you will need to install with make install.



16

Source Code Make II.The make utility uses the settings in the Makefile which in turn was created from the Makefile.in by the configure script.

$CLIBS/fsplit – a small program used to break large library.f files into smaller files – this was needed on older systems.

Make in the x-windows directory.The make in the x-windows directory will ignore errors in compiling programs and will carry on regardless.

Once complete you will need to install with make install.This will install the programs into $CBINMoves the library files into $CCP4_LIB / $CLIB

Make also has numerous other options.



17

Source Code Make IIImake [option]

Option Meaning

install Installs the components into the correct area

[program] Will make the program if its been updated

clean, realclean, distclean

Various degrees of severity of cleaning the directory structure.

uninstall What it says on the box…..

empty-targets Sets the size of e.g. $CPROG/[program] to zero



18

Source Code Make IV(reference only)

Make option

clean Removes *.o, *.~, etc

realcleanclean + any executable files in the directory

distcleanRealclean + removes Makefiles, config.status -> only distribution left

uninstallRemoves installed files created by the install step



19

Source Code Make V - Compilers

Under the majority of operating systems the native compilers should be used.

The GNU compilers gcc and g77 can used on many systemsRedhat 7.0 and 7.1 users should upgrade their gcc/g77Use gcc 3.0.<4

Linux users can also use a variety of other compilers.See recent ccp4bb conversations

(e.g Portland Group and Intel compilers).

Irix 6.5 users should not use f77 version 7.2.*We currently use 7.3.1.2m



20

Binary Code Make

You should have:$CCP4/BINARY.readme$CCP4/BINARY.setup$CCP4/bin/

You need to:Get normal distribution and unpack itEdit and source ccp4.setupRun the script BINARY.setup

See also http://www.ccp4.ac.uk/dist/INSTALL.html BINARY.about file in ‘binary’.tar.gz

(UNIX variants only)



21

WindowsTested on Windows 95/98/Me/NT(4.0)/2000You will need Admin privileges

Windows NT and 2000Use .exe auto install

Windows 95/98/Me variantsUse compressed .zip file

You will also need Tcl/tk and BLT for ccp4i (install before ccp4)

http://www.ccp4.ac.uk/dev/ccp4onnt.html

Cygwin is possible.



22

Installing on Multiple SystemsMultiple systems means e.g. Linux and Irix with shared/mounted disks

Install completely independent source treeUse duptree (found in $CCP4)

duptree must be run before any files are altered in $CCP4duptree creates a second identical source tree using links

Use --srcdir

Why? Size matters. You only need to bug fix one src.



23

Bug Fixing and Patching.

http://www.ccp4.ac.uk/problems.html



24

More Information

CCP4 manual.CCP4 website.

http://www.ccp4.ac.uk/dist/INSTALL.htmlCCP4 problems pages.

[email protected]



25

Now What Have You Got?

The programs

How they communicate

CCP4 Data formats

How to use the programs

The main programs

What is new in 4.2



26

The ProgramsMost programs written in standard(ish) fortran77

Over 150 programs mostly runnable from command line

Full listings on:http://www.ccp4.ac.uk/dist/html/INDEX.html http://www.ccp4.ac.uk/dist/html/FUNCTION.html



27

How They Communicate?Standard data formats that contain informationand informative log outputs.

The data formats are publicly known, documented and useable by anyone.

Nobody is forced to use our data formats as wesupply numerous conversion tools.

Some CCP4 programs already output keyinformation in XML files.



28

CCP4 Data Formats.Formats used are:

.mtz – reflection data.map .msk – map and mask formats.pdb – atom positions.cif – harvest files, refmac5 restraints

Binary formats were used for:SizeSecurity

MTZ/map files are binary but portable as they contain a machine stamp

More information in the FILE FORMATS section onhttp://www.ccp4.ac.uk/dist/html/INDEX.html



29

CCP4 Data Formats II – MTZ Reflection Files

Table of datacolumns - I, F, sigma, phase, FOM, etc. etc.rows - reflections

Columns grouped as datasets withproject name GerEdataset name se_inflection cell dimensions 108.7 61.6 71.6 90.0 97.1 90.0wavelength 0.981

In ccp4i, see:

Reflection Data Utilities -> Edit MTZ Project&Dataset



MTZ fileTitle/historySpacegroup

Crystal 1Crystal nameProject nameCell dimensions

Crystal 2Crystal nameProject nameCell dimensions

Dataset 1.1Dataset nameWavelength

Dataset 1.2Dataset nameWavelength

Column Column

Heirarchical view of reflection data.

CCP4 Data Formats III – MTZ Reflection Files II

The future ....



31

CCP4 Data Formats IV – MTZ Reflection Files III

Header information

Keyword – value pairs

TITLE – title NCOL – no. of data columns CELL – global cell parameters SYMINF – symmetry info COL – column information

RESO – resolution rangeNDIF – no. datasetsDATASET – dataset infoPROJECT – harvesting infoBATCH – batch images

TITLE make ccp4itut mtz inconspicuous Sfs from refined RNA_SANAT 8-7. NCOL 8 18077 0 CELL 64.8970 78.3230 38.7920 90.0000 90.0000 90.0000 SORT 1 2 3 0 0 SYMINF 4 4 P 19 P212121 PG222 COLUMN K H 0.0000 42.0000 COLUMN L H 0.0000 21.0000 COLUMN FP F 0.0000 1219.0000



32

CCP4 Data Formats V – MTZ Reflection Files IV

Viewing and manipulation

mtzdump – view ascii dump of mtz file

mtz2various – convert mtz file to MULTAN, SHELX, TNT, X-PLOR/CN* formats

mtzutils, cad, sftools – edit mtz files

scaleit, fhscal – column scaling

http://www.ccp4.ac.uk/dist/html/FUNCTION.html



33

CCP4 Data Formats VI – MAP Files - Densities

mtz map Fourierphases

fft –switch between electron density and reflection data

mapdump – give information on map

View map with – O, PyMol, Mapslicer, AstexViewer

Format – header + symops + data

Binary - but machine independent.



34

CCP4 Data Formats VII – Atom Positions

PDB files:Based on pdb version 2.1 draft

mmCIFCCP4 uses a subset of the mmCIF dictionaryUsed in;

Restraints library of Refmac5 and libcheckmmdb libraryHarvesting files



35

How To Run The Programsccp4i – the ccp4 interface

Command line options:program [<logical name> <file name>]

e.g. fft hklin native-Fs.mtz mapout 2Fo—Fc.map

Keyworded input:Keyword <options>

E.g. CELL a b c [ ]TITLE this job will work this time



36

How To Run The Programs IICommon Keywords

CELL a b c [ ]Cell dimensions in Å

RESOLUTION <limit> [<limit>]

Resolution limits

ENDTerminates input and starts program

SYMMETRY

<number> | <name> | <operator>HISTORY

Adds record to history stack of MTZ

LABIN <Program label>=<file label>…Associates the column labels that the program expects with MTZ cols

LABOUT <Program Label>=<file label>..Associates column labels used in output file with those in program.



37

The Main Programs

What do you get?

>150 programs encompassing the majority of protein crystallography – including:

Mosflm, Scala, Truncate, Mlphare, Rantan, Revise, Acorn, Oasis, SC, Amore, Molrep, Beast, DM, Fffear, Refmac5, Procheck, Sfcheck

See http://www.ccp4.ac.uk/dist/html/INDEX.html or http://www.ccp4.ac.uk/dist/html/FUNCTION.html



38

What Is New in 4.2ACORN ab initio procedure for the determination of protein structure at atomic resolution (Yao Jia-Xing)BEAST Brute-force molecular replacement with Ensemble Average Statistics, Maximum likelihood-based molecular replacement (Randy Read) PROFESSS determination of NCS operators from heavy atoms (Kevin Cowtan) ROTAMER list amino acids whose side chain torsion angles deviate from Richardson's Penultimate Rotamer Library (Dirk Kostrewa) ASTEXVIEWER Java application for viewing proteins, ligands and electron density maps (Mike Hartshorn)

Announcement on: http://www.ccp4.ac.uk/news/msg00098.html



39

An Introduction to CCP4iThe CCP4 Graphical User Interface

Author: Peter BriggsCCP4

CCP4i Introduction



40

• Offers user friendly interfaces to the programs

• Integrated help system

• Tools for file viewing & “project management”

How to get started

• (Unix/Linux) Type ccp4i at the command prompt

• (Windows/NT) Select ccp4i via the Start menu

Why Use CCP4i?



41

Tasks

Modules

Job Database Tools & Utilities

On-line help

Main CCP4i Window



42

• Access the module list by clicking on the gold bar displaying the current module

Modules



43

• Tasks used in a particular part of the structure determination process• Utility tasks for manipulating different types of data• Alphabetical list of programs/tasks

Modules II



44

• Click on the appropriate button in the list to start a particular task

• This starts the interface for that task

Tasks



45

Closed foldersAdvanced/infrequently

used options

Open foldersParameters that should be checked

by the user before runningHighlights indicate compulsory input

File folderSet input and output file names

Protocol folderMake the key decisions

WO

RK

FR

OM

TH

E T

OP

DO

WN

Run taskSave/restore parameters

Always add a title to distinguish different runs of the same task

Defaults - “If it’s not visible then it’s not important”

Example Task Interface



46

Watch the progress of the job in the Job Database window:

Running a Task



47

• View logfile in Netscape (if it contains HTML tags) and/or text browser

• View input and output files (.mtz, .pdb, CCP4 maps) using appropriate viewer- also use the View Any File option)

- or ccp4i -v <filename> at the prompt

• View graphs in logfile using loggraph- also use loggraph <filename> at the command prompt

View Output From Job



48

• Why bother?

• Benefits:- remind you what you did six months ago- keep track of multiple projects and associated data- facilitate “back-tracking”- make it easier to deposit your results/write your paper

Project Management Using CCP4i



49

“All data files relating to one crystallographic project should be in a single project directory”

One word alias ...… for project directory containing data files

Project Directories



50

• Each project directory has an associated job database accessed through the central panel in the main window:

• This displays:- which tasks were run, and when- their status (RUNNING, FINISHED, FAILED etc)- the title entered by the user

• The database also keeps a record of:- the parameters used to run the task- the input, output and log files associated with the task

Job Database



51

View files from any job in the database

Remove failed/unwanted jobs from the database and archive important data

Rerun any job in the database (with the option of changing the parameters first)

• Use this to review parameters used in an earlier run

Keep the database up-to-date• Add runs of “external” programs

Job Database Utilities



52

Notebook: record information for future reference

Edit data: e.g. update title of run, change file locations

Add external /non-interfaced program run to the database

Edit Job Data



53

General help from main window

Help with a particular option -Right hand mouse button click

over that option

Help for a particular task

Brings up relevant documentation in browser

On-Line Help



54

• Message line help

One-line help message appears in the bar at the top of the window

• Send e-mail to directly to CCP4(please send as much information as possible!)

More Help….



55

• CCP4 Roadmaps - accessed via program index (under General)• CCP4 Tutorial2000 - also from the program index

Navigating Modules



56

•On-line help:- help button on main window for general info- help button on task interfaces for help with individual tasks- right-hand mouse button click over part of the window for help on that option or feature.

• Roadmaps through the structure determination process:- $CCP4/ccp4i/help/roadmaps/index.html

• Tutorial material- $CCP4/examples/tutorial2000/html/index.html

• E-mail CCP4- [email protected]

CCP4 Help Summary



57

• Key programs in the structure solution procedure write out harvest files- in CCP4 these are SCALA, TRUNCATE, MLPHARE, REFMAC and RESTRAIN

• At deposition time these files represent an accurate record of how the final model was obtained

• Each file records details of the method used and the results obtained (e.g. heavy atom sites used in phasing)

• Harvest files can be sent directly to the deposition site, avoiding much manual processing

Data Harvesting



58

• Harvesting operates “invisibly”

• Harvesting requires MTZ files contain Project and Dataset names:- add these when data is first imported into CCP4 (Import Unscaled Data or Convert to MTZ & Standardise tasks in Data Reduction Module), or- edit Project name or Dataset name using Edit MTZ Project&Dataset (in the Reflection Data Utilities module)

• Switch on the harvesting options:- in individual tasks e.g. SCALA, REFMAC, or- set the harvesting defaults in the CCP4i Preferences window (RHS of the main window)

- you don’t need to think about it again until the end of the structure solution

Data Harvesting in CCP4i

The CCP4 Package.

Documents

Transcript of The CCP4 Package.