NASSP Masters 5003F - Computational Astronomy - 2009 Lecture 8 Some further unix & python tips Good...

NASSP Masters 5003F - Computational Astronomy - 2009

Lecture 8

• Some further unix & python tips

• Good programming habits

• Catalog cross-matching

• Bayesian variability test


More Unix tips and tricks: redirection.

How to redirect your output:– If I do

– I get

– etc. But if I do

– I apparently don’t get anything. But! Look inside file99:

– etc.

$ ls > file99

benne.html Desktop Examples pic26867.jpg

$ more file99benne.htmlDesktop

$ ls


More Unix tips and tricks: redirection.

• The > character redirects the output of ls (or any other command) from the screen to the named file.

• Careful! It will overwrite any pre-existing file of that name.

• If you just want to add stuff to the end of an existing file, use >> instead of >.

• To redirect errors, you have to be more elaborate. But we won’t worry about that.

• If you want to send the output to another command, use | (called a ‘pipe’). Eg

a2ps | lp -dljuct


Dots in Unix.

A dot . in Unix has several different functions, depending on context.

1. Here it is just part of a filename:

2. Here is the first character in the name of a ‘hidden’ file (ls won’t list these – but ls –a will):

3. Here it means run the script called fred:

4. Here it means the present working directory:

bignob.py

.bashrc

$ . fred

$ export PATH=/usr/bin:.$ ls ./*

Note the space!


Double dots in Unix.

Two dots in a row means the directory 1 higher up the tree from here. It can save you some typing. Eg the following:

– If I want to cd back to comp_astron, there’s two ways to do it. The long way:

– Or the short way:

$ pwd/home/ims/doc/teaching/comp_astron$ lscode latex figures$ cd code$ pwd/home/ims/doc/teaching/comp_astron/code

$ cd /home/ims/doc/teaching/comp_astron

$ cd ../


The tilde ~

• This is another symbol which has a meaning which depends on context.

1. Here it means the file is a backup. The shell makes these under certain conditions. They record the file before your last change. Obviously this can be useful!

2. Here it is shorthand for your home directory.

• As an aside, cd with no arguments always takes you direct to home:

$ cd ~/doc/teaching

some_file_name~

$ cd


Another few cm of the python.

• The range function – it is a bit tricky. Note the following:

• Function returns. Note:

• This is a tuple. It is better to be explicit...

>>> print range(3)[0, 1, 2]>>> print range(0,3)[0, 1, 2]

>>> def some_func(arg1,arg2):... a = arg1 + arg2... b = arg1 – arg2... return a, b...>>> c = some_func(42, 2)>>> print c(44, 40)


Another few cm of the python.

• Also in the function call:

– is better than

• You can also force it to return a list:

>>> def some_func(arg1,arg2):... a = arg1 + arg2... b = arg1 – arg2... return (a, b)

>>> (summ, diff) = some_func(42, 2)

>>> c = some_func(42, 2)

>>> def anuther_func(arg1,arg2):... a = arg1 + arg2... b = arg1 – arg2... return [a, b]...>>> c = anuther_func(20, 19)>>> print c[39, 1]


Another few cm of the python.• The for statement - most of you know this

way:

• But you can also do this:

• Saves a bit of typing.

>>> mylist = [‘peas’,’beans’,’dead dog’]>>> for i in range(len(mylist)):... item = mylist[i]... print item...peasbeansdead dog

>>> mylist = [‘peas’,’beans’,’dead dog’]>>> for item in mylist:... print item...peasbeansdead dog


Good vs.EVIL programming habits.• The good:

– Make your code as unspecialized as possible.– Chunk tasks into functions and modules.

• The bad:– GOTO – python has abolished this statement.– Equality testing against reals. Eg

• Only do this against zero.

– Leaving some values untested, eg:

if (myvar == 7.2):

if (nuddervar < 5.9): # do somethingelif (nuddervar > 5.9): # do something else


Good vs.EVIL programming habits.• The bad continued:

– Changing function arguments. Eg

– This is actually only ugly...BAD is doing it with lists. Try the following:

– Using reserved words for variable names.

• The ugly:– Too-short variable names.

def hobbit_state(bilbo, sam): sam = ‘hungry’

>>> def moonshine(list_of_moons):... list_of_moons.append(‘half’)...>>> mondlicht = [‘new’,’blue’]>>> moonshine(mondlicht)>>> print mondlicht


Catalog cross-matching

• It sometimes happens that you have 2 lists of objects, which you want to cross-match.– Maybe the lists are sources observed at

different frequencies.– The situation also arises in simulations.

• I’ll deal with the simulations situation first, because it is easier.– So: we start with a bunch of simulated

sources. Let’s keep it simple and assume they all have the same brightness.

– We add noise, then see how many we can find.


Catalog cross-matching– In order to know how well our source-

detection machinery is working, we need to match each detection with one of the input sources.

• How do we do this?• How do we know the ‘matched’ source is the ‘right

one’?

...I haven’t done a rigorous search of the literature yet – these arejust my own ideas.

CAVEAT:


Catalog cross-matchingBlack: simulated sourcesRed: 1 of many detections (with 68% confidence interval).

This case seems clear.



But what about these cases?

No matches insideconfidence interval.

Too many matchesinside confidence interval.



Or these?

Is any a good match? Which is ‘nearest’?



• My conclusion:1. The shape of the confidence intervals

affects which source is ‘nearest’.

2. The size of the confidence intervals has nothing to do with the probability that the ‘nearest’ match is non-random.

1. ‘Nearest neighbour’ turns out to be a slipperier concept than we at first think. To see this, imagine that we have now 1 spatial dimension and 1 flux dimension:



S

x

S

x

Which is the best match???

This makes more sense.Let’s then define r as:

1

3

4

5

7

1

8

9

2

6

7

3

5

4

1

9

6

21

8

Source 5? Or source 8?



2. As for the probability... well, what is the null hypothesis in this case?

– Answer: that the two catalogs have no relation to each other.

– So, we want the probability that, with a random distribution of the simulated sources, a source would lie as close or closer to the detected source than rnearest.

– This is given by:

– where ρ is the expected density of sim sources and V is the volume inside rnearest.

Pnull = 1 – exp(-ρV)



• So the procedure for matching to a simulated catalog is:

1. For each detection, find the input source for which r is smallest.

2. Calculate the probability of the null hypothesis from Pnull = 1 – exp(-ρV).

3. Discard those sources for which Pnull is greater than a pre-decided (low) cutoff.

• What about the general situation of matching between different catalogs?



ASCA data – M Akiyama et al (2003)


Bayesian variability analysis


Bayesian variability analysis

J D Scargle (1998)

NASSP Masters 5003F - Computational Astronomy - 2009 Lecture 8 Some further unix & python tips Good...

Documents

Transcript of NASSP Masters 5003F - Computational Astronomy - 2009 Lecture 8 Some further unix & python tips Good...