PI cataloging procedure

1. Photometry

a/ fast photometry

b/ asas photometry on 20 averaged frames

2. Astrometry

3. Cataloging

•   reduction - dark subtraction, division by flat frame
  
•  photometry - produces lists of stars ( so called mag files ) with the following information :
     x y   MAG mag0 mag1 mag2 ...magN
• astrometry - calculates transformation T : (x,y) -> (ra,dec) , and calculates  ( ra,dec )  for all stars in mag file , writes the results to ast file
  

1.     IMPORTANT : Drop indexes on table Measurements ( and some on Stars ) before loading
   
2.     frame information from header of ast file ( same like in fits header ) is added to database tables frame and frame_det ( table ObsFieldStat is
       automatically updated )
3. Stars from ast file are read to memory
   
4. Stars in the observed field are read from database table Stars to memory ( if not already read for ealier image of same field ), only
   stars with number of observations >=2 are read ( or >=1 for scan )
   
5. Normalization of magnitudo of stars read from ast file is done :
   
   •  frame is devided into 50x50 pixels squares
     
   •  in each sqaure stars are matched with catalog stars ( ASAS catalog used in astrometry is used : $DATADIR/cat/act )
   • median of the following variable is found in each sqaure :
            corr_value := cat_magnitude - ast_magnitude
     
   • correction frame is calculated in such a way that value of correction in each pixel is the value of neighbouring squares weighted
     by the distance to the particular square center
   • correction frame is applied to stars in ast , magnitudo of each star is recalculated using the following formula :
            mag_corr := ast_magnitude +  corr_value( star.x, star,y  ) 
6. ast-stars are matched to db-stars ( read from DB)  and measurements from ast are added to memory-table measurements_tab for
   every star, in case ast-star is not found in db-stars list it is added as new and field new_star in measurements table is filled with true value
   ( false otherwise ). Star measurements is matched to existing star in case it is closer then matching radius ( currently set to 120 arcsec ),
   otherwise new star record is created.
   
7. After ast from other field comes all stars/measurements tables stored in memory are dumped to sql file and it is loaded to database.
   Stars from database are read to memory according to ra,dec coordinates of the observed field and every new ast file is handled
   as desribed in points 1-5, unitil new field aperas or no more ast files are found
8. when loading stars to database measurements are checked, in case one star has more then one measurements on single frame then
   both are rejected, this seems to be main source of the problem of "bad measurements" - single points far from light curve .
   Maybe it will be changed so that only "wors  measurement" will be rejected and better will be kept, but this is not obvious how to
   guess which one is good , so now I reject both.
   This problem should not exist if photometry is good enough and does not identify same star as 2 or more different object, but
   asas photometry program is not resistant for such problems and sometimes single big star produces 2 objects to be cataloged.
   One have good magnitudo and second has much different ! Option in cataloging program piaddast2 -skip_dups enabled rejection and
   is enabled by default
   

1. Creation of indexes in database for faster queries
   
2. recalculation of values of several fields  in Star table like : magnitude (  average magnitudo ) , sigma_mag ( rms of the magnitude
   no_measurements ( number of measurements for this star ), this can be done during piaddast2 step but when loading lots of data
   from many nights it is better to do it afterwards
3. NEW : Stars from both cameras are matched together by field cam2_sstar_id - matching radius is currently 2 arcmin ( 120 arcsec ) - which is
   same value as in star matching in cataloging procedure, however for old database ( created ealier then 2006-01-11 ) it was 5 arcmin
   which is too much and can cause problem - that in case there is close star but not good it can overwrite good setting, my notes on this problem ( in polish ) :
                   jakis blad w linkowaniu gwiazd 20040918_asas@heplx48 :
                  gwiazda 30365 ma link 84146 a tak naprawde powinna miec : 84155
                 i o co chodzi ????
                juz chyba mam podejrzenie !!!
                Moze chodzi ze ta 84146 jest pozniej ale tez jest < 5 arcmin od tej
                i stad nadpisuje juz dobre ustawienie !
               5 arcmin to za duzo - zmieniam na 120 arcsec = 2 arcmin
              Tak czyli 5 arcmin = 300 arcsec to za duzo !
   
4. Optimize database , by creating cluster on tables Stars/Measurements , so that measurements of particular star are physicaly placed
   in the same place on the disk
   
5. fill SuperStar table and set sstar_id field on Stars table
   

1. links between camers - matching radius changed to 2 arcmin ( 20060111 )
2. skiping measurements in case >1 measurement of single star on single frame found ( 20051220 )
3. bugfix ( 20051220 )  : measurements from k2b could be wrongly linked to stars from camera k2a , check :
          select spathtofile,icamid from frame,measurements where
                   frame.id_frm=measurements.id_frm and star=1;
4. -min_star_count=15000 - option to specify minimal number of stars in ast file to be catalogued, otherwise
   number of stars is not enough to have good normalization to star catalog and measurements cause artificial flares, brigthness increase etc ...
5. 20060208 : Currently testing with min_stars_to_add_new=1 - to have measurements also of single measurements object - for moving objects finding

4. On-line cataloging

 5. Most important scripts/programs for cataloging

1. run_asas_pipeline_online_cat! - script for on-line pi-reduction , asas-photometry, asas-astrometry and pi-cataloging to database of frames averaged
   default number of averaged frames is 20, but this can be changed, example of usage can be found on heplx48 :
            [pi@heplx48 new_version]$ pwd
            /opt/pi/dev/pisys/daq/ndir/data/photo/new_version
            #!/bin/bash 
            nohup run_asas_pipeline_online_cat! 20050301 /scratch/pi_fits2004/results/fits/ /opt/pi/dev/pisys/daq/ndir/data/photo/new_version 20050301_new > out 2>&1 &
   
2. run_single_scan_cat! - script for pi-reduction, fast-photometry, asas-astrometry and pi-cataloging of scan single frames , example :
            cd /pi20/msok/scan
            #!/bin/bash 
            nohup run_single_scan_cat! 20050103/ /pi20/msok/scan/20050103/ /pi20/msok/scan/SCAN_PIPELINE/ scan_single > out 2>&1 &
   
3. run_scan_cat! - script for pi-reduction, asas-photometry, asas-astrometry and pi-cataloging of scan averaged frames
4. run_fast_pipeline_online_cat! - script for on-line fast-photometry/reduction , asas-astrometry and pi-cataloging to database of single frames, usage example ( tested ) :
             #!/bin/bash
             nohup run_fast_pipeline_online_cat! 20040928 /pi20/msok/fast/ONLINE/SMALL/fits/ /pi20/msok/fast/ONLINE/SMALL/ 20040928_fast > out 2>&1 &

6. Times and disks space of cataloging steps :

a/ fast photometry :

b/ pi-pipeline online ( 20 averaged frames )

c/ scan cataloging

7. Current settings

• border size - stars closer then border are not catalogued, stars close to edge of the chip are not very good, current border size = 100 pixels, I am
  checking if it is possbile to make it smaller without decrease of catalog quality , for more details and conclusions see here
  

8. New changes