CARMA FluxCal

This page is for those unfortunate Berkeley grad students that have been tasked with maintaining fluxcal in the absence of a Berkeley CARMA postdoc.

Meeting Notes

Fluxes.py Script Notes

Full CARMA fluxcal wiki page

• We should check how our list of accepted calibrators compares to the full list of calibrators in FluxSource.cat - see Fluxes.py Script Notes

The general strategy is to catch up each week on Fridays: ~order is Amber Statia Chat Katey

Upcoming out-of-town times:

Future Servitude (now 2012!):

To monitor fluxes of common calibrators…etc…

Wiki page for fluxcal activities status stuff (from Nikolaus, Nov 4 2011) here

Other fluxcal stuff:

• Woojin's reduction of dedicated fluxcal tracks
  • NOT done on science tracks
  • results are here - rather nice page (even has spectral indices for some calibrators!), but only linked to from the internal CARMA website
• John Carpenter's automatic reduction of all tracks
  • reduction script is /array/obs/bin/fluxes.py (on the highsite)
  • results are posted to this page
• Tom Plagge's cm-band dedicated fluxcal tracks
  • results posted here
  • uses this script
• the only calibration instructions given on the public CARMA site is a link to the current FluxSource.cat
  • also these fluxes are available through the xplore tool
• information on flux monitoring on internal CARMA site is woefully out-of-date (here)
  • I know Tom C wants to get this updated on the wiki somewhere, but I couldn't find where this will/should go

Amber
Statia
Chat
Katey

Update Sept 14,2011: due to network issues (at the request of Rick and Tom), we need to do our fluxcal analysis in a different directory. Specifically, Rick has asked that we work in (copy files to, run analysis in) the directory /mnt/szadata/obs/fluxcal on cedarflat4. Everyone now has his or her own subdirectory, and GetNewCalFlux.py has been edited to reflect the change. All you need to do is go to the new directory (either directly or via the symbolic link “fluxcal_new” that Chat put in everyone's /home/obs/ directory), and proceed with fluxcal as normal. [Note: when you're at CARMA, you'll have to login to cedarflat4 for the symbolic link to work.]

The day-to-day grunt work: fluxcal should be done ~ once a week to keep up with all the incoming tracks. The idea is to switch off among those of us involved in some way so that we each only have to do a week's worth of tracks once a month or so.

Full instructions on how to do all this can be found here, but you can find the short version (courtesy of Amber) below:

1. Log into the highsite computers:
   • ssh -Y obs@ssh.mmarray.org (-Y for Mac's, -X for Linux)
   • type 'hs' to get to the highsite
   • cd into your working directory
2. Figure out how many tracks there are since the last time fluxcal was done:
   • ./getLastN.csh last_track_run.mir
   • this will output the tracks since the last track run and tell you what to use for addLastN
3. Type 'python' to get into python and load the program:
   • import GetNewCalFlux
4. GetNewCalFlux.getTracksWithFlux(addLastN=10)
   • put whatever getLastN told you to here instead of 10
   • this will copy all the files you need to look at into your directory and create FluxTable.txt (a list of all the tracks you need to run doFluxCal on, listing the calibrators in each track. The calibrator list is not always accurate though, so you should check it with uvindex - see note below)
5. Figure out the appropriate GetNewCalFlux.doFluxCal commands to run for each track using the writeCommands.csh program:
   • writeCommands.csh reads the .mir files in your directory and finds the calibrators observed in each track (and their 'purpose') and then takes a stab at writing the doFluxCal command for you based on the purposes. It is interactive, so it tells you what it's doing and if you disagree with it's choices, you can tell it what to do.
   • run writeCommands.csh
   • this will create the file fluxcalCommands.txt, containing all the doFluxCal commands you need to run (including running the bandpass calibrator with plots='N') - just copy and paste into the command line from this file to complete the next step…
6. For each track, run
   • GetNewCalFlux.doFluxCal(track name, primary (ie Neptune), secondary, passband)
     • ie. GetNewCalFlux.doFluxCal('cx319.5E_90NGC253.5.mir','MARS','0132-169','3C454.3')
     • NOTE: make sure your primary calibrator (MARS, NEPTUNE, etc.) and the “C” in any passband calibrators is in all caps! Weird errors can result if not…
   • At the end of the script, you will be prompted for whether or not you want to add this value to science_fluxes.txt - if the data look good, then type 'Y' (capitals matter!) and hit enter. Failure to do both of these things will result in the flux not being added.
   • If the track seemed good (ie. you added the secondary calibrator flux to science_values.txt), then rerun it to calculate the flux of the passband as well by just changing the secondary name to the passband. At this point, you probably don't want to see the plots again, so I recommend adding plots='N' as well, ie.
     GetNewCalFlux.doFluxCal('cx319.5E_90NGC253.5.mir','MARS','3C454.3','3C454.3',plots='N')
7. When you're done, science_fluxes.txt should contain all the fluxes you chose to record. You first need to copy science_fluxes.txt to your machine where you have CVS set up. Then, you simply append these fluxes to the end of FluxSource_newadd.cat in your miriad install and commit it to CVS through the following 3 steps:
   • make sure your copy is up-to-date:

     cvs update FluxSource_newadd.cat

   • add the fluxes in science fluxes.txt to the end of FluxSource_newadd.cat using the text editor of your choice
   • commit your new version to the official repository, with your initials in the comment line, ie.:

     cvs commit -m "science fluxes added, AB" FluxSource_newadd.cat

A few suggestions on general workflow:

• It is good to have two terminals logged in to the highsite: a shell to run MIRIAD commands in, and a Python or iPython session (avoid running fluxcal in a SAC, where you run the risk of colliding with observing tasks).
• After you have created FluxTable.txt, you should check that the program actually got the right calibrators (it misses some sometimes) by running uvindex so that you can fill out the doFluxCal command appropriately (this is now done for you, interactively, by writeCommands.csh)
• The doFluxCal program will pop up several plots for you to check the quality of the data. If you see obvious problems (like an antenna is bad or a time chunk is bad), stop the script, flag the data and restart the script. However, you should not spend too long on a given track, so if the flagging is going to be involved or the data seem suspect for some other reason, just ignore that track and move on.

1. …on the high-site machines
   • Make your own directory (e.g. amberb/)
   • Copy necessary files from ~/amberb/:
     • getLastN.csh
     • writeCommands.csh
     • all .py files:
       • fileIOPython.py
       • GetNewCalFlux.py
       • shortCuts.py
       • FindSource.py
       • pymiriad.py
       • shortNew.py
   • Edit GetNewCalFlux.py to change directory ('amberb') to your working directory. You should do this on three lines:

     [obs@cedarflat4 ~/amberb]$ grep -n amberb GetNewCalFlux.py
     42:    fluxFile = '/home/obs/amberb/FluxTable.txt'
     94:        os.system("cp -rf %s /home/obs/amberb/" % i)
     261:           tedFile  = '/home/obs/amberb/science_fluxes.txt'

2. CVS on your local machine (according to Amber's experience - other instructions available in the fluxcal pdf document)
   1. If you have a real install of Miriad from CVS, you may already be set up
      • Look at the end of the fluxcal pdf document for instructions
      • Contact Peter Teuben if you have problems
   2. If you have Peter W's Miriad MacPorts install or if you don't have Miriad, then you can just do a CVS checkout of the catalogs folder without getting all of Miriad:
      • First, put the appropriate CVSROOT command in your .cshrc or .bashrc file. For .cshrc,
        

        setenv CVSROOT :pserver:fluxcal@cvs.astro.umd.edu:/home/cvsroot 
        setenv CVSEDITOR emacs

        For .bashrc,
        

        export CVSROOT=:pserver:fluxcal@cvs.astro.umd.edu:/home/cvsroot
        export CVSEDITOR=emacs

      • Then type

        cvs login

        in a terminal and enter the fluxcal password (ask someone for this). This creates a .cvspass file for you so that you shouldn't ever have to log in again.

      • Finally, you can check out the current version of the catalogs directory of Miriad. Simply go to the directory where you want to put it (e.g. ~/Documents/fluxcal/mycatcheckout/) and type:

        cvs checkout miriad/cat

        . Now, inside your desired director, there should be a 'Miriad' folder, and a 'cat' folder inside of that. This is where FluxSource_newadd.cat lives. To update it, simply go there (~/Documents/fluxcal/mycatcheckout/Miriad/cat/) and follow the directions in the How? section above.

• Script crashes:
  • Check that your primary flux calibrator name (MARS, NEPTUNE, etc.) is in all caps…don't know why this matters, but it seems to
  • Check that you have no spaces in your GetNewCalFlux.doFluxCal call… python seems to hate spaces?