CARMA FluxCal

This page is for those unfortunate Berkeley grad students that have been tasked with maintaining fluxcal in the absence of a Berkeley CARMA postdoc.

Meeting Notes

Fluxes.py Script Notes

Full CARMA fluxcal wiki page

To Do / Current Issues

  • We should check how our list of accepted calibrators compares to the full list of calibrators in FluxSource.cat - see Fluxes.py Script Notes

Sign Up / Future Servitude

The general strategy is to catch up each week on Fridays: ~order is Amber Statia Chat Katey

Upcoming out-of-town times:

Future Servitude (now 2012!):

??? ???

Record of Past Servitude

Date caught up to Peon Last Track Notes
Feb 17 Frank cx319.1E_90MACS04.4.mir no more Frank =(
Mar 17 Amber cx319.4D_90Abell2.1.mir not set up for CARMA23 - just flag ant(16-23) for now
Mar 29 Statia c0665.7SL_94macs17.16.mir
Apr 1 Amber c0722.1D_230SgrA.53.mir
Apr 8 Amber c0674.1D_115M31.17.mir
Apr 15 Statia c0713.1C_115NGC278.6.mir only 13 tracks to run?
Apr 22 Chat c0690.2C_88NGC126.2.mir
Apr 29 Amber c0699.1C_115OrionO.14.mir 19 tracks run
May 6 Katey c0753.3C_104HATLAS.4.mir 28 tracks
May 13 Statia c0685.2D_113MCG022.1.mir 10 tracks (bad weather at hs)
May 20 Chat c0672.2D_110DMTau.1.mir
May 27 Amber c0672.3D_110UXTau.1.mir
Jun 3 Katey didn't run Jun 3 - split with Amber Jun 10
Jun 10 Amber c0685.7D_113CGCG46.7.mir split May 27 - Jun 10 with Katey
June 17 Statia c0688.1D_100015028.5.mir
Jun 24 Chat c0685.7D_113CGCG46.13.mir
Jul 1 Katey c0712.1D_115NGC545.7.mir ~30 tracks
Jul 8 Amber c0657.2E_91HS0810.3.mir
Jul 15 Statia c0732.1E_245Epsilo.2.mir lots of tracks…
Jul 22 Chat c0684.18SL_30SAUS.1.mir Actually 7/26…Did half of the next person's tracks! That's what I get for taking so long…
Aug 6 Katey c0849.9SH_31RXJ175.3.mir did two weeks worth
Aug 12 Amber c0768.6E_107G12H34.6.mir
Aug 19 Statia c0857.5E_95PTF5.3.mir
Aug 26 Chat c0768.3D_96G12H89.2.mir My God that took a long time.
Sep 2 Amber c0834.12D_88C2.2.mir didn't take too long (addLastN=65), but moved this week
Sep 9 Statia cx327.1SH_31DQFlex.2.mir
Sep 16 Chat c0758.3D_114HE2149.6.mir The new era of fluxcal has begun…or at least, we're running old fluxcal in a new directory!
Sep 23 Katey Didn't run Sep 23, split Sep 30 with Statia
Sep 30 Statia c0817.14D_90MACSJ1.1.mir Split last two weeks with Katey
Oct 7 Chat c0843.5SH_90A1319.7.mir
Oct 14 Amber c0845.1SH_30Field1.12.mir
Oct 21 Katey c0834.13D_89C3.10.mir
Oct 28 Amber c0852.2D_230NGC545.5.mir kept notes on what passed/failed and why
Nov 4 Katey c0785.1D_86G03539.9.mir
Nov 11 Chat cx329.1D_90S140.4.mir Kept notes on what passed/failed and why. Also, it's SO MUCH BETTER doing fluxcal after a week of perfect weather! Mmmm…ALMA fluxcal…???
Nov 18 Statia c0857.1A_95PTF1.2.mir made up last Chat track (listed one is from Nov 7)
Nov 25 Amber c0857.1A_95PTF1.9.mir very few tracks with A array move and bad weather; also bootflux crash issue with MARS - bring up at telecon
Dec 2 Statia c0811.1A_30NGC452.PACS.11.mir almost all tracks failed at bootflux (mars, uranus and neptune
Dec 9 Chat c0811.1A_30NGC452.PACS.19.mir Didn't run 1mm tracks. One 3mm track succeeded (Uranus), as did PACS tracks with Uranus & Neptune (not Mars).
Dec 16 Katey c0773.2B_95GRB2.1.mir
Dec 23 Amber c0794.2B_30Mol160.PACS.5.mir after running out of space issue on Dec 23rd, cleared out some space and able to get tracks on Jan 3, 2012; fluxcal done on Jan 3 - bootflux crashed on MARS, 1mm and 3mm in B array
Dec 30 Chat cx334.1B_30GRB111.PACS.6.mir Space cleared, Amber put tracks in directory Jan 3, 2012
Jan 6 Statia c0811.1B_230NGC452.13.mir blah
Jan 13 Katey c0769Z.3SL_30MACSJ0.6.mir
Jan 20 Amber c0820.3C_86115049.3.mir
Jan 27 Statia c0820.3C_86115049.6.mir
Feb 3 Amber c0843.6SL_30A1622.1.mir C12 low amplitude issue - I just flagged C12 in all 1 and 3mm tracks
Feb 10 Katey c0834I.1C_97B1.2.mir
Feb 17 Chat c0834M.2C_96B2.3.mir
Feb 24 Statia c0859.2C_108G12H29.1.mir
Mar 2 Chat c0866.2C_100NGC624.4.mir
Mar 9 Katey c0888C.15C_230Ser1.1.mir Was out of town, had to do fluxcal on Monday
Mar 16 Amber c0933.29SL_30C58114.8.mir
Mar 23 Statia c0932.12SL_30RXJ223.3.mir Bad things happened when I was getting tracks
Mar 30 Chat c0933.26SL_30C57507.1.mir Bad things happened when I was getting tracks
Apr 6 Katey c0885.1D_91XMMM5.2.mir Didn't have any problems
Apr 13 Amber c0884.1SL_31MOO002.3.mir
Apr 20 Statia c0957.7D_115M31BRK.4.mir
Apr 27 Chat c0922.1D_98GC.2.mir
May 4 Katey c0915.1D_110Taurus.1.mir
May 11 Amber c0924V.3D_88Serpen.4.mir
May 25 Amber c0961.7SL_30PX03.1.mir OLD FLUXCAL IS DONE! Running 2 weeks of tracks with the new script!
Jun 5 Amber c0911.5D_90IRAS4F.3.mir new fluxcal at Illinois
Jun 14 Amber c0849Z.10SH_95Abell1.11.mir new fluxcal at high site pending fix of issue at Illinois
Jun 22 Amber c0933.2SH_30C13062.1.mir at high site from ~/amberb/newera/
Jul 3 Amber c0884.3SH_31MOO004.1.mir at high site; some weird Exception: 1 and 8…I don't know what this is coming from
Jul 6 Amber c0879.1E_90MS0451.2.mir at high site
Jul 16 Amber c0883.1SL_30ACTJ03.12.mir at high site
Aug 6 Amber c0892.12E_100MACSJ0.2.mir at high site
Aug 13 Amber c0973.30E_100HeLMS5.2.mir at high site
Aug 21 Amber cx340.1E_115gulf.4.mir at high site
Aug 31 Amber c1017.7E_100MACSJ0.1.mir at high site
Sep 28 Amber c1009.9SH_90ABELL1.7.mir at high site
Oct 8 Amber c0924.2D_88B1E.SL.1.miriad half at high site, half at Illinois

Why?

To monitor fluxes of common calibrators…etc…

Wiki page for fluxcal activities status stuff (from Nikolaus, Nov 4 2011) here

Other fluxcal stuff:

  • Woojin's reduction of dedicated fluxcal tracks
    • NOT done on science tracks
    • results are here - rather nice page (even has spectral indices for some calibrators!), but only linked to from the internal CARMA website
  • John Carpenter's automatic reduction of all tracks
    • reduction script is /array/obs/bin/fluxes.py (on the highsite)
    • results are posted to this page
  • Tom Plagge's cm-band dedicated fluxcal tracks
  • the only calibration instructions given on the public CARMA site is a link to the current FluxSource.cat
    • also these fluxes are available through the xplore tool
  • information on flux monitoring on internal CARMA site is woefully out-of-date (here)
    • I know Tom C wants to get this updated on the wiki somewhere, but I couldn't find where this will/should go

Who?

Amber
Statia
Chat
Katey

How?


Update Sept 14,2011: due to network issues (at the request of Rick and Tom), we need to do our fluxcal analysis in a different directory. Specifically, Rick has asked that we work in (copy files to, run analysis in) the directory /mnt/szadata/obs/fluxcal on cedarflat4. Everyone now has his or her own subdirectory, and GetNewCalFlux.py has been edited to reflect the change. All you need to do is go to the new directory (either directly or via the symbolic link “fluxcal_new” that Chat put in everyone's /home/obs/ directory), and proceed with fluxcal as normal. [Note: when you're at CARMA, you'll have to login to cedarflat4 for the symbolic link to work.]


The day-to-day grunt work: fluxcal should be done ~ once a week to keep up with all the incoming tracks. The idea is to switch off among those of us involved in some way so that we each only have to do a week's worth of tracks once a month or so.

Full instructions on how to do all this can be found here, but you can find the short version (courtesy of Amber) below:

  1. Log into the highsite computers:
    • ssh -Y obs@ssh.mmarray.org (-Y for Mac's, -X for Linux)
    • type 'hs' to get to the highsite
    • cd into your working directory
  2. Figure out how many tracks there are since the last time fluxcal was done:
    • ./getLastN.csh last_track_run.mir
    • this will output the tracks since the last track run and tell you what to use for addLastN
  3. Type 'python' to get into python and load the program:
    • import GetNewCalFlux
  4. GetNewCalFlux.getTracksWithFlux(addLastN=10)
    • put whatever getLastN told you to here instead of 10
    • this will copy all the files you need to look at into your directory and create FluxTable.txt (a list of all the tracks you need to run doFluxCal on, listing the calibrators in each track. The calibrator list is not always accurate though, so you should check it with uvindex - see note below)
  5. Figure out the appropriate GetNewCalFlux.doFluxCal commands to run for each track using the writeCommands.csh program:
    • writeCommands.csh reads the .mir files in your directory and finds the calibrators observed in each track (and their 'purpose') and then takes a stab at writing the doFluxCal command for you based on the purposes. It is interactive, so it tells you what it's doing and if you disagree with it's choices, you can tell it what to do.
    • run writeCommands.csh
    • this will create the file fluxcalCommands.txt, containing all the doFluxCal commands you need to run (including running the bandpass calibrator with plots='N') - just copy and paste into the command line from this file to complete the next step…
  6. For each track, run
    • GetNewCalFlux.doFluxCal(track name, primary (ie Neptune), secondary, passband)
      • ie. GetNewCalFlux.doFluxCal('cx319.5E_90NGC253.5.mir','MARS','0132-169','3C454.3')
      • NOTE: make sure your primary calibrator (MARS, NEPTUNE, etc.) and the “C” in any passband calibrators is in all caps! Weird errors can result if not…
    • At the end of the script, you will be prompted for whether or not you want to add this value to science_fluxes.txt - if the data look good, then type 'Y' (capitals matter!) and hit enter. Failure to do both of these things will result in the flux not being added.
    • If the track seemed good (ie. you added the secondary calibrator flux to science_values.txt), then rerun it to calculate the flux of the passband as well by just changing the secondary name to the passband. At this point, you probably don't want to see the plots again, so I recommend adding plots='N' as well, ie.
      GetNewCalFlux.doFluxCal('cx319.5E_90NGC253.5.mir','MARS','3C454.3','3C454.3',plots='N')
  7. When you're done, science_fluxes.txt should contain all the fluxes you chose to record. You first need to copy science_fluxes.txt to your machine where you have CVS set up. Then, you simply append these fluxes to the end of FluxSource_newadd.cat in your miriad install and commit it to CVS through the following 3 steps:
    • make sure your copy is up-to-date:
      cvs update FluxSource_newadd.cat
    • add the fluxes in science fluxes.txt to the end of FluxSource_newadd.cat using the text editor of your choice
    • commit your new version to the official repository, with your initials in the comment line, ie.:
      cvs commit -m "science fluxes added, AB" FluxSource_newadd.cat

A few suggestions on general workflow:

  • It is good to have two terminals logged in to the highsite: a shell to run MIRIAD commands in, and a Python or iPython session (avoid running fluxcal in a SAC, where you run the risk of colliding with observing tasks).
  • After you have created FluxTable.txt, you should check that the program actually got the right calibrators (it misses some sometimes) by running uvindex so that you can fill out the doFluxCal command appropriately (this is now done for you, interactively, by writeCommands.csh)
  • The doFluxCal program will pop up several plots for you to check the quality of the data. If you see obvious problems (like an antenna is bad or a time chunk is bad), stop the script, flag the data and restart the script. However, you should not spend too long on a given track, so if the flagging is going to be involved or the data seem suspect for some other reason, just ignore that track and move on.

Getting Set-Up

  1. …on the high-site machines
    • Make your own directory (e.g. amberb/)
    • Copy necessary files from ~/amberb/:
      • getLastN.csh
      • writeCommands.csh
      • all .py files:
        • fileIOPython.py
        • GetNewCalFlux.py
        • shortCuts.py
        • FindSource.py
        • pymiriad.py
        • shortNew.py
    • Edit GetNewCalFlux.py to change directory ('amberb') to your working directory. You should do this on three lines:
      [obs@cedarflat4 ~/amberb]$ grep -n amberb GetNewCalFlux.py
      42:    fluxFile = '/home/obs/amberb/FluxTable.txt'
      94:        os.system("cp -rf %s /home/obs/amberb/" % i)
      261:           tedFile  = '/home/obs/amberb/science_fluxes.txt'
  2. CVS on your local machine (according to Amber's experience - other instructions available in the fluxcal pdf document)
    1. If you have a real install of Miriad from CVS, you may already be set up
      • Look at the end of the fluxcal pdf document for instructions
      • Contact Peter Teuben if you have problems
    2. If you have Peter W's Miriad MacPorts install or if you don't have Miriad, then you can just do a CVS checkout of the catalogs folder without getting all of Miriad:
      • First, put the appropriate CVSROOT command in your .cshrc or .bashrc file. For .cshrc,
        setenv CVSROOT :pserver:fluxcal@cvs.astro.umd.edu:/home/cvsroot 
        setenv CVSEDITOR emacs

        For .bashrc,

        export CVSROOT=:pserver:fluxcal@cvs.astro.umd.edu:/home/cvsroot
        export CVSEDITOR=emacs
      • Then type
        cvs login

        in a terminal and enter the fluxcal password (ask someone for this). This creates a .cvspass file for you so that you shouldn't ever have to log in again.

      • Finally, you can check out the current version of the catalogs directory of Miriad. Simply go to the directory where you want to put it (e.g. ~/Documents/fluxcal/mycatcheckout/) and type:
        cvs checkout miriad/cat

        . Now, inside your desired director, there should be a 'Miriad' folder, and a 'cat' folder inside of that. This is where FluxSource_newadd.cat lives. To update it, simply go there (~/Documents/fluxcal/mycatcheckout/Miriad/cat/) and follow the directions in the How? section above.

Trouble-Shooting

  • Script crashes:
    • Check that your primary flux calibrator name (MARS, NEPTUNE, etc.) is in all caps…don't know why this matters, but it seems to
    • Check that you have no spaces in your GetNewCalFlux.doFluxCal call… python seems to hate spaces?