Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
carma_fluxcal [2012/09/28 20:45]
amberb [Record of Past Servitude]
carma_fluxcal [2012/10/08 22:02] (current)
amberb [Record of Past Servitude]
Line 1: Line 1:
 +====== CARMA FluxCal ======
  
 +This page is for those unfortunate Berkeley grad students that have been tasked with maintaining fluxcal in the absence of a Berkeley CARMA postdoc.
 +
 +[[fluxcal_notes|Meeting Notes]]
 +
 +[[fluxes_script|Fluxes.py Script Notes]]
 +
 +[[http://​www.mmarray.org/​twiki/​index.php/​Flux_calibration|Full CARMA fluxcal wiki page]] ​
 +===== To Do / Current Issues =====
 +  * **We should check how our list of accepted calibrators compares to the full list of calibrators in FluxSource.cat - see [[fluxes_script|Fluxes.py Script Notes]]**
 +
 +
 +
 +===== Sign Up / Future Servitude =====
 +
 +The general strategy is to catch up each week on Fridays:
 +~order is Amber Statia Chat Katey
 +
 +Upcoming out-of-town times:
 +
 +**Future Servitude (now 2012!):**
 +| ??? | ??? |
 +
 +===== Record of Past Servitude =====
 +
 +^ Date caught up to ^ Peon ^ Last Track ^ Notes ^
 +| Feb 17 | Frank | cx319.1E_90MACS04.4.mir | no more Frank =( |
 +| Mar 17 | Amber | cx319.4D_90Abell2.1.mir | not set up for CARMA23 - just flag ant(16-23) for now |
 +| Mar 29 | Statia ​  | c0665.7SL_94macs17.16.mir |                 |
 +| Apr 1 | Amber | c0722.1D_230SgrA.53.mir |  |
 +| Apr 8 | Amber | c0674.1D_115M31.17.mir |   |
 +| Apr 15 | Statia | c0713.1C_115NGC278.6.mir| only 13 tracks to run?  |
 +| Apr 22 | Chat | c0690.2C_88NGC126.2.mir |
 +| Apr 29 | Amber | c0699.1C_115OrionO.14.mir | 19 tracks run |
 +| May 6 | Katey | c0753.3C_104HATLAS.4.mir | 28 tracks |
 +| May 13 | Statia | c0685.2D_113MCG022.1.mir |10 tracks (bad weather at hs)|
 +| May 20 | Chat | c0672.2D_110DMTau.1.mir | |
 +| May 27 | Amber | c0672.3D_110UXTau.1.mir |   |
 +| Jun 3 | Katey | --- | didn't run Jun 3 - split with Amber Jun 10 |
 +| Jun 10 | Amber | c0685.7D_113CGCG46.7.mir | split May 27 - Jun 10 with Katey |
 +| June 17       | Statia | c0688.1D_100015028.5.mir| ​ |
 +| Jun 24 | Chat | c0685.7D_113CGCG46.13.mir | |
 +| Jul 1 | Katey | c0712.1D_115NGC545.7.mir | ~30 tracks |
 +| Jul 8 | Amber | c0657.2E_91HS0810.3.mir |   |
 +| Jul 15 | Statia | c0732.1E_245Epsilo.2.mir | lots of tracks... ​ |
 +| Jul 22 | Chat | c0684.18SL_30SAUS.1.mir| Actually 7/26...Did half of the next person'​s tracks! ​ That's what I get for taking so long... ​ | 
 +| Aug 6 | Katey | c0849.9SH_31RXJ175.3.mir | did two weeks worth |
 +| Aug 12 | Amber | c0768.6E_107G12H34.6.mir |  |
 +| Aug 19 | Statia | c0857.5E_95PTF5.3.mir | |
 +| Aug 26 | Chat | c0768.3D_96G12H89.2.mir | My God that took a long time. |
 +| Sep 2 | Amber | c0834.12D_88C2.2.mir | didn't take too long (addLastN=65),​ but moved this week |
 +| Sep 9 | Statia | cx327.1SH_31DQFlex.2.mir | |
 +| Sep 16 | Chat | c0758.3D_114HE2149.6.mir | The new era of fluxcal has begun...or at least, we're running old fluxcal in a new directory! |
 +| Sep 23 | Katey | |Didn'​t run Sep 23, split Sep 30 with Statia|
 +| Sep 30 | Statia | c0817.14D_90MACSJ1.1.mir | Split last two weeks with Katey|
 +| Oct 7 | Chat | c0843.5SH_90A1319.7.mir | |
 +| Oct 14 | Amber | c0845.1SH_30Field1.12.mir |  |
 +| Oct 21 | Katey | c0834.13D_89C3.10.mir | |
 +| Oct 28 | Amber | c0852.2D_230NGC545.5.mir | kept notes on what passed/​failed and why |
 +| Nov 4 | Katey | c0785.1D_86G03539.9.mir | |
 +| Nov 11 | Chat | cx329.1D_90S140.4.mir | Kept notes on what passed/​failed and why. Also, it's SO MUCH BETTER doing fluxcal after a week of perfect weather! ​ Mmmm...ALMA fluxcal...???​|
 +|Nov 18| Statia | c0857.1A_95PTF1.2.mir | made up last Chat track (listed one is from Nov 7)|
 +|Nov 25 | Amber | c0857.1A_95PTF1.9.mir | very few tracks with A array move and bad weather; also bootflux crash issue with MARS - bring up at telecon |
 +|Dec 2| Statia | c0811.1A_30NGC452.PACS.11.mir | almost all tracks failed at bootflux (mars, uranus and neptune|
 +|Dec 9| Chat| c0811.1A_30NGC452.PACS.19.mir | Didn't run 1mm tracks. ​ One 3mm track succeeded (Uranus), as did PACS tracks with Uranus & Neptune (not Mars). |
 +| Dec 16 | Katey | c0773.2B_95GRB2.1.mir | |
 +| Dec 23 | Amber | c0794.2B_30Mol160.PACS.5.mir | after running out of space issue on Dec 23rd, cleared out some space and able to get tracks on Jan 3, 2012; fluxcal done on Jan 3 - bootflux crashed on MARS, 1mm and 3mm in B array |
 +| Dec 30 | Chat | cx334.1B_30GRB111.PACS.6.mir | Space cleared, Amber put tracks in directory Jan 3, 2012 |
 +|Jan 6| Statia | c0811.1B_230NGC452.13.mir |blah |
 +| Jan 13 | Katey | c0769Z.3SL_30MACSJ0.6.mir | |
 +| Jan 20 | Amber | c0820.3C_86115049.3.mir | |
 +| Jan 27 | Statia | c0820.3C_86115049.6.mir | |
 +| Feb 3 | Amber | c0843.6SL_30A1622.1.mir | C12 low amplitude issue - I just flagged C12 in all 1 and 3mm tracks |
 +| Feb 10 | Katey | c0834I.1C_97B1.2.mir | |
 +| Feb 17 | Chat | c0834M.2C_96B2.3.mir | |
 +| Feb 24 | Statia | c0859.2C_108G12H29.1.mir| |
 +| Mar 2 | Chat | c0866.2C_100NGC624.4.mir |  |
 +| Mar 9 | Katey | c0888C.15C_230Ser1.1.mir | Was out of town, had to do fluxcal on Monday |
 +| Mar 16 | Amber | c0933.29SL_30C58114.8.mir |  |
 +| Mar 23 | Statia | c0932.12SL_30RXJ223.3.mir | Bad things happened when I was getting tracks |
 +| Mar 30 | Chat | c0933.26SL_30C57507.1.mir | Bad things happened when I was getting tracks |
 +| Apr 6 | Katey | c0885.1D_91XMMM5.2.mir | Didn't have any problems |
 +| Apr 13 | Amber | c0884.1SL_31MOO002.3.mir |  |
 +| Apr 20  | Statia |c0957.7D_115M31BRK.4.mir ​ | |
 +| Apr 27  | Chat | c0922.1D_98GC.2.mir ​ | |
 +| May 4 | Katey | c0915.1D_110Taurus.1.mir | |
 +| May 11 | Amber | c0924V.3D_88Serpen.4.mir |  |
 +| May 25 | Amber | c0961.7SL_30PX03.1.mir | OLD FLUXCAL IS DONE! Running 2 weeks of tracks with the new script! |
 +| Jun 5 | Amber | c0911.5D_90IRAS4F.3.mir | new fluxcal at Illinois |
 +| Jun 14 | Amber | c0849Z.10SH_95Abell1.11.mir | new fluxcal at high site pending fix of issue at Illinois |
 +| Jun 22 | Amber | c0933.2SH_30C13062.1.mir | at high site from ~/​amberb/​newera/​ |
 +| Jul 3 | Amber | c0884.3SH_31MOO004.1.mir | at high site; some weird Exception: 1 and 8...I don't know what this is coming from |
 +| Jul 6 | Amber | c0879.1E_90MS0451.2.mir | at high site |
 +| Jul 16 | Amber | c0883.1SL_30ACTJ03.12.mir | at high site |
 +| Aug 6 | Amber | c0892.12E_100MACSJ0.2.mir | at high site |
 +| Aug 13 | Amber | c0973.30E_100HeLMS5.2.mir | at high site |
 +| Aug 21 | Amber | cx340.1E_115gulf.4.mir | at high site |
 +| Aug 31 | Amber | c1017.7E_100MACSJ0.1.mir | at high site |
 +| Sep 28 | Amber | c1009.9SH_90ABELL1.7.mir | at high site |
 +| Oct 8 | Amber | c0924.2D_88B1E.SL.1.miriad | half at high site, half at Illinois |
 +
 +===== Why? =====
 +
 +To monitor fluxes of common calibrators...etc...
 +
 +Wiki page for fluxcal activities status stuff (from Nikolaus, Nov 4 2011) [[http://​www.mmarray.org/​twiki/​index.php/​Flux_calibration|here]]
 +
 +**Other fluxcal stuff:**
 +  * Woojin'​s reduction of dedicated fluxcal tracks
 +    * NOT done on science tracks
 +    * results are [[http://​haneul.astro.illinois.edu/​~wkwon/​CARMA/​fluxcal/​|here]] - rather nice page (even has spectral indices for some calibrators!),​ but only linked to from the **internal** CARMA website
 +  * John Carpenter'​s automatic reduction of **all** tracks
 +    * reduction script is /​array/​obs/​bin/​fluxes.py (on the highsite)
 +    * results are posted to [[http://​cedarflat.mmarray.org/​scripts/​calibrators/​|this page]]
 +  * Tom Plagge'​s cm-band dedicated fluxcal tracks ​
 +    * results posted [[http://​sza.uchicago.edu/​sza_pub/​mirgains|here]]
 +    * uses [[http://​sza.uchicago.edu/​sza_pub/​mirgains/​get_flux_cal.sh|this]] script
 +  * the only calibration instructions given on the public CARMA site is a link to the current [[http://​cedarflat.mmarray.org/​information/​carmaFluxSource.cat|FluxSource.cat]]
 +    * also these fluxes are available through the xplore tool
 +  * information on flux monitoring on internal CARMA site is **woefully** out-of-date ([[http://​cedarflat.mmarray.org/​arrayhealth/​arrayhealth.html#​flux|here]])
 +    * I know Tom C wants to get this updated on the wiki somewhere, but I couldn'​t find where this will/should go
 +
 +
 +===== Who? =====
 +
 +Amber\\ ​
 +Statia\\
 +Chat\\
 +Katey
 +
 +===== How? =====
 +
 +
 +----
 +**Update Sept 14,2011:** due to network issues (at the request of Rick and Tom), we need to do our fluxcal analysis in a different directory. Specifically,​ Rick has asked that we work in (copy files to, run analysis in) the **directory /​mnt/​szadata/​obs/​fluxcal on cedarflat4**. Everyone now has his or her own subdirectory,​ and GetNewCalFlux.py has been edited to reflect the change. ​ All you need to do is go to the new directory (either directly or via the symbolic link "​fluxcal_new"​ that Chat put in everyone'​s /home/obs/ directory), and proceed with fluxcal as normal. ​ [Note: when you're at CARMA, you'll have to login to cedarflat4 for the symbolic link to work.]
 +----
 +
 +
 +The day-to-day grunt work: fluxcal should be done ~ once a week to keep up with all the incoming tracks. The idea is to switch off among those of us involved in some way so that we each only have to do a week's worth of tracks once a month or so. 
 +
 +Full instructions on how to do all this can be found {{:​fluxcal.pdf|here}},​ but you can find the short version (courtesy of Amber) below:
 +
 +  - Log into the highsite computers:
 +     * ssh -Y obs@ssh.mmarray.org (-Y for Mac's, -X for Linux)
 +     * type '​hs'​ to get to the highsite
 +     * cd into your working directory ​
 +  - Figure out how many tracks there are since the last time fluxcal was done:
 +     * ./​getLastN.csh last_track_run.mir
 +     * this will output the tracks since the last track run and tell you what to use for addLastN
 +  - Type '​python'​ to get into python and load the program:
 +     * import GetNewCalFlux
 +  - GetNewCalFlux.getTracksWithFlux(addLastN=10)  ​
 +     * put whatever getLastN told you to here instead of 10
 +     * this will copy all the files you need to look at into your directory and create FluxTable.txt (a list of all the tracks you need to run doFluxCal on, listing the calibrators in each track. The calibrator list is not always accurate though, so you should check it with uvindex - see note below)
 +  - Figure out the appropriate GetNewCalFlux.doFluxCal commands to run for each track using the writeCommands.csh program:
 +     * writeCommands.csh reads the .mir files in your directory and finds the calibrators observed in each track (and their '​purpose'​) and then takes a stab at writing the doFluxCal command for you based on the purposes. It is interactive,​ so it tells you what it's doing and if you disagree with it's choices, you can tell it what to do.
 +     * run writeCommands.csh
 +     * this will create the file fluxcalCommands.txt,​ containing all the doFluxCal commands you need to run (including running the bandpass calibrator with plots='​N'​) - just copy and paste into the command line from this file to complete the next step... ​
 +  - For each track, run
 +     * GetNewCalFlux.doFluxCal(track name, primary (ie Neptune), secondary, passband)
 +       * ie. GetNewCalFlux.doFluxCal('​cx319.5E_90NGC253.5.mir','​MARS','​0132-169','​3C454.3'​)
 +       * NOTE: make sure your primary calibrator (MARS, NEPTUNE, etc.) and the "​C"​ in any passband calibrators is in all caps! Weird errors can result if not...
 +     * At the end of the script, you will be prompted for whether or not you want to add this value to science_fluxes.txt - if the data look good, then type '​Y'​ (capitals matter!) and hit enter. Failure to do both of these things will result in the flux not being added.
 +     * If the track seemed good (ie. you added the secondary calibrator flux to science_values.txt),​ then rerun it to calculate the flux of the passband as well by just changing the secondary name to the passband. At this point, you probably don't want to see the plots again, so I recommend adding plots='​N'​ as well, ie. \\ GetNewCalFlux.doFluxCal('​cx319.5E_90NGC253.5.mir','​MARS','​3C454.3','​3C454.3',​plots='​N'​)
 +  - When you're done, science_fluxes.txt should contain all the fluxes you chose to record. You first need to copy science_fluxes.txt to your machine where you have CVS set up. Then, you simply append these fluxes to the end of FluxSource_newadd.cat in your miriad install and commit it to CVS through the following 3 steps:  ​
 +     * make sure your copy is up-to-date: <​code>​cvs update FluxSource_newadd.cat</​code>​
 +     * add the fluxes in science fluxes.txt to the end of FluxSource_newadd.cat using the text editor of your choice
 +     * commit your new version to the official repository, with your initials in the comment line, ie.: <​code>​cvs commit -m "​science fluxes added, AB" FluxSource_newadd.cat</​code>​
 +
 +
 +**A few suggestions on general workflow:​** ​
 +  * It is good to have two terminals logged in to the highsite: a shell to run MIRIAD commands in, and a Python or iPython session (avoid running fluxcal in a SAC, where you run the risk of colliding with observing tasks).
 +  * After you have created FluxTable.txt,​ you should check that the program actually got the right calibrators (it misses some sometimes) by running uvindex so that you can fill out the doFluxCal command appropriately (this is now done for you, interactively,​ by writeCommands.csh)
 +  * The doFluxCal program will pop up several plots for you to check the quality of the data. If you see obvious problems (like an antenna is bad or a time chunk is bad), stop the script, flag the data and restart the script. However, you should not spend too long on a given track, so if the flagging is going to be involved or the data seem suspect for some other reason, just ignore that track and move on.
 +
 +===== Getting Set-Up =====
 +
 +  - ...on the high-site machines
 +    * Make your own directory (e.g. amberb/)
 +    * Copy necessary files from ~/amberb/:
 +      * getLastN.csh
 +      * writeCommands.csh
 +      * all .py files: ​
 +        * fileIOPython.py  ​
 +        * GetNewCalFlux.py  ​
 +        * shortCuts.py
 +        * FindSource.py ​   ​
 +        * pymiriad.py ​      
 +        * shortNew.py
 +    * Edit GetNewCalFlux.py to change directory ('​amberb'​) to your working directory. You should do this on three lines: <​code>​[obs@cedarflat4 ~/amberb]$ grep -n amberb GetNewCalFlux.py
 +42:    fluxFile = '/​home/​obs/​amberb/​FluxTable.txt'​
 +94:        os.system("​cp -rf %s /​home/​obs/​amberb/"​ % i)
 +261:           ​tedFile ​ = '/​home/​obs/​amberb/​science_fluxes.txt'</​code>​
 +  - CVS on your local machine (according to Amber'​s experience - other instructions available in the {{:​fluxcal.pdf|fluxcal pdf document}})
 +    -  If you have a real install of Miriad from CVS, you may already be set up
 +      * Look at the end of the {{:​fluxcal.pdf|fluxcal pdf document}} for instructions
 +      * Contact Peter Teuben if you have problems
 +    -  If you have Peter W's Miriad MacPorts install or if you don't have Miriad, then you can just do a CVS checkout of the catalogs folder without getting all of Miriad:
 +      * First, put the appropriate CVSROOT command in your .cshrc or .bashrc file. For .cshrc, \\ <​code>​
 +setenv CVSROOT :​pserver:​fluxcal@cvs.astro.umd.edu:/​home/​cvsroot ​
 +setenv CVSEDITOR emacs
 +</​code>​ For .bashrc, \\ <​code>​
 +export CVSROOT=:​pserver:​fluxcal@cvs.astro.umd.edu:/​home/​cvsroot
 +export CVSEDITOR=emacs
 +</​code>​
 +      * Then type <​code>​cvs login</​code>​ in a terminal and enter the fluxcal password (ask someone for this). This creates a .cvspass file for you so that you shouldn'​t ever have to log in again.
 +      * Finally, you can check out the current version of the catalogs directory of Miriad. Simply go to the directory where you want to put it (e.g. ~/​Documents/​fluxcal/​mycatcheckout/​) and type: <​code>​cvs checkout miriad/​cat</​code>​. Now, inside your desired director, there should be a '​Miriad'​ folder, and a '​cat'​ folder inside of that. This is where FluxSource_newadd.cat lives. To update it, simply go there (~/​Documents/​fluxcal/​mycatcheckout/​Miriad/​cat/​) and follow the directions in the How? section above.
 +===== Trouble-Shooting =====
 +
 +  * Script crashes:
 +    * Check that your primary flux calibrator name (MARS, NEPTUNE, etc.) is in all caps...don'​t know why this matters, but it seems to
 +    * Check that you have no spaces in your GetNewCalFlux.doFluxCal call... python seems to hate spaces?