Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| fluxcal_notes [2012/06/13 17:13] – amberb | fluxcal_notes [2019/09/06 22:52] (current) – removed casey | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== CARMA FluxCal Notes ====== | ||
| - | ===== To Do ===== | ||
| - | * Figure out algorithm for error estimation (currently 15%, or RMS of values) | ||
| - | * check that planet brightness temp is different for different windows | ||
| - | |||
| - | ===== Telecon, Jun 13 ===== | ||
| - | * online: Amber, Statia, Woojin, Manuel, Doug, Peter | ||
| - | * data transfer issue at Illinois: | ||
| - | * issue with script | ||
| - | * Peter - testing new code with Woojin | ||
| - | * 3 mars tables: | ||
| - | * 1 is marstb original increments of 25 days | ||
| - | * 2 is marstb2 stops at 115 GHz, increments of 6 hours | ||
| - | * 3 is marstb3 1cm to 260 GHz, increments of 1 hours | ||
| - | * this table is ~5 K cooler than old curves at 115 GHz - see Woojin' | ||
| - | * need to check at other freq and if spec index is the same, then ask Brian Butler | ||
| - | * can play with these in new miriad - marstb with table= (need to grab tar files for the tables first) | ||
| - | * will build new binaries for school students tomorrow but it is in CVS now | ||
| - | * fluxtest script is in Miriad now | ||
| - | * | ||
| - | |||
| - | ===== Telecon, Jun 6 ===== | ||
| - | * online: Amber, Statia, Chat, Nikolaus, Andreea, Woojin, Manuel, Peter, Doug | ||
| - | * Woojin: more 1cm data points and ' | ||
| - | * still trying to sort out what these new models are: 'most recent models of Glenn Orton and Rafael Moreno' | ||
| - | * Peter: | ||
| - | * new model from Brian Butler on Mars - Jan 2010 - Dec 2020 in 1 hour intervals at 30-1000 GHz (this is becoming the standard model in CASA) - 11 MB | ||
| - | * miriad program marstb will be able to read this soon - use table= to read 1hr cadence | ||
| - | * Uranus and Neptune models - European + North American collaboration. Hofstader model is more appropriate at 1cm, but this model is better at 3mm and 1mm. | ||
| - | * fluxcal on science: | ||
| - | * copy could take a minute or two but it does happen all in one go - check size of visdata | ||
| - | * multiple corr configs - Doug: sci1 track doppler tracking wasn't set up in first NOISE integration - this is nothing to worry about. sci2...?? wideband channels have a bandwidth of 0...maybe first antenna is being flagged by the system. maybe try re-running uvwide to recalculate the wideband channels from the spectral line channels. uvcat just spectral channels out options=nowide. | ||
| - | * Andreea - looking for comments on everything (incorporated my comments) | ||
| - | |||
| - | ===== Telecon, May 30 ===== | ||
| - | * online: Amber, Statia, Chat, John, Andreea, Doug, Peter, Woojin, Manuel, Tom | ||
| - | * Woojin: added new data points as well as JPL models (from Andreea) to plots | ||
| - | * new data point: Mars track not good (3C84 low elevation, pointing didn't converge) | ||
| - | * 1cm points: 5 or 6 tracks have been tried - planning to add these points soon | ||
| - | * Andreea: looking at 1cm data 2 tracks from May 18 | ||
| - | * estimate the flux of 3C454.3 using Uranus and Neptune separately with different fluxes | ||
| - | * JPL Nep 5.2 | ||
| - | * JPL Ura 5.3 | ||
| - | * CARMA Nep 6.7 (bootflux with nothing specified) | ||
| - | * CARMA Ura 6.6 (bootflux with nothing specified) | ||
| - | * looking now at dataset to figure out which of these fluxes is more appropriate using Mars | ||
| - | * Doug getting Illinois set up | ||
| - | * Peter uses Python package NTHOUGHT? | ||
| - | * fluxcal on science running automated on the high site now, will be running at Illinois soon | ||
| - | * pass on to Manuel once we have fully documented (coming soon) | ||
| - | * Peter contacted Mark Gurwell about Mars models | ||
| - | * passed on to Brian Butler - haven' | ||
| - | * been working on table reader in Miriad | ||
| - | * can't officially distribute marstb2.tab (only to 105) until we get okay from Brian Butler | ||
| - | ===== Telecon, May 23 ===== | ||
| - | * online: Amber, Statia, John, Andreea, Doug, Peter, Nikolaus, Woojin, Manuel, Tom | ||
| - | * Andreea: working on 1cm abstest data | ||
| - | * comparision of Uranus and Neptune models from Mark Holfstader show these models agree with eachother within 2% | ||
| - | * Woojin: still needs more abstest data at 1mm | ||
| - | * John points out that models are very similar at 1mm - 1mm data may not tell us much | ||
| - | * perhaps 1mm Uranus/ | ||
| - | * Mars model: | ||
| - | * best bet seems to be szaCalcMars at 1cm, 3mm (diurnal + seasonal var) and marstb at 1mm (seasonal var only) for now | ||
| - | * szaCalcMars uses a table that is almost identical to marstb2.tab, | ||
| - | * marstb uses marstb.tab - this is what is written into datasets by the online system (this table only covers 43 - 345 GHz) | ||
| - | * Peter is working on a miriad program to read marstb2.tab, | ||
| - | * Peter will contact Mark Gurwell about getting diurnal variation Mars model good out to 1mm | ||
| - | * fluxcal on science: | ||
| - | * Doug got permissions worked out - we should be getting new datasets today | ||
| - | * python install needed: Doug will check, but the best bet may be to put our own local install of python in our area | ||
| - | * getting julian dates on the high site: high site date calculations are done in C, so we should just use a local copy of a python package to do it. | ||
| - | |||
| - | ===== Telecon, May 16 ===== | ||
| - | * online: Amber, Statia, Chat, Woojin, Manuel, Doug, John, Andreea, Tom | ||
| - | * fluxcal on science: | ||
| - | * Doug gave us a username at Illinois and a directory to work in | ||
| - | * where should we put our scripts and other files? / | ||
| - | * where do the miriad files live? (so i can grab one for testing) | ||
| - | * would you like to copy files to our directory as they come in? script currently just makes a copy of the full path file to local directory, so this is not totally necessary. | ||
| - | * what is the best way to automate this? | ||
| - | * cron job, python script just always running in a screen, sleep for 30 min | ||
| - | * Woojin: | ||
| - | * more datasets - 3mm regime data looks good, large scatter in 1mm regime | ||
| - | * weather in 1mm data not terrible...why so much scatter? Woojin is doing a phase-only selfcal using the planet... | ||
| - | * Mars model: cross points cannot be explained by Mars diurnal variation - what is going on? lots of scatter in daytime pointing - could this be the issue? | ||
| - | * Tom: | ||
| - | * fluxcal_sci2 now gets abstest sources when they' | ||
| - | * eventually want to get just one script for all of this | ||
| - | * Peter: ? | ||
| - | * email obs and Nikolaus about how to run abstest | ||
| - | |||
| - | ===== Telecon, May 9 ===== | ||
| - | * online: Woojin, Manuel, Amber, Statia, Doug, John, Andreea, Peter | ||
| - | * Woojin - new plots of brightness temp versus freq | ||
| - | * want to sample higher frequencies in the 3mm band - change track for today to 105-110 GHz | ||
| - | * Nikolaus try to get abstest going in sci2 - will contact Tom offline | ||
| - | * actually - possible to make 1 fluxcal script work for sci1 and sci2? Nikolaus will take a look | ||
| - | * Peter - hasn't gotten to the online system model stuff | ||
| - | * Doug - has the go ahead from Leslie, tracking down the IT guy | ||
| - | |||
| - | ===== Telecon, May 2 ===== | ||
| - | * online: Woojin, Manuel, Amber, Statia, Chat, Doug, Tom, Andreea | ||
| - | * Woojin: plots from abstest last week | ||
| - | * how can we get brightness temperature from planet flux? | ||
| - | * Statia | ||
| - | * Woojin | ||
| - | * Andreea doing same as Woojin but using JPL models | ||
| - | * MARS pipeline model is identical to marstb != SZA model w/ diurnal variation | ||
| - | * TODO get good MARS model in online system (but Erik's szacalcmars only good at 3mm and 1cm) | ||
| - | * Doug will let us know about getting set up at Illinois this week - maybe we can bother Lisa | ||
| - | |||
| - | ===== Telecon, Apr 25 ===== | ||
| - | * Woojin - abstest: still needs 1mm | ||
| - | * U+N still give consistent values for other cals, but M gives slightly smaller value by 5% (but much closer than two weeks ago - 15%) | ||
| - | * reduction script applies different brightness temperatures in different windows | ||
| - | * Jupiter model - constant brightness temp (also too big to be used) - actually may be okay for sci2...? Might be worth taking a look at... | ||
| - | * Andreea - also working on abstest data from March | ||
| - | * smaflux - links don't work, Tom contacted guy (Eric Weistein) but hasn't gotten answers...odds of making sense of smaflux data are low | ||
| - | * Peter - nothing yet | ||
| - | ===== Telecon, Apr 18 ===== | ||
| - | * Woojin: abstest tracks from last week (3mm - 93.5 GHz) - | ||
| - | * Mars and Jup gave same values for 3C84, but 20% lower than values from U+N | ||
| - | * we should get brightness temp of U+N based on Mars calib | ||
| - | * Peter - same as last week, check against what Woojin has (he will email around) | ||
| - | * us: | ||
| - | * add a check of flux versus last entry and flag to check it if | ||
| - | ===== Telecon, Apr 11 ===== | ||
| - | * waiting to hear from Doug about getting set up at Illinois (emailed Apr 5) | ||
| - | * DRIP comp stalled b/c webpage hasn't been updated since Mar 16 - emailed Doug Apr 5 | ||
| - | * sent notes on webpage to Ted Yu (got short response) | ||
| - | * TO DO: | ||
| - | * Peter: sort out what planet models the online system is using: let us know the models used, the code that does this, any tables referenced, and where the code and tables live on the high-site. | ||
| - | * Andreea: re-work primay fluxcal document, send a summary of relevant information out to everyone for discussion at the next telecon | ||
| - | * Nikolaus/ | ||
| - | * Woojin: analyze fluxcal abstest data when they are taken | ||
| - | * Amber, | ||
| - | |||
| - | ===== Notes on Webpage (Telecon, Apr 4) ===== | ||
| - | * http:// | ||
| - | * source selection: | ||
| - | * ra dec input - text input should be the default | ||
| - | * searching by cal name | ||
| - | * extended sources? script writer knows! Depends on array. | ||
| - | * show all 3 wavelength band plots at once? | ||
| - | * SolSys date? what is this? | ||
| - | * error bars - how to handle points that don't have an error? very large error bar? a different color point? | ||
| - | * identify change over when we switch to new error reporting | ||
| - | * should print FluxSource.cat entries below the plot | ||
| - | * plot fluxes versus freq as well? calc rough spectral index? | ||
| - | * changing plot limits: | ||
| - | * okay to just enter min, not max | ||
| - | * flux val reporting: | ||
| - | * report avg and scatter for fluxes within plotting window instead of current value | ||
| - | |||
| - | |||
| - | ===== Telecon Jan 31 ===== | ||
| - | * set up telecon w/ DRIP people (Doug Freidel) to compare flux results there!!! | ||
| - | * SMA, PdBI values for polarization comparisons? | ||
| - | * 3C286 paper on archiv | ||
| - | |||
| - | ===== Day of Action: occupy the lounge to fight the 20% - Dec 16, 2011 ===== | ||
| - | Looking over compare3 results (Nov 25 - Dec 9): | ||
| - | * empty datasets: | ||
| - | * Why so few? Grading! quality script says a B grade but our script does not do a PACS reduction, so the straight grade was a D and failed the first check. (For 1mm and 3mm tracks this will be most relevant) | ||
| - | * mostly seems to be bootflux crashing (due to resolution issues) -> change new script to catch this bootflux failure and stop the reduction? | ||
| - | * just old datasets: | ||
| - | * sci2 tracks (PACS) weren' | ||
| - | * cx332 track has an F grade so didn't run through new | ||
| - | * datasets with fractional diff > 0.5 seem to mostly be due to bad data in pacs datasets because of bad baselines. We still don't know why the old script was able to handle this though... | ||
| - | * analysis of the 5-40% dataset - what have we learned? | ||
| - | * at 1cm, using all widebands is going to be an issue w/spectral index of the source - discuss this on the telecon perhaps. | ||
| - | * we need to update amp flagging in the new script - amp(500) insufficient - need to track down which bands it is happening in and flag those bands in all ants | ||
| - | * DATASET DETAILS: | ||
| - | * 1224+213 | ||
| - | * 1224+213 | ||
| - | * 0108+015 | ||
| - | * 0108+015 | ||
| - | * NOTE: the MFCAL failed when solving for the passband, though plotting the passband in calibrators.mir with smagpplt looked fine afterward... | ||
| - | * Before flagging: 3C84 2.500 / 2.280 (old/ | ||
| - | * After flagging: 3C84 5.977 / 5.816 (within 3%) | ||
| - | * Before flagging: 0108+015 0.500 / 0.440 | ||
| - | * After flagging: 0108+015 1.167 / 1.105 (within 6%) | ||
| - | * These numbers are much closer to the c0852.1D_230NGC062.3 numbers | ||
| - | * MORAL OF THE STORY: Flagging C7 was hugely important... | ||
| - | * 3C84 cx327.1SH_31DQFlex.15 -- no phase coherence on C21 for 1st 1.5 hours (inc. planet observation). Flagging C21 brought flux of 3C84 from old script within 2% of new script value. Note: For whatever reason, flagging C21 did not help the script' | ||
| - | * 3C84 c0782.7D_230L1551I.1 The amplitude vs. time plots vary for several antennas. The output of bootflux varies from 6.3-8.1. Not sure how to fix this without an amplitude selfcal. Tried flagging the worst offender (C12) but this did not improve matters. What to do? | ||
| - | ===== Telecon Dec 2, 2011 ===== | ||
| - | * Mark H on the line to discuss models | ||
| - | |||
| - | ===== Telecon Nov 18, 2011 ===== | ||
| - | |||
| - | * Planet models - | ||
| - | * Orton models may not be an option b/c may not be done soon and they want recognition in some way? | ||
| - | |||
| - | ===== Working-Time October 20, 2011 ===== | ||
| - | Sorting through the results from the new/old fluxcal comparison: | ||
| - | * 248 datasets total | ||
| - | * empty: 14 - IGNORE | ||
| - | * just old: 24 | ||
| - | * WHY did new fluxcal ignore?? | ||
| - | * new cuts are A or B and len >= 2 | ||
| - | * just new: 117 | ||
| - | * are they good? | ||
| - | * why didn't old do it? (cal list and prim flux cal) | ||
| - | * checked cal list and prim flux cal - still 65 that have prim cal and acceptable secondary cal...? | ||
| - | * both: 93 | ||
| - | * start by looking at frac diff > 0.5 datasets - which is right? | ||
| - | * explosions may be due to passband failing to converge | ||
| - | |||
| - | **Action Items:** | ||
| - | * calibrators list - should we only do certain cals? | ||
| - | * email John for a summary of how his program works | ||
| - | * add something to check for convergence of passband | ||
| - | * 1 bad baseline can kill it - maybe some flagging to check for crazy amps at the start? | ||
| - | * capitalization issue - **DONE** (Amber fixed comparison script) | ||
| - | * log file issue - **DONE** (Amber fixed fluxes_temp.py and run_gainCheck.csh to write two different log files for sci1 and sci2...hopefully it's working...gainCheck hasn't run yet..) | ||
| - | ===== Telecon October 7, 2011 ===== | ||
| - | |||
| - | * Error discussion: | ||
| - | * Big problem: some tracks have high variation in calculated fluxes from measurement to measurement (bad weather?), but others have small variations but a smooth change from high to low (changing tau and/or RMS?) | ||
| - | * We need to somehow reflect the weather during the track when we report errors on flux measurements...if there are 10 measurements of a calibrator' | ||
| - | * How do we robustly "pump up" the statistical errors to reflect the weather? | ||
| - | * What if there are phase variations on timescales smaller than the integration time, even if there' | ||
| - | * What about when tracks have a smooth change? | ||
| - | * Is T_sys being taken out???? | ||
| - | * If this really isn't being done, should we put a calculator on the CARMA calibration website that corrects for elevation? | ||
| - | * Maybe we need to make the weather-grade cut even higher, because presumably tracks that have a smooth change throughout show smooth changes in tau and/or RMS, which should get worse weather grades. | ||
| - | * This may be an unsolvable problem...we should probably just cut at a high grade, and then report tau and RMS in two columns next to the flux values and be done with it. | ||
| - | * We should report the weather grade (which takes array config. into account), and how the grade is calculated | ||
| - | * We should report the number of times fluxes were calculated | ||
| - | * For the passband calibrators, | ||
| - | * We need to put systematic (model) errors on a fluxcal website, but report our statistical (or modified statistical) errors for each flux measurement. | ||
| - | * Could val Vleck correction be a problem? | ||
| - | |||
| - | ===== Telecon August 31, 2011 ===== | ||
| - | |||
| - | * Getting John's script to work for us: | ||
| - | * error handling - what is actually a good error estimate? | ||
| - | * should this be run on bad datasets? | ||
| - | * need to check out what script is doing before going live with it | ||
| - | * automatic flagging reasonable? (flags on selfcal gains) | ||
| - | * things to add: | ||
| - | * length of track | ||
| - | * weather grade | ||
| - | * algorithm on bootflux output | ||
| - | |||
| - | |||
| - | ===== Telecon June 24, 2011 ===== | ||
| - | |||
| - | * Woojin: | ||
| - | * weekly list - 5 calibrators - purpose = secondary flux calibrators (passband) | ||
| - | * monthly = full list = 209 or 290? calibrators = all miriad calibrators | ||
| - | * mainly taken during array config change | ||
| - | * uses Miriad bootflux task - planetary models in source code directory: planet.for | ||
| - | * Planetary models (Andreea) | ||
| - | * compare WMAP planetary models with some of the models: | ||
| - | * Mars looks good | ||
| - | * Neptune, Uranus and Jupiter seem off | ||
| - | * Statia' | ||
| - | * Our stuff | ||
| - | * more thought needs to go into errors | ||
| - | * transfer to using John's scrips - uses all bands and also does sci2 properly | ||
| - | * uses all the continuum bands (USB and LSB) | ||
| - | * Woojin raises issue of averaging all continuum bands | ||
| - | * freq reported is LO1 - not good for sci2 especially | ||
| - | * Tom Plagge | ||
| - | * szaflux altered slightly version of smaflux | ||
| - | * has it's own planetary models - Tom added updated version of Mars model | ||
| - | * probably not necessary to switch to this from bootflux for sci2...just make a few tweaks to bootflux code? | ||
| - | * sci2 calibration stuff: | ||
| - | * corrected aperature efficiency for 3.5m's in miriad - fixed now | ||
| - | * MWC349 calibration off of Mars model seems to give consistent results (previous work using Neptune and Uranus yielded inconsistent results) | ||
| - | * Planetary models - compare between different institutions | ||
| - | * Uranus model from Gerwell that SZA has isn't good anymore? | ||
| - | * SMAflux uses Weinstein' | ||
| - | * Tom in contact with Weinstein - should get his code? | ||
| - | * PLAN GOING FORWARD: | ||
| - | * combine fluxcal on science with John's -> talk with John offline | ||
| - | * more simple website for PI's to check flux history - better interface to FluxSource.cat | ||
| - | * Xplore shows flux history?? | ||
| - | * calflux also does this? | ||
| - | * which/how many bands to use for fluxcal reduction -> need to learn about planetary models first | ||
| - | * document the origin of the planetary models | ||
| - | * see how Statia' | ||
| - | * what models should we be using?? - how do we get these into the Miriad header -> RTS group | ||
| - | * Andreea and Woojin will look at where all the planetary models come from | ||
| - | * Consistency between different bands on MWC349 and other calibrators | ||
| - | |||
| - | ===== Meeting June 8, 2011 ===== | ||
| - | |||
| - | * look at the not-passed tracks compared to John's | ||
| - | * comparing John's values to historical not conclusive - mostly 1cm. 3C454.3 is half of past average value at 1cm. 3mm data point reasonable. | ||
| - | * email John with our findings and suggestions | ||
| - | * better calculation of errors - 15%?? | ||
| - | * rejection of scripts based on weather and track length | ||
| - | * get the 1cm calibration working properly | ||
| - | * don't assume gains of 1 for tracks going into FluxSource.cat | ||
| - | |||
| - | ===== Meeting May 23, 2011 ===== | ||
| - | |||
| - | * check our results against Tom P's 1cm results? | ||
| - | * update list of calibrators - 1cm cals from Tom P? | ||
| - | * statia will look at neptune models from Apr 27 email | ||
| - | * amber will keep track of good and bad tracks this week to compare with John's calibrator table | ||
| - | * then send different accept/ | ||
| - | * perhaps accept/ | ||
| - | * if this is the case, produce a short plotting thing to determine whether to accept - then use what script?? | ||
| - | * look at John's script! | ||
| - | |||
| - | ===== Telecon April 19, 2011 ===== | ||
| - | |||
| - | * Talk to tom plagge about what 1cm sources should look like | ||
| - | * Are the planetary models any good at 1cm? | ||
| - | |||
| - | |||
| - | ===== Meeting April 6, 2011 ===== | ||
| - | |||
| - | Agenda: | ||
| - | * What we're doing | ||
| - | * weekly tasks | ||
| - | * status of script | ||
| - | * look for purpose (in select=) to grab the BP versus phase cals | ||
| - | * How is it used? | ||
| - | * (effort = usefulness? | ||
| - | * FluxSource.cat used by selfcal | ||
| - | * picks freq in band but does some time averaging? | ||
| - | * if you put a local version of FluxSource.cat, | ||
| - | * | ||
| - | * How can we make it more efficient? | ||
| - | * more useful | ||
| - | * something like SMA website? (show IDL program) | ||
| - | * script writer uses FluxSource.cat? | ||
| - | * less time consuming | ||
| - | * Support | ||
| - | * for code? | ||
| - | * Other issues: | ||
| - | * who else is adding entries to FluxSource_newadd.cat? | ||
| - | * what is happening to the Flux runs at CARMA? Why? | ||
| - | * errors good? | ||
| - | * could this be done in quality? what about John's script? | ||
| - | * perhaps when quality runs, if track is good, set it aside for fluxcal. Use grep to get grade and time? | ||
| - | * where do the coordinates come from? is this the list of ' | ||
| - | * THE PLAN: | ||
| - | * send less files to fluxcal' | ||
| - | * or...just making quality run this? Look at script (Katey) | ||
| - | * how is selfcal using this? is it smart? | ||
| - | * getting plots online? - purge obs webpage? put this on external site page (use at your own risk) website run by Nikolaus and Tom | ||
| - | * Talk to Woojin about what he does | ||
| - | * Meredith looking into planet/moon cal | ||
